LLM & VLM courseBlog
112Track 5: LLM inference and serving · session 35 of 65

compare RTN, GPTQ, AWQ, SmoothQuant and FP8 on one model

In this lab you take one 8B model, Llama 3.1 8B Instruct, and make five quantized copies of it: RTN, GPTQ and AWQ with 4-bit weights, SmoothQuant with 8-bit weights and activations, and FP8. You predict their memory and speed on an NVIDIA L4, then measure perplexity, two tasks, prefill latency and decode latency at batch 1 and batch 16, compare with the survey’s table, and write a one-page recommendation.

Time: 5 to 7 hours (reading 1.5 hours, quantization and GPU runs 3 to 4 hours, write-up 1 hour)Level: Intermediate · hands-on labBefore this: Sessions 103 to 108 (quantization math, formats, GPTQ, AWQ, SmoothQuant), session 82 (roofline), Python, one 24 GB GPU (L4) for the main runs

What you will learn

  • What RTN, GPTQ, AWQ, SmoothQuant and FP8 each do to one linear layer, written as one formula each
  • How to predict weight memory (16.06 GB in BF16, 5.70 GB in W4A16, 9.08 GB in W8A8) and prefill and decode latency on an L4 with the roofline
  • Why 4-bit weights speed up decode but not prefill, and why 8-bit activations speed up prefill
  • How to quantize an 8B model with llm-compressor, serve it with vLLM and evaluate it with lm-evaluation-harness
  • How to measure prefill and decode latency separately at batch 1 and batch 16
  • How to read the survey’s W4A16 speed-up table and explain the gap between a roofline estimate and a measurement
  • How to turn a results table into a one-page production recommendation

Why this matters

The last six sessions each taught one quantization method in isolation. A production team never meets them that way. It meets one model, one GPU, one latency target and one quality bar, and it must pick one recipe. In this lab you take one 8B model, make five quantized copies of it, predict how each will behave on an NVIDIA L4, then measure quality and speed the same way for all of them, and finish with a one-page recommendation that a team could act on.

The five recipes are the ones you already know: RTN (round to nearest, session 103), GPTQ (session 106), AWQ (session 107), SmoothQuant (session 108) and FP8 (session 105). The first three change only the weights, to 4 bits; the last two change weights and activations, to 8 bits. Session 104 explained why this difference matters: 4-bit weights move fewer bytes, which helps the memory-bound decode phase; 8-bit activations allow 8-bit matrix units, which helps the compute-bound prefill phase. This lab turns that sentence into numbers.

The work has four parts, one per key point of the session:

  1. Quantize Llama 3.1 8B Instruct with each method.
  2. Measure perplexity, two tasks (ARC-Challenge and GSM8K), prefill latency and decode latency on an L4.
  3. Repeat the speed runs at batch 1 and batch 16, and compare with the survey’s table of W4A16 speed-ups (Zhou et al., section 5.2.1).
  4. Write a one-page recommendation for your production model.

Where our numbers come from. We did not have an L4 when we wrote this page. So the page uses three kinds of numbers, and labels each one. (1) Predictions from the roofline, computed from the model’s config.json and NVIDIA’s spec sheet; we checked every one with a Python script. (2) A rehearsal: the same five recipes, simulated in PyTorch on a small model (Qwen2.5-0.5B) on a laptop, so you can see the quality ordering before you rent a GPU. (3) Published numbers: the survey’s measured speed-ups on an A100, and the quality scores that the makers of public quantized Llama 3.1 8B checkpoints report. The scripts for the real L4 runs are complete, but we did not run them; your L4 numbers are the result of the lab.

The lab at a glance

The model is Llama 3.1 8B Instruct. From its config.json [Certain]: width \(d = 4096\), \(L = 32\) layers, 32 query heads and 8 key-value heads of size 128 (grouped-query attention, session 97), a SwiGLU feed-forward layer of width 14,336, a vocabulary of 128,256 tokens, and an output layer that is not tied to the input embedding. That gives 8,030,261,248 parameters, of which 6,979,321,856 sit in the 224 linear layers of the blocks (7 per block). Only these linear layers are quantized; the embedding, the output layer (“lm_head”) and the norms stay in BF16, which is the usual practice [Certain for the public checkpoints in the table below].

RecipeWeightsActivationsCalibration dataMatmul in the kernel
BF16 (baseline)BF16BF16none\(\text{BF16} \times \text{BF16}\)
RTN W4A16INT4, one scale per group of 128BF16nonedequantize to BF16, then \(\text{BF16} \times \text{BF16}\)
GPTQ W4A16INT4, group 128, error-correctedBF16a few hundred sequencessame as RTN
AWQ W4A16INT4, group 128, salient channels scaled upBF16a few hundred sequencessame as RTN
SmoothQuant W8A8INT8, one scale per output channelINT8, one scale per tokena few hundred sequences\(\text{INT8} \times \text{INT8} \to \text{INT32}\)
FP8 W8A8FP8 E4M3, per channelFP8 E4M3, per token, dynamicnone for dynamic scales\(\text{FP8} \times \text{FP8} \to \text{FP32}\)

Why an L4. The L4 is a cheap, low-power data-center GPU (72 W) built on NVIDIA’s Ada architecture. Its spec sheet [Certain]: 24 GB of GDDR6 memory with 300 GB/s of bandwidth, 121 TFLOP/s of dense BF16 matrix math, and 242.5 TFLOP/s of dense FP8 or INT8 (NVIDIA lists 242, 485 and 485 “with sparsity”; dense is half). Ada has FP8 matrix units, so all five recipes have fast kernels on it. Its bandwidth is low next to its compute, so its ridge point is high: \(121 \times 10^{12} / 300 \times 10^{9} = 403\) FLOPs per byte in BF16. That makes the L4 a sharp test: many workloads that are compute-bound on an H100 are memory-bound here.

Five methods, one layer, one formula each

Every recipe changes one linear layer \(Y = X W^\top\), where \(X\) holds one row per token and \(W\) one row per output feature. We write each method as one formula, with the notation of the earlier sessions.

RTN rounds each weight to the nearest point of a grid. For 4-bit asymmetric quantization with groups of \(g = 128\) input channels, each group gets its own scale and zero point:

\[ \hat{w} = s\,\big(\operatorname{clamp}(\lfloor w/s \rceil + z,\ 0,\ 2^{b}-1) - z\big), \qquad s = \frac{\max(w) - \min(w)}{2^{b}-1},\quad z = \lfloor -\min(w)/s \rceil \tag{112.1} \]
Read the symbols
\(w, \hat{w}\)
one weight and its quantized value; max and min are taken over the group of 128 weights that share \(s\) and \(z\)
\(b\)
bits per weight: 4 here, so the integer codes are 0 to 15
\(\lfloor \cdot \rceil\)
round to the nearest integer
\(s, z\)
scale (16 bits) and integer zero point of the group

GPTQ quantizes the columns of \(W\) one at a time and, after each column, changes the columns not yet quantized so that the layer’s output error on calibration data stays small. The correction uses the inverse Hessian \(H^{-1}\) of the layer’s squared error:

\[ \min_{\hat{W}} \big\lVert X W^\top - X \hat{W}^\top \big\rVert_F^2, \qquad \mathbf{w}_{:,j\gt q} \leftarrow \mathbf{w}_{:,j\gt q} - \frac{\mathbf{w}_{:,q} - \hat{\mathbf{w}}_{:,q}}{[H^{-1}]_{qq}}\,[H^{-1}]_{q,\,j\gt q}, \qquad H = X^\top X \tag{112.2} \]
Read the symbols
\(\mathbf{w}_{:,q}\)
column \(q\) of \(W\): the weights of input channel \(q\) for all outputs
\(H\)
\(X^\top X\) over the calibration tokens, plus a small damping (1% of its mean diagonal) so that it can be inverted
\(j \gt q\)
the columns after \(q\), which absorb the rounding error of column \(q\)

AWQ notices that a few input channels carry large activations, so errors in their weights hurt most. It multiplies those weight columns by a scale \(s_j \gt 1\) before rounding (so they get relatively finer steps) and divides the activations by the same \(s_j\) (folded into the previous operation, so the layer is unchanged in exact arithmetic). It picks one exponent \(\alpha\) per layer by a small grid search:

\[ s_j = \bar{x}_j^{\,\alpha}, \qquad \alpha^* = \argmin_{\alpha \in \{0, 0.05, \dots, 0.95\}} \big\lVert X\, \big(Q(W \operatorname{diag}(\mathbf{s}))\operatorname{diag}(\mathbf{s})^{-1}\big)^\top - X W^\top \big\rVert_F^2 \tag{112.3} \]
Read the symbols
\(\bar{x}_j\)
the mean absolute activation of input channel \(j\) on the calibration data
\(Q(\cdot)\)
the RTN quantizer of (112.1)
\(\alpha\)
0 means plain RTN; larger values protect the high-activation channels more, at the cost of coarser steps for the others

SmoothQuant attacks the other problem: activations with outlier channels are hard to quantize to 8 bits, weights are easy. It moves part of the range from activations to weights with a per-channel scale, then quantizes both to INT8:

\[ Y = \big(X \operatorname{diag}(\mathbf{s})^{-1}\big)\big(W \operatorname{diag}(\mathbf{s})\big)^\top, \qquad s_j = \frac{\max_t |X_{tj}|^{\alpha}}{\max_i |W_{ij}|^{1-\alpha}} \tag{112.4} \]
Read the symbols
\(\max_t |X_{tj}|\)
the largest activation of channel \(j\) over the calibration tokens
\(\max_i |W_{ij}|\)
the largest weight of input channel \(j\) over the output rows
\(\alpha\)
the migration strength: 0.5 splits the difficulty evenly; the SmoothQuant paper uses 0.5 for most models and more for some

FP8 E4M3 needs no smoothing in most models, because a floating-point grid is fine near zero and coarse far from it, so one outlier does not waste the resolution of the small values. One scale per row brings the largest value to 448, the largest E4M3 number:

\[ \hat{\mathbf{x}} = s \cdot \operatorname{fp8}(\mathbf{x}/s), \qquad s = \frac{\max_j |x_j|}{448} \tag{112.5} \]
Read the symbols
\(\mathbf{x}\)
one row: a token’s activations (scale computed on the fly, “dynamic”) or an output channel’s weights (scale stored)
\(\operatorname{fp8}\)
round to the nearest E4M3 number: 1 sign bit, 4 exponent bits, 3 mantissa bits, so 8 steps per power of two
Algorithm 1: GPTQ for one linear layer (one column at a time)
  1. input: weights \(W\) (\(\text{out} \times \text{in}\)), calibration inputs \(X\), bits \(b\), group size \(g\)
  2. \(H \leftarrow X^\top X / n + \lambda I\) with \(\lambda = 0.01 \cdot \operatorname{mean}(\operatorname{diag} H)\)
  3. \(U \leftarrow\) upper Cholesky factor of \(H^{-1}\) // numerically stable form of the updates
  4. for \(q = 1, \dots, \text{in}\):
  5.   if \(q\) starts a group: compute \(s, z\) of (112.1) from the current (already corrected) columns \(q..q+g-1\)
  6.   \(\hat{\mathbf{w}}_{:,q} \leftarrow Q(\mathbf{w}_{:,q})\); \(\;\mathbf{e} \leftarrow (\mathbf{w}_{:,q} - \hat{\mathbf{w}}_{:,q}) / U_{qq}\)
  7.   \(\mathbf{w}_{:,q..} \leftarrow \mathbf{w}_{:,q..} - \mathbf{e}\, U_{q,\,q..}\) // push the error onto later columns
  8. return \(\hat{W}\)

Before you quantize a real model, look at what each method does to one layer. Figure 1 builds a small layer (128 inputs, 64 outputs) whose activations are correlated, like real ones, and have four outlier channels, like real LLM activations. It applies all five methods and reports the relative output error on held-out tokens:

\[ \varepsilon = \frac{\lVert X_{\text{test}} W^\top - \widehat{X_{\text{test}} W^\top} \rVert_F^2}{\lVert X_{\text{test}} W^\top \rVert_F^2} \tag{112.6} \]
Group size
4-bit or 3-bit weights, BF16 activations8-bit weights and activations
Figure 1. One linear layer (128 inputs, 64 outputs), 256 calibration tokens and 256 test tokens, with fixed random seeds. Each bar is the relative output error (112.6) on the test tokens, on a log axis: one step to the left is 10 times less error. The methods are the ones of (112.1) to (112.5), written as in the code below.
How to read Figure 1

With the defaults (outliers 20 times normal, 4-bit, group 32, \(\alpha = 0.5\)), RTN has an error of \(7.1 \times 10^{-3}\); GPTQ cuts it about 10 times, to \(7.1 \times 10^{-4}\); AWQ (best \(\alpha = 0.5\)) to \(1.6 \times 10^{-3}\). Among the 8-bit recipes, naive W8A8 gives \(4.2 \times 10^{-4}\), SmoothQuant \(4.9 \times 10^{-5}\), and FP8 \(1.0 \times 10^{-3}\). Now set the outliers to 1 (no outliers): naive W8A8 drops to \(8.4 \times 10^{-5}\), and SmoothQuant has almost nothing left to fix. Drag them to 50: AWQ now beats GPTQ, because protecting the outlier channels is exactly its idea. Move \(\alpha\) to 0 or 1: SmoothQuant gets worse at both ends, because all the difficulty sits either in the activations or in the weights. Switch to 3 bits: every 4-bit method gets several times worse, and RTN most.

Three lessons carry over to the real model. First, GPTQ and AWQ are much better than RTN at the same bits and the same memory; the bytes are identical, only the choice of codes changes. Second, naive INT8 activations suffer from outlier channels, and smoothing removes most of that damage. Third, FP8 without any calibration lands in between: its relative step is about \(2^{-4}\) everywhere, so its error is set by the mantissa, not by outliers. These are lessons about one toy layer; the ordering on a real model can differ, which is why we measure.

Part 1, before measuring: predict memory and speed

A good engineer predicts the result before running the experiment. Then a measurement that disagrees with the prediction teaches something. Start with memory. The linear layers take \(b\) bits per weight plus the group scales; everything else stays in BF16:

\[ M_{\text{w}} = N_{\text{lin}}\,\frac{b_{\text{eff}}}{8} + 2\,(N_{\text{emb}} + N_{\text{head}} + N_{\text{norm}}), \qquad M_{\text{kv}} = B\,(T + n)\,\text{kv}_{\text{tok}}, \qquad \text{kv}_{\text{tok}} = 2 L\, n_{kv}\, d_h \cdot 2 \tag{112.7} \]
Read the symbols
\(b_{\text{eff}}\)
bits per weight including the scales: 16 for BF16; \(4 + 16/128 = 4.125\) for INT4 with one 16-bit scale per group of 128; 8 for INT8 and FP8 (one scale per output channel adds only 5.5 MB in total)
\(N_{\text{lin}}, N_{\text{emb}}, N_{\text{head}}\)
6.98 billion weights in the linear layers, 525 million in the embedding and 525 million in the output layer
\(B, T, n\)
batch size, prompt length and generated tokens
\(\text{kv}_{\text{tok}}\)
BF16 KV cache per token: 131,072 bytes (128 KiB) for this model
Worked example: the memory of the five copies

BF16: \(2 \times 8{,}030{,}261{,}248 = 16.06\) GB. W4A16 (RTN, GPTQ, AWQ): \(6.979 \times 10^9 \times 4.125/8 = 3.60\) GB for the linear layers, plus 2.10 GB for the BF16 embedding, output layer and norms, so 5.70 GB. (An asymmetric format also stores a 4-bit zero point per group: 5.73 GB.) W8A8 (SmoothQuant, FP8): \(6.98 + 2.10 = \) 9.08 GB.

The KV cache: a batch of 16 with 2,048-token prompts and 128 new tokens needs \(16 \times 2{,}176 \times 131{,}072 = 4.56\) GB. With vLLM’s default memory budget (90% of the 24 GB, so 21.6 GB), the BF16 model leaves room for about 42,000 tokens of cache, W8A8 for about 95,500, and W4A16 for about 121,000 [Certain for the arithmetic; the real number is a little lower, because the runtime also needs memory for activations and CUDA graphs]. So on an L4, quantization is first a capacity win: almost three times more concurrent context.

Now the time. Use the bound of session 82: a pass cannot be faster than its FLOPs at peak speed, nor faster than its bytes at full bandwidth. The weights are read once per pass. Only the 8-bit recipes get the faster 8-bit matrix units; W4A16 dequantizes to BF16 and uses the BF16 units:

\[ t_{\text{pre}}(B, T) \ge \max\!\left( \frac{2 N_{mm} B T}{F_{\text{fmt}}} + \frac{2 L d\, B T^2}{F_{16}},\ \ \frac{M_{\text{read}} + B T\, \text{kv}_{\text{tok}}}{\text{BW}} \right) \tag{112.8} \]
\[ t_{\text{dec}}(B, c) \ge \max\!\left( \frac{2 N_{mm} B}{F_{\text{fmt}}} + \frac{4 L d\, B c}{F_{16}},\ \ \frac{M_{\text{read}} + B c\, \text{kv}_{\text{tok}}}{\text{BW}} \right) \tag{112.9} \]
Read the symbols
\(N_{mm}\)
weights used in matmuls each pass: the linear layers plus the output layer, 7.50 billion
\(F_{\text{fmt}}\)
matmul peak of the format: 121 TFLOP/s for BF16 and W4A16, 242.5 for INT8 and FP8 on the L4
\(F_{16}\)
the BF16 peak, used for attention, which stays in BF16 in all five recipes
\(M_{\text{read}}\)
weight bytes read per pass: the linear layers in their format plus the BF16 output layer (15.01 GB, 4.65 GB, 8.03 GB)
\(c\)
context length of the decode step; we use the middle of generation, \(c = T + n/2\)
\(\text{BW}\)
memory bandwidth, 300 GB/s
Worked example: one request of 512 prompt tokens and 128 new tokens on an L4

Prefill, batch 1. FLOPs: \(2 \times 7.50 \times 10^9 \times 512 = 7.69\) TFLOP plus a little attention. In BF16 that takes 64.1 ms at 121 TFLOP/s; reading 15 GB of weights takes 50 ms. Compute wins, so BF16 prefill is about 64 ms. W4A16 has the same FLOPs on the same units: also 64 ms, no speed-up. W8A8 doubles the matmul peak: 32 ms, a speed-up of 1.99.

Decode, batch 1, at context 576. Bytes: 15.01 GB of weights plus 75 MB of cache. At 300 GB/s that is 50.3 ms per token in BF16, about 20 tokens per second. W4A16 reads 4.65 GB: 15.7 ms, a speed-up of 3.19. W8A8 reads 8.03 GB: 27.0 ms, 1.86. The FLOPs are tiny (15 GFLOP, 0.12 ms), so decode is memory-bound by a factor of about 400.

Batch 16. Prefill time grows 16 times (1.03 s in BF16; 0.52 s in W8A8). Decode grows only from 50.3 to 54.1 ms in BF16, because the weights are shared; W4A16 still wins, 19.5 ms, a speed-up of 2.77. The speed-up shrinks because the KV cache, which W4A16 does not compress, is now a larger part of the bytes. With 2,048-token prompts at batch 16 it drops to 2.14.

One surprise hides in the short prompts. At 128 tokens, BF16 prefill is memory-bound on an L4 (50 ms to read the weights, 16 ms of math), because 128 is below the ridge of 403. So W4A16 does speed up a short prefill there, by up to 3.15 in the bound. On an H100, with a ridge of about 295, the same holds only for even shorter prompts. Figure 2 lets you explore all cases.

BF16W4A16 (RTN, GPTQ, AWQ)W8A8 (SmoothQuant INT8, FP8)
Figure 2. Predicted memory (112.7), prefill time (112.8) and decode time per token (112.9) of Llama 3.1 8B on an L4 (24 GB, 300 GB/s, 121 TFLOP/s BF16, 242.5 TFLOP/s INT8/FP8). These are best-case bounds from the spec sheet, not measurements. Bars are scaled to the largest value in each panel. RTN, GPTQ and AWQ share one bar: they produce the same format, so they have the same speed.
How to read Figure 2

With the defaults (batch 1, 512-token prompt, 128 new tokens), W4A16 cuts decode from 50.3 to 15.7 ms per token but leaves prefill at 64.1 ms; W8A8 halves prefill (32.3 ms) and cuts decode to 27.0 ms. Lower the prompt to 128: BF16 prefill becomes memory-bound and W4A16 helps prefill too. Raise the batch to 16 and the prompt to 2,048: the W4A16 decode advantage drops to 2.14, because the uncompressed KV cache grows to 4.56 GB. Raise the batch to 64 and the prompt to 1,024: the BF16 copy needs 25.7 GB and no longer fits in 21.6 GB, while W8A8 (18.7 GB) and W4A16 still do. The readout prints every number and says which bound (compute or memory) is active.

Part 2: quantize, serve and measure on the L4

The tools: LLM Compressor (from the vLLM project) applies the recipes and saves checkpoints that vLLM loads directly; lm-evaluation-harness measures quality through vLLM; vLLM’s benchmark command measures latency. TensorRT-LLM supports the same formats and is the other common choice on NVIDIA GPUs.

Algorithm 2: one fair comparison of quantization recipes
  1. input: base model \(M\), recipes \(R_1..R_5\), calibration set \(C\) (same 512 sequences for all), GPU, workloads \(\{(B, T, n)\}\)
  2. for each recipe \(R\): \(M_R \leftarrow\) quantize\((M, R, C)\); save; record time and peak memory
  3. for each model in \(\{M, M_{R_1}, \dots\}\): // same engine, same flags, same GPU for all
  4.   quality: WikiText-2 perplexity, ARC-Challenge accuracy, GSM8K exact match (same prompts, same few-shot, greedy)
  5.   for each \((B, T)\) in \(\{1, 16\} \times \{128, 512, 2048\}\):
  6.     \(t_1 \leftarrow\) latency of \(n = 1\) (prefill and one token); \(\;t_n \leftarrow\) latency of \(n = 128\)
  7.     TTFT \(\approx t_1\); \(\;\)TPOT \(\leftarrow (t_n - t_1)/(n - 1)\) // formula (112.10)
  8.   record weight memory and the KV-cache capacity that the engine reports
  9. report every number next to its prediction from (112.7) to (112.9); explain each gap larger than 30%

Step 1: make the five checkpoints

The script below is written for LLM Compressor and vLLM as of late 2026. Their module paths change between versions; if an import fails, copy the import line from the examples folder of your installed version. Quantizing an 8B model needs the BF16 model in memory: on a 24 GB L4, LLM Compressor moves layers between CPU and GPU, so give the machine at least 64 GB of CPU memory [Likely]. We did not run this script.

# quantize.py: five copies of Llama 3.1 8B Instruct (LLM Compressor; not run by us)
import sys
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.awq import AWQModifier
from llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier

BASE = "meta-llama/Llama-3.1-8B-Instruct"
SKIP = ["lm_head"]                                  # keep the output layer in BF16
RECIPES = {
    "rtn-w4a16":  [QuantizationModifier(targets="Linear", scheme="W4A16", ignore=SKIP)],
    "gptq-w4a16": [GPTQModifier(targets="Linear", scheme="W4A16", ignore=SKIP)],
    "awq-w4a16":  [AWQModifier(), QuantizationModifier(targets="Linear", scheme="W4A16_ASYM", ignore=SKIP)],
    "sq-w8a8":    [SmoothQuantModifier(smoothing_strength=0.8),
                   QuantizationModifier(targets="Linear", scheme="W8A8", ignore=SKIP)],
    "fp8-dynamic":[QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=SKIP)],
}
name = sys.argv[1]
model = AutoModelForCausalLM.from_pretrained(BASE, dtype="auto")
tok = AutoTokenizer.from_pretrained(BASE)
needs_data = name in ("gptq-w4a16", "awq-w4a16", "sq-w8a8")
oneshot(model=model, recipe=RECIPES[name],
        **(dict(dataset="ultrachat_200k", splits="train_sft[:512]", max_seq_length=2048,
                num_calibration_samples=512) if needs_data else {}))
model.save_pretrained(f"ckpt/{name}", save_compressed=True); tok.save_pretrained(f"ckpt/{name}")

Two choices in this script deserve a sentence. SmoothQuant with RTN, not with GPTQ: the public W8A8 recipes stack SmoothQuant and GPTQ, which is usually a little better, but then you compare two methods at once. Here we want to see what smoothing alone buys. The same calibration data for all: 512 chat sequences of up to 2,048 tokens. Calibration data that does not look like your traffic is a common reason for a disappointing GPTQ or AWQ result.

Step 2: quality

# perplexity on WikiText-2 and two tasks, through vLLM (one model at a time)
for m in meta-llama/Llama-3.1-8B-Instruct ckpt/rtn-w4a16 ckpt/gptq-w4a16 ckpt/awq-w4a16 ckpt/sq-w8a8 ckpt/fp8-dynamic; do
  lm_eval --model vllm --model_args pretrained=$m,dtype=auto,max_model_len=4096,gpu_memory_utilization=0.9 \
          --tasks wikitext,arc_challenge,gsm8k --batch_size auto --seed 0 \
          --output_path results/$(basename $m)
done

Perplexity is the exponential of the mean negative log-likelihood per token:

\[ \text{PPL} = \exp\!\Big(-\frac{1}{N}\sum_{i=1}^{N} \log p_\theta(x_i \mid x_{\lt i})\Big), \qquad \text{TPOT} \approx \frac{t(n) - t(1)}{n - 1}, \qquad \text{TTFT} \approx t(1) \tag{112.10} \]
Read the symbols
\(N\)
number of scored tokens (or words: the harness’s wikitext task reports perplexity per word, which is a larger number than per token; compare only numbers of the same kind)
\(t(n)\)
measured end-to-end latency of one batch that generates \(n\) tokens per sequence after the prompt

Perplexity is a sensitive smoke test: a broken quantization raises it a lot. But small perplexity differences do not predict small task differences, and the reverse. GSM8K (grade-school math with chain-of-thought, scored by exact match of the final number) is a good second test because long reasoning chains amplify small errors; ARC-Challenge (multiple-choice science questions) checks knowledge with little generation. Note that the instruct model is usually scored on these tasks with its chat template; use the same flags for all six models, or the comparison is meaningless.

Step 3: latency at batch 1 and batch 16

# prefill and decode latency, measured separately (vLLM; not run by us)
for m in ...; do                      # the same six models
  for B in 1 16; do for T in 128 512 2048; do
    vllm bench latency --model $m --batch-size $B --input-len $T --output-len 1   --num-iters 20 \
         --output-json lat/$(basename $m)-$B-$T-1.json      # t(1): TTFT
    vllm bench latency --model $m --batch-size $B --input-len $T --output-len 128 --num-iters 20 \
         --output-json lat/$(basename $m)-$B-$T-128.json    # t(128)
  done; done
done
# then: TPOT = (t(128) - t(1)) / 127, with the median latencies from the two JSON files

Why two runs instead of one? A single end-to-end time mixes the prefill with 127 decode steps. The difference of two runs with the same prompt removes the prefill and leaves the decode steps, so you get both phases cleanly, as the survey’s table does. The benchmark warms up and repeats for you; keep everything else fixed (CUDA graphs on, same max_model_len, same vLLM version) and write the versions into your report.

A rehearsal on a laptop: the quality ordering

Before you pay for GPU hours, rehearse the quality side on a small model. We applied the five recipes to Qwen2.5-0.5B (24 layers, width 896) in PyTorch, as fake quantization: each weight (and, for the 8-bit recipes, each activation) is rounded to the target grid and stored back in FP32, so the arithmetic is exactly that of the quantized model, but nothing gets faster (session 111 used the same trick). We scored perplexity on the WikiText-2 test set (the raw version, 146 windows of 2,048 tokens, 298,862 scored tokens) on an Apple laptop. The calibration set for GPTQ, AWQ and SmoothQuant was 32 random windows of 512 tokens from the WikiText-2 training set, the same for all three. [Certain, our run]

Our versions are simplified, and you should know where: GPTQ is the algorithm of the paper (blocks of 128 columns, 1% damping, group 128, asymmetric) without act-order; AWQ is the scale search of (112.3) per linear layer, without the weight-clipping search of the paper; SmoothQuant uses \(\alpha = 0.5\) and smooths all seven linear layers (a real deployment can fold the scale only where a norm or a linear layer comes before, so it usually smooths the QKV and the gate-up projections only); FP8 uses dynamic per-token and per-channel scales.

Show
Figure 3. Our rehearsal: WikiText-2 test perplexity (per token, 2,048-token windows) of Qwen2.5-0.5B with each recipe, simulated in PyTorch (fake quantization). The third view shows what the same formats would weigh for Llama 3.1 8B, from (112.7). Small models are more sensitive to quantization than 8B models, so expect smaller gaps on the real model [Likely].
How to read Figure 3

The bars are sorted by format: the 4-bit weight recipes on the left, the 8-bit recipes on the right. Compare within a format first: GPTQ and AWQ should sit below RTN at the same 4 bits. Then compare across formats with the third view: the 4-bit models are 37% smaller than the 8-bit ones, so a 4-bit recipe that comes close to the 8-bit quality is the better deal for decode-heavy traffic. Click a bar for its exact numbers.

What the rehearsal says. [Certain, our run] The BF16 model scores 13.07. At 4 bits with group 128, RTN raises perplexity to 15.54 (+18.9%), AWQ to 14.73 (+12.7%) and GPTQ to 14.19 (+8.6%): GPTQ removes 55% of the RTN damage, AWQ 33%. At 8 bits, naive INT8 gives 13.33 (+2.0%), SmoothQuant 13.21 (+1.1%, removing 45% of the naive damage) and FP8 13.26 (+1.4%) without any calibration. So the ordering of Figure 1 survives on a real model: within each format, the calibrated methods win, and the 8-bit recipes are much closer to BF16 than the 4-bit ones. Our AWQ is weaker than the paper’s, since it has no clipping search and chose a mean \(lpha\) of only 0.33; on an 8B model with the full method, expect AWQ and GPTQ to be close.

Two practical numbers from the same run: quantizing took about 5 minutes for GPTQ, 3.4 minutes for AWQ and 2.3 minutes for SmoothQuant (statistics plus rounding, on a shared laptop CPU), and seconds for RTN and FP8. On an 8B model expect tens of minutes for GPTQ and AWQ on one GPU [Likely]. Quantization is a one-time cost, but it is part of your release pipeline, so write it down.

The code of Figure 1, runnable

This file reproduces the one-layer experiment in PyTorch. It is deterministic (fixed seeds, float64) and runs on a CPU in about a second. Its random numbers come from PyTorch, not from the figure’s generator, so its errors differ a little from Figure 1, but the ordering is the same. We ran it; its output follows.

import torch
torch.manual_seed(0)
torch.set_default_dtype(torch.float64)
D_IN, D_OUT, N = 128, 64, 256
OUT = [7, 33, 70, 101]                       # outlier input channels

def acts(n, mix):                            # correlated activations + outliers
    x = torch.randn(n, 8) @ mix + 0.3 * 8 ** 0.5 * torch.randn(n, D_IN)
    x[:, OUT] *= 20
    return x

W = 0.05 * torch.randn(D_OUT, D_IN)
mix = torch.randn(8, D_IN)
Xc, Xt = acts(N, mix), acts(N, mix)          # calibration set, test set
Y = Xt @ W.T

def q_group(w, bits=4, gs=32):               # asymmetric RTN, one scale per group (112.1)
    o, i = w.shape
    g = w.reshape(o, i // gs, gs)
    lo = g.amin(-1, keepdim=True).clamp(max=0)
    hi = g.amax(-1, keepdim=True).clamp(min=0)
    s = ((hi - lo) / (2 ** bits - 1)).clamp(min=1e-12)
    z = torch.round(-lo / s)
    q = (torch.round(g / s) + z).clamp(0, 2 ** bits - 1)
    return ((q - z) * s).reshape(o, i)

def q_rows(a, bits=8):                       # symmetric INT8, one scale per row
    s = a.abs().amax(-1, keepdim=True) / (2 ** (bits - 1) - 1)
    return torch.round(a / s).clamp(-127, 127) * s

def gptq(w, x, bits=4, gs=32):               # Algorithm 1, formula (112.2)
    H = x.T @ x / len(x)
    H += 0.01 * H.diag().mean() * torch.eye(len(H))
    U = torch.linalg.cholesky(torch.linalg.inv(H)).T   # upper factor of H^-1
    w, Q = w.clone(), torch.zeros_like(w)
    for i in range(w.shape[1]):
        if i % gs == 0:                      # group scale from the updated weights
            g = w[:, i:i + gs]
            lo, hi = g.amin(1).clamp(max=0), g.amax(1).clamp(min=0)
            s = (hi - lo) / (2 ** bits - 1); z = torch.round(-lo / s)
        q = ((torch.round(w[:, i] / s) + z).clamp(0, 2 ** bits - 1) - z) * s
        Q[:, i] = q
        err = (w[:, i] - q) / U[i, i]
        w[:, i:] -= err[:, None] * U[i, i:][None, :]   # push the error to later columns
    return Q

def awq(w, x, bits=4, gs=32):                # grid search of (112.3)
    m, ref, best = x.abs().mean(0), x @ w.T, None
    for k in range(20):
        s = m ** (k / 20); s = s / (s.max() * s.min()).sqrt()
        wq = q_group(w * s, bits, gs) / s     # scale up, quantize, scale back
        e = ((x @ wq.T - ref) ** 2).sum().item()
        if best is None or e < best[0]: best = (e, k / 20, wq)
    return best[2], best[1]

def smooth(w, x, alpha=0.5):                 # SmoothQuant scale (112.4)
    return x.abs().amax(0) ** alpha / w.abs().amax(0) ** (1 - alpha)

def fp8(a):                                  # per-row scale, then E4M3 rounding (112.5)
    s = a.abs().amax(-1, keepdim=True) / 448
    return (a / s).to(torch.float8_e4m3fn).to(a.dtype) * s

rel = lambda yh: (((yh - Y) ** 2).sum() / (Y ** 2).sum()).item()
Wa, alpha = awq(W, Xc)
s = smooth(W, Xc)
print(f"RTN  W4 g32     {rel(Xt @ q_group(W).T):.2e}")
print(f"GPTQ W4 g32     {rel(Xt @ gptq(W, Xc).T):.2e}")
print(f"AWQ  W4 g32     {rel(Xt @ Wa.T):.2e}  (alpha={alpha})")
print(f"W8A8 naive      {rel(q_rows(Xt) @ q_rows(W).T):.2e}")
print(f"SmoothQuant     {rel(q_rows(Xt / s) @ q_rows(W * s).T):.2e}")
print(f"FP8 E4M3        {rel(fp8(Xt) @ fp8(W).T):.2e}")
RTN  W4 g32     5.77e-03
GPTQ W4 g32     5.09e-04
AWQ  W4 g32     1.67e-03  (alpha=0.35)
W8A8 naive      3.97e-04
SmoothQuant     5.27e-05
FP8 E4M3        7.45e-04

Part 3: compare with the survey’s table

Zhou et al. measured what W4A16 buys on a real GPU (their Table IV, in section 5.2.1). They quantized Llama-2-7B and Llama-2-13B to 4-bit weights with AWQ, served them with TensorRT-LLM and with LMDeploy on one A100, and reported the speed-up over FP16 of three latencies, prefill / decode / end-to-end, for batch sizes 1 to 16 and prompt lengths 128 to 2,048. [Certain] Their four observations, in short: (1) weight-only quantization speeds up decode a lot, and so end-to-end latency; (2) it can make prefill slower, because prefill is compute-bound and the weights must be dequantized; (3) the gain shrinks as batch and prompt length grow; (4) larger models gain more.

Your job is to predict their table from (112.9) and explain the gap. For decode, the bound is a ratio of bytes, so the bandwidth cancels:

\[ \text{speed-up}_{\text{dec}} \le \frac{2 N_{mm} + B c\, \text{kv}_{\text{tok}}}{N_{\text{lin}} \cdot 4.125/8 + 2 N_{\text{head}} + B c\, \text{kv}_{\text{tok}}} \tag{112.11} \]

Llama-2-7B has no grouped-query attention, so its cache is large: 512 KiB per token, four times that of Llama 3.1 8B. At batch 1 and 128 tokens the bound is 3.62; the table says 2.40 (TensorRT-LLM) and 2.11 (LMDeploy), so the kernels reach 58 to 66% of the ideal. At batch 16 and 2,048 tokens the bound has fallen to 1.46, because 17.2 GB of FP16 cache now dominate the 13.2 GB of FP16 weights, and the table says 1.16 and 1.28. The trend of observation (3) is entirely in the formula; the level is the kernels.

EngineModelPhase
Figure 4. The survey’s Table IV (measured on one A100, AWQ W4A16 against FP16), one cell per batch size and prompt length. The big number is the measured speed-up; the small number below is our bound: (112.11) for decode, and (112.8) with A100 80GB SXM spec numbers (312 TFLOP/s, 2.039 TB/s) for prefill. Click a cell for the details. “OOM”: the run did not fit in GPU memory.
How to read Figure 4

Start with decode, TensorRT-LLM, 7B. Along the top row (batch 1) the measured speed-up falls slowly from 2.40 to 2.00 as the prompt grows; down the left column (128 tokens) it falls faster, to 1.43 at batch 8. The bounds fall the same way, from 3.62 to 1.46. Click any cell: the readout gives the ratio measured / bound: between 0.43 and 0.79 for TensorRT-LLM on the 7B model, lowest at batch 4 to 16 with short prompts. Now switch to prefill: the measured values are close to 1 or below it, while the bound says 1.0 (compute-bound) except at batch 1 and 128 tokens, where it allows 1.20. Switch to 13B: decode speed-ups are a little higher, as observation (4) says. Two oddities are in the source table, not in our figure: the batch-8 and batch-16 rows of Llama-2-7B are identical in both engines, and some decode speed-ups are lower than end-to-end ones for 13B, which cannot happen if end-to-end is a mix of prefill and decode. Treat them as likely copy errors [Likely].

What should you expect on an L4? Run your own version of this table with the latency script above. The bound of (112.9) for Llama 3.1 8B on an L4 is 3.19 at batch 1 and 512 tokens and 2.14 at batch 16 and 2,048 tokens. If your kernels reach the same 45 to 80% of the bound as the A100 runs, you would measure roughly 1.5 to 2.5 at batch 1 [Speculation, until you measure]. Two differences point in opposite directions: Llama 3.1 has a cache four times smaller per token, which keeps the decode bound high at large batches; and the L4 has much less bandwidth, so every byte saved is worth more milliseconds.

Published quality for Llama 3.1 8B

You will compare your quality numbers with public quantized checkpoints of the same model. Their makers (Neural Magic, now Red Hat AI) report scores from their own fork of lm-evaluation-harness with the chat template, so use the same settings if you want to compare exact numbers. [Certain, quoted from the three model cards]

Checkpoint (recipe)ARC-Challenge, 0-shotGSM8K, CoT 8-shotMMLU, 5-shot
BF16 baseline (as reported on the W4A16 and W8A8 cards)81.482.868.3
W4A16, GPTQ, symmetric, group 128, act-order80.2 (98.0%)82.9 (100.2%)66.9 (97.9%)
W8A8 INT8 (GPTQ, per-channel W, dynamic per-token A)81.7 (100.3%)84.8 (102.5%)67.8 (99.3%)
FP8, static per-tensor scales (its own baseline: 82.00, 81.96, 67.95)81.66 (99.6%)81.12 (99.0%)67.97 (100.0%)

Read this table with care. Differences of one point are within the noise of these benchmarks (ARC-Challenge has 1,172 test questions, so one point is about 12 questions; GSM8K has 1,319), and a quantized model “beating” the baseline on GSM8K is noise, not magic. The real message: at 8 bits all recipes are lossless within noise; at 4 bits a good recipe (GPTQ with group 128) loses one to two points on knowledge tasks. Your lab should confirm or refute this for RTN and AWQ, which these cards do not cover.

Part 4: the one-page recommendation

A results table is not a decision. The last step turns it into one page that a team lead can read in three minutes. Use this structure:

Algorithm 3: from a results table to a recommendation
  1. input: the workload (prompt and output lengths, batch or concurrency, TTFT and TPOT targets), the quality bar (which tasks, how many points of loss are acceptable), the GPU
  2. drop every recipe that misses the quality bar on your tasks // quality first: speed cannot fix wrong answers
  3. drop every recipe that does not fit the memory needed for the target concurrency (112.7)
  4. split the time per request into prefill and decode: \(t \approx \text{TTFT} + (n-1)\,\text{TPOT}\)
  5. if decode dominates (short prompts, long answers, small batches): prefer W4A16 (the best of GPTQ and AWQ)
  6. if prefill dominates (long prompts, RAG, short answers, large batches): prefer W8A8 (FP8 where the GPU supports it, else SmoothQuant INT8)
  7. write: the choice, the numbers that decided it, the risks, and the test that would change the decision
Worked example: an example recommendation, from the predictions

Workload. A support chatbot: prompts of about 512 tokens, answers of about 256 tokens, up to 16 users at once per GPU, TPOT target 40 ms.

Time per request at batch 16 (bounds of Figure 2, with \(n = 256\)): BF16 \(1.03\) s prefill + \(255 \times 54.5\) ms decode = 14.9 s; W4A16 \(1.03 + 255 \times 20.0\) ms = 6.1 s; W8A8 \(0.52 + 255 \times 31.2\) ms = 8.5 s. Decode is 93% of the BF16 time, so the byte-saving recipe wins. BF16 also misses the TPOT target (54.5 ms), the other two meet it.

Recommendation (to be confirmed by your L4 measurements): serve GPTQ or AWQ W4A16, whichever scores better on GSM8K and on 200 real support conversations; keep the FP8 copy as the fallback if the 4-bit model loses more than 2 points on any task that matters. Risks: W4A16 kernels may be slower than predicted at batch 16; long-tail prompts of several thousand tokens shift the balance toward FP8. What would change the decision: a measured W4A16 TPOT above 40 ms at batch 16, or a quality loss above the bar.

Your page should have five short blocks: the decision in one sentence; the table of measured numbers (quality, memory, TTFT and TPOT at batch 1 and 16) with the predictions next to them; why the winner wins, in terms of bytes and FLOPs; risks and open questions; and the next experiment. Keep the raw logs and versions in an appendix, not on the page.

Where this is used

WhereTypical choiceWhy
Chat on small GPUs (L4, A10G, consumer cards)W4A16 with GPTQ or AWQDecode-heavy, batch small: bytes decide; 8B fits with room for cache
High-throughput serving on H100 or newerFP8 W8A8 (often with an FP8 KV cache)Large batches make linear layers compute-bound; FP8 doubles matmul speed with little loss [Likely]
GPUs without FP8 (A100, older)SmoothQuant INT8 for throughput, W4A16 for latencyINT8 units exist; FP8 does not
Long-prompt workloads: RAG, document QA, Video-RAGW8A8Prefill dominates; thousands of visual or retrieved tokens per request make it compute-bound
Video understanding with a VLM (video QA, key-moment detection)W8A8 for the language model; vision encoder often kept in BF16 or FP8Many visual tokens per frame make prefill the cost; the encoder is small and sensitive [Likely]
On-device (laptops, phones)4-bit weights, often 3 or 4 bits with groupsMemory size and bandwidth are the limits; batch is 1
Model releases on model hubsSeveral checkpoints (W4A16, W8A8, FP8) with recovery tablesUsers pick by GPU and workload, as in this lab

Common mistakes

  • Comparing recipes with different calibration data or sequence lengths. Then you compare datasets, not methods. Use the same calibration set for GPTQ, AWQ and SmoothQuant, and say what it is.
  • Reporting only perplexity. A 0.1 perplexity difference can hide a 3-point drop on GSM8K, or the opposite. Always add at least one task that generates long answers.
  • Mixing word and token perplexity, or different window lengths. lm-evaluation-harness reports word perplexity for wikitext; many papers report token perplexity with 2,048-token windows. The numbers are not comparable.
  • One end-to-end latency instead of two phases. W4A16 can be 3 times faster in decode and 10% slower in prefill; the end-to-end number hides both. Measure \(t(1)\) and \(t(n)\).
  • Measuring only batch 1. The W4A16 advantage shrinks with batch and context (Figure 4), and production runs at batch 16 or more.
  • Expecting W4A16 to speed up prefill. It uses the same BF16 matrix units plus dequantization work. Only 8-bit activations make long prefills faster.
  • Forgetting the KV cache. None of the five recipes compresses it. At large batch and long context it becomes the main cost of a decode step; that is the topic of session 110.
  • Comparing engines instead of recipes. Run all six models in the same engine version with the same flags; a kernel change between versions can be larger than the difference between two recipes.

Interview questions

Why does 4-bit weight-only quantization speed up decode but not prefill?basic
Decode processes one token per sequence per step, so its intensity is about \(B\) FLOPs per weight read: far below the ridge of any GPU. Its time is the bytes divided by bandwidth, and W4A16 reads about 3.2 times fewer bytes for the linear layers of an 8B model (15.0 GB against 4.65 GB with the BF16 output layer). Prefill processes hundreds or thousands of tokens per weight read, so it is compute-bound, and W4A16 does not change the FLOPs: the kernels dequantize to BF16 and use the same BF16 matrix units. The dequantization even adds work, which is why the survey measured prefill speed-ups below 1 in most cells.
What is the weight memory of Llama 3.1 8B in BF16, W4A16 (group 128) and W8A8, and why is W4A16 not exactly a quarter?basic
BF16: \(2 \times 8.03 \times 10^9 = 16.06\) GB. W4A16: the 6.98 billion linear weights take \(4.125/8\) bytes each (4 bits plus a 16-bit scale per 128 weights), 3.60 GB, and the embedding, output layer and norms stay in BF16, 2.10 GB: 5.70 GB, about 35% of BF16, not 25%. W8A8: \(6.98 + 2.10 = 9.08\) GB. The un-quantized embedding and output layer are large here because the vocabulary has 128,256 tokens: they hold 13% of the parameters.
How do GPTQ and AWQ differ, and when would you expect each to win?core
Both produce the same INT4 format. GPTQ quantizes one column at a time and corrects the remaining columns with the inverse Hessian \(H = X^\top X\) to minimize the layer’s output error (112.2); it uses second-order information about all channels. AWQ only rescales input channels before rounding, with a scale from the mean activation magnitude and one exponent found by grid search (112.3); it protects the few salient channels and has no per-weight updates, so it is faster, needs less calibration data and overfits less to it. In our toy layer GPTQ won with moderate outliers and AWQ with very large ones. On real models both are close at 4 bits with group 128; GPTQ can overfit to calibration data that differs from the test domain, AWQ is more robust to that [Likely]. Measure on your tasks.
Why does SmoothQuant help INT8 activations, and what does the \(\alpha\) parameter do?core
LLM activations have a few channels with values 10 to 100 times larger than the rest. With one INT8 scale per token, the outliers set the scale and the normal channels get only a few integer levels. SmoothQuant divides each activation channel by \(s_j\) and multiplies the matching weight column by \(s_j\) (112.4); the product is unchanged, but the activation range becomes flat and the weights, which are easy to quantize, absorb the range. \(\alpha\) sets how much moves: 0 leaves the activations as they are, 1 moves all the range into the weights, which then get outlier columns. Values around 0.5 to 0.8 work; in our toy layer \(\alpha = 0.5\) cut the error 8.6 times compared with naive W8A8.
Why does FP8 often need no smoothing, while INT8 does?core
INT8 has uniform steps: after scaling the largest value to 127, a value 50 times smaller lands on only about 2 or 3 levels. E4M3 has steps proportional to the magnitude (3 mantissa bits, so at most about 6% relative error for any value between \(2^{-6}\) and 448), so small values keep their relative precision even when one outlier sets the scale. The cost is a larger error for the typical value than a well-scaled INT8 grid (6% against less than 1%), which is why SmoothQuant INT8 beat FP8 in our toy layer. In practice FP8 with per-tensor or per-channel scales is close to lossless for 8B models, which the public recovery tables show [Certain for the cited cards].
You measure a W4A16 decode speed-up of 1.4 at batch 16 with 2,048-token prompts. Is something wrong?advanced
Not necessarily. Compute the bound (112.11). For Llama-2-7B the KV cache at that point is 17.2 GB in FP16 against 13.2 GB of weights, and W4A16 does not compress the cache, so the bound is only 1.46; the survey measured 1.16 to 1.28. For Llama 3.1 8B with grouped-query attention the cache is four times smaller and the bound is 2.14, so 1.4 would be 65% of the bound, a typical kernel efficiency. Then check: is the W4A16 kernel compute-limited at that batch (dequantization work)? Is the cache in BF16? Quantizing the KV cache to FP8 would raise the bound.
Why do you measure TPOT as \((t(128) - t(1))/127\) instead of dividing one end-to-end time by the number of tokens?basic
Because the end-to-end time includes the prefill, which can be longer than many decode steps (1 s for 16 prompts of 512 tokens on an L4 in BF16). Dividing by the tokens spreads the prefill over the decode steps and makes TPOT depend on the prompt length and batch in a misleading way. The difference of two runs with the same prompt cancels the prefill and the fixed overheads, and leaves the decode steps. It is still an average over contexts from \(T\) to \(T + 127\), which is fine because decode time changes slowly with context.
Which recipe would you choose for a RAG service on L4s with 4,000-token prompts and 100-token answers?advanced
First compute both phases, because the answer depends on the batch. 4,000 tokens is 10 times the L4 ridge, so prefill is compute-bound: about 0.53 s per request in BF16 by the bound (112.8). At batch 1, 100 decode steps take about 5.2 s, so decode dominates and W4A16 wins: \(0.53 + 1.73 = 2.3\) s per request against \(0.28 + 2.85 = 3.1\) s for W8A8. At batch 16, prefill is 8.5 s per batch against 7.8 s of decode; W8A8 halves the prefill and gives \(4.5 + 5.5 = 10.0\) s, W4A16 gives \(8.5 + 4.4 = 12.9\) s, so FP8 (the L4 supports it) wins. So: FP8 for a high-concurrency service, W4A16 for a low-concurrency one, and measure both near the crossover. Also check memory: \(16 \times 4{,}100\) tokens \(\times\) 128 KiB = 8.6 GB of cache, which does not fit next to the BF16 weights in vLLM’s default 21.6 GB but fits with either quantized copy.
Your 4-bit model has the same perplexity as BF16 within 2%, but users complain about math answers. What happened, and how would you check?core
Perplexity averages over all tokens; most are easy and barely change. Multi-step reasoning depends on a few critical tokens, and an error there ruins the whole answer, so task accuracy can fall much more than perplexity suggests. Check with a generative task such as GSM8K with chain-of-thought, scored by exact match, and on a sample of real user prompts. Remedies: a better recipe (GPTQ or AWQ instead of RTN, smaller groups), calibration data that contains math and chat, keeping sensitive layers in higher precision, or moving to 8 bits.
Why is the L4 a harder test for these recipes than an H100?advanced
Its ratio of compute to bandwidth is higher: 121 TFLOP/s BF16 over 300 GB/s gives a ridge of 403 FLOPs per byte (808 for FP8), against about 295 for the H100. So more workloads are memory-bound on an L4, including short prefills, and the byte savings of W4A16 matter more; at the same time its 24 GB make capacity tight, so the memory savings of all recipes decide how much batch you can run. A recipe ranking measured on an H100 does not transfer automatically.

Exercises

  1. Compute the weight memory of Llama 3.1 8B in W4A16 with group size 32 instead of 128 (one 16-bit scale per group, symmetric). How much does it grow?
  2. Using (112.9), compute the decode time per token at batch 1 and context 576 for a W3A16 copy (3 bits, group 128) on an L4. What speed-up over BF16 does the bound allow?
  3. In Figure 1, set the outliers to 1. Why does SmoothQuant barely improve on naive W8A8 there, and why is GPTQ still much better than RTN?
  4. From the survey’s table (Figure 4), take TensorRT-LLM, Llama-2-7B, batch 1, 128 tokens. Compute the ratio of the measured decode speed-up to the bound (112.11). Repeat for batch 16 and 2,048 tokens. What does the trend tell you?
  5. At batch 16 with 2,048-token prompts and 128 new tokens, how much GPU memory does each of the three formats need (weights plus KV cache)? Which fit in vLLM’s default budget of 21.6 GB on an L4? Up to what batch does the BF16 copy fit?
  6. Run the lab (or, without an L4, the rehearsal) and write the one-page recommendation for a workload of your choice. Include at least one number that disagreed with the prediction and your explanation.
Answerscheck
1. Bits per weight become \(4 + 16/32 = 4.5\): \(6.979 \times 10^9 \times 4.5/8 = 3.93\) GB, plus 2.10 GB, so 6.03 GB instead of 5.70 GB: 0.33 GB (6%) more. 2. Linear layers: \(6.979 \times 10^9 \times 3.125/8 = 2.73\) GB; plus 1.05 GB output layer and 75.5 MB of cache: 3.85 GB, so 12.8 ms per token; speed-up \(50.3 / 12.8 = 3.92\) (against 3.19 for 4 bits). 3. Without outliers the activations already have a flat range, so naive per-token INT8 is accurate (\(8.4 \times 10^{-5}\)) and there is little to smooth (\(7.2 \times 10^{-5}\)). GPTQ still wins over RTN (\(1.1 \times 10^{-3}\) against \(7.1 \times 10^{-3}\)) because its benefit comes from correlations between input channels, which the Hessian captures, not from outliers; AWQ, whose benefit does come from salient channels, loses most of its advantage (\(5.4 \times 10^{-3}\)). 4. \(2.40 / 3.62 = 0.66\); \(1.16 / 1.46 = 0.79\). The ratio is not constant: over the table it goes from 0.43 (batch 8, 128 tokens) to 0.79. It is highest when the cache traffic is a large part of the step, because that part is the same attention kernel in both models and runs at the same speed; it is lowest at medium batch with short prompts, where the step is mostly the W4A16 matmul, whose unpack and dequantize work grows with the batch. So the gap to the bound is a property of the 4-bit matmul kernel. 5. KV: \(16 \times 2{,}176 \times 131{,}072 = 4.56\) GB. BF16: 20.62 GB, W8A8: 13.64 GB, W4A16: 10.26 GB; all three fit in 21.6 GB, BF16 only just. BF16 leaves room for \(42{,}262\) tokens of cache, so for 2,176 tokens per sequence the largest batch is 19. 6. Open.

Key takeaways

  • RTN, GPTQ and AWQ make the same INT4 format (5.70 GB for Llama 3.1 8B with group 128); they differ only in which codes they pick. GPTQ corrects errors with the Hessian, AWQ protects salient channels. Both beat RTN clearly.
  • SmoothQuant and FP8 quantize activations too (9.08 GB). SmoothQuant moves outlier range from activations to weights; FP8’s floating-point grid tolerates outliers by itself.
  • Predict before measuring. On an L4, W4A16 should cut decode from 50.3 to 15.7 ms per token at batch 1 and leave long prefills unchanged; W8A8 should halve prefill and cut decode to 27.0 ms.
  • The W4A16 decode advantage shrinks with batch and context because the KV cache is not compressed; the survey’s A100 table shows this trend, at about 45 to 80% of the byte bound.
  • Measure quality with perplexity and generative tasks, and latency as two phases (\(t(1)\) and \(t(n)\)) at batch 1 and 16, with the same engine, data and flags for every model.
  • Decide from the workload: decode-heavy traffic favors W4A16, prefill-heavy or high-batch traffic favors W8A8, and the quality bar on your own tasks comes first.

Key formulas

\[ \hat{w} = s\big(\operatorname{clamp}(\lfloor w/s \rceil + z, 0, 2^b - 1) - z\big), \qquad Y = (X\operatorname{diag}(\mathbf{s})^{-1})(W\operatorname{diag}(\mathbf{s}))^\top \]
\[ M_{\text{w}} = N_{\text{lin}} \frac{b_{\text{eff}}}{8} + 2(N_{\text{emb}} + N_{\text{head}}), \qquad M_{\text{kv}} = B(T+n)\,\text{kv}_{\text{tok}} \]
\[ t_{\text{dec}} \ge \frac{M_{\text{read}} + Bc\,\text{kv}_{\text{tok}}}{\text{BW}}, \qquad t_{\text{pre}} \ge \frac{2N_{mm}BT}{F_{\text{fmt}}}, \qquad \text{TPOT} \approx \frac{t(n) - t(1)}{n-1} \]

References for this session

  1. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  2. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  3. Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  4. NVIDIA (code). TensorRT-LLM
  5. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  6. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  7. Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  8. Micikevicius et al. (2022). FP8 Formats for Deep Learning
  9. NVIDIA (code). TensorRT-LLM
  10. vLLM project (docs). vLLM documentation
  11. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  12. EleutherAI (code). LM Evaluation Harness
  13. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  14. vLLM project (code). LLM Compressor: a library for quantizing models for vLLM
  15. Meta (2024). Llama 3.1 8B Instruct: model card and config.json
  16. NVIDIA (docs). NVIDIA L4 Tensor Core GPU: product specifications
  17. Merity et al. (2016). Pointer Sentinel Mixture Models (the WikiText datasets)
  18. Clark et al. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
  19. Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems (GSM8K)
  20. Neural Magic / Red Hat AI (2024). Meta-Llama-3.1-8B-Instruct-quantized.w4a16 (model card with evaluation table)
  21. Neural Magic / Red Hat AI (2024). Meta-Llama-3.1-8B-Instruct-quantized.w8a8 (model card with evaluation table)
  22. Neural Magic / Red Hat AI (2024). Meta-Llama-3.1-8B-Instruct-FP8 (model card with evaluation table)
  23. Qwen Team (2024). Qwen2.5 Technical Report