compare RTN, GPTQ, AWQ, SmoothQuant and FP8 on one model
In this lab you take one 8B model, Llama 3.1 8B Instruct, and make five quantized copies of it: RTN, GPTQ and AWQ with 4-bit weights, SmoothQuant with 8-bit weights and activations, and FP8. You predict their memory and speed on an NVIDIA L4, then measure perplexity, two tasks, prefill latency and decode latency at batch 1 and batch 16, compare with the survey’s table, and write a one-page recommendation.
What you will learn
- What RTN, GPTQ, AWQ, SmoothQuant and FP8 each do to one linear layer, written as one formula each
- How to predict weight memory (16.06 GB in BF16, 5.70 GB in W4A16, 9.08 GB in W8A8) and prefill and decode latency on an L4 with the roofline
- Why 4-bit weights speed up decode but not prefill, and why 8-bit activations speed up prefill
- How to quantize an 8B model with llm-compressor, serve it with vLLM and evaluate it with lm-evaluation-harness
- How to measure prefill and decode latency separately at batch 1 and batch 16
- How to read the survey’s W4A16 speed-up table and explain the gap between a roofline estimate and a measurement
- How to turn a results table into a one-page production recommendation
Why this matters
The last six sessions each taught one quantization method in isolation. A production team never meets them that way. It meets one model, one GPU, one latency target and one quality bar, and it must pick one recipe. In this lab you take one 8B model, make five quantized copies of it, predict how each will behave on an NVIDIA L4, then measure quality and speed the same way for all of them, and finish with a one-page recommendation that a team could act on.
The five recipes are the ones you already know: RTN (round to nearest, session 103), GPTQ (session 106), AWQ (session 107), SmoothQuant (session 108) and FP8 (session 105). The first three change only the weights, to 4 bits; the last two change weights and activations, to 8 bits. Session 104 explained why this difference matters: 4-bit weights move fewer bytes, which helps the memory-bound decode phase; 8-bit activations allow 8-bit matrix units, which helps the compute-bound prefill phase. This lab turns that sentence into numbers.
The work has four parts, one per key point of the session:
- Quantize Llama 3.1 8B Instruct with each method.
- Measure perplexity, two tasks (ARC-Challenge and GSM8K), prefill latency and decode latency on an L4.
- Repeat the speed runs at batch 1 and batch 16, and compare with the survey’s table of W4A16 speed-ups (Zhou et al., section 5.2.1).
- Write a one-page recommendation for your production model.
Where our numbers come from. We did not have an L4 when we wrote this page. So the page uses three kinds of numbers, and labels each one. (1) Predictions from the roofline, computed from the model’s config.json and NVIDIA’s spec sheet; we checked every one with a Python script. (2) A rehearsal: the same five recipes, simulated in PyTorch on a small model (Qwen2.5-0.5B) on a laptop, so you can see the quality ordering before you rent a GPU. (3) Published numbers: the survey’s measured speed-ups on an A100, and the quality scores that the makers of public quantized Llama 3.1 8B checkpoints report. The scripts for the real L4 runs are complete, but we did not run them; your L4 numbers are the result of the lab.
The lab at a glance
The model is Llama 3.1 8B Instruct. From its config.json [Certain]: width \(d = 4096\), \(L = 32\) layers, 32 query heads and 8 key-value heads of size 128 (grouped-query attention, session 97), a SwiGLU feed-forward layer of width 14,336, a vocabulary of 128,256 tokens, and an output layer that is not tied to the input embedding. That gives 8,030,261,248 parameters, of which 6,979,321,856 sit in the 224 linear layers of the blocks (7 per block). Only these linear layers are quantized; the embedding, the output layer (“lm_head”) and the norms stay in BF16, which is the usual practice [Certain for the public checkpoints in the table below].
| Recipe | Weights | Activations | Calibration data | Matmul in the kernel |
|---|---|---|---|---|
| BF16 (baseline) | BF16 | BF16 | none | \(\text{BF16} \times \text{BF16}\) |
| RTN W4A16 | INT4, one scale per group of 128 | BF16 | none | dequantize to BF16, then \(\text{BF16} \times \text{BF16}\) |
| GPTQ W4A16 | INT4, group 128, error-corrected | BF16 | a few hundred sequences | same as RTN |
| AWQ W4A16 | INT4, group 128, salient channels scaled up | BF16 | a few hundred sequences | same as RTN |
| SmoothQuant W8A8 | INT8, one scale per output channel | INT8, one scale per token | a few hundred sequences | \(\text{INT8} \times \text{INT8} \to \text{INT32}\) |
| FP8 W8A8 | FP8 E4M3, per channel | FP8 E4M3, per token, dynamic | none for dynamic scales | \(\text{FP8} \times \text{FP8} \to \text{FP32}\) |
Why an L4. The L4 is a cheap, low-power data-center GPU (72 W) built on NVIDIA’s Ada architecture. Its spec sheet [Certain]: 24 GB of GDDR6 memory with 300 GB/s of bandwidth, 121 TFLOP/s of dense BF16 matrix math, and 242.5 TFLOP/s of dense FP8 or INT8 (NVIDIA lists 242, 485 and 485 “with sparsity”; dense is half). Ada has FP8 matrix units, so all five recipes have fast kernels on it. Its bandwidth is low next to its compute, so its ridge point is high: \(121 \times 10^{12} / 300 \times 10^{9} = 403\) FLOPs per byte in BF16. That makes the L4 a sharp test: many workloads that are compute-bound on an H100 are memory-bound here.
Five methods, one layer, one formula each
Every recipe changes one linear layer \(Y = X W^\top\), where \(X\) holds one row per token and \(W\) one row per output feature. We write each method as one formula, with the notation of the earlier sessions.
RTN rounds each weight to the nearest point of a grid. For 4-bit asymmetric quantization with groups of \(g = 128\) input channels, each group gets its own scale and zero point:
- Read the symbols
- \(w, \hat{w}\)
- one weight and its quantized value; max and min are taken over the group of 128 weights that share \(s\) and \(z\)
- \(b\)
- bits per weight: 4 here, so the integer codes are 0 to 15
- \(\lfloor \cdot \rceil\)
- round to the nearest integer
- \(s, z\)
- scale (16 bits) and integer zero point of the group
GPTQ quantizes the columns of \(W\) one at a time and, after each column, changes the columns not yet quantized so that the layer’s output error on calibration data stays small. The correction uses the inverse Hessian \(H^{-1}\) of the layer’s squared error:
- Read the symbols
- \(\mathbf{w}_{:,q}\)
- column \(q\) of \(W\): the weights of input channel \(q\) for all outputs
- \(H\)
- \(X^\top X\) over the calibration tokens, plus a small damping (1% of its mean diagonal) so that it can be inverted
- \(j \gt q\)
- the columns after \(q\), which absorb the rounding error of column \(q\)
AWQ notices that a few input channels carry large activations, so errors in their weights hurt most. It multiplies those weight columns by a scale \(s_j \gt 1\) before rounding (so they get relatively finer steps) and divides the activations by the same \(s_j\) (folded into the previous operation, so the layer is unchanged in exact arithmetic). It picks one exponent \(\alpha\) per layer by a small grid search:
- Read the symbols
- \(\bar{x}_j\)
- the mean absolute activation of input channel \(j\) on the calibration data
- \(Q(\cdot)\)
- the RTN quantizer of (112.1)
- \(\alpha\)
- 0 means plain RTN; larger values protect the high-activation channels more, at the cost of coarser steps for the others
SmoothQuant attacks the other problem: activations with outlier channels are hard to quantize to 8 bits, weights are easy. It moves part of the range from activations to weights with a per-channel scale, then quantizes both to INT8:
- Read the symbols
- \(\max_t |X_{tj}|\)
- the largest activation of channel \(j\) over the calibration tokens
- \(\max_i |W_{ij}|\)
- the largest weight of input channel \(j\) over the output rows
- \(\alpha\)
- the migration strength: 0.5 splits the difficulty evenly; the SmoothQuant paper uses 0.5 for most models and more for some
FP8 E4M3 needs no smoothing in most models, because a floating-point grid is fine near zero and coarse far from it, so one outlier does not waste the resolution of the small values. One scale per row brings the largest value to 448, the largest E4M3 number:
- Read the symbols
- \(\mathbf{x}\)
- one row: a token’s activations (scale computed on the fly, “dynamic”) or an output channel’s weights (scale stored)
- \(\operatorname{fp8}\)
- round to the nearest E4M3 number: 1 sign bit, 4 exponent bits, 3 mantissa bits, so 8 steps per power of two
- input: weights \(W\) (\(\text{out} \times \text{in}\)), calibration inputs \(X\), bits \(b\), group size \(g\)
- \(H \leftarrow X^\top X / n + \lambda I\) with \(\lambda = 0.01 \cdot \operatorname{mean}(\operatorname{diag} H)\)
- \(U \leftarrow\) upper Cholesky factor of \(H^{-1}\) // numerically stable form of the updates
- for \(q = 1, \dots, \text{in}\):
- if \(q\) starts a group: compute \(s, z\) of (112.1) from the current (already corrected) columns \(q..q+g-1\)
- \(\hat{\mathbf{w}}_{:,q} \leftarrow Q(\mathbf{w}_{:,q})\); \(\;\mathbf{e} \leftarrow (\mathbf{w}_{:,q} - \hat{\mathbf{w}}_{:,q}) / U_{qq}\)
- \(\mathbf{w}_{:,q..} \leftarrow \mathbf{w}_{:,q..} - \mathbf{e}\, U_{q,\,q..}\) // push the error onto later columns
- return \(\hat{W}\)
Before you quantize a real model, look at what each method does to one layer. Figure 1 builds a small layer (128 inputs, 64 outputs) whose activations are correlated, like real ones, and have four outlier channels, like real LLM activations. It applies all five methods and reports the relative output error on held-out tokens:
How to read Figure 1
With the defaults (outliers 20 times normal, 4-bit, group 32, \(\alpha = 0.5\)), RTN has an error of \(7.1 \times 10^{-3}\); GPTQ cuts it about 10 times, to \(7.1 \times 10^{-4}\); AWQ (best \(\alpha = 0.5\)) to \(1.6 \times 10^{-3}\). Among the 8-bit recipes, naive W8A8 gives \(4.2 \times 10^{-4}\), SmoothQuant \(4.9 \times 10^{-5}\), and FP8 \(1.0 \times 10^{-3}\). Now set the outliers to 1 (no outliers): naive W8A8 drops to \(8.4 \times 10^{-5}\), and SmoothQuant has almost nothing left to fix. Drag them to 50: AWQ now beats GPTQ, because protecting the outlier channels is exactly its idea. Move \(\alpha\) to 0 or 1: SmoothQuant gets worse at both ends, because all the difficulty sits either in the activations or in the weights. Switch to 3 bits: every 4-bit method gets several times worse, and RTN most.
Three lessons carry over to the real model. First, GPTQ and AWQ are much better than RTN at the same bits and the same memory; the bytes are identical, only the choice of codes changes. Second, naive INT8 activations suffer from outlier channels, and smoothing removes most of that damage. Third, FP8 without any calibration lands in between: its relative step is about \(2^{-4}\) everywhere, so its error is set by the mantissa, not by outliers. These are lessons about one toy layer; the ordering on a real model can differ, which is why we measure.
Part 1, before measuring: predict memory and speed
A good engineer predicts the result before running the experiment. Then a measurement that disagrees with the prediction teaches something. Start with memory. The linear layers take \(b\) bits per weight plus the group scales; everything else stays in BF16:
- Read the symbols
- \(b_{\text{eff}}\)
- bits per weight including the scales: 16 for BF16; \(4 + 16/128 = 4.125\) for INT4 with one 16-bit scale per group of 128; 8 for INT8 and FP8 (one scale per output channel adds only 5.5 MB in total)
- \(N_{\text{lin}}, N_{\text{emb}}, N_{\text{head}}\)
- 6.98 billion weights in the linear layers, 525 million in the embedding and 525 million in the output layer
- \(B, T, n\)
- batch size, prompt length and generated tokens
- \(\text{kv}_{\text{tok}}\)
- BF16 KV cache per token: 131,072 bytes (128 KiB) for this model
BF16: \(2 \times 8{,}030{,}261{,}248 = 16.06\) GB. W4A16 (RTN, GPTQ, AWQ): \(6.979 \times 10^9 \times 4.125/8 = 3.60\) GB for the linear layers, plus 2.10 GB for the BF16 embedding, output layer and norms, so 5.70 GB. (An asymmetric format also stores a 4-bit zero point per group: 5.73 GB.) W8A8 (SmoothQuant, FP8): \(6.98 + 2.10 = \) 9.08 GB.
The KV cache: a batch of 16 with 2,048-token prompts and 128 new tokens needs \(16 \times 2{,}176 \times 131{,}072 = 4.56\) GB. With vLLM’s default memory budget (90% of the 24 GB, so 21.6 GB), the BF16 model leaves room for about 42,000 tokens of cache, W8A8 for about 95,500, and W4A16 for about 121,000 [Certain for the arithmetic; the real number is a little lower, because the runtime also needs memory for activations and CUDA graphs]. So on an L4, quantization is first a capacity win: almost three times more concurrent context.
Now the time. Use the bound of session 82: a pass cannot be faster than its FLOPs at peak speed, nor faster than its bytes at full bandwidth. The weights are read once per pass. Only the 8-bit recipes get the faster 8-bit matrix units; W4A16 dequantizes to BF16 and uses the BF16 units:
- Read the symbols
- \(N_{mm}\)
- weights used in matmuls each pass: the linear layers plus the output layer, 7.50 billion
- \(F_{\text{fmt}}\)
- matmul peak of the format: 121 TFLOP/s for BF16 and W4A16, 242.5 for INT8 and FP8 on the L4
- \(F_{16}\)
- the BF16 peak, used for attention, which stays in BF16 in all five recipes
- \(M_{\text{read}}\)
- weight bytes read per pass: the linear layers in their format plus the BF16 output layer (15.01 GB, 4.65 GB, 8.03 GB)
- \(c\)
- context length of the decode step; we use the middle of generation, \(c = T + n/2\)
- \(\text{BW}\)
- memory bandwidth, 300 GB/s
Prefill, batch 1. FLOPs: \(2 \times 7.50 \times 10^9 \times 512 = 7.69\) TFLOP plus a little attention. In BF16 that takes 64.1 ms at 121 TFLOP/s; reading 15 GB of weights takes 50 ms. Compute wins, so BF16 prefill is about 64 ms. W4A16 has the same FLOPs on the same units: also 64 ms, no speed-up. W8A8 doubles the matmul peak: 32 ms, a speed-up of 1.99.
Decode, batch 1, at context 576. Bytes: 15.01 GB of weights plus 75 MB of cache. At 300 GB/s that is 50.3 ms per token in BF16, about 20 tokens per second. W4A16 reads 4.65 GB: 15.7 ms, a speed-up of 3.19. W8A8 reads 8.03 GB: 27.0 ms, 1.86. The FLOPs are tiny (15 GFLOP, 0.12 ms), so decode is memory-bound by a factor of about 400.
Batch 16. Prefill time grows 16 times (1.03 s in BF16; 0.52 s in W8A8). Decode grows only from 50.3 to 54.1 ms in BF16, because the weights are shared; W4A16 still wins, 19.5 ms, a speed-up of 2.77. The speed-up shrinks because the KV cache, which W4A16 does not compress, is now a larger part of the bytes. With 2,048-token prompts at batch 16 it drops to 2.14.
One surprise hides in the short prompts. At 128 tokens, BF16 prefill is memory-bound on an L4 (50 ms to read the weights, 16 ms of math), because 128 is below the ridge of 403. So W4A16 does speed up a short prefill there, by up to 3.15 in the bound. On an H100, with a ridge of about 295, the same holds only for even shorter prompts. Figure 2 lets you explore all cases.
How to read Figure 2
With the defaults (batch 1, 512-token prompt, 128 new tokens), W4A16 cuts decode from 50.3 to 15.7 ms per token but leaves prefill at 64.1 ms; W8A8 halves prefill (32.3 ms) and cuts decode to 27.0 ms. Lower the prompt to 128: BF16 prefill becomes memory-bound and W4A16 helps prefill too. Raise the batch to 16 and the prompt to 2,048: the W4A16 decode advantage drops to 2.14, because the uncompressed KV cache grows to 4.56 GB. Raise the batch to 64 and the prompt to 1,024: the BF16 copy needs 25.7 GB and no longer fits in 21.6 GB, while W8A8 (18.7 GB) and W4A16 still do. The readout prints every number and says which bound (compute or memory) is active.
Part 2: quantize, serve and measure on the L4
The tools: LLM Compressor (from the vLLM project) applies the recipes and saves checkpoints that vLLM loads directly; lm-evaluation-harness measures quality through vLLM; vLLM’s benchmark command measures latency. TensorRT-LLM supports the same formats and is the other common choice on NVIDIA GPUs.
- input: base model \(M\), recipes \(R_1..R_5\), calibration set \(C\) (same 512 sequences for all), GPU, workloads \(\{(B, T, n)\}\)
- for each recipe \(R\): \(M_R \leftarrow\) quantize\((M, R, C)\); save; record time and peak memory
- for each model in \(\{M, M_{R_1}, \dots\}\): // same engine, same flags, same GPU for all
- quality: WikiText-2 perplexity, ARC-Challenge accuracy, GSM8K exact match (same prompts, same few-shot, greedy)
- for each \((B, T)\) in \(\{1, 16\} \times \{128, 512, 2048\}\):
- \(t_1 \leftarrow\) latency of \(n = 1\) (prefill and one token); \(\;t_n \leftarrow\) latency of \(n = 128\)
- TTFT \(\approx t_1\); \(\;\)TPOT \(\leftarrow (t_n - t_1)/(n - 1)\) // formula (112.10)
- record weight memory and the KV-cache capacity that the engine reports
- report every number next to its prediction from (112.7) to (112.9); explain each gap larger than 30%
Step 1: make the five checkpoints
The script below is written for LLM Compressor and vLLM as of late 2026. Their module paths change between versions; if an import fails, copy the import line from the examples folder of your installed version. Quantizing an 8B model needs the BF16 model in memory: on a 24 GB L4, LLM Compressor moves layers between CPU and GPU, so give the machine at least 64 GB of CPU memory [Likely]. We did not run this script.
# quantize.py: five copies of Llama 3.1 8B Instruct (LLM Compressor; not run by us)
import sys
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.awq import AWQModifier
from llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier
BASE = "meta-llama/Llama-3.1-8B-Instruct"
SKIP = ["lm_head"] # keep the output layer in BF16
RECIPES = {
"rtn-w4a16": [QuantizationModifier(targets="Linear", scheme="W4A16", ignore=SKIP)],
"gptq-w4a16": [GPTQModifier(targets="Linear", scheme="W4A16", ignore=SKIP)],
"awq-w4a16": [AWQModifier(), QuantizationModifier(targets="Linear", scheme="W4A16_ASYM", ignore=SKIP)],
"sq-w8a8": [SmoothQuantModifier(smoothing_strength=0.8),
QuantizationModifier(targets="Linear", scheme="W8A8", ignore=SKIP)],
"fp8-dynamic":[QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=SKIP)],
}
name = sys.argv[1]
model = AutoModelForCausalLM.from_pretrained(BASE, dtype="auto")
tok = AutoTokenizer.from_pretrained(BASE)
needs_data = name in ("gptq-w4a16", "awq-w4a16", "sq-w8a8")
oneshot(model=model, recipe=RECIPES[name],
**(dict(dataset="ultrachat_200k", splits="train_sft[:512]", max_seq_length=2048,
num_calibration_samples=512) if needs_data else {}))
model.save_pretrained(f"ckpt/{name}", save_compressed=True); tok.save_pretrained(f"ckpt/{name}")
Two choices in this script deserve a sentence. SmoothQuant with RTN, not with GPTQ: the public W8A8 recipes stack SmoothQuant and GPTQ, which is usually a little better, but then you compare two methods at once. Here we want to see what smoothing alone buys. The same calibration data for all: 512 chat sequences of up to 2,048 tokens. Calibration data that does not look like your traffic is a common reason for a disappointing GPTQ or AWQ result.
Step 2: quality
# perplexity on WikiText-2 and two tasks, through vLLM (one model at a time)
for m in meta-llama/Llama-3.1-8B-Instruct ckpt/rtn-w4a16 ckpt/gptq-w4a16 ckpt/awq-w4a16 ckpt/sq-w8a8 ckpt/fp8-dynamic; do
lm_eval --model vllm --model_args pretrained=$m,dtype=auto,max_model_len=4096,gpu_memory_utilization=0.9 \
--tasks wikitext,arc_challenge,gsm8k --batch_size auto --seed 0 \
--output_path results/$(basename $m)
done
Perplexity is the exponential of the mean negative log-likelihood per token:
- Read the symbols
- \(N\)
- number of scored tokens (or words: the harness’s
wikitexttask reports perplexity per word, which is a larger number than per token; compare only numbers of the same kind) - \(t(n)\)
- measured end-to-end latency of one batch that generates \(n\) tokens per sequence after the prompt
Perplexity is a sensitive smoke test: a broken quantization raises it a lot. But small perplexity differences do not predict small task differences, and the reverse. GSM8K (grade-school math with chain-of-thought, scored by exact match of the final number) is a good second test because long reasoning chains amplify small errors; ARC-Challenge (multiple-choice science questions) checks knowledge with little generation. Note that the instruct model is usually scored on these tasks with its chat template; use the same flags for all six models, or the comparison is meaningless.
Step 3: latency at batch 1 and batch 16
# prefill and decode latency, measured separately (vLLM; not run by us)
for m in ...; do # the same six models
for B in 1 16; do for T in 128 512 2048; do
vllm bench latency --model $m --batch-size $B --input-len $T --output-len 1 --num-iters 20 \
--output-json lat/$(basename $m)-$B-$T-1.json # t(1): TTFT
vllm bench latency --model $m --batch-size $B --input-len $T --output-len 128 --num-iters 20 \
--output-json lat/$(basename $m)-$B-$T-128.json # t(128)
done; done
done
# then: TPOT = (t(128) - t(1)) / 127, with the median latencies from the two JSON files
Why two runs instead of one? A single end-to-end time mixes the prefill with 127 decode steps. The difference of two runs with the same prompt removes the prefill and leaves the decode steps, so you get both phases cleanly, as the survey’s table does. The benchmark warms up and repeats for you; keep everything else fixed (CUDA graphs on, same max_model_len, same vLLM version) and write the versions into your report.
A rehearsal on a laptop: the quality ordering
Before you pay for GPU hours, rehearse the quality side on a small model. We applied the five recipes to Qwen2.5-0.5B (24 layers, width 896) in PyTorch, as fake quantization: each weight (and, for the 8-bit recipes, each activation) is rounded to the target grid and stored back in FP32, so the arithmetic is exactly that of the quantized model, but nothing gets faster (session 111 used the same trick). We scored perplexity on the WikiText-2 test set (the raw version, 146 windows of 2,048 tokens, 298,862 scored tokens) on an Apple laptop. The calibration set for GPTQ, AWQ and SmoothQuant was 32 random windows of 512 tokens from the WikiText-2 training set, the same for all three. [Certain, our run]
Our versions are simplified, and you should know where: GPTQ is the algorithm of the paper (blocks of 128 columns, 1% damping, group 128, asymmetric) without act-order; AWQ is the scale search of (112.3) per linear layer, without the weight-clipping search of the paper; SmoothQuant uses \(\alpha = 0.5\) and smooths all seven linear layers (a real deployment can fold the scale only where a norm or a linear layer comes before, so it usually smooths the QKV and the gate-up projections only); FP8 uses dynamic per-token and per-channel scales.
How to read Figure 3
The bars are sorted by format: the 4-bit weight recipes on the left, the 8-bit recipes on the right. Compare within a format first: GPTQ and AWQ should sit below RTN at the same 4 bits. Then compare across formats with the third view: the 4-bit models are 37% smaller than the 8-bit ones, so a 4-bit recipe that comes close to the 8-bit quality is the better deal for decode-heavy traffic. Click a bar for its exact numbers.
What the rehearsal says. [Certain, our run] The BF16 model scores 13.07. At 4 bits with group 128, RTN raises perplexity to 15.54 (+18.9%), AWQ to 14.73 (+12.7%) and GPTQ to 14.19 (+8.6%): GPTQ removes 55% of the RTN damage, AWQ 33%. At 8 bits, naive INT8 gives 13.33 (+2.0%), SmoothQuant 13.21 (+1.1%, removing 45% of the naive damage) and FP8 13.26 (+1.4%) without any calibration. So the ordering of Figure 1 survives on a real model: within each format, the calibrated methods win, and the 8-bit recipes are much closer to BF16 than the 4-bit ones. Our AWQ is weaker than the paper’s, since it has no clipping search and chose a mean \(lpha\) of only 0.33; on an 8B model with the full method, expect AWQ and GPTQ to be close.
Two practical numbers from the same run: quantizing took about 5 minutes for GPTQ, 3.4 minutes for AWQ and 2.3 minutes for SmoothQuant (statistics plus rounding, on a shared laptop CPU), and seconds for RTN and FP8. On an 8B model expect tens of minutes for GPTQ and AWQ on one GPU [Likely]. Quantization is a one-time cost, but it is part of your release pipeline, so write it down.
The code of Figure 1, runnable
This file reproduces the one-layer experiment in PyTorch. It is deterministic (fixed seeds, float64) and runs on a CPU in about a second. Its random numbers come from PyTorch, not from the figure’s generator, so its errors differ a little from Figure 1, but the ordering is the same. We ran it; its output follows.
import torch
torch.manual_seed(0)
torch.set_default_dtype(torch.float64)
D_IN, D_OUT, N = 128, 64, 256
OUT = [7, 33, 70, 101] # outlier input channels
def acts(n, mix): # correlated activations + outliers
x = torch.randn(n, 8) @ mix + 0.3 * 8 ** 0.5 * torch.randn(n, D_IN)
x[:, OUT] *= 20
return x
W = 0.05 * torch.randn(D_OUT, D_IN)
mix = torch.randn(8, D_IN)
Xc, Xt = acts(N, mix), acts(N, mix) # calibration set, test set
Y = Xt @ W.T
def q_group(w, bits=4, gs=32): # asymmetric RTN, one scale per group (112.1)
o, i = w.shape
g = w.reshape(o, i // gs, gs)
lo = g.amin(-1, keepdim=True).clamp(max=0)
hi = g.amax(-1, keepdim=True).clamp(min=0)
s = ((hi - lo) / (2 ** bits - 1)).clamp(min=1e-12)
z = torch.round(-lo / s)
q = (torch.round(g / s) + z).clamp(0, 2 ** bits - 1)
return ((q - z) * s).reshape(o, i)
def q_rows(a, bits=8): # symmetric INT8, one scale per row
s = a.abs().amax(-1, keepdim=True) / (2 ** (bits - 1) - 1)
return torch.round(a / s).clamp(-127, 127) * s
def gptq(w, x, bits=4, gs=32): # Algorithm 1, formula (112.2)
H = x.T @ x / len(x)
H += 0.01 * H.diag().mean() * torch.eye(len(H))
U = torch.linalg.cholesky(torch.linalg.inv(H)).T # upper factor of H^-1
w, Q = w.clone(), torch.zeros_like(w)
for i in range(w.shape[1]):
if i % gs == 0: # group scale from the updated weights
g = w[:, i:i + gs]
lo, hi = g.amin(1).clamp(max=0), g.amax(1).clamp(min=0)
s = (hi - lo) / (2 ** bits - 1); z = torch.round(-lo / s)
q = ((torch.round(w[:, i] / s) + z).clamp(0, 2 ** bits - 1) - z) * s
Q[:, i] = q
err = (w[:, i] - q) / U[i, i]
w[:, i:] -= err[:, None] * U[i, i:][None, :] # push the error to later columns
return Q
def awq(w, x, bits=4, gs=32): # grid search of (112.3)
m, ref, best = x.abs().mean(0), x @ w.T, None
for k in range(20):
s = m ** (k / 20); s = s / (s.max() * s.min()).sqrt()
wq = q_group(w * s, bits, gs) / s # scale up, quantize, scale back
e = ((x @ wq.T - ref) ** 2).sum().item()
if best is None or e < best[0]: best = (e, k / 20, wq)
return best[2], best[1]
def smooth(w, x, alpha=0.5): # SmoothQuant scale (112.4)
return x.abs().amax(0) ** alpha / w.abs().amax(0) ** (1 - alpha)
def fp8(a): # per-row scale, then E4M3 rounding (112.5)
s = a.abs().amax(-1, keepdim=True) / 448
return (a / s).to(torch.float8_e4m3fn).to(a.dtype) * s
rel = lambda yh: (((yh - Y) ** 2).sum() / (Y ** 2).sum()).item()
Wa, alpha = awq(W, Xc)
s = smooth(W, Xc)
print(f"RTN W4 g32 {rel(Xt @ q_group(W).T):.2e}")
print(f"GPTQ W4 g32 {rel(Xt @ gptq(W, Xc).T):.2e}")
print(f"AWQ W4 g32 {rel(Xt @ Wa.T):.2e} (alpha={alpha})")
print(f"W8A8 naive {rel(q_rows(Xt) @ q_rows(W).T):.2e}")
print(f"SmoothQuant {rel(q_rows(Xt / s) @ q_rows(W * s).T):.2e}")
print(f"FP8 E4M3 {rel(fp8(Xt) @ fp8(W).T):.2e}")
RTN W4 g32 5.77e-03 GPTQ W4 g32 5.09e-04 AWQ W4 g32 1.67e-03 (alpha=0.35) W8A8 naive 3.97e-04 SmoothQuant 5.27e-05 FP8 E4M3 7.45e-04
Part 3: compare with the survey’s table
Zhou et al. measured what W4A16 buys on a real GPU (their Table IV, in section 5.2.1). They quantized Llama-2-7B and Llama-2-13B to 4-bit weights with AWQ, served them with TensorRT-LLM and with LMDeploy on one A100, and reported the speed-up over FP16 of three latencies, prefill / decode / end-to-end, for batch sizes 1 to 16 and prompt lengths 128 to 2,048. [Certain] Their four observations, in short: (1) weight-only quantization speeds up decode a lot, and so end-to-end latency; (2) it can make prefill slower, because prefill is compute-bound and the weights must be dequantized; (3) the gain shrinks as batch and prompt length grow; (4) larger models gain more.
Your job is to predict their table from (112.9) and explain the gap. For decode, the bound is a ratio of bytes, so the bandwidth cancels:
Llama-2-7B has no grouped-query attention, so its cache is large: 512 KiB per token, four times that of Llama 3.1 8B. At batch 1 and 128 tokens the bound is 3.62; the table says 2.40 (TensorRT-LLM) and 2.11 (LMDeploy), so the kernels reach 58 to 66% of the ideal. At batch 16 and 2,048 tokens the bound has fallen to 1.46, because 17.2 GB of FP16 cache now dominate the 13.2 GB of FP16 weights, and the table says 1.16 and 1.28. The trend of observation (3) is entirely in the formula; the level is the kernels.
How to read Figure 4
Start with decode, TensorRT-LLM, 7B. Along the top row (batch 1) the measured speed-up falls slowly from 2.40 to 2.00 as the prompt grows; down the left column (128 tokens) it falls faster, to 1.43 at batch 8. The bounds fall the same way, from 3.62 to 1.46. Click any cell: the readout gives the ratio measured / bound: between 0.43 and 0.79 for TensorRT-LLM on the 7B model, lowest at batch 4 to 16 with short prompts. Now switch to prefill: the measured values are close to 1 or below it, while the bound says 1.0 (compute-bound) except at batch 1 and 128 tokens, where it allows 1.20. Switch to 13B: decode speed-ups are a little higher, as observation (4) says. Two oddities are in the source table, not in our figure: the batch-8 and batch-16 rows of Llama-2-7B are identical in both engines, and some decode speed-ups are lower than end-to-end ones for 13B, which cannot happen if end-to-end is a mix of prefill and decode. Treat them as likely copy errors [Likely].
What should you expect on an L4? Run your own version of this table with the latency script above. The bound of (112.9) for Llama 3.1 8B on an L4 is 3.19 at batch 1 and 512 tokens and 2.14 at batch 16 and 2,048 tokens. If your kernels reach the same 45 to 80% of the bound as the A100 runs, you would measure roughly 1.5 to 2.5 at batch 1 [Speculation, until you measure]. Two differences point in opposite directions: Llama 3.1 has a cache four times smaller per token, which keeps the decode bound high at large batches; and the L4 has much less bandwidth, so every byte saved is worth more milliseconds.
Published quality for Llama 3.1 8B
You will compare your quality numbers with public quantized checkpoints of the same model. Their makers (Neural Magic, now Red Hat AI) report scores from their own fork of lm-evaluation-harness with the chat template, so use the same settings if you want to compare exact numbers. [Certain, quoted from the three model cards]
| Checkpoint (recipe) | ARC-Challenge, 0-shot | GSM8K, CoT 8-shot | MMLU, 5-shot |
|---|---|---|---|
| BF16 baseline (as reported on the W4A16 and W8A8 cards) | 81.4 | 82.8 | 68.3 |
| W4A16, GPTQ, symmetric, group 128, act-order | 80.2 (98.0%) | 82.9 (100.2%) | 66.9 (97.9%) |
| W8A8 INT8 (GPTQ, per-channel W, dynamic per-token A) | 81.7 (100.3%) | 84.8 (102.5%) | 67.8 (99.3%) |
| FP8, static per-tensor scales (its own baseline: 82.00, 81.96, 67.95) | 81.66 (99.6%) | 81.12 (99.0%) | 67.97 (100.0%) |
Read this table with care. Differences of one point are within the noise of these benchmarks (ARC-Challenge has 1,172 test questions, so one point is about 12 questions; GSM8K has 1,319), and a quantized model “beating” the baseline on GSM8K is noise, not magic. The real message: at 8 bits all recipes are lossless within noise; at 4 bits a good recipe (GPTQ with group 128) loses one to two points on knowledge tasks. Your lab should confirm or refute this for RTN and AWQ, which these cards do not cover.
Part 4: the one-page recommendation
A results table is not a decision. The last step turns it into one page that a team lead can read in three minutes. Use this structure:
- input: the workload (prompt and output lengths, batch or concurrency, TTFT and TPOT targets), the quality bar (which tasks, how many points of loss are acceptable), the GPU
- drop every recipe that misses the quality bar on your tasks // quality first: speed cannot fix wrong answers
- drop every recipe that does not fit the memory needed for the target concurrency (112.7)
- split the time per request into prefill and decode: \(t \approx \text{TTFT} + (n-1)\,\text{TPOT}\)
- if decode dominates (short prompts, long answers, small batches): prefer W4A16 (the best of GPTQ and AWQ)
- if prefill dominates (long prompts, RAG, short answers, large batches): prefer W8A8 (FP8 where the GPU supports it, else SmoothQuant INT8)
- write: the choice, the numbers that decided it, the risks, and the test that would change the decision
Workload. A support chatbot: prompts of about 512 tokens, answers of about 256 tokens, up to 16 users at once per GPU, TPOT target 40 ms.
Time per request at batch 16 (bounds of Figure 2, with \(n = 256\)): BF16 \(1.03\) s prefill + \(255 \times 54.5\) ms decode = 14.9 s; W4A16 \(1.03 + 255 \times 20.0\) ms = 6.1 s; W8A8 \(0.52 + 255 \times 31.2\) ms = 8.5 s. Decode is 93% of the BF16 time, so the byte-saving recipe wins. BF16 also misses the TPOT target (54.5 ms), the other two meet it.
Recommendation (to be confirmed by your L4 measurements): serve GPTQ or AWQ W4A16, whichever scores better on GSM8K and on 200 real support conversations; keep the FP8 copy as the fallback if the 4-bit model loses more than 2 points on any task that matters. Risks: W4A16 kernels may be slower than predicted at batch 16; long-tail prompts of several thousand tokens shift the balance toward FP8. What would change the decision: a measured W4A16 TPOT above 40 ms at batch 16, or a quality loss above the bar.
Your page should have five short blocks: the decision in one sentence; the table of measured numbers (quality, memory, TTFT and TPOT at batch 1 and 16) with the predictions next to them; why the winner wins, in terms of bytes and FLOPs; risks and open questions; and the next experiment. Keep the raw logs and versions in an appendix, not on the page.
Where this is used
| Where | Typical choice | Why |
|---|---|---|
| Chat on small GPUs (L4, A10G, consumer cards) | W4A16 with GPTQ or AWQ | Decode-heavy, batch small: bytes decide; 8B fits with room for cache |
| High-throughput serving on H100 or newer | FP8 W8A8 (often with an FP8 KV cache) | Large batches make linear layers compute-bound; FP8 doubles matmul speed with little loss [Likely] |
| GPUs without FP8 (A100, older) | SmoothQuant INT8 for throughput, W4A16 for latency | INT8 units exist; FP8 does not |
| Long-prompt workloads: RAG, document QA, Video-RAG | W8A8 | Prefill dominates; thousands of visual or retrieved tokens per request make it compute-bound |
| Video understanding with a VLM (video QA, key-moment detection) | W8A8 for the language model; vision encoder often kept in BF16 or FP8 | Many visual tokens per frame make prefill the cost; the encoder is small and sensitive [Likely] |
| On-device (laptops, phones) | 4-bit weights, often 3 or 4 bits with groups | Memory size and bandwidth are the limits; batch is 1 |
| Model releases on model hubs | Several checkpoints (W4A16, W8A8, FP8) with recovery tables | Users pick by GPU and workload, as in this lab |
Common mistakes
- Comparing recipes with different calibration data or sequence lengths. Then you compare datasets, not methods. Use the same calibration set for GPTQ, AWQ and SmoothQuant, and say what it is.
- Reporting only perplexity. A 0.1 perplexity difference can hide a 3-point drop on GSM8K, or the opposite. Always add at least one task that generates long answers.
- Mixing word and token perplexity, or different window lengths. lm-evaluation-harness reports word perplexity for
wikitext; many papers report token perplexity with 2,048-token windows. The numbers are not comparable. - One end-to-end latency instead of two phases. W4A16 can be 3 times faster in decode and 10% slower in prefill; the end-to-end number hides both. Measure \(t(1)\) and \(t(n)\).
- Measuring only batch 1. The W4A16 advantage shrinks with batch and context (Figure 4), and production runs at batch 16 or more.
- Expecting W4A16 to speed up prefill. It uses the same BF16 matrix units plus dequantization work. Only 8-bit activations make long prefills faster.
- Forgetting the KV cache. None of the five recipes compresses it. At large batch and long context it becomes the main cost of a decode step; that is the topic of session 110.
- Comparing engines instead of recipes. Run all six models in the same engine version with the same flags; a kernel change between versions can be larger than the difference between two recipes.
Interview questions
Why does 4-bit weight-only quantization speed up decode but not prefill?basic
What is the weight memory of Llama 3.1 8B in BF16, W4A16 (group 128) and W8A8, and why is W4A16 not exactly a quarter?basic
How do GPTQ and AWQ differ, and when would you expect each to win?core
Why does SmoothQuant help INT8 activations, and what does the \(\alpha\) parameter do?core
Why does FP8 often need no smoothing, while INT8 does?core
You measure a W4A16 decode speed-up of 1.4 at batch 16 with 2,048-token prompts. Is something wrong?advanced
Why do you measure TPOT as \((t(128) - t(1))/127\) instead of dividing one end-to-end time by the number of tokens?basic
Which recipe would you choose for a RAG service on L4s with 4,000-token prompts and 100-token answers?advanced
Your 4-bit model has the same perplexity as BF16 within 2%, but users complain about math answers. What happened, and how would you check?core
Why is the L4 a harder test for these recipes than an H100?advanced
Exercises
- Compute the weight memory of Llama 3.1 8B in W4A16 with group size 32 instead of 128 (one 16-bit scale per group, symmetric). How much does it grow?
- Using (112.9), compute the decode time per token at batch 1 and context 576 for a W3A16 copy (3 bits, group 128) on an L4. What speed-up over BF16 does the bound allow?
- In Figure 1, set the outliers to 1. Why does SmoothQuant barely improve on naive W8A8 there, and why is GPTQ still much better than RTN?
- From the survey’s table (Figure 4), take TensorRT-LLM, Llama-2-7B, batch 1, 128 tokens. Compute the ratio of the measured decode speed-up to the bound (112.11). Repeat for batch 16 and 2,048 tokens. What does the trend tell you?
- At batch 16 with 2,048-token prompts and 128 new tokens, how much GPU memory does each of the three formats need (weights plus KV cache)? Which fit in vLLM’s default budget of 21.6 GB on an L4? Up to what batch does the BF16 copy fit?
- Run the lab (or, without an L4, the rehearsal) and write the one-page recommendation for a workload of your choice. Include at least one number that disagreed with the prediction and your explanation.
Answerscheck
Key takeaways
- RTN, GPTQ and AWQ make the same INT4 format (5.70 GB for Llama 3.1 8B with group 128); they differ only in which codes they pick. GPTQ corrects errors with the Hessian, AWQ protects salient channels. Both beat RTN clearly.
- SmoothQuant and FP8 quantize activations too (9.08 GB). SmoothQuant moves outlier range from activations to weights; FP8’s floating-point grid tolerates outliers by itself.
- Predict before measuring. On an L4, W4A16 should cut decode from 50.3 to 15.7 ms per token at batch 1 and leave long prefills unchanged; W8A8 should halve prefill and cut decode to 27.0 ms.
- The W4A16 decode advantage shrinks with batch and context because the KV cache is not compressed; the survey’s A100 table shows this trend, at about 45 to 80% of the byte bound.
- Measure quality with perplexity and generative tasks, and latency as two phases (\(t(1)\) and \(t(n)\)) at batch 1 and 16, with the same engine, data and flags for every model.
- Decide from the workload: decode-heavy traffic favors W4A16, prefill-heavy or high-batch traffic favors W8A8, and the quality bar on your own tasks comes first.
Key formulas
References for this session
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- NVIDIA (code). TensorRT-LLM
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Micikevicius et al. (2022). FP8 Formats for Deep Learning
- NVIDIA (code). TensorRT-LLM
- vLLM project (docs). vLLM documentation
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- EleutherAI (code). LM Evaluation Harness
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- vLLM project (code). LLM Compressor: a library for quantizing models for vLLM
- Meta (2024). Llama 3.1 8B Instruct: model card and config.json
- NVIDIA (docs). NVIDIA L4 Tensor Core GPU: product specifications
- Merity et al. (2016). Pointer Sentinel Mixture Models (the WikiText datasets)
- Clark et al. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems (GSM8K)
- Neural Magic / Red Hat AI (2024). Meta-Llama-3.1-8B-Instruct-quantized.w4a16 (model card with evaluation table)
- Neural Magic / Red Hat AI (2024). Meta-Llama-3.1-8B-Instruct-quantized.w8a8 (model card with evaluation table)
- Neural Magic / Red Hat AI (2024). Meta-Llama-3.1-8B-Instruct-FP8 (model card with evaluation table)
- Qwen Team (2024). Qwen2.5 Technical Report