Where quantization helps: weight-only vs weight–activation
Quantization does not make every part of inference faster. Weight-only quantization (W4A16, W8A16) makes the weights smaller, so it speeds up decode, which waits for memory. Only weight–activation quantization (W8A8, FP8) makes the arithmetic faster, so only it can speed up prefill. We derive this with a roofline model and check it against the full table of measured AWQ speed-ups on an A100 from the efficient-inference survey.
What you will learn
- What W4A16, W8A16, W8A8 and FP8 mean, and how the weight-only and weight–activation workflows differ inside one linear layer
- Why decode (GEMV, memory-bound) gains from weight-only quantization and prefill (GEMM, compute-bound) does not
- How to compute the roofline speed-up of each recipe for one layer, including the effective bits of group-wise scales and the batch where the gain ends
- How to read the survey’s measured W4A16 results: about 2.0× to 2.5× in decode at batch 1, much less at batch 16, and 0.75× to 1.32× in prefill
- Why the gain shrinks with batch and context (the KV cache stays FP16) and grows with model size
- How to choose a recipe for chat, RAG, VLM and video workloads, and where W4A8, FP8 and KV-cache quantization fit
Why this matters
Quantize Llama 2 7B to 4-bit weights with AWQ and serve it on one A100 GPU. Each new token comes about 2.4 times faster. But the prompt is read slightly slower than before. The same quantized model speeds up one phase and slows down the other. This session explains why, and what to do about it.
In session 103 you learned how to quantize a tensor: a scale, a zero point, a granularity, and the error that comes with them. This session asks which tensors to quantize, and why that choice decides where the speed-up appears. There are two main recipes:
- Weight-only quantization: store the weights in 4 or 8 bits, keep the activations (the inputs of each layer) in 16 bits, and do the math in 16 bits. Names: W4A16, W8A16 (W = weight bits, A = activation bits).
- Weight–activation quantization: quantize both the weights and the activations, and do the math in low precision on faster tensor cores. Names: W8A8 (INT8) and FP8.
From session 82 you know the two phases of generation behave very differently. Prefill (reading the prompt) is one big matrix–matrix product (GEMM) per layer, and it is limited by arithmetic: it is compute-bound. Decode (making one token per step) is a matrix–vector product (GEMV) per layer, and it is limited by how fast the weights arrive from memory: it is memory-bound. The main message of this session follows directly: [Certain]
- Weight-only quantization makes the weights smaller, so it speeds up the memory-bound phase: decode.
- Only weight–activation quantization makes the arithmetic faster, so only it can speed up the compute-bound phase: prefill.
We follow the efficient-inference survey of Zhou et al. (section 5.2.1), which states this rule and tests it: W4A16 with AWQ on LLaMA-2-7B and 13B, on one NVIDIA A100, in two serving frameworks (TensorRT-LLM and LMDeploy), over 5 batch sizes and 5 context lengths. We use their full table in Figure 3, and we check it against a roofline model that you can build yourself.
Two recipes: weight-only and weight–activation
Every linear layer computes \(Y = X W^\top\). Here \(X\) holds the activations of the \(m\) tokens that pass through the layer at the same time (\(m\) = batch size in decode, or prompt length times batch in prefill), and \(W\) is the weight matrix. The two recipes quantize different parts of this product.
Weight-only (W4A16, W8A16)
The weights are quantized once, offline, for example with GPTQ or AWQ (session 106, session 107). They are stored as small integers \(q\) plus a scale \(s\) and a zero point \(z\) for each group of \(g\) weights (often \(g = 128\)). At run time the kernel loads the small integers, turns them back into FP16 numbers on the chip, and then does an ordinary FP16 matrix product:
- Read the symbols
- \(q_{ij}\)
- the stored integer for weight \(i, j\): 0 to 15 for 4 bits
- \(G(j)\)
- the group that input column \(j\) belongs to: groups of \(g\) neighbouring columns in the same row share one scale and one zero point
- \(s, z\)
- the FP16 scale and zero point of that group (from session 103)
- \(\hat W\)
- the dequantized weights: FP16 numbers close to the original \(W\). They exist only inside the chip, a small tile at a time; they are never written back to memory
So the math is unchanged: same FP16 tensor cores, same number of FLOPs, plus a little extra work to dequantize. What changes is the number of bytes read from memory. With 4-bit integers and an FP16 scale and zero for every 128 weights, each weight costs
instead of 16. The weights become \(16 / 4.25 = 3.76\) times smaller. [Certain] Smaller groups cost more: \(g = 64\) gives 4.5 bits.
Weight–activation (W8A8, FP8)
Here the activations are quantized too. The weights are still quantized offline (INT8 with one scale per output channel, for example). But the activations change with every input, so they must be quantized online, before each matrix product, usually with one scale per token. Then the product runs on low-precision tensor cores, which accumulate in INT32 (for INT8) and rescale once at the end:
- Read the symbols
- \(X^{q}, W^{q}\)
- INT8 activations and weights (−127 to 127)
- \(s^{x}_{t}\)
- scale of token \(t\): computed at run time, \(\max_j |X_{tj}| / 127\)
- \(s^{w}_{n}\)
- scale of output channel \(n\): fixed offline
- \(\sum_j\)
- the integer dot product, done by INT8 tensor cores and summed in INT32, so no overflow
The scales factor out of the sum because one scale belongs to a row of \(X\) and one to a row of \(W\). That is why per-token and per-channel scales are the standard choice: finer groups along \(j\) would break the single integer sum. [Certain] The hard part is accuracy: activations of LLMs have large outliers in a few channels, so naive INT8 activations lose accuracy; LLM.int8() and SmoothQuant were built to fix this (session 108). The survey’s Figure 9 draws the two workflows side by side; Figure 1 below follows the same idea.
How to read Figure 1
Step through the five stages. The highlighted box is the current stage; FP16 data is drawn in the query colour, low-bit data in the value colour. Watch two things: the bytes read in the “load” stage (23.95 MB for W4A16, 45.1 MB for W8A8, against 90.2 MB in FP16), and the tensor cores used in the “multiply” stage (FP16 for W4A16, INT8 for W8A8). Switch between \(m = 1\) and \(m = 2048\) and look at the last stage: which recipe wins depends on \(m\).
- offline, once: for each row of \(W\) and each group of \(g = 128\) columns, compute \(s, z\); store \(q = \mathrm{clip}(\mathrm{round}(W/s) + z, 0, 15)\), two values per byte
- input: FP16 activations \(X\) (\(m \times k\)), packed \(q\), FP16 \(s, z\)
- for each output tile of \(Y\) (in parallel over thread blocks) do
- for each slice of \(k\) do
- load a tile of packed \(q\) and its \(s, z\) from HBM // 4.25 bits per weight: the saving
- unpack and compute \(\hat W = s\,(q - z)\) in registers // extra work on CUDA cores
- accumulate \(X_{\text{tile}} \hat W_{\text{tile}}^\top\) on FP16 tensor cores, FP32 accumulator
- write the FP16 output tile
- return \(Y\) // same FLOPs as FP16; 3.76× fewer weight bytes
- offline, once: \(s^w_n = \max_j |W_{nj}| / 127\); \(W^q = \mathrm{round}(W / s^w)\) as INT8
- input: FP16 activations \(X\) (\(m \times k\))
- \(s^x_t \leftarrow \max_j |X_{tj}| / 127\) for every token \(t\) // online, every call; often fused into the previous kernel
- \(X^q \leftarrow \mathrm{round}(X / s^x)\) as INT8
- \(A \leftarrow X^q (W^q)^\top\) on INT8 tensor cores, INT32 accumulator // 2× the FP16 rate on A100 and H100
- \(Y \leftarrow A \odot (s^x (s^w)^\top)\) in FP16 // one rescale per output, fused into the GEMM epilogue
- return \(Y\) // half the weight bytes and twice the math speed
One linear layer on the roofline
Now make the rule exact. Take one linear layer with input size \(k\), output size \(n\), and \(m\) tokens. Count its FLOPs and its bytes moved from memory (HBM):
- Read the symbols
- \(F\)
- FLOPs: one multiply and one add for each weight and each token
- \(Q\)
- bytes: weights read once, activations read, FP16 outputs written
- \(b_w, b_a\)
- bits per weight (16, 8.25 or 4.25 with group scales, 8 with per-channel scales) and bits per activation (16 or 8)
- \(m, k, n\)
- tokens in the product, input size, output size
As in session 82, the step takes at least the larger of compute time and memory time:
- Read the symbols
- \(P_{\text{fmt}}\)
- dense tensor-core speed for the math format. A100: 312 TFLOP/s in FP16, 624 TOPS in INT8. H100 SXM: 989 TFLOP/s in BF16/FP16, 1,979 TFLOP/s in FP8
- \(B\)
- memory bandwidth: 2,039 GB/s on the A100 80 GB SXM, 3.35 TB/s on the H100 SXM
The A100 and H100 numbers are from NVIDIA’s product pages; the pages list “with sparsity” values that are twice these dense ones. [Certain] When \(m\) is much smaller than \(k\) and \(n\), the weight bytes dominate \(Q\), and the arithmetic intensity (FLOPs per byte) is
The layer is memory-bound while \(I\) is below the ridge point \(P/B\) of the GPU (153 FLOP/byte for FP16 on an A100). In FP16 (\(b_w = 16\)), \(I = m\): the layer is memory-bound until \(m \approx 153\). Now compare the recipes. Write \(m^\ast = P_{16}/B\) for the FP16 ridge. Ignoring activation bytes, the speed-ups over FP16 are
- Read the symbols
- \(m^\ast\)
- the token count where FP16 becomes compute-bound: 153 on an A100, 295 on an H100
- \(m^\ast b_{\text{eff}}/16\)
- where W4A16 becomes compute-bound: 40.6 tokens on an A100, 78.4 on an H100
- \(P_8 / P_{16}\)
- the speed ratio of 8-bit and 16-bit tensor cores: 2 on both GPUs
Read the left formula in three parts. For small \(m\) (decode), \(S = 16/b_{\text{eff}} = 3.76\): all the gain of the smaller weights. Between \(m^\ast b_{\text{eff}}/16\) and \(m^\ast\) it falls, because W4A16 has already hit the compute roof while FP16 has not. Above \(m^\ast\) (prefill) it is exactly 1: both run the same FP16 math at the same speed. W8A8 is different: it halves the bytes and doubles the math speed, so it gives 2× in both regimes. [Certain, as a roofline bound]
The roofline ignores one thing: the dequantization in Algorithm 1 is extra work on the general-purpose CUDA cores. In the memory-bound regime it hides behind the memory wait. In the compute-bound regime it adds to the time, so the real speed-up of W4A16 in prefill is below 1. The QServe paper reports dequantization overheads of 20% to 90% of runtime for existing INT4 kernels in large-batch serving. [Certain] Figure 2 lets you add such an overhead.
How to read Figure 2
Start at \(m = 1\): W4A16 is at 3.76×, W8A16 at 1.94×, W8A8 at 2×. Move right. The W4A16 curve stays high until about 32 tokens, then falls and reaches 1.0 near the FP16 ridge (153 on the A100). From there on, only W8A8 still gives a speed-up. Now raise the dequantization cost: the right part of the weight-only curves drops below 1, which is what the survey measured in prefill. The left part does not move, because there the dequantization hides behind the memory wait. Switch to the H100: its ridge is higher (295), so weight-only helps up to larger batches.
What the survey measured
The survey tested exactly this. LLaMA-2-7B and LLaMA-2-13B, weights quantized to 4 bits with AWQ (W4A16), one NVIDIA A100, served by TensorRT-LLM and by LMDeploy. For each batch size \(B \in \{1, 2, 4, 8, 16\}\) and context length \(\in \{128, 256, 512, 1024, 2048\}\), it reports three speed-ups over FP16: prefill latency, decode latency and end-to-end latency (their Table IV). [Certain] Figure 3 shows all 98 measured cells (two are out of memory).
How to read Figure 3
In the decode view, the top-left corner (batch 1, context 128) is the darkest green and the colour fades toward the bottom right: the gain shrinks as batch and context grow. Switch to prefill: almost every cell is red (slower than FP16). The only green prefill cells sit in the first column (context 128) at batch 1 or 2, where even prefill is small enough to be memory-bound. Switch to 13B: the decode greens are darker. The readout summarises the grid you are viewing.
Four observations, in the survey’s own order:
- Decode speeds up a lot. At batch 1 the decode speed-up is 2.00× to 2.40× (TensorRT-LLM, 7B), 1.92× to 2.11× (LMDeploy, 7B), 1.94× to 2.51× and 2.01× to 2.34× for 13B. The end-to-end latency follows decode, because decode takes most of the time. [Certain]
- Prefill does not speed up and often slows down. Prefill speed-ups range from 0.75× to 1.32×; 92 of the 98 cells are below 1. The survey gives two reasons: prefill is compute-bound, and dequantizing the weights adds work. [Certain]
- The gain shrinks as batch and context grow. At batch 16, the decode speed-up of the 7B model drops to 1.16× to 1.43× (TensorRT-LLM) and 1.28× to 1.92× (LMDeploy). [Certain]
- Bigger models gain more. Averaged over the same cells, decode speed-up is 1.71× (7B) against 1.94× (13B) in TensorRT-LLM, and 1.88× against 2.04× in LMDeploy; 13B wins in 46 of 48 matched cells. [Certain, computed from the table]
The six prefill cells at or above 1.0 are all at context 128 and batch 1 or 2: 1.06 and 1.24 (TensorRT-LLM, batch 1), 1.30, 1.03, 1.32, 1.06 (LMDeploy). Our roofline predicts this: a 128-token prefill of LLaMA-2-7B at batch 1 needs 5.43 ms of FP16 math but 6.51 ms to read the FP16 weights, so it is still memory-bound, and W4A16 should give about 1.20×. At batch 2 the math doubles to about 10.9 ms, the pass becomes compute-bound, and the bound is 1.00×. [Certain, as a roofline bound]
Why the gain shrinks, and why bigger models gain more
One linear layer is not the whole decode step. A decode step also reads the KV cache: the stored keys and values of every past token of every sequence (session 80). W4A16 shrinks the weights but not the KV cache, which stays FP16. For a memory-bound decode step of a model with \(N_{\text{lin}}\) weights in quantized layers and \(N_{\text{head}}\) weights in the output head (often kept in FP16), the ideal speed-up is
- Read the symbols
- \(B, c\)
- batch size and context length (tokens already in each sequence’s cache)
- \(\text{kv}_{\text{tok}}\)
- KV-cache bytes per token in FP16: keys and values, all \(L\) layers, \(H_{kv}\) KV heads of size \(d_h\)
- numerator, denominator
- bytes read in one step in FP16 and in W4A16. The KV term is the same in both, so it pulls the ratio toward 1
This explains the survey’s observations 3 and 4. The KV term grows with \(B \cdot c\), so the speed-up falls with both batch and context. And at larger \(B\), the compute term also grows (formula 104.7), which caps the gain further. A bigger model has more weight bytes per byte of KV cache: LLaMA-2-7B reads 13.2 GB of weights per step and 0.5 MiB of KV per token (both use plain multi-head attention, \(H_{kv} = 32\) and 40); LLaMA-2-13B reads 25.7 GB and 0.78 MiB. So its weights equal the KV cache of about 25,200 tokens for 7B and 31,400 tokens for 13B: the 13B weights stay dominant longer, and the speed-up stays higher. [Certain, from the configs]
How to read Figure 4
Each bar is one decode step, drawn as time (bytes divided by bandwidth). The green part is weights, the orange part KV cache. At batch 1, context 128, the FP16 bar is almost all green and W4A16 removes most of it: 3.52× in theory. Raise the batch and the context: the orange part, which W4A16 does not shrink, takes over, and the ratio falls. The measured numbers are always lower than the bound (between 44% and 88% of it over the survey grid, median 65%), because real kernels do not reach full bandwidth and parts of the step (attention, normalization, sampling, kernel launches) do not shrink with the weights. The trend, however, is the same.
Worked example: LLaMA-2-7B on an A100
Shapes. From the model’s config.json: \(d = 4096\), MLP width 11,008, \(L = 32\) layers, 32 heads with 32 KV heads, vocabulary 32,000, embeddings not tied. The 7 projections of all layers hold \(N_{\text{lin}} = 32 \times (4 \cdot 4096^2 + 3 \cdot 4096 \cdot 11008) = 6{,}476{,}005{,}376\) weights; the output head holds \(N_{\text{head}} = 131{,}072{,}000\). All parameters: 6,738,415,616, which is 13.48 GB in FP16. [Certain]
Weights read per decode step. FP16: \(2 \times (6.476 + 0.131) \times 10^9 = 13.21\) GB. W4A16 with \(g = 128\) (head kept in FP16): \(6.476 \times 10^9 \times 4.25 / 8 + 0.262 \times 10^9 = 3.70\) GB.
Decode, batch 1, context 128. KV cache: \(128 \times 524{,}288\) bytes = 0.067 GB. FP16 step: \((13.21 + 0.07) / 2039 = 6.51\) ms. W4A16 step: \((3.70 + 0.07) / 2039 = 1.85\) ms. Compute is only 0.043 ms, so both steps are memory-bound. Bound on the speed-up: \(6.51 / 1.85 = 3.52\times\). Measured: 2.40× (TensorRT-LLM) and 2.11× (LMDeploy), that is 68% and 60% of the bound.
Decode, batch 16, context 2048. KV cache: \(16 \times 2048 \times 524{,}288\) bytes = 17.18 GB, more than the FP16 weights. FP16 step 14.91 ms, W4A16 step 10.24 ms: bound 1.46×. Measured: 1.16× and 1.28×.
Prefill, batch 1, context 2048. FLOPs: \(2 \times 6.607 \times 10^9 \times 2048 + 2 \times 32 \times 4096 \times 2048^2 = 28.2\) TFLOP, so 90.3 ms of FP16 math against 7.0 ms of memory traffic. Compute-bound; W4A16 does the same FP16 math, so the bound is 1.00×. Measured: 0.91× and 0.94×: with the dequantization, speed is 6% to 9% lower than FP16.
What W8A8 would give. INT8 tensor cores run at 624 TOPS, twice FP16. The same prefill would need at least 45.1 ms of math: a bound of 2.0×. In decode at batch 1, 8-bit per-channel weights make the step read \(6.48 + 0.26 + 0.07 = 6.81\) GB: a bound of \(13.28 / 6.81 = 1.95\times\), less than W4A16’s 3.52×. So for this model: INT4 weight-only for latency-critical decode, 8-bit weight–activation for prefill-heavy or large-batch work.
The survey also shows that the end-to-end speed-up tracks decode (2.37× at batch 1, context 128, TensorRT-LLM, 7B). That is because generating many tokens takes far longer than reading the prompt. For a workload with long prompts and short answers (classification, retrieval reranking, a VLM reading many image tokens and answering “yes”), prefill dominates, and W4A16 can make the request slower overall. [Likely]
Beyond the two recipes
Real systems mix the ideas. Here is the map; the details come in the next sessions.
- FP8 (W8A8 in floating point). On H100-class GPUs, FP8 tensor cores run at twice the BF16 rate (1,979 against 989 TFLOP/s dense). FP8 keeps an exponent, so outliers in activations hurt less than with INT8; ZeroQuant-FP found FP formats better than INT for activations. [Certain] It is a common default for serving on Hopper GPUs. [Likely] The formats themselves are in session 105.
- W4A8. 4-bit weights for decode bandwidth plus 8-bit activations for prefill math. QServe (W4A8KV4) reports 1.2× higher maximum serving throughput for Llama-3-8B and 2.4× for Qwen1.5-72B on an A100, compared with TensorRT-LLM. [Certain, as reported by the authors] The catch is that the 4-bit weights must be converted to 8 bits inside the GEMM cheaply, without slowing the tensor cores.
- W4A4 and lower. 4-bit tensor cores (on A100, and FP4 on newer GPUs) double the math speed again, but 4-bit activations are hard to keep accurate; Atom and rotation methods (session 109) try. [Likely]
- KV-cache quantization. Formula (104.8) shows that at large \(B \cdot c\) the KV cache, not the weights, dominates decode bytes. Quantizing the cache (KIVI, KVQuant) attacks that term directly (session 110). [Certain]
- Memory size. Even when W4A16 gives no speed-up, it frees memory. LLaMA-2-7B drops from 13.48 GB to about 3.97 GB of stored weights, leaving room for a larger batch or a longer context, which raises throughput (session 84). AWQ’s TinyChat uses this to run Llama-2-70B on mobile GPUs. [Certain]
- input: workload (prompt length \(T\), answer length \(n\), batch \(B\)), GPU (\(P_{16}, P_8, B_{\text{mem}}\), memory size), accuracy budget
- if the FP16 model does not fit in GPU memory then use weight-only INT4 first // memory size beats everything
- estimate prefill time \(t_p\) and total decode time \(t_d\) with formula (104.5)
- if \(t_d \gg t_p\) and \(B\) is small (below about \(m^\ast b_{\text{eff}}/16\)) then weight-only W4A16 // chat, agents, long answers
- else if \(t_p\) dominates or \(B\) is large then W8A8: FP8 on Hopper or newer, INT8 with SmoothQuant on Ampere // RAG, long prompts, big batches
- else consider W4A8 kernels (QServe-style), or W8A8 plus KV-cache quantization
- if \(B\,c\,\text{kv}_{\text{tok}} \gt\) weight bytes then add KV-cache quantization // formula (104.8)
- check accuracy on your task, then measure prefill and decode separately // never trust one end-to-end number
In code
The script quantizes one \(4096 \times 4096\) layer both ways and checks the output error, then evaluates the roofline of a \(4096 \to 11008\) layer on an A100 for several \(m\). It needs only PyTorch on a CPU and is deterministic (seed 0). The dequantization here makes a full FP16 copy of the weights; real kernels (Marlin, AWQ, TensorRT-LLM) dequantize small tiles inside the GEMM instead, or there would be no memory saving at all.
import torch torch.manual_seed(0) # ---- 1) Weight-only W4A16: 4-bit weights, groups of 128, FP16 scale and zero per group ---- def quant_w4(W, g=128): # W: (out, in) o, i = W.shape Wg = W.reshape(o, i // g, g) lo, hi = Wg.amin(-1, keepdim=True), Wg.amax(-1, keepdim=True) s = (hi - lo).clamp(min=1e-8) / 15 # 16 levels: 0..15 z = torch.round(-lo / s) # integer zero point q = torch.clamp(torch.round(Wg / s) + z, 0, 15).to(torch.uint8) packed = q[..., 0::2] | (q[..., 1::2] << 4) # two 4-bit values per byte return packed, s.half(), z.half() def dequant_w4(packed, s, z): q = torch.stack([packed & 15, packed >> 4], -1).flatten(-2).float() return ((q - z.float()) * s.float()).flatten(1) # back to (out, in) # ---- 2) Weight-activation W8A8: per-token activation scale, per-channel weight scale ---- def quant_sym8(t, dim): s = t.abs().amax(dim=dim, keepdim=True).clamp(min=1e-8) / 127 return torch.clamp(torch.round(t / s), -127, 127).to(torch.int8), s def linear_w8a8(X, W): Xq, sx = quant_sym8(X, dim=1) # online, every forward: one scale per token Wq, sw = quant_sym8(W, dim=1) # offline: one scale per output channel acc = Xq.to(torch.int32) @ Wq.to(torch.int32).T # integer matmul, int32 accumulate return acc.float() * sx * sw.T # rescale the output once d_in, d_out = 4096, 4096 W = torch.randn(d_out, d_in) * 0.02 X = torch.randn(8, d_in) Y = X @ W.T packed, s, z = quant_w4(W) Y_w4 = X @ dequant_w4(packed, s, z).T # real kernels dequantize inside the GEMM Y_w8a8 = linear_w8a8(X, W) rel = lambda A: ((A - Y).norm() / Y.norm()).item() bytes_fp16 = W.numel() * 2 bytes_w4 = packed.numel() + s.numel() * 2 + z.numel() * 2 print(f"W4A16 rel. error {rel(Y_w4):.4f} weight bytes {bytes_w4/2**20:.2f} MiB vs {bytes_fp16/2**20:.0f} MiB" f" ({bytes_fp16/bytes_w4:.2f}x smaller, {8*bytes_w4/W.numel():.2f} bits/weight)") print(f"W8A8 rel. error {rel(Y_w8a8):.4f}") # ---- 3) Roofline estimate for one linear layer: time = max(FLOPs / P, bytes / BW) ---- def layer_time(m, k, n, wbits, P, BW, abytes=2): flops = 2 * m * k * n bytes_ = k * n * wbits / 8 + abytes * m * k + 2 * m * n return max(flops / P, bytes_ / BW) P16, P8, BW = 312e12, 624e12, 2.039e12 # A100 SXM: dense FP16 and INT8 tensor cores, HBM for m in [1, 16, 64, 256, 2048]: # tokens in the matmul: decode batch or prompt length t16 = layer_time(m, 4096, 11008, 16, P16, BW) tw4 = layer_time(m, 4096, 11008, 4.25, P16, BW) # W4A16 still does FP16 math tw8a8 = layer_time(m, 4096, 11008, 8, P8, BW, abytes=1) print(f"m={m:5d} W4A16 speedup {t16/tw4:4.2f} W8A8 speedup {t16/tw8a8:4.2f}") # W4A16 rel. error 0.1001 weight bytes 8.50 MiB vs 32 MiB (3.76x smaller, 4.25 bits/weight) # W8A8 rel. error 0.0124 # m= 1 W4A16 speedup 3.76 W8A8 speedup 2.00 # m= 16 W4A16 speedup 3.71 W8A8 speedup 1.99 # m= 64 W4A16 speedup 2.44 W8A8 speedup 1.97 # m= 256 W4A16 speedup 1.00 W8A8 speedup 2.00 # m= 2048 W4A16 speedup 1.00 W8A8 speedup 2.00
Two notes on the output. The W4A16 error of 10% is large because these are random Gaussian weights quantized by plain rounding: real weights quantized with GPTQ or AWQ keep perplexity close to FP16 even when the per-layer error looks like this. And the roofline part reproduces the shape of Figure 2: 3.76× for weight-only at \(m = 1\), falling to 1.00× by \(m = 256\), while W8A8 stays near 2×.
Where this is used
| Setting | Phase that dominates | Typical choice |
|---|---|---|
| Local or single-user chat (llama.cpp, laptops, one GPU) | Decode at batch 1: memory-bound | Weight-only 4-bit (GPTQ, AWQ, GGUF formats); also needed just to fit in memory |
| Cloud chat serving at high load | Decode at large batch, plus many prefills | FP8 W8A8 on H100; W4A8 or W4A16 with Marlin-style kernels at moderate batch; KV-cache quantization |
| RAG and long-document QA | Prefill of long prompts: compute-bound | W8A8 / FP8; weight-only helps little for TTFT |
| Classification, reranking, embedding models | Prefill only (one forward pass, no decode) | W8A8 / FP8, or INT8 with SmoothQuant |
| VLM video understanding (classification, action recognition, key-moment detection) | Prefill of thousands of visual tokens, short answers | FP8 or INT8 activations for the LLM and the vision encoder; fewer visual tokens matter even more |
| Dense video captioning, long video summaries | Decode of long answers after a long visual prefill | Mixed: FP8 for prefill speed; weight-only 4-bit helps the decode part at small batch |
| Video-RAG over long recordings | Prefill of retrieved clips plus a large KV cache | W8A8 plus KV-cache quantization; weight-only to free memory for the cache |
| Edge and phones | Decode, and memory size first | 4-bit (or lower) weight-only; INT8 activations where the NPU supports it |
The book’s chapter on efficient VLMs (Noyan et al., chapter 7) treats quantization as one of the main tools for deploying VLMs; the rule of this page tells you which variant fits which part of a VLM pipeline. A vision encoder processes all image patches in one pass, so it behaves like prefill. [Certain]
Common mistakes
- Expecting W4A16 to cut time to first token. Prefill is compute-bound and the math is still FP16. The survey measured 0.75× to 1.32×, mostly below 1.
- Quoting the ideal 3.76× (or 4×) as the decode speed-up. The survey measured 1.9× to 2.5× at batch 1. The KV cache, attention, normalization, sampling and launch overheads do not shrink with the weights.
- Benchmarking at batch 1 only. A W4A16 model that is 2.4× faster at batch 1 can be only 1.2× faster at batch 16, and slower at batch 128. Measure at the batch you will serve.
- Reporting one end-to-end number. Measure prefill and decode separately (TTFT and TPOT, session 81); the two can move in opposite directions.
- Dequantizing the whole matrix into memory. A kernel that writes \(\hat W\) back to HBM reads and writes more bytes than FP16. The dequantization must happen tile by tile inside the matrix product.
- Using per-group scales for activations in W8A8. Scales along the summed dimension do not factor out of the integer sum (formula 104.3). Use per-token scales for activations and per-channel scales for weights, or a kernel built for groups.
- Forgetting the hardware. INT8 W8A8 needs INT8 tensor cores (A100, H100); FP8 needs Hopper or newer. On a GPU without fast low-precision math, W8A8 gives no compute gain.
- Ignoring accuracy of activations. Weight-only INT4 is usually easier to keep accurate than INT8 activations, because LLM activations have large outlier channels. This is why weight-only methods came first.
Interview questions
What do W4A16 and W8A8 mean?basic
Why does weight-only quantization speed up decode?basic
Why does W4A16 not speed up prefill, and why can it slow it down?core
Which kind of quantization speeds up prefill, and by how much at most?core
Derive the batch size where W4A16 stops helping a decode GEMM on an A100.advanced
The decode speed-up of W4A16 drops from 2.4× at batch 1 to about 1.2× at batch 16 with long contexts. Why?core
Why do bigger models gain more from weight-only quantization?core
Why are W8A8 scales per token for activations and per channel for weights?advanced
A team quantizes their RAG service to INT4 weight-only and sees TTFT get worse. What do you suggest?advanced
Can weight-only quantization raise throughput even when it gives no speed-up per step?advanced
Exercises
- With 4-bit weights, an FP16 scale and no zero point, what is \(b_{\text{eff}}\) for \(g = 128\) and for \(g = 32\)? What is the ideal decode speed-up of each over FP16?
- On an H100 SXM (989 TFLOP/s BF16, 3.35 TB/s), up to what \(m\) is a W4A16 (4.25-bit) GEMM memory-bound? Up to what \(m\) is a BF16 GEMM memory-bound?
- Use formula (104.8): what is the ideal W4A16 decode speed-up of LLaMA-2-13B at batch 1 and context 2048? Compare with the survey (TensorRT-LLM 1.94×, LMDeploy 2.01×).
- For LLaMA-2-7B, at what batch × context product does the FP16 KV cache equal the FP16 weights read per step?
- A W8A16 model (8 bits, per-channel scales) on an A100: what is the roofline speed-up of a decode GEMM at \(m = 1\), and of a prefill GEMM at \(m = 2048\)? And for W8A8?
- Use Figure 3: in LMDeploy, 7B, which batch size first shows a decode speed-up below 1.5× at context 2048?
Answerscheck
Key takeaways
- Weight-only quantization (W4A16, W8A16) makes weights smaller but keeps FP16 math. It helps memory-bound work: decode at small batch. Bound: \(16/b_{\text{eff}}\), 3.76× for 4-bit with groups of 128.
- Weight–activation quantization (W8A8, FP8) also makes the math faster, so it is the only one of the two that speeds up compute-bound prefill: about 2× on A100 (INT8) and H100 (FP8).
- The survey (A100, AWQ W4A16): decode 1.9× to 2.5× faster at batch 1; at batch 16 down to 1.16× to 1.43× for 7B in TensorRT-LLM; prefill 0.75× to 1.32×, below 1 in 92 of 98 cells.
- The gain shrinks as batch and context grow (KV cache stays FP16, GEMMs move toward the compute roof) and grows with model size (weights dominate the bytes longer).
- Always measure prefill and decode separately, at the batch you will serve; weight-only quantization also frees memory, which can raise throughput even without a per-step speed-up.
Key formulas
References for this session
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Micikevicius et al. (2022). FP8 Formats for Deep Learning
- Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
- Frantar, Castro, Chen, Hoefler, Alistarh (2024). MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models
- Lin, Tang, Yang, Zhang, Xiao, Gan, Han (2024). QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
- Wu, Yao, He (2023). ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
- Zhao et al. (2023). Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
- Liu et al. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Meta (mirrored by NousResearch on Hugging Face) (2023). Llama-2-7b-hf: config.json
- NVIDIA (docs). NVIDIA A100 Tensor Core GPU: product specifications
- NVIDIA (docs). NVIDIA H100 Tensor Core GPU: product specifications
- NVIDIA (code). TensorRT-LLM
- InternLM team (code). LMDeploy: a toolkit for compressing, deploying and serving LLMs