Weight PTQ 2: AWQ, outliers and mixed precision
At 3 or 4 bits, a few weights decide most of the quality loss: the ones that meet large activations, and a handful of very large or very sensitive values. This session shows three ways to protect them: scale them up before rounding (AWQ), keep them in 16 bits (OWQ, SpQR), or place the quantization levels where they matter (SqueezeLLM). It ends with LUT-GEMM, a kernel that runs such formats without turning the weights back into floats.
What you will learn
- Explain why the output error of a quantized layer is the weight error weighted by the size of the activations, and why 1% of the input channels can cause most of it
- Derive the AWQ scaling trick: multiply a salient weight column by s, divide its input by s, and see the error shrink by about 1/s until the group maximum changes
- Run the AWQ search over s = s_X^α, know where the 1/s is folded (RMSNorm, v_proj, up_proj) and why it needs no backpropagation
- Pick weights to keep in FP16 with a Hessian-based sensitivity (OWQ columns, SpQR single weights) and count the extra bits per weight
- Build non-uniform levels with Fisher-weighted k-means and a dense-and-sparse split (SqueezeLLM), and say what they cost
- Explain how LUT-GEMM multiplies binary-coded weights with table lookups, and why weight-only formats speed up decoding but not large-batch prefill
- Read the published perplexity numbers of RTN, GPTQ, AWQ, OWQ, SpQR and SqueezeLLM without mixing up their settings
Why this matters
A 7B model has about 6.74 billion weights. In FP16 they take 13.5 GB. At 4 bits with one 16-bit scale and one 16-bit zero point per group of 128 weights, they take about 3.6 GB. During decoding the GPU must read every weight once per step, so fewer bytes means more tokens per second.
The previous session (session 106) showed GPTQ: round one column at a time and let the not-yet-rounded columns correct the error, using second-order (Hessian) information. This session asks a different question: which weights matter most? At 4 bits plain rounding is almost fine. At 3 bits it is not, and most of the damage comes from a very small set of weights. There are three practical answers, and each became a widely used method:
- Scale the important weights up before rounding, so they get relatively finer steps. This is AWQ (Activation-aware Weight Quantization, Lin et al.), which won the MLSys 2024 best paper award. [Certain]
- Keep the important weights in 16 bits and quantize the rest: OWQ keeps whole columns, SpQR keeps single weights in a sparse list.
- Put the quantization levels where the important weights are: SqueezeLLM learns non-uniform levels with a weighted k-means and keeps outliers in a sparse list.
Non-uniform or binary-coded weights then need a special kernel. LUT-GEMM replaces multiplications by table lookups. In the efficient-inference survey (Zhou et al.) these methods sit in Section 5.2.1, weight-only post-training quantization.
This is not only a research topic. The AWQ authors list integrations in vLLM, Hugging Face TGI, TensorRT-LLM, LMDeploy and Hugging Face transformers (from_pretrained). [Certain] Many 4-bit checkpoints on the Hugging Face Hub are AWQ or GPTQ files. [Likely]
Not all weights are equal: the output view
A quantizer rounds each weight \(w_j\) to a nearby value \(Q(w_j)\). We do not care about the weight error itself. We care about the error of the layer output. For one output \(y = \sum_j w_j x_j\):
- Read the symbols
- \(w_j, x_j\)
- the weight on input channel \(j\), and the activation that arrives on that channel
- \(Q(w_j) - w_j\)
- the rounding error of that weight
- \(\E[x_j^2]\)
- how large channel \(j\)’s activations are on average (over tokens of a calibration set)
- \(\approx\)
- we drop the cross terms; this is exact if different channels are uncorrelated, and a good first picture otherwise
Formula (107.1) says: a weight’s rounding error counts in proportion to the size of its input. In LLMs a few input channels carry activations that are 10 to 100 times larger than the rest. This is the same outlier effect that LLM.int8() found (session 108). Weights in those channels are the salient weights. The same weight error there costs 100 to 10,000 times more output error.
The AWQ paper tested this directly on OPT models at INT3 with groups of 128. It kept a small share of input channels in FP16 and rounded the rest, choosing the channels three ways:
| OPT-6.7B, WikiText-2 perplexity (FP16: 10.86, plain INT3 rounding: 23.54) | keep 0.1% | keep 1% | keep 3% |
|---|---|---|---|
| channels with the largest activations | 11.58 | 11.39 | 11.36 |
| channels with the largest weight norm | 23.41 | 22.37 | 22.45 |
| random channels | 23.54 | 24.23 | 24.22 |
Source: AWQ paper, Table 1. [Certain] Keeping 1% of the channels picked by activation size removes almost the whole gap. Picking by weight size is no better than random. The weights themselves look ordinary; it is their inputs that make them important.
But keeping 1% of the columns in FP16 is a problem for the hardware: the matrix now has two data types, and the kernel must handle both. AWQ’s key idea is to get the same protection without mixed precision.
AWQ: scale the salient weights before rounding
Start with the quantizer. For a group of weights (for example 128 weights of one output row), a symmetric \(b\)-bit quantizer uses one step size \(\Delta\):
(The AWQ paper writes \(2^{b-1}\) in the denominator; the exact integer range does not change the argument.) The rounding error is at most \(\Delta/2\), and on average its root mean square is \(\Delta/\sqrt{12}\). Its size depends only on \(\Delta\), not on \(w\).
Now the trick. Multiply weight column \(j\) by a number \(s \gt 1\) and divide its input by the same \(s\). The product does not change:
The error of the scaled weight is still about the size of the step \(\Delta'\) of its group. But in the output it is divided by \(s\). If scaling one weight in a group of 128 does not change the group maximum, then \(\Delta' = \Delta\), and the output error from this weight shrinks by the factor \(1/s\). In general:
- Read the symbols
- \(s\)
- the scale applied to one input channel: its weight column is multiplied by \(s\), its activation divided by \(s\)
- \(\Delta, \Delta'\)
- the step size of the group before and after scaling; \(\Delta' \gt \Delta\) only if the scaled weight becomes the new group maximum
So scaling helps as long as \(\Delta'/\Delta\) stays near 1. If \(s\) is too large, the scaled salient weight becomes the largest in its group, \(\Delta'\) grows, and every other weight in the group gets a coarser grid. The AWQ paper measured this on OPT-6.7B (1% salient channels, INT3):
| s | 1 | 1.25 | 1.5 | 2 | 4 |
|---|---|---|---|---|---|
| groups where \(\Delta' \ne \Delta\) | 0% | 2.8% | 4.4% | 8.2% | 21.2% |
| average \(\Delta'/\Delta\) | 1 | 1.005 | 1.013 | 1.038 | 1.213 |
| average \((\Delta'/\Delta)\cdot(1/s)\) | 1 | 0.804 | 0.676 | 0.519 | 0.303 |
| WikiText-2 perplexity | 23.54 | 12.87 | 12.48 | 11.92 | 12.36 |
Source: AWQ paper, Table 2. [Certain] Scaling by 2 brings perplexity from 23.54 to 11.92, close to the 11.39 of keeping 1% in FP16, with a single data type. At \(s = 4\) the salient error is smaller still, but the other weights suffer and perplexity goes up again.
A group has largest weight \(|w| = 0.75\). With INT3 symmetric levels \(-3 \dots 3\), \(\Delta = 0.75/3 = 0.25\). One weight, \(w_1 = 0.3\), sits on a channel whose activations have size 10 (the others have size 1).
No scaling. The RMS rounding error is \(0.25/\sqrt{12} = 0.0722\). In the output it is multiplied by 10: \(0.722\).
\(s = 2\). \(s w_1 = 0.6 \lt 0.75\), so \(\Delta' = 0.25\). The weight error in original units is \(0.0722/2 = 0.0361\); in the output \(0.361\). Half the error, and no other weight changed.
\(s = 4\). \(s w_1 = 1.2 \gt 0.75\): the group maximum changes, \(\Delta' = 1.2/3 = 0.4\), which is 1.6 times larger. The salient error falls to \(0.4/\sqrt{12}/4 = 0.0289\) (output \(0.289\)), but the RMS error of all 127 other weights grows by 1.6, so their squared error grows by \(1.6^2 = 2.56\). This is the trade-off of formula (107.4).
Figure 1 runs this on 128 rows of 128 random weights, with input channel 0 salient. Move \(s\) and watch the two parts of the error.
How to read Figure 1
At \(s = 1\) the one salient weight causes 47.1% of the total error (with 10× activations), although it is 1 of 128 weights. As \(s\) grows, the dark red curve falls (formula 107.4), while the orange curve stays flat until the scaled weight starts to become the group maximum, then rises fast. With 10× activations the best grid point is \(s = 2\): total error ×0.670, with 18.8% of rows having a changed \(\Delta\), average \(\Delta'/\Delta = 1.048\) and \((\Delta'/\Delta)/s = 0.524\). These are close to the paper’s 1.038 and 0.519. With 20× the best \(s\) moves to 2.5; with 5× it moves to 1.5. The more salient the channel, the more scaling it can use.
Where does the 1/s go?
Dividing the activation by \(s\) must cost nothing at inference, or the trick is useless. AWQ folds the division into the operation that produces the activation. In a Llama block, the reference code does this (llm-awq, auto_scale.py): [Certain]
- q, k and v projections: the 1/s goes into the weight of the RMSNorm before attention (
input_layernorm). - gate and up projections: into the RMSNorm before the MLP (
post_attention_layernorm). - down projection: into the output rows of the up projection (the SwiGLU product is linear in the up output).
- output projection o: into the output rows of v, but only when v and o have matching shapes. With grouped-query attention, v has fewer output rows than o has inputs, and the code skips this pair.
After folding, the model has the same architecture and the same number of weights. Only the quantized matrices change. This is why AWQ needs no special kernel beyond normal INT4 group dequantization.
Searching the scale
For each layer, AWQ looks for the per-input-channel scale vector \(\mathbf{s}\) that makes the quantized output closest to the original output on a small calibration set:
\(Q\) is not differentiable (it rounds), so AWQ does not use gradients. It shrinks the search to one number \(\alpha\) per layer, using the average activation size per channel:
- Read the symbols
- \(W\)
- the weight matrix, one row per output, one column per input channel
- \(X\)
- calibration inputs of the layer, \(T\) tokens; \(X_{tj}\) is the value of channel \(j\) at token \(t\)
- \(\mathbf{s}_X\)
- the mean absolute activation of each input channel
- \(\alpha\)
- how strongly to follow \(\mathbf{s}_X\): \(\alpha = 0\) means no scaling (plain rounding), \(\alpha = 1\) means scale each channel by its full activation size
- \(\mathcal{L}\)
- the output error of formula (107.5)
The reference code tries 20 values, \(\alpha = 0, 0.05, \dots, 0.95\), and keeps the one with the smallest mean squared output error of the whole block (for example, the full attention module for q, k and v). It divides \(\mathbf{s}\) by \(\sqrt{\max \mathbf{s} \cdot \min \mathbf{s}}\) to keep numbers near 1; with absmax quantization this does not change the result, because a common factor on a whole group cancels. [Certain] After the scales, AWQ also searches a clipping range per group: shrinking the group maximum a little can lower the mean squared error, because it gives finer steps to the many small weights. [Certain]
Figure 2 runs the search on a toy layer: 64 rows, 128 input channels with log-normal activation sizes, and two outlier channels whose size you can change.
How to read Figure 2
With the defaults (INT3, outliers 10×) the best \(\alpha\) is 0.35 and the error is 0.360 times that of plain rounding; full scaling (\(\alpha = 1\)) is worse than no scaling (1.81×), because the outlier columns become the group maximum and coarsen everything else. Set the outliers to 1× (no outliers): the best \(\alpha\) is 0.2 and the gain is small (0.915×). Set them to 50×: the gain is large (0.091×) and the best \(\alpha\) moves up to 0.45. The curve is bowl-shaped, so a coarse grid of 20 points is enough.
Why AWQ generalizes well
GPTQ changes the remaining weights to fit the calibration data. AWQ only chooses one \(\alpha\) per layer (and clipping ranges) from activation statistics. It does no backpropagation and no reconstruction, so it has little room to overfit. The paper reports that AWQ needs a smaller calibration set (16 sequences in an OPT-6.7B experiment, against 192 for GPTQ) and is less sensitive to which domain the calibration text comes from. [Certain] It also reports good results on instruction-tuned models and on multimodal models such as VILA and OpenFlamingo. [Likely]
- input: block weights, calibration activations from the previous (already quantized) blocks, bits \(b\), group size \(g = 128\)
- for each group of linear layers that read one activation \(X\) (qkv; o; gate+up; down) do
- \(\mathbf{s}_X \leftarrow\) mean of \(|X|\) over tokens, per channel // formula (107.6)
- for \(\alpha\) in \(0, 0.05, \dots, 0.95\) do
- \(\mathbf{s} \leftarrow \mathbf{s}_X^{\alpha}\), normalized; \(\hat W \leftarrow Q(W \operatorname{diag}(\mathbf{s}))\operatorname{diag}(\mathbf{s})^{-1}\) for every layer in the group
- \(\mathcal{L}(\alpha) \leftarrow\) MSE between the module output with \(\hat W\) and with \(W\) // formula (107.5)
- keep the best \(\alpha\); multiply the weights by \(\mathbf{s}\); divide the previous op (RMSNorm, up, v) by \(\mathbf{s}\)
- for each linear layer: search a clipping ratio per group that minimizes output MSE
- quantize every weight with RTN in groups of \(g\), store INT codes + FP16 scales and zeros
Results on Llama 2
| WikiText-2 perplexity | Llama-2 7B | Llama-2 13B | Llama-2 70B |
|---|---|---|---|
| FP16 | 5.47 | 4.88 | 3.32 |
| INT3-g128 RTN | 6.66 | 5.52 | 3.98 |
| INT3-g128 GPTQ | 6.43 | 5.48 | 3.88 |
| INT3-g128 GPTQ with reordering | 6.42 | 5.41 | 3.86 |
| INT3-g128 AWQ | 6.24 | 5.32 | 3.74 |
| INT4-g128 RTN | 5.73 | 4.98 | 3.46 |
| INT4-g128 GPTQ | 5.69 | 4.98 | 3.42 |
| INT4-g128 GPTQ with reordering | 5.63 | 4.99 | 3.43 |
| INT4-g128 AWQ | 5.60 | 4.97 | 3.41 |
Source: AWQ paper, Table 4. [Certain] Two lessons. At INT4-g128 all methods are close to FP16, and even plain rounding loses only 0.26 perplexity on the 7B model; the method matters much more at 3 bits. And the bigger the model, the smaller the loss.
Keep the outliers in high precision
AWQ avoids mixed precision. The other family accepts it: keep a tiny set of weights in FP16 and quantize the rest more aggressively. The questions are which weights, and how to store them cheaply.
Which weights: a second-order sensitivity
Formula (107.1) only looks at activations. A better score also looks at how much the weight actually loses when rounded. With the layer Hessian \(H = 2XX^\top\) from the previous session, OWQ (Lee et al.) gives each input column \(j\) a sensitivity:
\(\Delta W_{:,j}\) is the rounding error of column \(j\). Because \(\lambda_j\) is the sum of squared activations of channel \(j\), outlier channels get large scores; OWQ calls their columns weak columns. Formula (107.7) is formula (107.1) summed over the rows, so it is the exact contribution of column \(j\) when channels are uncorrelated. SpQR (Dettmers et al.) works at the level of single weights and uses the Optimal Brain Surgeon form from GPTQ, which also counts the help that other weights can give by compensating:
- Read the symbols
- \(H\)
- the Hessian of the layer’s squared output error, \(2XX^\top\), one row and column per input channel
- \(\lambda_j = H_{jj}\)
- its diagonal entry: twice the total squared activation of channel \(j\)
- \([H^{-1}]_{jj}\)
- diagonal of the inverse Hessian: small when channel \(j\) is hard to compensate for with other channels, which makes the weight more sensitive
OWQ: whole columns in FP16
OWQ keeps the weak columns in FP16 and quantizes the rest with GPTQ (it moves the weak columns to the end of the matrix so GPTQ’s error compensation is absorbed by them). Whole columns keep the memory access regular. If a fraction \(f\) of the weights is kept in FP16 instead of \(b\) bits, the average is about
So OWQ’s “3.01-bit” setting means \(0.01/13 \approx 0.077\%\) of the weights are in FP16. OWQ’s low-bit matrix also stores zeros in those places, which adds \(f \cdot b \approx 0.002\) bits; the paper calls this 3.012 bits. [Certain] On LLaMA it gives these WikiText-2 perplexities (OWQ paper; the OPTQ rows are GPTQ with its options turned on): [Certain]
| LLaMA | 7B | 13B | 30B | 65B |
|---|---|---|---|---|
| FP16 | 5.68 | 5.09 | 4.10 | 3.53 |
| OPTQ (GPTQ), 3 bits | 8.13 | 6.67 | 5.67 | 5.41 |
| OWQ, 3.01 bits | 6.66 | 5.66 | 4.75 | 4.25 |
| OPTQ (GPTQ), 4 bits | 6.10 | 5.36 | 4.45 | 4.10 |
| OWQ, 4.01 bits | 5.94 | 5.25 | 4.25 | 3.74 |
OWQ also proposes weak column tuning: for task adaptation, fine-tune only the FP16 weak columns and leave the low-bit part frozen, a cheap alternative to QLoRA (session 58).
SpQR: small groups plus a sparse list
SpQR (“Sparse-Quantized Representation”) goes further, aiming at near-lossless 3 to 4 bits. It combines three ideas:
- Very small groups, \(\beta_1 = 8\) to 32 weights, so one outlier spoils few neighbors.
- Bilevel quantization: the per-group scales and zero points are themselves quantized to 3 bits, in groups of \(\beta_2\), with FP16 second-level statistics. Small groups would be too expensive otherwise.
- Outliers: weights whose sensitivity (107.8) is above a threshold are stored in FP16 in a CSR-like sparse list: a 16-bit value and a 16-bit column index each, about 32 bits per outlier. The paper finds an outlier rate of about 0.2% to 0.5% works best, below 1%. [Certain]
The average cost per weight is (SpQR paper, Eq. 4):
- Read the symbols
- \(b_w\)
- bits per weight code
- \(b_s, b_z\)
- bits of each quantized scale and zero point
- \(\beta_1, \beta_2\)
- first-level group size (weights), second-level group size (scales)
- \(64\)
- two FP16 second-level statistics for scales and two for zeros, shared by \(\beta_1\beta_2\) weights
- \(r_o\)
- the share of weights that are outliers, each costing about 32 bits
With \(b_w = 3\), \(b_s = b_z = 3\), \(\beta_1 = 16\), \(\beta_2 = 32\) and \(r_o = 0.4\%\): \(3 + 6/16 + 64/512 + 32 \times 0.004 = 3 + 0.375 + 0.125 + 0.128 = 3.628\) bits per weight. The outliers cost only 0.128 bits, although each one takes 32 bits. Compare with plain INT3 with groups of 128 and FP16 scale and zero: \(3 + 32/128 = 3.25\) bits.
SpQR on LLaMA (WikiText-2; FP16 5.68 / 5.09 / 4.10 / 3.53 for 7B / 13B / 30B / 65B): at about 3.9 bits it reaches 5.87 / 5.22 / 4.25 / 3.68, and at about 4.6 to 4.7 bits it reaches 5.73 / 5.13 / 4.14 / 3.57. GPTQ at 4 bits gave 6.13 / 5.40 / 4.48 / 3.83 in the same table. [Certain] The paper reports that a 33B model fits on one 24 GB GPU this way.
Figure 3 shows the selection problem on a toy layer: 256 input channels, 128 outputs, INT3 with groups of 128, and four outlier channels.
How to read Figure 3
With outliers at 20×, the four outlier channels cause 78.0% of the error. Keeping 4 columns (1.56% of the weights, 3.5 bits per weight) chosen by activation or by sensitivity leaves 0.220 of the error; chosen by weight norm or at random it leaves 0.997. This is the AWQ table in miniature. Now set the outliers to 1×: no column stands out, and even 16 FP16 columns (4.25 bits) only bring the error to 0.822 (sensitivity) or 0.827 (activation). Mixed precision is powerful only when the error is concentrated. The sensitivity rule is never worse than the activation rule, because it also counts the column’s own rounding error.
- input: weights \(W\), calibration inputs \(X\), base bits \(b\), outlier budget (columns \(k\) or rate \(r_o\))
- \(H \leftarrow 2XX^\top\) (plus a small damping on the diagonal)
- \(\Delta W \leftarrow W - Q(W)\) with a first quantization pass
- OWQ: \(\text{sens}_j \leftarrow H_{jj}\lVert\Delta W_{:,j}\rVert^2\) // formula (107.7); keep the top \(k\) columns in FP16; move them to the end; run GPTQ on the rest
- SpQR: run GPTQ column by column with small groups; for each weight compute \(\text{sens}_{ij}\) // formula (107.8); if above the threshold \(\tau\), store \((w_{ij}, j)\) in the sparse list and do not round it
- quantize the group scales and zeros again (SpQR, bilevel)
- at inference: \(y = \hat W_{\text{dense}}\,\mathbf{x} + W_{\text{sparse}}\,\mathbf{x}\) with a dense low-bit kernel plus a sparse FP16 kernel
Non-uniform levels: SqueezeLLM
Uniform INT levels are evenly spaced. But weights are not evenly spread: most are near 0, a few are far out. A few outliers stretch the range, so most levels sit where almost no weights are. SqueezeLLM (Kim et al.) fixes both problems.
Sensitivity-weighted k-means
Give each output row its own table of \(2^b\) levels (8 levels for 3 bits), and choose the levels by k-means, so they follow the weight distribution. Plain k-means minimizes the weight error. SqueezeLLM weights each weight’s error by its sensitivity, estimated by the diagonal of the Fisher information (the average squared gradient):
- Read the symbols
- \(c_1, \dots, c_K\)
- the \(K = 2^b\) levels of one row; \(c(w_i)\) is the level closest to \(w_i\)
- \(F_{ii}\)
- sensitivity of weight \(i\): its squared gradient of the training loss, averaged over calibration samples \(D\)
- \(g_{d,i}\)
- gradient of the loss with respect to weight \(i\) on sample \(d\)
The Fisher diagonal approximates the Hessian of the final loss, not of one layer’s output. It needs one backward pass per calibration sample (the paper uses 100 samples and reports that about 10 are usually enough). [Certain] The weighted k-means is Lloyd’s algorithm where each center is a weighted average: \(c_k = \sum_{i \in k} F_{ii} w_i / \sum_{i \in k} F_{ii}\). Sensitive weights pull the levels toward themselves.
Dense-and-sparse decomposition
\(S\) is a sparse FP16 matrix (CSR format) holding the outliers (the largest-magnitude weights, about 0.4%) and the most sensitive weights (0.05%); \(D\) holds the rest. Taking the outliers out makes the range of \(D\) much smaller, so the levels can be packed where the weights are. SqueezeLLM keeps 0.45% of the weights in \(S\) in its main sparse setting. [Certain]
How to read Figure 4
At 3 bits without the sparse part, uniform levels have error 0.4837: the outliers stretch the range and most levels fall in empty space. k-means brings it to 0.0642, and Fisher-weighted k-means to 0.0587. Turn on dense-and-sparse: 9 values (0.45%) move to FP16, and the errors drop to 0.0801, 0.0285 and 0.0240. The weighted version wins every time, but by less than the jump from uniform to k-means. Try 2 bits: with only 4 levels, removing outliers matters most.
- input: weights \(W\), Fisher diagonal \(F\) (squared gradients on about 10 to 100 samples), bits \(b\), sparse rates (0.4% outliers, 0.05% sensitive)
- \(S \leftarrow\) weights beyond the outlier thresholds, plus the top 0.05% by \(F\); \(D \leftarrow W - S\) // formula (107.12)
- for each output row \(r\) of \(D\) do
- initialize \(2^b\) levels (for example at quantiles); repeat Lloyd steps:
- assign each weight to its nearest level
- \(c_k \leftarrow \sum_{i \in k} F_{ii} w_i / \sum_{i \in k} F_{ii}\) // formula (107.11)
- store \(b\)-bit indices and a table of \(2^b\) FP16 levels for row \(r\)
- store \(S\) in CSR (FP16 values, column indices, row pointers)
On LLaMA-7B (FP16 WikiText-2: 5.68), the SqueezeLLM paper reports 6.32 at 3.02 bits and 6.13 at 3.24 bits with the 0.45% sparse part; at about 4 bits, 5.79 (4.05 bits) and 5.77 (4.27 bits). [Certain] Its 3-bit kernel ran up to about 2.3× faster than FP16 on an A6000 at batch size 1 (the abstract says 2.3×, the main text 2.4×). [Certain]
Kernels for non-uniform formats: LUT-GEMM
Uniform INT4 has a simple kernel: load 4-bit codes, dequantize with scale and zero inside the matrix product, multiply in FP16. A non-uniform format needs a table lookup per weight to dequantize. LUT-GEMM (Park et al.) goes one step further: it removes the dequantization and most multiplications.
It uses binary-coding quantization (BCQ). Each weight is a sum of \(q\) signed bits with real scales:
Uniform quantization is a special case: choose \(\alpha_i = 2^{i-1}\, s\) (half the bit weights of the INT code times the scale) and absorb the rest into the offset \(z\) (LUT-GEMM paper, Appendix C). [Certain] Non-uniform levels can be fitted directly with free \(\alpha_i\).
Now the dot product of one row with the input \(\mathbf{x}\). Cut \(\mathbf{x}\) into chunks of \(\mu\) inputs. For each bit-plane, the row’s bits inside one chunk are just a sign pattern, and there are only \(2^\mu\) possible patterns. So precompute, once per chunk, all \(2^\mu\) signed sums \(\pm x_1 \pm \dots \pm x_\mu\). Then every row and every bit-plane only looks up its partial sum:
Building the tables costs about \(2^\mu \cdot n/\mu\) additions and is shared by all \(m\) rows; reading costs \(m \cdot n/\mu \cdot q\) lookups, against \(m \cdot n\) multiply-adds for a dense product. The saving is about \(\mu/q\) when \(m\) is large. [Certain] The tables live in GPU shared memory and are shared by the threads of a block.
How to read Figure 5
Step through. The table is built once for the chunk (8 entries). Row 1 then needs only two reads: LUT[101] = 3.5 and LUT[110] = −2.5, so \(y_1 = 0.6 \times 3.5 + 0.2 \times (-2.5) = 1.6\), the same as the direct product with weights 0.8, −0.4, 0.4. With 12,288 rows, the same 8 entries would be reused 12,288 times per bit-plane; that reuse is where the saving comes from.
One FFN-sized matrix with \(m = n = 12{,}288\), \(q = 3\) bits and \(\mu = 8\). Dense: \(m n = 151{,}000{,}000\) multiply-adds (rounded). Tables: \(2^8 \times 12{,}288/8 = 393{,}216\) entries. Lookups: \(12{,}288 \times 1{,}536 \times 3 = 56{,}623{,}104\). The ratio is 2.65, close to \(\mu/q = 8/3\). In the paper’s measurement for the first FFN layer of OPT-175B on an A100 (\(m = 12{,}288\), group size 128), cuBLAS FP16 took 0.7256 ms, the GPTQ 3-bit kernel 0.3599 ms and LUT-GEMM 3-bit 0.2250 ms (3.22× faster than FP16). [Certain]
End to end, LUT-GEMM ran OPT-175B in 3 bits on a single A100 at 51.6 ms per token, against 106.5 ms for the GPTQ kernel (about 2.1×). [Certain]
Why weight-only helps decoding, not prefill
During decoding with a small batch, each weight is read once and used for only a few tokens, so the matrix product is limited by memory bandwidth (session 82). Reading 3 or 4 bits instead of 16 is then a direct speedup, and the extra dequantization or lookup work is hidden. During prefill, or at large batch, the same weight is reused by hundreds of tokens; the product becomes compute-bound and runs on FP16 tensor cores after dequantization. Weight-only formats then save memory but little time, and can even be slower. To speed up compute-bound phases you must also quantize activations (session 108). [Certain]
All methods side by side
| Method | What it protects | How | Storage | Kernel |
|---|---|---|---|---|
| RTN | nothing | round each group | INT3/4 + scale, zero per group | standard dequant GEMV/GEMM |
| GPTQ | the output, via error feedback | Hessian-based updates of unrounded weights | same as RTN | same as RTN |
| AWQ | weights on large-activation channels | per-channel scale \(\mathbf{s}_X^\alpha\), folded into the previous op; clipping search | same as RTN | same as RTN |
| OWQ | weak columns (outlier channels) | Hessian column sensitivity (107.7); FP16 columns + GPTQ | INT + a few FP16 columns | dense low-bit + small FP16 part |
| SpQR | single sensitive weights | OBS sensitivity (107.8); small groups, bilevel scales | INT3 + 3-bit stats + CSR outliers | custom dense + sparse |
| SqueezeLLM | sensitive and outlier weights | Fisher-weighted k-means levels; dense-and-sparse | 3/4-bit indices + per-row LUT + CSR | LUT dequant + sparse |
| LUT-GEMM | (a kernel, not a selection rule) | binary coding, table of signed sums | bit-planes + scales per group | lookups, no dequant |
The numbers in different papers are not directly comparable. For LLaMA-7B at INT3 with groups of 128, the AWQ paper reports 6.35 for AWQ while the SqueezeLLM paper reports 6.44 for AWQ; the SpQR paper reports 4-bit RTN at 6.43, but the AWQ paper reports INT4-g128 RTN at 5.96. [Certain] The most likely reason is the group size: SpQR’s RTN and GPTQ baselines appear to use one scale per row. [Likely] Always check the bits (including scales), the group size, the calibration set and the evaluation context length before comparing.
A runnable experiment
The code builds a linear layer with 1% outlier input channels, then compares plain INT3 rounding, keeping 1% of the columns in FP16 by four rules, and the AWQ search. It runs on a CPU in a few seconds and prints the same numbers every time.
import torch torch.manual_seed(0) def quantize(w, bits=3, group=128): """Asymmetric min-max round-to-nearest, one scale and one zero point per group of `group` inputs.""" rows, cols = w.shape g = w.reshape(rows, cols // group, group) lo, hi = g.amin(-1, keepdim=True), g.amax(-1, keepdim=True) scale = (hi - lo).clamp(min=1e-8) / (2**bits - 1) zero = torch.round(-lo / scale) q = torch.clamp(torch.round(g / scale) + zero, 0, 2**bits - 1) return ((q - zero) * scale).reshape(rows, cols) # a toy linear layer y = W x with a few "outlier" input channels d_in, d_out, n = 1024, 512, 2048 W = torch.randn(d_out, d_in) * 0.02 mag = torch.exp(0.3 * torch.randn(d_in)) # normal channels: size about 1 out_ch = torch.randperm(d_in)[:10] # 1% of the channels ... mag[out_ch] *= 20.0 # ... are 20x larger X = torch.randn(d_in, n) * mag[:, None] # calibration activations Y = W @ X def err(W_hat): # relative output error return ((W_hat @ X - Y).pow(2).mean() / Y.pow(2).mean()).item() bits, k = 3, d_in // 100 W_rtn = quantize(W, bits) print(f"RTN INT{bits} g128 : {err(W_rtn):.4f}") def keep_fp16(idx): # mixed precision: some columns stay exact W_hat = W_rtn.clone(); W_hat[:, idx] = W[:, idx]; return W_hat act = X.abs().mean(1) # s_X, formula (107.6) lam = (X * X).sum(1) # diagonal of X X^T (H up to a factor 2) sens = lam * (W - W_rtn).pow(2).sum(0) # OWQ sensitivity, formula (107.7) print(f"keep 1% by activation : {err(keep_fp16(act.topk(k).indices)):.4f}") print(f"keep 1% by OWQ sensitivity : {err(keep_fp16(sens.topk(k).indices)):.4f}") print(f"keep 1% by weight norm : {err(keep_fp16(W.norm(dim=0).topk(k).indices)):.4f}") print(f"keep 1% at random : {err(keep_fp16(torch.randperm(d_in)[:k])):.4f}") # AWQ: s = s_X^alpha, search alpha on a grid of 20 values in [0, 1) best = (float("inf"), None) for i in range(20): alpha = i / 20 s = act.pow(alpha) s = s / (s.max() * s.min()).sqrt() # keeps numbers in a safe range W_hat = quantize(W * s[None, :], bits) / s[None, :] # scale columns, quantize, fold 1/s back best = min(best, (err(W_hat), alpha)) print(f"AWQ, best alpha = {best[1]:.2f} : {best[0]:.4f}")
Output with PyTorch 2.8 on a CPU: RTN 0.0463; keep 1% by activation 0.0087, by OWQ sensitivity 0.0087, by weight norm 0.0462, at random 0.0462; AWQ best \(\alpha = 0.35\), 0.0176. Scaling alone (one data type) removes about 62% of the error; keeping the 10 outlier columns exact removes 81%. In this toy, activation and sensitivity pick the same columns, because the outliers dominate everything. Other PyTorch versions may differ in the last digits.
Where this is used
| Setting | Typical choice | Why |
|---|---|---|
| Serving a 4-bit chat model with vLLM, TGI or TensorRT-LLM | AWQ or GPTQ, INT4, groups of 128 | Same storage as RTN, standard kernels, near-FP16 quality at 4 bits |
| A model on a laptop or a Jetson board | AWQ INT4 with TinyChat or similar runtimes | Batch size 1 decoding is memory-bound; the AWQ paper reports 2.7 to 3.9× over FP16 on an RTX 4090 [Certain] |
| Video captioning, video question answering or key-moment detection with a VLM on an edge device | AWQ on the language model of a VLM such as VILA | The paper reports VILA-7B on Jetson Orin going from 11.5 to 35.6 tokens/s [Certain]; the vision encoder often stays in FP16 [Likely] |
| Fitting a larger model into one consumer GPU | SpQR or SqueezeLLM near 3 to 4 bits | Near-lossless quality per bit; SpQR reports a 33B model on one 24 GB GPU |
| Cheap task adaptation of a quantized model | OWQ weak-column tuning | Train only the FP16 columns; the low-bit part stays frozen |
| Video-RAG or video retrieval: a quantized LLM reads retrieved clip captions | 4-bit AWQ/GPTQ LLM, FP16 embedding model | Frees memory for long contexts and the KV cache; embeddings need precise distances [Likely] |
| Very low bits (2 to 3) on non-standard hardware | Binary-coded weights with LUT-GEMM | No dequantization; work scales with the number of bit-planes |
Common mistakes
- Choosing salient weights by weight size. The AWQ table shows this is no better than random. Use activation size or a Hessian or Fisher sensitivity.
- Scaling too hard. \(\alpha = 1\) is often worse than no scaling (Figure 2): once the scaled column becomes the group maximum, the whole group loses resolution.
- Forgetting to fold 1/s. If you scale the weights but do not divide the RMSNorm (or v, up) by the same \(\mathbf{s}\), the model output changes. Check that the FP16 model with scales folded gives the same logits before you quantize.
- Different scales for layers that share an input. q, k and v read one normalized activation; they must use one \(\mathbf{s}\), because there is only one RMSNorm to fold it into.
- Counting bits without the overhead. “3-bit” with groups of 128 and FP16 scale and zero is 3.25 bits; SpQR’s small groups and outliers add more (formula 107.10). Compare methods at equal total bits.
- Comparing perplexities across papers. RTN at 4 bits is 6.43 in one paper and 5.96 in another for the same LLaMA-7B, most likely because of group size. Re-run baselines yourself.
- Expecting speedups at large batch. Weight-only formats speed up memory-bound decoding. Prefill and large-batch serving are compute-bound; there the benefit is mostly memory.
- Calibrating on the wrong layer inputs. Collect the inputs of block \(\ell\) from the already quantized blocks \(1 \dots \ell-1\), so each block corrects for the errors before it.
Interview questions
Why does AWQ choose salient channels by activation size and not by weight size?basic
Explain mathematically why scaling a weight by s before quantization reduces its error.core
Where does the 1/s go at inference time? Does AWQ change the model architecture?core
How does AWQ search the scales, and why not optimize s with gradients?core
AWQ or GPTQ: how do they differ and when would you pick each?core
What is a “weak column” in OWQ, and how many bits does OWQ 3.01 really use?core
How does SpQR keep the bit cost low while using groups as small as 16?advanced
Why does SqueezeLLM use the Fisher information instead of the layer Hessian, and what does weighted k-means change?advanced
How does LUT-GEMM compute a matrix-vector product without dequantizing?advanced
Your INT4 AWQ model decodes 3× faster but prefill is no faster. Why?core
You must quantize a model to about 3 bits and quality drops badly with RTN. What do you try, in order?advanced
Exercises
- A group has largest weight 1.2 and uses INT4 symmetric (levels \(-7 \dots 7\)). What is \(\Delta\)? A salient weight is 0.4. What is the largest \(s\) that does not change \(\Delta\)?
- Using the AWQ Table 2, by what factor does \((\Delta'/\Delta)(1/s)\) improve from \(s = 2\) to \(s = 4\), and why does perplexity still get worse?
- In a Llama 3 block with 8 KV heads and 32 query heads (head size 128, hidden size 4096), can AWQ fold a scale for o_proj into v_proj? Give the two shapes.
- Compute the SpQR bit cost (formula 107.10) for \(b_w = 3\), \(b_s = b_z = 3\), \(\beta_1 = \beta_2 = 16\), \(r_o = 0.3\%\).
- OWQ uses 4-bit base weights and wants 4.05 bits on average. What share of the weights can stay in FP16?
- LUT-GEMM with \(\mu = 8\): how many table entries per chunk, and how many lookups does a 4,096-wide row need for \(q = 4\)?
- In Figure 1, set the salient activation to 20×. Read the best \(s\) and the total error. Why can it use a larger \(s\) than at 10×?
Answerscheck
Key takeaways
- The output error of a quantized layer is the weight error weighted by the activation size (107.1). A few outlier channels cause most of it; keeping 1% of channels by activation fixes most of the INT3 loss, choosing by weight size does not.
- AWQ scales salient columns up and their inputs down (107.3); the error falls by \((\Delta'/\Delta)/s\) (107.4). The scale \(\mathbf{s}_X^\alpha\) is found by a 20-point grid search (107.6) and folded into RMSNorm, up or v, so storage and kernels stay standard.
- Mixed precision keeps sensitive weights in FP16: OWQ by columns (107.7), SpQR by single weights with an OBS score (107.8), small groups and bilevel scales; count the bits with (107.9) and (107.10).
- SqueezeLLM places non-uniform levels with Fisher-weighted k-means (107.11) and moves 0.45% of weights to a sparse FP16 matrix (107.12).
- LUT-GEMM turns binary-coded weights (107.13) into table lookups (107.14), saving about \(\mu/q\) of the work. All weight-only formats help memory-bound decoding most, not compute-bound prefill.
Key formulas
References for this session
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Dettmers et al. (2023). SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
- Lee et al. (2023). OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models
- Kim et al. (2023). SqueezeLLM: Dense-and-Sparse Quantization
- Park et al. (2022). LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Dettmers et al. (2023). SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
- Lee et al. (2023). OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models
- Kim et al. (2023). SqueezeLLM: Dense-and-Sparse Quantization
- Park et al. (2022). LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- MIT HAN Lab (code). llm-awq: AWQ and TinyChat reference implementation (awq/quantize/auto_scale.py)