LLM & VLM courseBlog
107Track 5: LLM inference and serving · session 30 of 65

Weight PTQ 2: AWQ, outliers and mixed precision

At 3 or 4 bits, a few weights decide most of the quality loss: the ones that meet large activations, and a handful of very large or very sensitive values. This session shows three ways to protect them: scale them up before rounding (AWQ), keep them in 16 bits (OWQ, SpQR), or place the quantization levels where they matter (SqueezeLLM). It ends with LUT-GEMM, a kernel that runs such formats without turning the weights back into floats.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 82, 103 and 106 (why inference is slow, quantization math, GPTQ)

What you will learn

  • Explain why the output error of a quantized layer is the weight error weighted by the size of the activations, and why 1% of the input channels can cause most of it
  • Derive the AWQ scaling trick: multiply a salient weight column by s, divide its input by s, and see the error shrink by about 1/s until the group maximum changes
  • Run the AWQ search over s = s_X^α, know where the 1/s is folded (RMSNorm, v_proj, up_proj) and why it needs no backpropagation
  • Pick weights to keep in FP16 with a Hessian-based sensitivity (OWQ columns, SpQR single weights) and count the extra bits per weight
  • Build non-uniform levels with Fisher-weighted k-means and a dense-and-sparse split (SqueezeLLM), and say what they cost
  • Explain how LUT-GEMM multiplies binary-coded weights with table lookups, and why weight-only formats speed up decoding but not large-batch prefill
  • Read the published perplexity numbers of RTN, GPTQ, AWQ, OWQ, SpQR and SqueezeLLM without mixing up their settings

Why this matters

A 7B model has about 6.74 billion weights. In FP16 they take 13.5 GB. At 4 bits with one 16-bit scale and one 16-bit zero point per group of 128 weights, they take about 3.6 GB. During decoding the GPU must read every weight once per step, so fewer bytes means more tokens per second.

The previous session (session 106) showed GPTQ: round one column at a time and let the not-yet-rounded columns correct the error, using second-order (Hessian) information. This session asks a different question: which weights matter most? At 4 bits plain rounding is almost fine. At 3 bits it is not, and most of the damage comes from a very small set of weights. There are three practical answers, and each became a widely used method:

  • Scale the important weights up before rounding, so they get relatively finer steps. This is AWQ (Activation-aware Weight Quantization, Lin et al.), which won the MLSys 2024 best paper award. [Certain]
  • Keep the important weights in 16 bits and quantize the rest: OWQ keeps whole columns, SpQR keeps single weights in a sparse list.
  • Put the quantization levels where the important weights are: SqueezeLLM learns non-uniform levels with a weighted k-means and keeps outliers in a sparse list.

Non-uniform or binary-coded weights then need a special kernel. LUT-GEMM replaces multiplications by table lookups. In the efficient-inference survey (Zhou et al.) these methods sit in Section 5.2.1, weight-only post-training quantization.

This is not only a research topic. The AWQ authors list integrations in vLLM, Hugging Face TGI, TensorRT-LLM, LMDeploy and Hugging Face transformers (from_pretrained). [Certain] Many 4-bit checkpoints on the Hugging Face Hub are AWQ or GPTQ files. [Likely]

Not all weights are equal: the output view

A quantizer rounds each weight \(w_j\) to a nearby value \(Q(w_j)\). We do not care about the weight error itself. We care about the error of the layer output. For one output \(y = \sum_j w_j x_j\):

\[ \Delta y = \sum_{j} \big(Q(w_j) - w_j\big)\, x_j, \qquad \E\big[(\Delta y)^2\big] \approx \sum_{j} \big(Q(w_j) - w_j\big)^2\, \E\big[x_j^2\big] \tag{107.1} \]
Read the symbols
\(w_j, x_j\)
the weight on input channel \(j\), and the activation that arrives on that channel
\(Q(w_j) - w_j\)
the rounding error of that weight
\(\E[x_j^2]\)
how large channel \(j\)’s activations are on average (over tokens of a calibration set)
\(\approx\)
we drop the cross terms; this is exact if different channels are uncorrelated, and a good first picture otherwise

Formula (107.1) says: a weight’s rounding error counts in proportion to the size of its input. In LLMs a few input channels carry activations that are 10 to 100 times larger than the rest. This is the same outlier effect that LLM.int8() found (session 108). Weights in those channels are the salient weights. The same weight error there costs 100 to 10,000 times more output error.

The AWQ paper tested this directly on OPT models at INT3 with groups of 128. It kept a small share of input channels in FP16 and rounded the rest, choosing the channels three ways:

OPT-6.7B, WikiText-2 perplexity (FP16: 10.86, plain INT3 rounding: 23.54)keep 0.1%keep 1%keep 3%
channels with the largest activations11.5811.3911.36
channels with the largest weight norm23.4122.3722.45
random channels23.5424.2324.22

Source: AWQ paper, Table 1. [Certain] Keeping 1% of the channels picked by activation size removes almost the whole gap. Picking by weight size is no better than random. The weights themselves look ordinary; it is their inputs that make them important.

But keeping 1% of the columns in FP16 is a problem for the hardware: the matrix now has two data types, and the kernel must handle both. AWQ’s key idea is to get the same protection without mixed precision.

AWQ: scale the salient weights before rounding

Start with the quantizer. For a group of weights (for example 128 weights of one output row), a symmetric \(b\)-bit quantizer uses one step size \(\Delta\):

\[ Q(w) = \Delta \cdot \operatorname{round}\!\left(\frac{w}{\Delta}\right), \qquad \Delta = \frac{\max_{\text{group}} |w|}{2^{b-1} - 1} \tag{107.2} \]

(The AWQ paper writes \(2^{b-1}\) in the denominator; the exact integer range does not change the argument.) The rounding error is at most \(\Delta/2\), and on average its root mean square is \(\Delta/\sqrt{12}\). Its size depends only on \(\Delta\), not on \(w\).

Now the trick. Multiply weight column \(j\) by a number \(s \gt 1\) and divide its input by the same \(s\). The product does not change:

\[ w_j x_j = (s\, w_j) \cdot \frac{x_j}{s} \quad\Longrightarrow\quad \text{quantized: } \; Q(s\, w_j) \cdot \frac{x_j}{s} \tag{107.3} \]

The error of the scaled weight is still about the size of the step \(\Delta'\) of its group. But in the output it is divided by \(s\). If scaling one weight in a group of 128 does not change the group maximum, then \(\Delta' = \Delta\), and the output error from this weight shrinks by the factor \(1/s\). In general:

\[ \frac{\text{error with scaling}}{\text{error without}} = \frac{\Delta'}{\Delta} \cdot \frac{1}{s} \tag{107.4} \]
Read the symbols
\(s\)
the scale applied to one input channel: its weight column is multiplied by \(s\), its activation divided by \(s\)
\(\Delta, \Delta'\)
the step size of the group before and after scaling; \(\Delta' \gt \Delta\) only if the scaled weight becomes the new group maximum

So scaling helps as long as \(\Delta'/\Delta\) stays near 1. If \(s\) is too large, the scaled salient weight becomes the largest in its group, \(\Delta'\) grows, and every other weight in the group gets a coarser grid. The AWQ paper measured this on OPT-6.7B (1% salient channels, INT3):

s11.251.524
groups where \(\Delta' \ne \Delta\)0%2.8%4.4%8.2%21.2%
average \(\Delta'/\Delta\)11.0051.0131.0381.213
average \((\Delta'/\Delta)\cdot(1/s)\)10.8040.6760.5190.303
WikiText-2 perplexity23.5412.8712.4811.9212.36

Source: AWQ paper, Table 2. [Certain] Scaling by 2 brings perplexity from 23.54 to 11.92, close to the 11.39 of keeping 1% in FP16, with a single data type. At \(s = 4\) the salient error is smaller still, but the other weights suffer and perplexity goes up again.

Worked example: one salient weight in a group

A group has largest weight \(|w| = 0.75\). With INT3 symmetric levels \(-3 \dots 3\), \(\Delta = 0.75/3 = 0.25\). One weight, \(w_1 = 0.3\), sits on a channel whose activations have size 10 (the others have size 1).

No scaling. The RMS rounding error is \(0.25/\sqrt{12} = 0.0722\). In the output it is multiplied by 10: \(0.722\).

\(s = 2\). \(s w_1 = 0.6 \lt 0.75\), so \(\Delta' = 0.25\). The weight error in original units is \(0.0722/2 = 0.0361\); in the output \(0.361\). Half the error, and no other weight changed.

\(s = 4\). \(s w_1 = 1.2 \gt 0.75\): the group maximum changes, \(\Delta' = 1.2/3 = 0.4\), which is 1.6 times larger. The salient error falls to \(0.4/\sqrt{12}/4 = 0.0289\) (output \(0.289\)), but the RMS error of all 127 other weights grows by 1.6, so their squared error grows by \(1.6^2 = 2.56\). This is the trade-off of formula (107.4).

Figure 1 runs this on 128 rows of 128 random weights, with input channel 0 salient. Move \(s\) and watch the two parts of the error.

salient activation size
salient weight’s partthe other 127 weights’ parttotal
Figure 1. Scaling one salient weight per group (INT3, one group of 128 per row, 128 rows of Gaussian weights). The error is formula (107.1) with activation size 1 on every channel except the salient one. All values are divided by the total error at \(s = 1\).
How to read Figure 1

At \(s = 1\) the one salient weight causes 47.1% of the total error (with 10× activations), although it is 1 of 128 weights. As \(s\) grows, the dark red curve falls (formula 107.4), while the orange curve stays flat until the scaled weight starts to become the group maximum, then rises fast. With 10× activations the best grid point is \(s = 2\): total error ×0.670, with 18.8% of rows having a changed \(\Delta\), average \(\Delta'/\Delta = 1.048\) and \((\Delta'/\Delta)/s = 0.524\). These are close to the paper’s 1.038 and 0.519. With 20× the best \(s\) moves to 2.5; with 5× it moves to 1.5. The more salient the channel, the more scaling it can use.

Where does the 1/s go?

Dividing the activation by \(s\) must cost nothing at inference, or the trick is useless. AWQ folds the division into the operation that produces the activation. In a Llama block, the reference code does this (llm-awq, auto_scale.py): [Certain]

  • q, k and v projections: the 1/s goes into the weight of the RMSNorm before attention (input_layernorm).
  • gate and up projections: into the RMSNorm before the MLP (post_attention_layernorm).
  • down projection: into the output rows of the up projection (the SwiGLU product is linear in the up output).
  • output projection o: into the output rows of v, but only when v and o have matching shapes. With grouped-query attention, v has fewer output rows than o has inputs, and the code skips this pair.

After folding, the model has the same architecture and the same number of weights. Only the quantized matrices change. This is why AWQ needs no special kernel beyond normal INT4 group dequantization.

Searching the scale

For each layer, AWQ looks for the per-input-channel scale vector \(\mathbf{s}\) that makes the quantized output closest to the original output on a small calibration set:

\[ \mathbf{s}^{*} = \argmin_{\mathbf{s}} \Big\lVert\, Q\big(W \operatorname{diag}(\mathbf{s})\big)\big(\operatorname{diag}(\mathbf{s})^{-1} X\big) - W X \,\Big\rVert^2 \tag{107.5} \]

\(Q\) is not differentiable (it rounds), so AWQ does not use gradients. It shrinks the search to one number \(\alpha\) per layer, using the average activation size per channel:

\[ \mathbf{s} = \mathbf{s}_X^{\,\alpha}, \qquad s_{X,j} = \frac{1}{T}\sum_{t=1}^{T} |X_{t j}|, \qquad \alpha^{*} = \argmin_{\alpha \in [0, 1]} \mathcal{L}\big(\mathbf{s}_X^{\,\alpha}\big) \tag{107.6} \]
Read the symbols
\(W\)
the weight matrix, one row per output, one column per input channel
\(X\)
calibration inputs of the layer, \(T\) tokens; \(X_{tj}\) is the value of channel \(j\) at token \(t\)
\(\mathbf{s}_X\)
the mean absolute activation of each input channel
\(\alpha\)
how strongly to follow \(\mathbf{s}_X\): \(\alpha = 0\) means no scaling (plain rounding), \(\alpha = 1\) means scale each channel by its full activation size
\(\mathcal{L}\)
the output error of formula (107.5)

The reference code tries 20 values, \(\alpha = 0, 0.05, \dots, 0.95\), and keeps the one with the smallest mean squared output error of the whole block (for example, the full attention module for q, k and v). It divides \(\mathbf{s}\) by \(\sqrt{\max \mathbf{s} \cdot \min \mathbf{s}}\) to keep numbers near 1; with absmax quantization this does not change the result, because a common factor on a whole group cancels. [Certain] After the scales, AWQ also searches a clipping range per group: shrinking the group maximum a little can lower the mean squared error, because it gives finer steps to the many small weights. [Certain]

Figure 2 runs the search on a toy layer: 64 rows, 128 input channels with log-normal activation sizes, and two outlier channels whose size you can change.

bits
Figure 2. The AWQ grid search of formula (107.6). Filled dots are the 20 grid values the reference code tries; the green dot is the best one. The hollow dot at \(\alpha = 1\) is shown for comparison. The error is formula (107.1) summed over 64 rows.
How to read Figure 2

With the defaults (INT3, outliers 10×) the best \(\alpha\) is 0.35 and the error is 0.360 times that of plain rounding; full scaling (\(\alpha = 1\)) is worse than no scaling (1.81×), because the outlier columns become the group maximum and coarsen everything else. Set the outliers to 1× (no outliers): the best \(\alpha\) is 0.2 and the gain is small (0.915×). Set them to 50×: the gain is large (0.091×) and the best \(\alpha\) moves up to 0.45. The curve is bowl-shaped, so a coarse grid of 20 points is enough.

Why AWQ generalizes well

GPTQ changes the remaining weights to fit the calibration data. AWQ only chooses one \(\alpha\) per layer (and clipping ranges) from activation statistics. It does no backpropagation and no reconstruction, so it has little room to overfit. The paper reports that AWQ needs a smaller calibration set (16 sequences in an OPT-6.7B experiment, against 192 for GPTQ) and is less sensitive to which domain the calibration text comes from. [Certain] It also reports good results on instruction-tuned models and on multimodal models such as VILA and OpenFlamingo. [Likely]

Algorithm 1: AWQ for one transformer block
  1. input: block weights, calibration activations from the previous (already quantized) blocks, bits \(b\), group size \(g = 128\)
  2. for each group of linear layers that read one activation \(X\) (qkv; o; gate+up; down) do
  3.   \(\mathbf{s}_X \leftarrow\) mean of \(|X|\) over tokens, per channel // formula (107.6)
  4.   for \(\alpha\) in \(0, 0.05, \dots, 0.95\) do
  5.     \(\mathbf{s} \leftarrow \mathbf{s}_X^{\alpha}\), normalized; \(\hat W \leftarrow Q(W \operatorname{diag}(\mathbf{s}))\operatorname{diag}(\mathbf{s})^{-1}\) for every layer in the group
  6.     \(\mathcal{L}(\alpha) \leftarrow\) MSE between the module output with \(\hat W\) and with \(W\) // formula (107.5)
  7.   keep the best \(\alpha\); multiply the weights by \(\mathbf{s}\); divide the previous op (RMSNorm, up, v) by \(\mathbf{s}\)
  8. for each linear layer: search a clipping ratio per group that minimizes output MSE
  9. quantize every weight with RTN in groups of \(g\), store INT codes + FP16 scales and zeros

Results on Llama 2

WikiText-2 perplexityLlama-2 7BLlama-2 13BLlama-2 70B
FP165.474.883.32
INT3-g128 RTN6.665.523.98
INT3-g128 GPTQ6.435.483.88
INT3-g128 GPTQ with reordering6.425.413.86
INT3-g128 AWQ6.245.323.74
INT4-g128 RTN5.734.983.46
INT4-g128 GPTQ5.694.983.42
INT4-g128 GPTQ with reordering5.634.993.43
INT4-g128 AWQ5.604.973.41

Source: AWQ paper, Table 4. [Certain] Two lessons. At INT4-g128 all methods are close to FP16, and even plain rounding loses only 0.26 perplexity on the 7B model; the method matters much more at 3 bits. And the bigger the model, the smaller the loss.

Keep the outliers in high precision

AWQ avoids mixed precision. The other family accepts it: keep a tiny set of weights in FP16 and quantize the rest more aggressively. The questions are which weights, and how to store them cheaply.

Which weights: a second-order sensitivity

Formula (107.1) only looks at activations. A better score also looks at how much the weight actually loses when rounded. With the layer Hessian \(H = 2XX^\top\) from the previous session, OWQ (Lee et al.) gives each input column \(j\) a sensitivity:

\[ \text{sens}_j = \lambda_j \, \big\lVert \Delta W_{:,j} \big\rVert_2^2, \qquad \lambda_j = H_{jj} = 2 \sum_{t} X_{tj}^2 \tag{107.7} \]

\(\Delta W_{:,j}\) is the rounding error of column \(j\). Because \(\lambda_j\) is the sum of squared activations of channel \(j\), outlier channels get large scores; OWQ calls their columns weak columns. Formula (107.7) is formula (107.1) summed over the rows, so it is the exact contribution of column \(j\) when channels are uncorrelated. SpQR (Dettmers et al.) works at the level of single weights and uses the Optimal Brain Surgeon form from GPTQ, which also counts the help that other weights can give by compensating:

\[ \text{sens}_{ij} = \frac{\big(w_{ij} - Q(w_{ij})\big)^2}{2\,\big[H^{-1}\big]_{jj}} \tag{107.8} \]
Read the symbols
\(H\)
the Hessian of the layer’s squared output error, \(2XX^\top\), one row and column per input channel
\(\lambda_j = H_{jj}\)
its diagonal entry: twice the total squared activation of channel \(j\)
\([H^{-1}]_{jj}\)
diagonal of the inverse Hessian: small when channel \(j\) is hard to compensate for with other channels, which makes the weight more sensitive

OWQ: whole columns in FP16

OWQ keeps the weak columns in FP16 and quantizes the rest with GPTQ (it moves the weak columns to the end of the matrix so GPTQ’s error compensation is absorbed by them). Whole columns keep the memory access regular. If a fraction \(f\) of the weights is kept in FP16 instead of \(b\) bits, the average is about

\[ \bar b \approx b + f\,(16 - b) \tag{107.9} \]

So OWQ’s “3.01-bit” setting means \(0.01/13 \approx 0.077\%\) of the weights are in FP16. OWQ’s low-bit matrix also stores zeros in those places, which adds \(f \cdot b \approx 0.002\) bits; the paper calls this 3.012 bits. [Certain] On LLaMA it gives these WikiText-2 perplexities (OWQ paper; the OPTQ rows are GPTQ with its options turned on): [Certain]

LLaMA7B13B30B65B
FP165.685.094.103.53
OPTQ (GPTQ), 3 bits8.136.675.675.41
OWQ, 3.01 bits6.665.664.754.25
OPTQ (GPTQ), 4 bits6.105.364.454.10
OWQ, 4.01 bits5.945.254.253.74

OWQ also proposes weak column tuning: for task adaptation, fine-tune only the FP16 weak columns and leave the low-bit part frozen, a cheap alternative to QLoRA (session 58).

SpQR: small groups plus a sparse list

SpQR (“Sparse-Quantized Representation”) goes further, aiming at near-lossless 3 to 4 bits. It combines three ideas:

  • Very small groups, \(\beta_1 = 8\) to 32 weights, so one outlier spoils few neighbors.
  • Bilevel quantization: the per-group scales and zero points are themselves quantized to 3 bits, in groups of \(\beta_2\), with FP16 second-level statistics. Small groups would be too expensive otherwise.
  • Outliers: weights whose sensitivity (107.8) is above a threshold are stored in FP16 in a CSR-like sparse list: a 16-bit value and a 16-bit column index each, about 32 bits per outlier. The paper finds an outlier rate of about 0.2% to 0.5% works best, below 1%. [Certain]

The average cost per weight is (SpQR paper, Eq. 4):

\[ \bar b \approx b_w + \frac{b_s + b_z}{\beta_1} + \frac{64}{\beta_1 \beta_2} + 32\, r_o \tag{107.10} \]
Read the symbols
\(b_w\)
bits per weight code
\(b_s, b_z\)
bits of each quantized scale and zero point
\(\beta_1, \beta_2\)
first-level group size (weights), second-level group size (scales)
\(64\)
two FP16 second-level statistics for scales and two for zeros, shared by \(\beta_1\beta_2\) weights
\(r_o\)
the share of weights that are outliers, each costing about 32 bits
Worked example: SpQR’s bit budget

With \(b_w = 3\), \(b_s = b_z = 3\), \(\beta_1 = 16\), \(\beta_2 = 32\) and \(r_o = 0.4\%\): \(3 + 6/16 + 64/512 + 32 \times 0.004 = 3 + 0.375 + 0.125 + 0.128 = 3.628\) bits per weight. The outliers cost only 0.128 bits, although each one takes 32 bits. Compare with plain INT3 with groups of 128 and FP16 scale and zero: \(3 + 32/128 = 3.25\) bits.

SpQR on LLaMA (WikiText-2; FP16 5.68 / 5.09 / 4.10 / 3.53 for 7B / 13B / 30B / 65B): at about 3.9 bits it reaches 5.87 / 5.22 / 4.25 / 3.68, and at about 4.6 to 4.7 bits it reaches 5.73 / 5.13 / 4.14 / 3.57. GPTQ at 4 bits gave 6.13 / 5.40 / 4.48 / 3.83 in the same table. [Certain] The paper reports that a 33B model fits on one 24 GB GPU this way.

Figure 3 shows the selection problem on a toy layer: 256 input channels, 128 outputs, INT3 with groups of 128, and four outlier channels.

Figure 3. Mixed precision by columns, as in OWQ. Each rule ranks the 256 input columns and keeps the top ones in FP16; the rest stay INT3. The error is formula (107.7) summed over the remaining columns, divided by the error of plain INT3. The four outlier channels get \(1 + (F-1)\cdot r\) times their normal size, with \(F\) the slider value and \(r = 1, 0.75, 0.6, 0.5\). With strong outliers the activation and sensitivity lines lie on top of each other, and so do the weight-norm and random lines.
How to read Figure 3

With outliers at 20×, the four outlier channels cause 78.0% of the error. Keeping 4 columns (1.56% of the weights, 3.5 bits per weight) chosen by activation or by sensitivity leaves 0.220 of the error; chosen by weight norm or at random it leaves 0.997. This is the AWQ table in miniature. Now set the outliers to 1×: no column stands out, and even 16 FP16 columns (4.25 bits) only bring the error to 0.822 (sensitivity) or 0.827 (activation). Mixed precision is powerful only when the error is concentrated. The sensitivity rule is never worse than the activation rule, because it also counts the column’s own rounding error.

Algorithm 2: outlier-aware quantization (OWQ columns, SpQR weights)
  1. input: weights \(W\), calibration inputs \(X\), base bits \(b\), outlier budget (columns \(k\) or rate \(r_o\))
  2. \(H \leftarrow 2XX^\top\) (plus a small damping on the diagonal)
  3. \(\Delta W \leftarrow W - Q(W)\) with a first quantization pass
  4. OWQ: \(\text{sens}_j \leftarrow H_{jj}\lVert\Delta W_{:,j}\rVert^2\) // formula (107.7); keep the top \(k\) columns in FP16; move them to the end; run GPTQ on the rest
  5. SpQR: run GPTQ column by column with small groups; for each weight compute \(\text{sens}_{ij}\) // formula (107.8); if above the threshold \(\tau\), store \((w_{ij}, j)\) in the sparse list and do not round it
  6. quantize the group scales and zeros again (SpQR, bilevel)
  7. at inference: \(y = \hat W_{\text{dense}}\,\mathbf{x} + W_{\text{sparse}}\,\mathbf{x}\) with a dense low-bit kernel plus a sparse FP16 kernel

Non-uniform levels: SqueezeLLM

Uniform INT levels are evenly spaced. But weights are not evenly spread: most are near 0, a few are far out. A few outliers stretch the range, so most levels sit where almost no weights are. SqueezeLLM (Kim et al.) fixes both problems.

Sensitivity-weighted k-means

Give each output row its own table of \(2^b\) levels (8 levels for 3 bits), and choose the levels by k-means, so they follow the weight distribution. Plain k-means minimizes the weight error. SqueezeLLM weights each weight’s error by its sensitivity, estimated by the diagonal of the Fisher information (the average squared gradient):

\[ \min_{c_1, \dots, c_K} \; \sum_{i} F_{ii} \, \big(w_i - c(w_i)\big)^2, \qquad F_{ii} = \frac{1}{|D|} \sum_{d \in D} g_{d,i}^2 \tag{107.11} \]
Read the symbols
\(c_1, \dots, c_K\)
the \(K = 2^b\) levels of one row; \(c(w_i)\) is the level closest to \(w_i\)
\(F_{ii}\)
sensitivity of weight \(i\): its squared gradient of the training loss, averaged over calibration samples \(D\)
\(g_{d,i}\)
gradient of the loss with respect to weight \(i\) on sample \(d\)

The Fisher diagonal approximates the Hessian of the final loss, not of one layer’s output. It needs one backward pass per calibration sample (the paper uses 100 samples and reports that about 10 are usually enough). [Certain] The weighted k-means is Lloyd’s algorithm where each center is a weighted average: \(c_k = \sum_{i \in k} F_{ii} w_i / \sum_{i \in k} F_{ii}\). Sensitive weights pull the levels toward themselves.

Dense-and-sparse decomposition

\[ W = D + S, \qquad W\mathbf{x} = D\mathbf{x} + S\mathbf{x} \tag{107.12} \]

\(S\) is a sparse FP16 matrix (CSR format) holding the outliers (the largest-magnitude weights, about 0.4%) and the most sensitive weights (0.05%); \(D\) holds the rest. Taking the outliers out makes the range of \(D\) much smaller, so the levels can be packed where the weights are. SqueezeLLM keeps 0.45% of the weights in \(S\) in its main sparse setting. [Certain]

Figure 4. One row of 2,000 weights: Gaussian, with 1% of them 6 times larger (heavy tails). Each weight has a Fisher sensitivity (log-normal, with 3% of the weights 40 times more sensitive). The bars at the bottom compare the sensitivity-weighted error of formula (107.11) for the three ways to place levels.
How to read Figure 4

At 3 bits without the sparse part, uniform levels have error 0.4837: the outliers stretch the range and most levels fall in empty space. k-means brings it to 0.0642, and Fisher-weighted k-means to 0.0587. Turn on dense-and-sparse: 9 values (0.45%) move to FP16, and the errors drop to 0.0801, 0.0285 and 0.0240. The weighted version wins every time, but by less than the jump from uniform to k-means. Try 2 bits: with only 4 levels, removing outliers matters most.

Algorithm 3: SqueezeLLM for one weight matrix
  1. input: weights \(W\), Fisher diagonal \(F\) (squared gradients on about 10 to 100 samples), bits \(b\), sparse rates (0.4% outliers, 0.05% sensitive)
  2. \(S \leftarrow\) weights beyond the outlier thresholds, plus the top 0.05% by \(F\); \(D \leftarrow W - S\) // formula (107.12)
  3. for each output row \(r\) of \(D\) do
  4.   initialize \(2^b\) levels (for example at quantiles); repeat Lloyd steps:
  5.     assign each weight to its nearest level
  6.     \(c_k \leftarrow \sum_{i \in k} F_{ii} w_i / \sum_{i \in k} F_{ii}\) // formula (107.11)
  7.   store \(b\)-bit indices and a table of \(2^b\) FP16 levels for row \(r\)
  8. store \(S\) in CSR (FP16 values, column indices, row pointers)

On LLaMA-7B (FP16 WikiText-2: 5.68), the SqueezeLLM paper reports 6.32 at 3.02 bits and 6.13 at 3.24 bits with the 0.45% sparse part; at about 4 bits, 5.79 (4.05 bits) and 5.77 (4.27 bits). [Certain] Its 3-bit kernel ran up to about 2.3× faster than FP16 on an A6000 at batch size 1 (the abstract says 2.3×, the main text 2.4×). [Certain]

Kernels for non-uniform formats: LUT-GEMM

Uniform INT4 has a simple kernel: load 4-bit codes, dequantize with scale and zero inside the matrix product, multiply in FP16. A non-uniform format needs a table lookup per weight to dequantize. LUT-GEMM (Park et al.) goes one step further: it removes the dequantization and most multiplications.

It uses binary-coding quantization (BCQ). Each weight is a sum of \(q\) signed bits with real scales:

\[ w \approx \sum_{i=1}^{q} \alpha_i\, b_i + z, \qquad b_i \in \{-1, +1\} \tag{107.13} \]

Uniform quantization is a special case: choose \(\alpha_i = 2^{i-1}\, s\) (half the bit weights of the INT code times the scale) and absorb the rest into the offset \(z\) (LUT-GEMM paper, Appendix C). [Certain] Non-uniform levels can be fitted directly with free \(\alpha_i\).

Now the dot product of one row with the input \(\mathbf{x}\). Cut \(\mathbf{x}\) into chunks of \(\mu\) inputs. For each bit-plane, the row’s bits inside one chunk are just a sign pattern, and there are only \(2^\mu\) possible patterns. So precompute, once per chunk, all \(2^\mu\) signed sums \(\pm x_1 \pm \dots \pm x_\mu\). Then every row and every bit-plane only looks up its partial sum:

\[ y_r = \sum_{i=1}^{q} \alpha_{r,i} \sum_{c=1}^{n/\mu} \mathrm{LUT}_c\big[\, \mathbf{b}_{r,i,c} \,\big] \;+\; z_r \sum_j x_j \tag{107.14} \]

Building the tables costs about \(2^\mu \cdot n/\mu\) additions and is shared by all \(m\) rows; reading costs \(m \cdot n/\mu \cdot q\) lookups, against \(m \cdot n\) multiply-adds for a dense product. The saving is about \(\mu/q\) when \(m\) is large. [Certain] The tables live in GPU shared memory and are shared by the threads of a block.

Figure 5. LUT-GEMM on a tiny example: \(\mu = 3\) inputs, 4 rows, 2 bit-planes with scales 0.6 and 0.2 (offset \(z = 0\)). The last two columns show the result from the table and from a normal dot product; they always agree.
How to read Figure 5

Step through. The table is built once for the chunk (8 entries). Row 1 then needs only two reads: LUT[101] = 3.5 and LUT[110] = −2.5, so \(y_1 = 0.6 \times 3.5 + 0.2 \times (-2.5) = 1.6\), the same as the direct product with weights 0.8, −0.4, 0.4. With 12,288 rows, the same 8 entries would be reused 12,288 times per bit-plane; that reuse is where the saving comes from.

Worked example: how much work does the table save?

One FFN-sized matrix with \(m = n = 12{,}288\), \(q = 3\) bits and \(\mu = 8\). Dense: \(m n = 151{,}000{,}000\) multiply-adds (rounded). Tables: \(2^8 \times 12{,}288/8 = 393{,}216\) entries. Lookups: \(12{,}288 \times 1{,}536 \times 3 = 56{,}623{,}104\). The ratio is 2.65, close to \(\mu/q = 8/3\). In the paper’s measurement for the first FFN layer of OPT-175B on an A100 (\(m = 12{,}288\), group size 128), cuBLAS FP16 took 0.7256 ms, the GPTQ 3-bit kernel 0.3599 ms and LUT-GEMM 3-bit 0.2250 ms (3.22× faster than FP16). [Certain]

End to end, LUT-GEMM ran OPT-175B in 3 bits on a single A100 at 51.6 ms per token, against 106.5 ms for the GPTQ kernel (about 2.1×). [Certain]

Why weight-only helps decoding, not prefill

During decoding with a small batch, each weight is read once and used for only a few tokens, so the matrix product is limited by memory bandwidth (session 82). Reading 3 or 4 bits instead of 16 is then a direct speedup, and the extra dequantization or lookup work is hidden. During prefill, or at large batch, the same weight is reused by hundreds of tokens; the product becomes compute-bound and runs on FP16 tensor cores after dequantization. Weight-only formats then save memory but little time, and can even be slower. To speed up compute-bound phases you must also quantize activations (session 108). [Certain]

All methods side by side

MethodWhat it protectsHowStorageKernel
RTNnothinground each groupINT3/4 + scale, zero per groupstandard dequant GEMV/GEMM
GPTQthe output, via error feedbackHessian-based updates of unrounded weightssame as RTNsame as RTN
AWQweights on large-activation channelsper-channel scale \(\mathbf{s}_X^\alpha\), folded into the previous op; clipping searchsame as RTNsame as RTN
OWQweak columns (outlier channels)Hessian column sensitivity (107.7); FP16 columns + GPTQINT + a few FP16 columnsdense low-bit + small FP16 part
SpQRsingle sensitive weightsOBS sensitivity (107.8); small groups, bilevel scalesINT3 + 3-bit stats + CSR outlierscustom dense + sparse
SqueezeLLMsensitive and outlier weightsFisher-weighted k-means levels; dense-and-sparse3/4-bit indices + per-row LUT + CSRLUT dequant + sparse
LUT-GEMM(a kernel, not a selection rule)binary coding, table of signed sumsbit-planes + scales per grouplookups, no dequant

The numbers in different papers are not directly comparable. For LLaMA-7B at INT3 with groups of 128, the AWQ paper reports 6.35 for AWQ while the SqueezeLLM paper reports 6.44 for AWQ; the SpQR paper reports 4-bit RTN at 6.43, but the AWQ paper reports INT4-g128 RTN at 5.96. [Certain] The most likely reason is the group size: SpQR’s RTN and GPTQ baselines appear to use one scale per row. [Likely] Always check the bits (including scales), the group size, the calibration set and the evaluation context length before comparing.

A runnable experiment

The code builds a linear layer with 1% outlier input channels, then compares plain INT3 rounding, keeping 1% of the columns in FP16 by four rules, and the AWQ search. It runs on a CPU in a few seconds and prints the same numbers every time.

import torch
torch.manual_seed(0)

def quantize(w, bits=3, group=128):
    """Asymmetric min-max round-to-nearest, one scale and one zero point per group of `group` inputs."""
    rows, cols = w.shape
    g = w.reshape(rows, cols // group, group)
    lo, hi = g.amin(-1, keepdim=True), g.amax(-1, keepdim=True)
    scale = (hi - lo).clamp(min=1e-8) / (2**bits - 1)
    zero = torch.round(-lo / scale)
    q = torch.clamp(torch.round(g / scale) + zero, 0, 2**bits - 1)
    return ((q - zero) * scale).reshape(rows, cols)

# a toy linear layer y = W x with a few "outlier" input channels
d_in, d_out, n = 1024, 512, 2048
W = torch.randn(d_out, d_in) * 0.02
mag = torch.exp(0.3 * torch.randn(d_in))          # normal channels: size about 1
out_ch = torch.randperm(d_in)[:10]                 # 1% of the channels ...
mag[out_ch] *= 20.0                                # ... are 20x larger
X = torch.randn(d_in, n) * mag[:, None]            # calibration activations
Y = W @ X

def err(W_hat):                                    # relative output error
    return ((W_hat @ X - Y).pow(2).mean() / Y.pow(2).mean()).item()

bits, k = 3, d_in // 100
W_rtn = quantize(W, bits)
print(f"RTN INT{bits} g128            : {err(W_rtn):.4f}")

def keep_fp16(idx):                                # mixed precision: some columns stay exact
    W_hat = W_rtn.clone(); W_hat[:, idx] = W[:, idx]; return W_hat
act = X.abs().mean(1)                              # s_X, formula (107.6)
lam = (X * X).sum(1)                               # diagonal of X X^T (H up to a factor 2)
sens = lam * (W - W_rtn).pow(2).sum(0)             # OWQ sensitivity, formula (107.7)
print(f"keep 1% by activation       : {err(keep_fp16(act.topk(k).indices)):.4f}")
print(f"keep 1% by OWQ sensitivity  : {err(keep_fp16(sens.topk(k).indices)):.4f}")
print(f"keep 1% by weight norm      : {err(keep_fp16(W.norm(dim=0).topk(k).indices)):.4f}")
print(f"keep 1% at random           : {err(keep_fp16(torch.randperm(d_in)[:k])):.4f}")

# AWQ: s = s_X^alpha, search alpha on a grid of 20 values in [0, 1)
best = (float("inf"), None)
for i in range(20):
    alpha = i / 20
    s = act.pow(alpha)
    s = s / (s.max() * s.min()).sqrt()             # keeps numbers in a safe range
    W_hat = quantize(W * s[None, :], bits) / s[None, :]   # scale columns, quantize, fold 1/s back
    best = min(best, (err(W_hat), alpha))
print(f"AWQ, best alpha = {best[1]:.2f}     : {best[0]:.4f}")

Output with PyTorch 2.8 on a CPU: RTN 0.0463; keep 1% by activation 0.0087, by OWQ sensitivity 0.0087, by weight norm 0.0462, at random 0.0462; AWQ best \(\alpha = 0.35\), 0.0176. Scaling alone (one data type) removes about 62% of the error; keeping the 10 outlier columns exact removes 81%. In this toy, activation and sensitivity pick the same columns, because the outliers dominate everything. Other PyTorch versions may differ in the last digits.

Where this is used

SettingTypical choiceWhy
Serving a 4-bit chat model with vLLM, TGI or TensorRT-LLMAWQ or GPTQ, INT4, groups of 128Same storage as RTN, standard kernels, near-FP16 quality at 4 bits
A model on a laptop or a Jetson boardAWQ INT4 with TinyChat or similar runtimesBatch size 1 decoding is memory-bound; the AWQ paper reports 2.7 to 3.9× over FP16 on an RTX 4090 [Certain]
Video captioning, video question answering or key-moment detection with a VLM on an edge deviceAWQ on the language model of a VLM such as VILAThe paper reports VILA-7B on Jetson Orin going from 11.5 to 35.6 tokens/s [Certain]; the vision encoder often stays in FP16 [Likely]
Fitting a larger model into one consumer GPUSpQR or SqueezeLLM near 3 to 4 bitsNear-lossless quality per bit; SpQR reports a 33B model on one 24 GB GPU
Cheap task adaptation of a quantized modelOWQ weak-column tuningTrain only the FP16 columns; the low-bit part stays frozen
Video-RAG or video retrieval: a quantized LLM reads retrieved clip captions4-bit AWQ/GPTQ LLM, FP16 embedding modelFrees memory for long contexts and the KV cache; embeddings need precise distances [Likely]
Very low bits (2 to 3) on non-standard hardwareBinary-coded weights with LUT-GEMMNo dequantization; work scales with the number of bit-planes

Common mistakes

  • Choosing salient weights by weight size. The AWQ table shows this is no better than random. Use activation size or a Hessian or Fisher sensitivity.
  • Scaling too hard. \(\alpha = 1\) is often worse than no scaling (Figure 2): once the scaled column becomes the group maximum, the whole group loses resolution.
  • Forgetting to fold 1/s. If you scale the weights but do not divide the RMSNorm (or v, up) by the same \(\mathbf{s}\), the model output changes. Check that the FP16 model with scales folded gives the same logits before you quantize.
  • Different scales for layers that share an input. q, k and v read one normalized activation; they must use one \(\mathbf{s}\), because there is only one RMSNorm to fold it into.
  • Counting bits without the overhead. “3-bit” with groups of 128 and FP16 scale and zero is 3.25 bits; SpQR’s small groups and outliers add more (formula 107.10). Compare methods at equal total bits.
  • Comparing perplexities across papers. RTN at 4 bits is 6.43 in one paper and 5.96 in another for the same LLaMA-7B, most likely because of group size. Re-run baselines yourself.
  • Expecting speedups at large batch. Weight-only formats speed up memory-bound decoding. Prefill and large-batch serving are compute-bound; there the benefit is mostly memory.
  • Calibrating on the wrong layer inputs. Collect the inputs of block \(\ell\) from the already quantized blocks \(1 \dots \ell-1\), so each block corrects for the errors before it.

Interview questions

Why does AWQ choose salient channels by activation size and not by weight size?basic
The output error is the weight error times the input (formula 107.1). A weight’s rounding error has about the same size no matter how large the weight is (it depends on the group’s step \(\Delta\)), but it is multiplied by the activation of its channel. Channels with 10 to 100 times larger activations therefore cause most of the output error. The AWQ paper shows that keeping 1% of channels in FP16 chosen by activation brings OPT-6.7B INT3 from 23.54 to 11.39 perplexity, while choosing by weight norm gives 22.37, no better than random.
Explain mathematically why scaling a weight by s before quantization reduces its error.core
Write \(w x = (s w)(x/s)\). After quantization we compute \(Q(sw)\, x/s\). The rounding error of \(sw\) is about the step \(\Delta'\) of its group, independent of the value. In the output it is divided by \(s\), so the error ratio is \((\Delta'/\Delta)(1/s)\) (formula 107.4). If scaling one weight among 128 does not change the group maximum, \(\Delta' = \Delta\) and the error falls by \(1/s\). If \(s\) is too large, the weight becomes the group maximum, \(\Delta'\) grows and all other weights in the group get coarser steps. That is why the best \(s\) in the paper is about 2, not 4.
Where does the 1/s go at inference time? Does AWQ change the model architecture?core
It is folded into the operation that produces the activation: the RMSNorm weight before q/k/v and before gate/up, the up projection’s output rows before down, and the v projection’s output rows before o (only when v and o shapes match, so not with GQA). After folding the architecture is unchanged and the quantized weights use the normal INT4 group format, so any AWQ-compatible INT4 kernel can run them.
How does AWQ search the scales, and why not optimize s with gradients?core
It uses \(\mathbf{s} = \mathbf{s}_X^\alpha\), where \(\mathbf{s}_X\) is the mean absolute activation per channel, and tries 20 values of \(\alpha\) in \([0, 1)\), keeping the one with the smallest output MSE of the module on calibration data. Quantization contains rounding, which has zero gradient almost everywhere, so direct gradient descent does not work without tricks like the straight-through estimator. A one-parameter grid search is cheap, stable, and hard to overfit, which is why AWQ needs only a small calibration set.
AWQ or GPTQ: how do they differ and when would you pick each?core
GPTQ rounds columns one by one and updates the remaining weights with the inverse Hessian to compensate; it reconstructs the layer output on calibration data. AWQ only rescales channels and clips, using activation statistics, with no reconstruction. Both produce the same storage format. AWQ is faster to run, needs less calibration data and is reported to be more robust to calibration domain shift; on the AWQ paper’s Llama-2 tables it is slightly better at INT3 and INT4. GPTQ can be stronger at very low bits on some models. They are complementary: the scaling can be applied first and GPTQ run afterwards. [Likely]
What is a “weak column” in OWQ, and how many bits does OWQ 3.01 really use?core
A weak column is an input column whose sensitivity \(H_{jj}\lVert\Delta W_{:,j}\rVert^2\) is large, usually because its input channel has activation outliers (\(H_{jj} = 2\sum_t X_{tj}^2\)). OWQ keeps those columns in FP16 and quantizes the rest with GPTQ. The average is \(b + f(16-b)\); for 3.01 bits, \(f = 0.01/13 \approx 0.077\%\) of weights. Column indices and the zero-filled low-bit entries add a little more (the paper mentions 3.012 bits).
How does SpQR keep the bit cost low while using groups as small as 16?advanced
Small groups normally cost a lot: two FP16 numbers per 16 weights is 2 bits per weight. SpQR quantizes the scales and zero points themselves to 3 bits in second-level groups, with FP16 second-level statistics. With \(\beta_1 = 16\), \(\beta_2 = 32\), the statistics cost \(6/16 + 64/512 = 0.5\) bits. Outliers (about 0.2 to 0.5% of weights) are stored in a sparse list at about 32 bits each, so 0.4% costs 0.128 bits. Total with 3-bit codes: about 3.63 bits per weight (formula 107.10).
Why does SqueezeLLM use the Fisher information instead of the layer Hessian, and what does weighted k-means change?advanced
The layer Hessian \(2XX^\top\) measures the effect on one layer’s output; the Fisher diagonal (mean squared gradient of the final loss) measures the effect on the model’s loss, which is what we care about, and it is per weight, so it fits a 1-D k-means. Weighted k-means replaces each center by the sensitivity-weighted mean of its weights, pulling levels toward sensitive weights. The cost is a few backward passes on calibration samples. In Figure 4 the weighting lowers the weighted error from 0.0642 to 0.0587 at 3 bits; the larger gains come from non-uniform levels and from removing outliers.
How does LUT-GEMM compute a matrix-vector product without dequantizing?advanced
Weights are stored as bit-planes: \(w = \sum_i \alpha_i b_i\) with \(b_i \in \{\pm1\}\). For each chunk of \(\mu\) inputs, it precomputes all \(2^\mu\) signed sums and keeps them in shared memory. For every row and bit-plane, the \(\mu\) bits form an index into that table, so the partial dot product is one lookup, then multiplied by \(\alpha_i\). Cost: \(m \cdot n/\mu \cdot q\) lookups plus a table build shared by all rows, against \(m n\) multiply-adds, a saving of about \(\mu/q\). Uniform INT codes can be written in this binary form too.
Your INT4 AWQ model decodes 3× faster but prefill is no faster. Why?core
At batch size 1, decoding reads every weight for one token: the matrix-vector product is memory-bound, and reading 4.25 bits instead of 16 cuts time almost proportionally. Prefill multiplies the same weights with hundreds or thousands of tokens at once; it is compute-bound, and the kernel dequantizes to FP16 and uses the same tensor-core math as before, plus dequantization work. To speed it up you need low-precision activations too (INT8 or FP8 weight–activation quantization).
You must quantize a model to about 3 bits and quality drops badly with RTN. What do you try, in order?advanced
First, smaller groups (128 or 64) and count the extra bits. Second, AWQ scaling and clipping, cheap and kernel-compatible. Third, GPTQ (possibly after AWQ scaling). If the quality is still not enough and you can afford a custom kernel, keep outliers in FP16 (OWQ columns, SpQR sparse weights) or use non-uniform levels with a dense-and-sparse split (SqueezeLLM). Measure perplexity and task accuracy at equal total bits, and check the speed of the kernel you will actually deploy. Rotation methods (session 109) are another option for 2 to 3 bits.

Exercises

  1. A group has largest weight 1.2 and uses INT4 symmetric (levels \(-7 \dots 7\)). What is \(\Delta\)? A salient weight is 0.4. What is the largest \(s\) that does not change \(\Delta\)?
  2. Using the AWQ Table 2, by what factor does \((\Delta'/\Delta)(1/s)\) improve from \(s = 2\) to \(s = 4\), and why does perplexity still get worse?
  3. In a Llama 3 block with 8 KV heads and 32 query heads (head size 128, hidden size 4096), can AWQ fold a scale for o_proj into v_proj? Give the two shapes.
  4. Compute the SpQR bit cost (formula 107.10) for \(b_w = 3\), \(b_s = b_z = 3\), \(\beta_1 = \beta_2 = 16\), \(r_o = 0.3\%\).
  5. OWQ uses 4-bit base weights and wants 4.05 bits on average. What share of the weights can stay in FP16?
  6. LUT-GEMM with \(\mu = 8\): how many table entries per chunk, and how many lookups does a 4,096-wide row need for \(q = 4\)?
  7. In Figure 1, set the salient activation to 20×. Read the best \(s\) and the total error. Why can it use a larger \(s\) than at 10×?
Answerscheck
1. \(\Delta = 1.2/7 = 0.171\). The scaled weight must stay at most 1.2: \(0.4\,s \le 1.2\), so \(s \le 3\). 2. \(0.519 / 0.303 = 1.71\) times smaller for the salient weights, but at \(s = 4\) 21.2% of the groups have a larger \(\Delta\) (average \(\Delta'/\Delta = 1.213\)), so all the other weights in those groups get coarser steps; perplexity goes from 11.92 to 12.36. 3. No. v_proj has \(8 \times 128 = 1024\) output rows, while o_proj has \(32 \times 128 = 4096\) input columns; each v row feeds 4 query heads’ positions in o’s input, so one row scale cannot give 4096 different column scales. The reference code skips the pair when the shapes differ. 4. \(3 + 6/16 + 64/256 + 32 \times 0.003 = 3 + 0.375 + 0.25 + 0.096 = 3.721\) bits. 5. \(f = 0.05/12 \approx 0.42\%\). 6. \(2^8 = 256\) entries; \(4096/8 = 512\) chunks, times 4 bit-planes = 2,048 lookups (against 4,096 multiply-adds). 7. Best \(s = 2.5\), total ×0.347 (salient share at \(s = 1\) is 78.1%). The salient part is a bigger share of the error, so paying a larger price on the other weights is worth it.

Key takeaways

  • The output error of a quantized layer is the weight error weighted by the activation size (107.1). A few outlier channels cause most of it; keeping 1% of channels by activation fixes most of the INT3 loss, choosing by weight size does not.
  • AWQ scales salient columns up and their inputs down (107.3); the error falls by \((\Delta'/\Delta)/s\) (107.4). The scale \(\mathbf{s}_X^\alpha\) is found by a 20-point grid search (107.6) and folded into RMSNorm, up or v, so storage and kernels stay standard.
  • Mixed precision keeps sensitive weights in FP16: OWQ by columns (107.7), SpQR by single weights with an OBS score (107.8), small groups and bilevel scales; count the bits with (107.9) and (107.10).
  • SqueezeLLM places non-uniform levels with Fisher-weighted k-means (107.11) and moves 0.45% of weights to a sparse FP16 matrix (107.12).
  • LUT-GEMM turns binary-coded weights (107.13) into table lookups (107.14), saving about \(\mu/q\) of the work. All weight-only formats help memory-bound decoding most, not compute-bound prefill.

Key formulas

\[ \E[(\Delta y)^2] \approx \sum_j (Q(w_j)-w_j)^2\,\E[x_j^2], \quad \frac{\text{err}'}{\text{err}} = \frac{\Delta'}{\Delta}\cdot\frac1s, \quad \mathbf{s} = \mathbf{s}_X^{\alpha} \]
\[ \text{sens}_j = H_{jj}\lVert\Delta W_{:,j}\rVert^2, \quad \bar b \approx b_w + \frac{b_s+b_z}{\beta_1} + \frac{64}{\beta_1\beta_2} + 32\,r_o, \quad \min_c \sum_i F_{ii}(w_i - c(w_i))^2 \]

References for this session

  1. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  2. Dettmers et al. (2023). SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
  3. Lee et al. (2023). OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models
  4. Kim et al. (2023). SqueezeLLM: Dense-and-Sparse Quantization
  5. Park et al. (2022). LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
  6. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  7. Dettmers et al. (2023). SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
  8. Lee et al. (2023). OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models
  9. Kim et al. (2023). SqueezeLLM: Dense-and-Sparse Quantization
  10. Park et al. (2022). LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
  11. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  12. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  13. Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  14. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  15. MIT HAN Lab (code). llm-awq: AWQ and TinyChat reference implementation (awq/quantize/auto_scale.py)