LLM & VLM courseBlog
108Track 5: LLM inference and serving · session 31 of 65

Weight–activation quantization: LLM.int8(), SmoothQuant, ZeroQuant, OmniQuant

Weight-only quantization makes the weights small, but the matrix multiply still runs in 16-bit. To use INT8 or INT4 tensor cores, the activations must be quantized too, and that is hard because a few channels carry huge outliers. This session shows four answers: keep the outlier channels in FP16 (LLM.int8()), move the difficulty into the weights with a per-channel scale (SmoothQuant), use finer scales per token and per group (ZeroQuant), and learn the clipping and the scales (OmniQuant), up to Atom's 4-bit mixed-precision recipe.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 103 to 107 (quantization math, where quantization helps, number formats, GPTQ, AWQ)

What you will learn

  • Explain why activation outliers sit in a few fixed channels, and why that breaks per-tensor and per-token INT8 but not per-channel scales
  • Write the integer GEMM with per-token and per-output-channel scales, and say which scales an INT8 kernel can and cannot handle
  • Apply LLM.int8(): threshold the outlier channels, run them in FP16, and know when it is slower than FP16
  • Derive SmoothQuant's per-channel factor s, fold it into the previous layer, and choose the migration strength α
  • Describe ZeroQuant's group-wise weights, dynamic per-token activations, kernel fusion and layer-by-layer distillation
  • Describe OmniQuant's learnable weight clipping and learnable equivalent transformation, and Atom's 4-bit recipe (128 INT8 outlier channels, groups of 128)
  • Read the papers' accuracy and speed tables and pick a method for a given model, batch size and GPU

Why this matters

Weight-only quantization (GPTQ, AWQ) makes a model small, but the matrix multiply still runs in 16-bit: the weights are turned back into FP16 just before they are used. To run the multiply itself on fast integer units (INT8 or INT4 tensor cores), the activations must be quantized too. Activations of large language models are hard to quantize, because a few channels carry values that are 20 to 100 times larger than the rest. This session is about the four standard answers to that problem.

Session 104 showed when this matters: weight-only quantization helps when decoding is limited by memory traffic (small batch), and weight–activation quantization (written W8A8, W4A4, …: weight bits, activation bits) helps when the multiply is limited by compute (prefill, large batches, serving many users). An A100 does 312 TFLOPS in FP16, 624 TOPS in INT8 and 1,248 TOPS in INT4 on its tensor cores. [Certain, as quoted in the Atom paper] Those factors of 2 and 4 are only available if both inputs of the multiply are integers.

  • Outliers live in a few channels. We look at the data first: where the large values are, and why the usual scales fail (Figure 1).
  • LLM.int8() keeps the few outlier channels in FP16 and runs more than 99.9% of the values in INT8 (Figures 2 and 3).
  • SmoothQuant divides each activation channel by a factor and multiplies the matching weight row by the same factor. The output does not change, but both sides become easy to quantize (Figures 4 and 5).
  • ZeroQuant uses finer scales (groups for weights, one scale per token for activations), fuses them into kernels, and adds a cheap layer-by-layer distillation.
  • OmniQuant learns the clipping range and the smoothing factors by gradient descent (Figure 6); Atom combines mixed precision, groups and reordering to reach 4-bit weights and activations (Figure 7).

In the efficient-inference survey this is the weight–activation part of post-training quantization (its section 5.2.1). [Certain] Typical interview questions: “Why are activations harder to quantize than weights?”, “Why can’t you just use a per-channel scale for activations?”, “Derive the SmoothQuant scale”, “When is LLM.int8() slower than FP16?”, “What does W4A4 need to work?”. They all come back to one fact: outliers sit in fixed channels, and a fast integer GEMM can only take scales on its outer dimensions.

Outliers live in a few channels

The quantizer and the integer GEMM

We use the symmetric quantizer from session 103. A group of numbers shares one scale \(\Delta\) (a step size). Each number becomes a small integer:

\[ \bar{x} = \operatorname{clamp}\!\left(\left\lfloor \frac{x}{\Delta} \right\rceil, -Q, Q\right), \qquad \Delta = \frac{\max |x|}{Q}, \qquad Q = 2^{b-1} - 1 \tag{108.1} \]
Read the symbols
\(\bar{x}\)
the stored integer; we read it back as \(\Delta \bar{x}\)
\(\lfloor \cdot \rceil\)
round to the nearest integer
\(b\)
the number of bits: \(Q = 127\) for INT8, \(Q = 7\) for INT4
\(\max |x|\)
the largest magnitude in the group that shares the scale (this rule is called absmax)

The rounding error of one number is at most \(\Delta / 2\). So the error of every number in a group is set by the largest number in that group. That is the whole problem of this session.

A linear layer computes \(Y = X W\) with activations \(X \in \R^{T \times C}\) (\(T\) tokens, \(C\) input channels) and weights \(W \in \R^{C \times O}\). Which groups can share a scale? Give each row of \(X\) (one token) its own scale \(\Delta^x_t\) and each column of \(W\) (one output channel) its own scale \(\Delta^w_o\). Then

\[ Y_{to} = \sum_{j=1}^{C} X_{tj} W_{jo} \approx \Delta^x_t \, \Delta^w_o \sum_{j=1}^{C} \bar{X}_{tj} \bar{W}_{jo} \tag{108.2} \]
Read the symbols
\(\Delta^x_t\)
the scale of token \(t\) (per-token scale); one scale for all of \(X\) is per-tensor
\(\Delta^w_o\)
the scale of output channel \(o\) (per-channel weight scale)
\(\sum_j \bar{X}_{tj} \bar{W}_{jo}\)
a pure integer dot product, accumulated in INT32 on the tensor cores

Both scales sit outside the sum, so the GEMM runs on integers and the scales are applied once per output, in the epilogue (the last step of the kernel that writes the result). LLM.int8() calls this vector-wise quantization. [Certain]

Now try a scale per input channel \(j\) of \(X\), \(\Delta^x_j\). It would sit inside the sum: \(\sum_j \Delta^x_j \bar{X}_{tj} \bar{W}_{jo}\). The kernel would have to multiply by a different float inside its inner loop, and the integer GEMM is gone.

\[ \sum_{j} \Delta^x_j \, \bar{X}_{tj} \bar{W}_{jo} \ \ne\ (\text{scale}) \times \sum_{j} \bar{X}_{tj} \bar{W}_{jo} \quad \text{unless all } \Delta^x_j \text{ are equal} \tag{108.3} \]

So the rule is: scales may depend on the token or on the output channel, not on the input channel. Keep this in mind; every method below works around it.

What the activations look like

Dettmers et al. measured the hidden states that enter the linear layers of OPT-style models and found emergent outlier features. [Certain, from the LLM.int8() paper]

  • Most values are small: the non-outlier feature dimensions usually stay in about \([-3.5, 3.5]\). The outliers are 3 to 20 times larger than that.
  • They sit in fixed feature dimensions (columns of \(X\)): in a 6.7B model about 150,000 outlier values per 2,048-token sequence, but in only 6 hidden dimensions. Up to 13B parameters, at most 7 dimensions.
  • Around 6.7B parameters there is a phase shift: the same outlier dimensions appear in all layers and in about 75% of all tokens. Smaller models have them in some layers only.
  • They are important: setting those few dimensions to zero drops the mean top-1 softmax probability from about 40% to about 20% and raises validation perplexity by 600 to 1000%. Removing 7 random dimensions changes perplexity by about 0.1%.
  • They are mostly one-sided: in most models an outlier dimension is always positive or always negative.

The SmoothQuant paper adds two more observations: outliers are about 100 times larger than most activation values, and weights are flat and easy to quantize; INT8 and even INT4 weights were already known to keep accuracy. [Certain] Why the outliers exist is not fully settled; later work links them to attention heads that learn to “do nothing” and to the residual stream needing a few huge directions. [Speculation, as far as one cause is concerned]

What do outliers do to the quantizer? With one scale for the whole tensor, \(\Delta = m / Q\) with \(m\) the global maximum. A normal channel \(j\) with maximum \(m_j\) only uses a few of the integer levels:

\[ \text{levels}_j \approx 2 Q \, \frac{m_j}{m} \tag{108.4} \]
Read the symbols
\(m_j\)
the largest magnitude in input channel \(j\) (over the tokens of a calibration set)
\(m\)
the largest magnitude in the whole quantization group (here the whole tensor)
\(\text{levels}_j\)
the number of distinct integers channel \(j\) can reach; INT8 has \(2Q + 1 = 255\)

With \(m_j = 1.2\) and \(m = 45\), INT8 gives channel \(j\) only \(2 \times 127 \times 1.2 / 45 \approx 6.8\) levels: the channel is quantized with less than 3 bits. The SmoothQuant paper reports 2 to 3 effective levels for non-outlier channels of real models. [Certain] A per-token scale does not help, because the outlier channel is large in almost every token, so every row still has an outlier in it. Only a per-input-channel scale would fix it, and (108.3) says the INT8 GEMM cannot use one.

Figure 1. A synthetic activation matrix: 16 tokens × 64 channels, normal values from a seeded Gaussian (largest magnitude 3.77), plus a few outlier channels whose values have the same sign in every token. Top: the magnitude of each value (darker = larger). Bottom: the relative quantization error, measured on the normal channels only, for three choices of scale (108.1).
How to read Figure 1

Start with 0 outlier channels: all three scales give under 1% error in INT8. Now move to 4 outlier channels of size 40: the dark columns appear, and the per-tensor and per-token errors jump to about 9%, ten times higher, while the per-channel error stays at 0.5%. The readout shows why: the median normal channel now gets only about 12 of the 255 INT8 levels (108.4). Per-token does not help, because every token contains the outliers. Switch to INT4: with outliers, per-tensor and per-token errors are about 98%, which means almost every normal value is rounded to zero. Per-channel would work, but its scale sits on the inner dimension of the GEMM (108.3).

LLM.int8(): keep the outlier channels in FP16

The idea

If at most 7 of thousands of channels are the problem, take them out. LLM.int8() (Dettmers, Lewis, Belkada, Zettlemoyer, 2022) splits the multiply by input channel: the outlier channels go through an FP16 GEMM, and the rest through an INT8 GEMM with vector-wise scales (108.2). [Certain]

\[ Y \approx X_{:,\mathcal{O}} \, W_{\mathcal{O},:} \;+\; \operatorname{diag}(\Delta^x)\,\big(\bar{X}_{:,\bar{\mathcal{O}}}\, \bar{W}_{\bar{\mathcal{O}},:}\big)\,\operatorname{diag}(\Delta^w), \qquad \mathcal{O} = \{\, j : \max_t |X_{tj}| \ge \alpha \,\} \tag{108.5} \]
Read the symbols
\(\mathcal{O}\)
the set of outlier input channels: any channel with at least one value of magnitude \(\alpha\) or more in the current input
\(\bar{\mathcal{O}}\)
all other channels; they are quantized to INT8 with per-token and per-output-channel scales
\(X_{:,\mathcal{O}}, W_{\mathcal{O},:}\)
the outlier columns of \(X\) and the matching rows of \(W\), kept in FP16
\(\alpha\)
the threshold; the paper uses \(\alpha = 6.0\)

Because both halves split the sum over \(j\), their outputs simply add: no approximation comes from the split itself. The outlier set is found at run time from the current input, so no calibration data is needed. The weights are stored in INT8; the FP16 rows for the outlier channels are dequantized on the fly. [Certain for the method; the storage detail follows the bitsandbytes implementation, Likely]

Worked example: one tiny LLM.int8() multiply

Two tokens, four channels, two outputs. Channel 3 is an outlier channel:

\(X = \begin{pmatrix} 0.5 & -1.2 & 45 & 0.8 \\ 1.1 & 0.3 & -38 & -0.6 \end{pmatrix}\), \(\ W = \begin{pmatrix} 0.2 & -0.1 \\ 0.4 & 0.3 \\ 0.05 & -0.02 \\ -0.3 & 0.25 \end{pmatrix}\), exact \(Y = \begin{pmatrix} 1.63 & -1.11 \\ -1.38 & 0.59 \end{pmatrix}\).

Vector-wise INT8 only. Token 1 has \(\Delta^x_1 = 45/127 = 0.3543\). Its normal values become \(0.5 \to 1 \to 0.354\), \(-1.2 \to -3 \to -1.063\), \(0.8 \to 2 \to 0.709\): errors of up to 29%. The result is \(\begin{pmatrix} 1.701 & -1.027 \\ -1.379 & 0.539 \end{pmatrix}\), a relative error of 4.85%.

LLM.int8(). Channel 3 has \(\max |x| = 45 \ge 6\), so it goes to FP16. The other three channels now have \(\Delta^x_1 = 1.2/127\) and \(\Delta^x_2 = 1.1/127\), so they are quantized finely. The result is \(\begin{pmatrix} 1.629 & -1.109 \\ -1.382 & 0.592 \end{pmatrix}\), a relative error of 0.13%: 37 times smaller, with 3 of 4 channels still in INT8.

Figure 2. LLM.int8() run on the worked example above, one operation per step. Orange columns of \(X\) and orange rows of \(W\) are outlier channels: they stay in FP16. Blue cells go through the integer path; after quantization they show their stored integers. The output \(Y\) first receives the FP16 part, then the rescaled integer part; the bars at the bottom compare the final error with plain vector-wise quantization of all four channels. In this toy, the outlier rows of \(W\) are kept in FP16 and the weight scales are taken over the other rows only.
How to read Figure 2

Press Play, or Step through by hand. Steps 2 to 5 scan the four channels: only channel 3 has \(\max |x| = 45 \ge 6\), so it turns orange. Step 6 runs the FP16 product of that one column: \(Y_{16} = (2.25, -0.9; -1.9, 0.76)\). Then each token gets a scale from its normal channels only: \(\Delta^x_1 = 1.2/127 = 0.00945\), so the stored integers of token 1 are 53, −127 and 85. The integer dot products are exact integers, for example \(A_{11} = 53 \cdot 63 + (-127) \cdot 127 + 85 \cdot (-95) = -20865\). The last step adds the two parts: error 0.13% against 4.85% for plain vector-wise INT8. Switch to INT4: the decomposition gives 1.92%, plain INT4 gives 71%. Move \(\alpha\) to 1: three channels go to FP16 and the result is exact, but most of the work is now in the slow path. Move it to 50: no channel is an outlier and you get the plain vector-wise result.

What the threshold does

A low \(\alpha\) sends many channels to FP16: very accurate, but the FP16 part grows and the speed-up shrinks. A high \(\alpha\) keeps everything in INT8 and the outliers come back. Because the gap between normal values (at most about 3.5) and outliers (often 20 to 60) is wide, any \(\alpha\) in a wide middle range finds the same few channels. Figure 3 shows this.

Figure 3. Mixed-precision decomposition (108.5) on the matrix of Figure 1 (4 outlier channels of size about 40) times a seeded weight matrix (64 × 32). Bars: \(\max_t |X_{tj}|\) for each input channel, log scale; the dashed line is \(\alpha\). The readout gives the output error of LLM.int8() against the plain vector-wise INT8 multiply (108.2).
How to read Figure 3

At the paper’s \(\alpha = 6\) the 4 outlier channels turn orange and go to FP16; 93.75% of the channels stay in INT8, and the output error falls from 6.78% (vector-wise) to 0.60%, about 11 times lower. Move \(\alpha\) anywhere from 4 to 30: nothing changes, because no channel lies in that range. Below 4 the normal channels start to leave INT8 (at \(\alpha = 2\), 30 channels are FP16: better accuracy, but half the work is now in the slow path). Above 40, the outliers return to INT8 and the error is back to 6.78%. In a real 4,096-wide model, 6 or 7 FP16 channels are 0.15 to 0.17% of the channels.

What it achieved, and what it costs

On C4 perplexity, plain INT8 methods broke down as models grew, while LLM.int8() matched the 32-bit model at every size. [Certain, Table 1 of the paper]

C4 perplexity (lower is better)125M1.3B2.7B6.7B13B
32-bit float25.6515.9114.4313.3012.45
INT8 absmax, per-tensor87.7616.5515.1114.5919.08
INT8 absmax, vector-wise35.8416.8214.9814.1316.48
INT8 zero-point, vector-wise25.7215.9414.3613.3813.47
LLM.int8() (vector-wise + decomposition)25.8315.9314.4413.2412.45

Asymmetric (zero-point) quantization does better than absmax because outliers are one-sided: an always-positive channel wastes half of a symmetric range. It still fails at 13B.

The price is speed. The method was built to save memory (a 175B model in INT8 needs about 175 GB instead of 350 GB, so OPT-175B or BLOOM-176B fits on one server of consumer GPUs). Its kernels are slower than FP16 for small matrices. Measured speed of the first feed-forward layer of GPT-3-sized models, relative to FP16 [Certain, Table 5 of the paper]:

Model size (model dimension)2.7B (2,560)6.7B (4,096)13B (5,140)175B (12,288)
INT8 GEMM alone, no quantization overhead1.63×1.67×2.13×2.29×
Vector-wise INT8 (with quantize and dequantize)0.94×1.18×1.59×2.00×
LLM.int8() (vector-wise + decomposition)0.64×0.86×1.22×1.81×

Numbers below 1 are slow-downs. The overhead comes from finding the outlier columns, gathering them, running two GEMMs and adding the results. End to end, BLOOM-176B in INT8 was “slightly slower but close” to 16-bit in the paper’s Hugging Face benchmark. [Certain] So LLM.int8() is a memory method: use it to fit a model, not to make it faster. It is the 8-bit mode of the bitsandbytes library (load_in_8bit=True in Hugging Face Transformers). [Certain]

SmoothQuant: move the difficulty into the weights

The idea: an exact rescaling

LLM.int8() needs two GEMMs because the scale for the outlier channels cannot go inside one INT8 GEMM (108.3). SmoothQuant (Xiao, Lin, Seznec, Wu, Demouth, Han, 2022) takes the per-input-channel scale and moves it out of the kernel: it divides input channel \(j\) of \(X\) by a factor \(s_j\) and multiplies row \(j\) of \(W\) by the same factor. [Certain]

\[ Y = X W = \big(X \operatorname{diag}(s)^{-1}\big)\,\big(\operatorname{diag}(s)\, W\big) = \hat{X} \hat{W} \tag{108.6} \]

This is exact in real numbers. After the change, the outlier channel of \(\hat{X}\) is \(s_j\) times smaller, and row \(j\) of \(\hat{W}\) is \(s_j\) times larger. Weights are easy to quantize, so they can absorb some of the difficulty. Both \(\hat{X}\) and \(\hat{W}\) then go through a normal W8A8 GEMM with scales on the outer dimensions only.

The division by \(s\) is free at run time. \(X\) is the output of a LayerNorm (or RMSNorm), whose last step multiplies each channel by a learned weight \(\gamma_j\). Replace \(\gamma_j\) by \(\gamma_j / s_j\) once, offline, and the norm produces \(\hat{X}\) directly. The weight \(\hat{W}\) is also computed once and then quantized. Inputs that come from a residual add need an extra scaling on that branch. [Certain]

Choosing the factor

If we choose \(s_j = \max |X_j|\), all activation channels end with maximum 1: perfect for the activations, but the outlier is now in the weights, and the weight error is large. If we choose \(s_j = 1 / \max |W_j|\), the weights are perfect and the activations are untouched. SmoothQuant interpolates between the two with a migration strength \(\alpha\):

\[ s_j = \frac{\max_t |X_{tj}|^{\alpha}}{\max_o |W_{jo}|^{1-\alpha}} \tag{108.7} \]
Read the symbols
\(s_j\)
the smoothing factor of input channel \(j\); \(\hat{X}_{:j} = X_{:j} / s_j\), \(\hat{W}_{j:} = s_j W_{j:}\)
\(\max_t |X_{tj}|\)
the largest activation in channel \(j\), measured on calibration data (512 random sentences from the Pile in the paper)
\(\max_o |W_{jo}|\)
the largest weight in row \(j\) (the row that multiplies channel \(j\))
\(\alpha\)
how much of the difficulty moves to the weights: 0 moves none to the weights, 1 moves all

With \(\alpha = 0.5\) the two maxima become equal:

\[ \max_t |\hat{X}_{tj}| = \max_o |\hat{W}_{jo}| = \sqrt{\max_t |X_{tj}| \cdot \max_o |W_{jo}|} \qquad (\alpha = 0.5) \tag{108.8} \]

Each channel then splits its “difficulty” evenly between the two sides. The paper uses \(\alpha = 0.5\) for all OPT and BLOOM models and 0.75 for GLM-130B, whose activations are harder. For Llama-2 and Mistral models with per-token activations it reports 0.8 to 0.9. [Certain] \(\alpha\) is chosen by a small grid search on held-out data.

Worked example: smoothing two channels

Channel A is an outlier channel: \(\max |X_A| = 64\), \(\max |W_A| = 0.25\). Channel B is normal: \(\max |X_B| = 2\), \(\max |W_B| = 0.5\).

\(\alpha = 0.5\): \(s_A = \sqrt{64} / \sqrt{0.25} = 8 / 0.5 = 16\). Then \(\max |\hat{X}_A| = 64/16 = 4\) and \(\max |\hat{W}_A| = 0.25 \times 16 = 4\). For B: \(s_B = \sqrt{2}/\sqrt{0.5} = 2\), giving 1 and 1.

Per-tensor INT8 activations before smoothing: the global maximum is 64, so channel B gets \(127 \times 2 / 64 \approx 3.97\) steps on each side of zero. After smoothing the maximum is 4, so B gets \(127 \times 1 / 4 = 31.75\) steps: 8 times finer, 3 more bits of resolution.

\(\alpha = 0.75\): \(s_A = 64^{0.75} / 0.25^{0.25} = 22.63 / 0.7071 = 32\); \(\hat{X}_A\) has maximum 2 and \(\hat{W}_A\) has maximum 8. The activations are easier still; the weight row of channel A is now 16 times larger than channel B’s (8 against 0.5), and if the weights use one scale per output column, the other rows lose resolution. That is the trade-off \(\alpha\) controls.

Figure 4. SmoothQuant (108.6)–(108.7), one channel at a time, on a small layer: 6 tokens, 8 input channels, 4 outputs. Channel 3 (values 18 to 31.7) and channel 6 (always negative, −8.1 to −12.1) are outlier channels, and their weight rows are small (largest 0.05 and 0.188), as in real models. Top: per-channel maxima on a log scale; the outlined bars are the values before smoothing, the filled bars the current ones, and the dashed lines are the largest values, which set the per-tensor scales. Bottom: the relative output error \(\lVert \hat{X}_q \hat{W}_q - XW \rVert / \lVert XW \rVert\) after 0, 1, …, 8 channels have been smoothed; weights use one per-tensor scale.
How to read Figure 4

Press Play. Steps 1 and 2 measure \(m^x\) and \(m^w\): the activation maxima range from 0.59 to 31.7, the weight maxima from 0.05 to 0.65. Then each channel is smoothed in turn; the live box shows (108.7) with the numbers. Channel 3 at \(\alpha = 0.5\): \(s_3 = \sqrt{31.7} / \sqrt{0.05} = 25.18\), so its activations shrink from 31.7 to 1.26 and its weight row grows from 0.05 to 1.26 (108.8). Watch the error points: smoothing the first normal channels changes little (it even rises a bit), but the error drops from 12.8% to 6.2% when channel 3 is smoothed, and to 0.88% after channel 6. The last two normal channels move it a little, to 1.37% with all 8 channels smoothed, against 9.31% without smoothing. The product \(\hat{X}\hat{W}\) equals \(XW\) at every step; only the quantization error changes. Move \(\alpha\) during the animation: \(\alpha = 0\) gives flat weights, \(\alpha = 1\) flat activations. On this small layer the best W8A8 per-tensor point is \(\alpha = 0.65\) (0.79%). Switch to W4A4: 128% without smoothing, 24% at \(\alpha = 0.5\).

Figure 5. SmoothQuant (108.6)–(108.7) on the same layer as Figure 3. The weights of the outlier rows are 10 times smaller than the others, as is common when a channel is large. Top: per-channel maxima after smoothing, log scale (orange marks the 4 outlier channels). Bottom: relative output error \(\lVert \hat{X}_q \hat{W}_q - XW \rVert / \lVert XW \rVert\) against \(\alpha\); weights use one per-tensor scale, as in SmoothQuant-O3; the dashed line is the same quantization without smoothing.
How to read Figure 5

Start at W8A8, per-tensor, \(\alpha = 0.5\): both rows of bars line up (108.8) at a common maximum of 0.90, and the output error is 1.10%, against 7.81% without smoothing (dashed line). Move \(\alpha\) to 0: the activations keep their outliers relative to the others (error 1.38%); move it to 1: all activations become flat, but the weights inherit the outliers (1.53%). The curve is a shallow valley: the best point here is \(\alpha = 0.4\). Switch to W4A4: smoothing still helps (95% error without, 20% with), but 20% is far from usable. SmoothQuant alone is an 8-bit method; Figure 7 and the next session show what 4 bits need.

Three levels of speed: O1, O2, O3

After smoothing, the paper offers three settings that trade accuracy for speed. [Certain, Table 2 of the paper] Weights always use one scale per tensor.

SettingWeightsActivationsWhat it costs at run time
SmoothQuant-O1per-tensorper-token, dynamiccompute a max for each token
SmoothQuant-O2per-tensorper-tensor, dynamiccompute one max per tensor
SmoothQuant-O3per-tensorper-tensor, staticnothing: the scale is fixed from calibration

Static means the activation scale is computed once on calibration data and frozen; dynamic means it is computed from the actual input at run time. O3 is the fastest because the kernel needs no reduction over the input before it starts.

Results

On OPT-175B, zero-shot accuracy averaged over 7 tasks, and WikiText perplexity [Certain, Table 3 of the paper]:

OPT-175BAverage accuracyWikiText perplexity
FP1666.9%10.99
W8A8, naive per-tensor35.5%93,080
ZeroQuant35.8%84,648
LLM.int8()66.7%11.10
SmoothQuant-O166.5%11.11
SmoothQuant-O366.8%11.17

Naive W8A8 destroys the model; SmoothQuant even in its fastest setting stays within 0.1 points of FP16. On Llama-2-7B with per-token activations and per-channel weights, WikiText-2 perplexity went from 5.474 to 5.515 (\(\alpha = 0.85\)); on Llama-2-70B from 3.320 to 3.359 (\(\alpha = 0.9\)). [Certain] Speed: up to 1.51× faster and 1.96× less memory in PyTorch, up to 1.56× faster with half the memory in FasterTransformer, OPT-175B served on half the GPUs, and a 530B model on one 8-GPU node. [Certain, as reported] SmoothQuant’s W8A8 path is supported in production engines such as TensorRT-LLM and vLLM. [Likely]

ZeroQuant: finer scales and kernel fusion

ZeroQuant (Yao et al., Microsoft, 2022) came a few months before SmoothQuant and took a different route: keep the data as it is, but use finer scales, and make those scales cheap with custom kernels. [Certain] It has three parts.

  • Group-wise weights. The weight matrix is split into groups that each get their own scale (rows of the matrix grouped so that they fit the tensor-core tiles of the A100).
  • Token-wise activations, dynamic. One scale per token, computed at run time, exactly the per-token scale of (108.2). It needs no calibration of activation ranges. Computing it naively costs extra memory traffic, so ZeroQuant fuses the quantization into the previous operator (for example the LayerNorm) and the dequantization into the GEMM epilogue, which scales the INT32 result by both scales before writing FP16.
  • Layer-by-layer knowledge distillation (LKD). For lower bits (INT4 weights in the feed-forward layers), each quantized layer is trained to copy the original layer’s output, one layer at a time, using the original model’s outputs of the previous layers as inputs.

Group scales on the weights are fine for an integer GEMM when each group spans a block of the inner dimension: the kernel multiplies a block in integers, scales that partial sum in float, and accumulates. This costs more than (108.2), but much less than one scale per element. With \(g\) values per group and a 16-bit scale per group, storage per weight is

\[ b_{\text{eff}} = b + \frac{16}{g} \tag{108.9} \]

so INT4 with \(g = 128\) costs 4.125 bits per weight. The LKD loss for layer \(k\) is

\[ \mathcal{L}_k = \operatorname{MSE}\Big( L_k\big(L_{k-1} \cdots L_1(X)\big) \,-\, \hat{L}_k\big(L_{k-1} \cdots L_1(X)\big) \Big) \tag{108.10} \]
Read the symbols
\(L_k, \hat{L}_k\)
the original layer \(k\) and its quantized copy, the only part that is trained
\(L_{k-1} \cdots L_1(X)\)
the input of layer \(k\), computed with the original earlier layers
\(\operatorname{MSE}\)
mean squared error; the paper notes KL divergence can be used too

Only one layer and its optimizer states are in memory at a time, no labels are needed, and the data does not have to be the original training data. [Certain] This idea, “minimize the output error of one block”, reappears in GPTQ, AWQ’s scale search and OmniQuant.

Reported results [Certain, as reported in the ZeroQuant paper]: INT8 weights and activations with up to 5.19× (BERT-base) and 4.16× (GPT-3-style 350M) speed-up over FP16 on A100; INT4 feed-forward weights with INT8 attention weights and INT8 activations, with LKD, for 3× less memory; GPT-J-6B 3.67× faster; GPT-NeoX-20B served on 1 GPU instead of 2 with latency from 65 ms to 25 ms. But per-token scales do not fix channel outliers: in the SmoothQuant paper’s test, ZeroQuant’s W8A8 failed on OPT-175B (35.8% accuracy, table above). ZeroQuant works on models without strong emergent outliers. [Likely, as a reading of the two papers together]

OmniQuant: learn the clipping and the scales

SmoothQuant’s \(s\) comes from a formula with one hand-tuned \(\alpha\). The clipping range of the weights is usually the full min–max. OmniQuant (Shao et al., 2023) makes both learnable and trains them, block by block, to minimize the output error of each transformer block (the same objective as (108.10), applied to a whole block). [Certain] The model weights themselves stay frozen; only a few quantization parameters are trained, so it keeps the cost of post-training quantization.

Learnable weight clipping (LWC)

For each output channel, OmniQuant learns how much of the range to keep. With \(N\) bits and asymmetric (zero-point) quantization:

\[ h = \frac{\gamma \max(W) - \beta \min(W)}{2^N - 1}, \qquad z = -\left\lfloor \frac{\beta \min(W)}{h} \right\rceil, \qquad W_q = \operatorname{clamp}\!\left(\left\lfloor \frac{W}{h} \right\rceil + z,\ 0,\ 2^N - 1\right) \tag{108.11} \]
Read the symbols
\(\gamma, \beta \in [0, 1]\)
clipping strengths for the upper and the lower bound; stored as sigmoids of free parameters, so they stay in \((0, 1)\)
\(h\)
the step size; \(\gamma = \beta = 1\) gives ordinary min–max quantization
\(z\)
the zero point: the integer that represents the real number 0
\(W_q\)
the stored integers; read back as \((W_q - z) h\)

Clipping trades two errors. A smaller range gives a smaller step, so normal weights are rounded more finely, but the few largest weights are cut off. The gradient through the rounding is passed with the straight-through estimator (treat \(\lfloor \cdot \rceil\) as the identity in the backward pass; see session 111).

Figure 6. Learnable weight clipping (108.11) on one column of 128 seeded weights (standard deviation 0.02) with three large weights (0.11, −0.085, 0.075). Top: the weights (dots) and the quantization levels (ticks) inside the clipping range. Bottom: the mean squared quantization error against the clipping strength, with one shared \(\gamma = \beta\). “Learn the clipping” starts from \(\operatorname{sigmoid}(4) = 0.982\) and runs 200 Adam steps (learning rate 0.05) with the straight-through gradient.
How to read Figure 6

At 1.000 (min–max) the grid spans all weights, and the step is wide. In 4-bit, lowering the clipping to about 0.905 reduces the error by 12% (ratio 1.13): the largest weight is clipped, but all the others are rounded more finely. In 3-bit, the gain is larger: the best clipping is about 0.67, and the error is 1.55 times lower than min–max. Press “Learn the clipping”: gradient descent finds 0.898 (4-bit) or 0.711 (3-bit), close to the grid optimum. Real OmniQuant minimizes the block output error, not the weight error, so it also takes into account which weights meet large activations.

Learnable equivalent transformation (LET)

For the activations, OmniQuant generalizes SmoothQuant with a learnable scale \(s\) and a learnable shift \(\delta\) per channel (the shift idea comes from Outlier Suppression+):

\[ Y = XW + B = \big[(X - \delta) \oslash s\big]\,\big[s \odot W\big] + \big[B + \delta W\big] \tag{108.12} \]
Read the symbols
\(\delta, s \in \R^{1 \times C}\)
per-channel shift and scale, trained by gradient descent
\(\oslash, \odot\)
element-wise division of each column of \(X - \delta\) by \(s_j\), and multiplication of each row \(j\) of \(W\) by \(s_j\)
\(B + \delta W\)
the new bias; it absorbs the shift exactly

The shift helps one-sided outliers: an always-positive channel is moved to be centred around zero, so a symmetric quantizer stops wasting half its range. OmniQuant also applies a scale between queries and keys inside attention (\(Q K^\top = (Q \oslash s)(s \odot K)^\top\)), which helps when the KV cache is quantized. [Certain]

Recipe and results [Certain, from the paper]: 128 random 2,048-token segments of WikiText-2 as calibration data, 20 epochs, learning rate 5e-3 for LWC and 1e-2 for LET, on a single A100; 1 to 16 hours for LLaMA-2 7B to 70B. WikiText-2 perplexity of LLaMA-1-7B (FP16 5.68): W6A6 SmoothQuant 6.03, OmniQuant 5.96; W4A4 SmoothQuant 25.25, OmniQuant 11.26. At W4A4, learning the parameters halves the perplexity, but 11.26 is still twice the FP16 value. 4-bit activations need more than smoothing.

Atom: 4-bit weights and activations

Atom (Zhao et al., 2023) asks what it takes to make W4A4 accurate and fast for serving, where large batches make the GEMMs compute-bound. It combines the ideas above into one system. [Certain]

  • Mixed precision with reordering. 128 outlier channels, chosen offline as the channels with the largest sum of squares on 128 calibration sentences, are kept in INT8; all other channels use INT4. Irregular mixed precision is slow (that was LLM.int8()’s problem), so Atom reorders channels: outlier channels are moved to the end of the activation (by a reorder fused into the previous operator, under 0.5% of run time) and the weight rows are permuted the same way, once, offline. Now the INT4 part and the INT8 part are both dense blocks.
  • Group quantization with groups of 128 for both weights and activations, with the group dequantization fused into the GEMM pipeline.
  • Dynamic per-group activation scales, clipping (factors 0.9 for activations and 0.85 for weights, found by grid search), GPTQ for the INT4 weights, and an INT4 KV cache.

The storage per value with \(C\) channels, \(k\) INT8 outlier channels and one 16-bit scale per group of \(g\):

\[ b_{\text{eff}} = \frac{4\,(C - k) + 8\,k}{C} + \frac{16}{g} = \frac{4 \cdot 3968 + 8 \cdot 128}{4096} + \frac{16}{128} = 4.25 \tag{108.13} \]

for Llama-7B (\(C = 4{,}096\)); the paper reports 4.251 effective bits. [Certain for the reported value; our formula is a simplification]

The ablation on Llama-7B (WikiText-2 perplexity, FP16 = 5.68) shows what each step buys [Certain, Table 3 of the Atom paper]:

StepPerplexity
W4A4 round-to-nearest2,315.52
+ keep 128 outlier channels in FP1611.34
+ quantize those outliers to INT811.39
+ group size 1286.22
+ clipping6.13
+ GPTQ for the weights6.04
+ INT4 KV cache6.16

Two lessons. First, outliers in INT8 are almost as good as FP16 (+0.05), which is what makes the dense mixed-precision kernel possible. Second, at 4 bits, outlier handling alone is not enough (11.39); the fine group scales close most of the remaining gap. In serving, Atom reports up to 7.73× the throughput of FP16 and 2.53× that of INT8 at the same latency target, with a 1.4% average zero-shot accuracy drop and 0.3 perplexity increase on Llama-65B. [Certain, as reported] These speeds need INT4 tensor cores (A100 and RTX-class GPUs); NVIDIA’s Hopper H100 dropped INT4 tensor-core support, which pushed later 4-bit work toward FP8 and FP4 formats. [Likely]

Figure 7. Atom’s recipe in miniature, on the layer of Figures 3 and 5 (64 input channels, 4 of them outliers). Top: the layout of the activation matrix’s channels after each step (orange = 8-bit, crimson = 4-bit, each box one scale group; the 4 outlier channels are drawn darker). Bottom: relative output error of each step; bars appear as you step. The W8A8 vector-wise bar is the reference.
How to read Figure 7

Step 1 is the W8A8 vector-wise reference: 6.78% error. Step 2 drops to 4 bits with the same per-token and per-channel scales: the error is 94%; almost every normal activation rounds to zero, because each token’s scale is set by its outliers. Step 3 reorders the 4 channels with the largest sum of squares to the end and keeps them in INT8: 11.8%. Step 4 splits the other 60 channels into 4 groups of 15 with their own scales for activations and weights: 9.5%, now below the INT8 error without outlier handling (step 1). The order of the steps and the size of each jump follow the Atom ablation table; the numbers are from this toy layer, not from Llama.

Putting the methods side by side

MethodWeightsActivationsHow it handles outliersNeeds data?Main benefit
LLM.int8()INT8 per-channelINT8 per-token, dynamicFP16 GEMM for channels with \(|x| \ge 6\)nomemory (2× smaller), no accuracy loss
SmoothQuantINT8 (per-tensor or per-channel)INT8 per-tensor static (O3) to per-tokenper-channel factor \(s\) moved into the weightscalibration (512 sentences)speed (up to 1.56×) and memory
ZeroQuantINT8 or INT4, group-wiseINT8 per-token, dynamicfiner scales only; LKD for INT4optional (LKD)speed via fused kernels
OmniQuantINT4 to INT6, learned clippingINT4 to INT6 per-tokenlearned scale and shift (LET)128 segments, 1–16 GPU hoursbest PTQ accuracy at W6A6 in 2023
AtomINT4 groups of 128 (GPTQ)INT4 groups of 128 + 128 INT8 channelsreorder + mixed precision + clipping128 sentencesserving throughput (up to 7.73× FP16)

All quality numbers above come from the papers themselves, with different models and tasks; compare methods only inside one table. [Certain that these are the reported numbers] The next session, session 109, takes the other road: instead of handling outliers channel by channel, it rotates the activations so that no channel is an outlier. That is how QuaRot reaches 4-bit weights, activations and KV cache without mixed precision.

Pseudo-code

Algorithm 1: LLM.int8() forward pass of one linear layer
  1. input: activations \(X\) (FP16, \(T \times C\)); INT8 weights \(\bar{W}\) with column scales \(\Delta^w\); threshold \(\alpha = 6\)
  2. \(\mathcal{O} \leftarrow \{ j : \max_t |X_{tj}| \ge \alpha \}\) // found per input, no calibration
  3. \(Y_{16} \leftarrow X_{:,\mathcal{O}} \cdot \operatorname{dequant}(\bar{W}_{\mathcal{O},:})\) // FP16 GEMM, few columns
  4. \(\Delta^x_t \leftarrow \max_{j \notin \mathcal{O}} |X_{tj}| / 127\) for every token \(t\); \(\bar{X} \leftarrow \lfloor X_{:,\bar{\mathcal{O}}} / \Delta^x \rceil\)
  5. \(A \leftarrow \bar{X}\, \bar{W}_{\bar{\mathcal{O}},:}\) // INT8 GEMM, INT32 accumulate
  6. return \(Y_{16} + \operatorname{diag}(\Delta^x)\, A\, \operatorname{diag}(\Delta^w)\) // formula (108.5)
Algorithm 2: SmoothQuant calibration and conversion (offline)
  1. input: model, calibration texts (for example 512 sentences), migration strength \(\alpha\)
  2. run the model on the texts; for each linear layer record \(m^x_j = \max_t |X_{tj}|\) over all tokens
  3. for each norm followed by linear layers (QKV after the attention norm, up/gate after the FFN norm) do
  4. \(m^w_j \leftarrow \max_o |W_{jo}|\) over all the linear layers that read this norm // they share one s
  5. \(s_j \leftarrow (m^x_j)^\alpha / (m^w_j)^{1-\alpha}\) // formula (108.7)
  6. norm weight \(\gamma_j \leftarrow \gamma_j / s_j\) (and bias \(\leftarrow\) bias \(/ s_j\)); each \(W_{j:} \leftarrow s_j W_{j:}\) // formula (108.6), exact
  7. quantize all weights to INT8; for O3, run calibration again to fix static activation scales
  8. grid-search \(\alpha \in \{0.4, 0.45, \dots, 0.9\}\) on held-out perplexity and keep the best

The ideas of this session in PyTorch, on a synthetic layer with 4 outlier channels. The script is deterministic and runs on a CPU in a few seconds. Integers are kept in float tensors, which is exact here because the integer products are small. The printed values are at the end.

import torch
torch.manual_seed(0)
T, C, O = 64, 256, 128
X = torch.randn(T, C) * (0.6 + 0.4 * torch.rand(C))          # 256 normal channels
out = torch.tensor([3, 50, 120, 200])                          # 4 outlier channels
X[:, out] = torch.tensor([-1., 1., -1., -1.]) * 40 * (0.7 + 0.3 * torch.rand(T, 4))
W = 0.05 * torch.randn(C, O)
W[out] *= 0.1                                                  # small weights read the outliers
Y = X @ W

def q_absmax(x, bits=8, dim=None):
    """Symmetric absmax quantization. dim=None: one scale; dim=1: per row; dim=0: per column."""
    qmax = 2 ** (bits - 1) - 1
    m = x.abs().amax() if dim is None else x.abs().amax(dim=dim, keepdim=True)
    d = m.clamp(min=1e-12) / qmax
    return (x / d).round().clamp(-qmax, qmax), d               # integers (kept as float), scales

def rel(a, b):
    return ((a - b).norm() / b.norm()).item()

def wxax(X, W, act="tensor", wgt="tensor", bits=8):
    xq, dx = q_absmax(X, bits, None if act == "tensor" else 1)     # per-token = one scale per row
    wq, dw = q_absmax(W, bits, None if wgt == "tensor" else 0)     # per-channel = one per output column
    return (xq @ wq) * dx * dw              # integer GEMM, then scale the outer dims (108.2)

def llm_int8(X, W, alpha=6.0):
    o = X.abs().amax(0) >= alpha                                   # outlier channels (108.5)
    return X[:, o] @ W[o] + wxax(X[:, ~o], W[~o], "token", "channel"), int(o.sum())

def smooth(X, W, a=0.5):
    s = X.abs().amax(0).pow(a) / W.abs().amax(1).pow(1 - a)      # (108.7)
    return X / s, W * s[:, None]                                   # (108.6): same product

def group_fake_quant(W, bits=4, g=64):
    """Group-wise weights: one scale per output column and per group of g input rows."""
    Wg = W.reshape(C // g, g, O)
    q, d = q_absmax(Wg, bits, 1)
    return (q * d).reshape(C, O)

print("1) per-tensor W8A8 :", round(rel(wxax(X, W), Y), 4))
print("   per-token  W8A8 :", round(rel(wxax(X, W, "token", "channel"), Y), 4))
Yd, n = llm_int8(X, W)
print("2) LLM.int8()      :", round(rel(Yd, Y), 4), "with", n, "FP16 channels")
for a in [0.25, 0.5, 0.75]:
    Xs, Ws = smooth(X, W, a)
    print(f"3) SmoothQuant a={a}: per-tensor", round(rel(wxax(Xs, Ws), Y), 4),
          " per-token", round(rel(wxax(Xs, Ws, "token"), Y), 4))

# 4) 4-bit weights and activations: the Atom recipe in miniature
def a4_token(X, bits=4):
    q, d = q_absmax(X, bits, 1); return q * d
def a4_group(X, bits=4, g=64):
    q, d = q_absmax(X.reshape(T, C // g, g), bits, 2); return (q * d).reshape(T, C)
print("4) W4A4 per-token/per-channel:", round(rel(wxax(X, W, "token", "channel", 4), Y), 4))
keep = X.abs().amax(0).topk(8).indices                             # 8 outlier channels (Atom: 128)
rest = torch.ones(C, dtype=torch.bool); rest[keep] = False
Xo, Wo = wxax(X[:, keep], W[keep], "token", "channel", 8), None
Y_mix = Xo + wxax(X[:, rest], W[rest], "token", "channel", 4)
print("   + outliers in INT8       :", round(rel(Y_mix, Y), 4))
Xr, Wr = X[:, rest], W[rest]
Cr = Xr.shape[1]
def grp(M, bits, g, dim):
    shp = M.shape
    if dim == 1:   # activations: groups along the channel axis of each token
        q, d = q_absmax(M.reshape(shp[0], -1, g), bits, 2)
    else:          # weights: groups along the input axis of each output column
        q, d = q_absmax(M.reshape(-1, g, shp[1]), bits, 1)
    return (q * d).reshape(shp)
Y_grp = Xo + grp(Xr, 4, 31, 1) @ grp(Wr, 4, 31, 0)               # 248 = 8 groups of 31
print("   + group quantization     :", round(rel(Y_grp, Y), 4))

# 5) OmniQuant-style learnable weight clipping (one strength per output column), 4-bit
def lwc_quant(W, gamma, beta, bits=4):
    L = 2 ** bits - 1
    hi, lo = torch.sigmoid(gamma) * W.amax(0), torch.sigmoid(beta) * W.amin(0)
    h = (hi - lo) / L                                              # (108.11)
    z = (-lo / h).round()
    ste = lambda v: (v.round() - v).detach() + v                   # straight-through round
    q = torch.clamp(ste(W / h) + z, 0, L)
    return (q - z) * h
gamma = torch.full((O,), 4.0, requires_grad=True)                  # sigmoid(4) = 0.982
beta = torch.full((O,), 4.0, requires_grad=True)
opt = torch.optim.AdamW([gamma, beta], lr=5e-2, weight_decay=0.0)
with torch.no_grad():
    print("5) W4 MinMax output error  :", round(rel(X @ lwc_quant(W, gamma + 99, beta + 99), Y), 4))
for step in range(300):
    loss = ((X @ lwc_quant(W, gamma, beta) - Y) ** 2).mean()       # output error, not weight error
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    print("   W4 learned clipping     :", round(rel(X @ lwc_quant(W, gamma, beta), Y), 4),
          " mean clip strength", round(torch.sigmoid(gamma).mean().item(), 3))

# Printed (PyTorch 2.8, CPU):
# 1) per-tensor W8A8 : 0.1045
#    per-token  W8A8 : 0.0939
# 2) LLM.int8()      : 0.0089 with 4 FP16 channels
# 3) SmoothQuant a=0.25: per-tensor 0.0115  per-token 0.0107
# 3) SmoothQuant a=0.5: per-tensor 0.0114  per-token 0.0109
# 3) SmoothQuant a=0.75: per-tensor 0.0118  per-token 0.0115
# 4) W4A4 per-token/per-channel: 0.9843
#    + outliers in INT8       : 0.159
#    + group quantization     : 0.1206
# 5) W4 MinMax output error  : 0.3888
#    W4 learned clipping     : 0.2565  mean clip strength 0.665

Where this is used

ApplicationMethod and whyLink
Loading a model too big for your GPULLM.int8() through bitsandbytes halves weight memory with no calibration; accept the slower GEMMs.session 58
High-throughput chat servingW8A8 (SmoothQuant-style) uses INT8 tensor cores in prefill and in large decode batches, where GEMMs are compute-bound.session 135
Long-prompt prefill: RAG, long documentsPrefill is compute-bound even at batch 1, so W8A8 speeds up time to first token; weight-only quantization does not.session 104
Video understanding: video QA, Video-RAG, key-moment detectionA video adds thousands of visual tokens per request; prefill over them is a large compute-bound GEMM where W8A8 or W4A4 pays off. Vision encoders have activation outliers too, so smoothing is applied there as well. [Likely]session 112
Video retrieval, classification and fingerprinting at scaleEmbedding millions of clips is pure batched throughput: integer GEMMs with static (O3-style) scales give the lowest cost per clip.session 140
Edge and mobile inferenceNPUs often run integer arithmetic only; static per-tensor activation scales after smoothing match what such hardware supports. [Likely]session 140
4-bit serving on INT4-capable GPUsAtom-style mixed precision with groups and reordering, or rotations (QuaRot) to remove outliers first.session 109

Common mistakes

  • Expecting LLM.int8() to be faster. It is a memory method. For 6.7B-sized layers its GEMM ran at 0.86× of FP16 in the paper; only the 13B and 175B shapes were faster.
  • Using a per-input-channel activation scale in an INT8 GEMM. It sits inside the dot product (108.3). Either migrate it into the weights (SmoothQuant) or split the GEMM (LLM.int8()).
  • Thinking per-token scales fix outliers. Outlier channels are large in almost every token, so each token’s scale is still set by them (Figure 1: 8.6% error per-token against 9.2% per-tensor).
  • Forgetting that several linear layers share one smoothing factor. Q, K and V all read the same norm output, so they must use the same \(s\), computed from the maximum over all three weight matrices. A different \(s\) per projection cannot be folded into one norm.
  • Calibrating on the wrong data. Static (O3) scales are frozen from calibration; if the deployed traffic (code, another language, long contexts) has larger activations, values clip. Use dynamic per-token scales when traffic is unknown.
  • Pushing \(\alpha\) to 1. All activation channels become flat, but the outliers move into the weights (Figure 5, right end of the curve).
  • Mixed precision without reordering. Gathering scattered outlier channels at run time makes the kernel irregular and slow; Atom reorders once offline and fuses the activation reorder.
  • Comparing W4A4 results across papers. Some papers keep some layers or the KV cache in higher precision, or use groups; check exactly what was quantized before comparing perplexities.

Interview questions

Why are activations harder to quantize than weights in LLMs?basic
Weights are fixed and roughly Gaussian with few extremes, so a per-channel scale fits them well, and they can be quantized offline with care (GPTQ). Activations change with every input and, in large models, a few fixed channels carry values 20 to 100 times larger than the rest. With absmax scaling, the error of all values in a group is set by its largest value (108.1), so the outliers ruin the resolution of every other channel in the same token (108.4).
Why can you use per-token activation scales and per-channel weight scales in an INT8 GEMM, but not per-channel activation scales?core
The GEMM sums over the input channel \(j\). A scale that depends on the token \(t\) or the output channel \(o\) is constant inside that sum and can be applied once per output element in the epilogue (108.2). A scale that depends on \(j\) changes inside the sum (108.3), so the products cannot be accumulated as plain integers.
Explain LLM.int8(). How does it choose the outliers and what does it cost?basic
Vector-wise INT8 (per-token, per-output-channel scales) plus mixed-precision decomposition: any input channel with a value of magnitude at least \(\alpha = 6\) in the current input goes into a separate FP16 GEMM, the rest into INT8, and the outputs are added (108.5). It needs no calibration and kept full accuracy up to 175B (OPT-175B average 66.7% against 66.9%). Cost: finding and gathering outlier columns and running two GEMMs; in the paper it was slower than FP16 below the 13B shape (0.86× at 6.7B), 1.81× faster only at 175B.
Derive the SmoothQuant factor and show that the output is unchanged.core
\(XW = X \operatorname{diag}(s)^{-1} \operatorname{diag}(s) W\) for any positive \(s\) (108.6). We want both \(\hat{X} = X/s\) and \(\hat{W} = sW\) to have small channel maxima: \(\max |\hat{X}_j| = m^x_j / s_j\) and \(\max |\hat{W}_j| = s_j m^w_j\). Choosing \(s_j = (m^x_j)^\alpha / (m^w_j)^{1-\alpha}\) (108.7) interpolates; at \(\alpha = 0.5\), both maxima equal \(\sqrt{m^x_j m^w_j}\) (108.8). The division is folded into the previous norm’s weights and the multiplication into \(W\), offline, so there is no run-time cost.
How do you pick \(\alpha\) for SmoothQuant, and what happens at the extremes?core
Grid search on held-out perplexity. At \(\alpha = 0\) with \(s_j = 1/m^w_j\) the weights are flat and all difficulty stays in the activations; at \(\alpha = 1\) the activations are flat and the outliers move to the weights. The paper used 0.5 for OPT and BLOOM, 0.75 for GLM-130B and 0.8 to 0.9 for Llama-2 and Mistral with per-token activations. Models with stronger outliers need larger \(\alpha\).
What are SmoothQuant O1, O2 and O3?basic
Three activation granularities, all with per-tensor INT8 weights: O1 per-token dynamic, O2 per-tensor dynamic, O3 per-tensor static (scale fixed from calibration). O3 is fastest because the kernel needs no reduction over the input; on OPT-175B it kept 66.8% average accuracy against 66.9% for FP16.
ZeroQuant reports 5× speed-ups but fails on OPT-175B. Why?advanced
Its speed comes from fused kernels for token-wise activation quantization and group-wise weights, tested mainly on BERT and GPT-3-style models up to 20B. Per-token scales do not remove channel outliers, which are present in every token of large OPT models after the phase shift around 6.7B parameters. In the SmoothQuant paper’s test, ZeroQuant W8A8 on OPT-175B gave 35.8% average accuracy, as bad as naive W8A8.
What does OmniQuant learn, and how is it trained?advanced
Two sets of quantization parameters, with the model weights frozen: LWC learns per-channel clipping strengths \(\gamma, \beta\) through sigmoids (108.11), and LET learns per-channel scales and shifts for activations (108.12), including a scale between queries and keys. They are trained block by block to minimize the MSE between the full-precision and the quantized block outputs, with straight-through gradients for rounding; 128 calibration segments, 1 to 16 hours on one A100 for LLaMA-2 7B to 70B.
Why does clipping weights ever help, if it throws away information?core
With \(N\) bits, the step is the range divided by \(2^N - 1\). A few large weights make the range, and thus the step, large for all weights. Clipping them costs a large error on a few values but reduces the rounding error of all the others. The best clipping balances the two; it is stronger at fewer bits (Figure 6: about 0.9 in 4-bit, about 0.67 in 3-bit).
What does Atom do to make W4A4 accurate and fast?advanced
It keeps 128 outlier channels (largest sum of squares on calibration data) in INT8 and the rest in INT4; it reorders channels so both parts are dense blocks (weights permuted offline, activation reorder fused into the previous operator); it uses groups of 128 with fused dequantization, dynamic activation scales, clipping, GPTQ for weights and an INT4 KV cache. Effective storage is about 4.25 bits (108.13). On Llama-7B the ablation goes from 2,315 perplexity (RTN W4A4) to 6.16 (FP16 5.68); in serving it reported up to 7.73× FP16 throughput.
You serve an 8B model at batch 1 with short prompts. Should you use W8A8 or W4A16?core
W4A16 (weight-only). At batch 1, decode is memory-bound: each step reads all weights to produce one token, so time scales with weight bytes. 4-bit weights move 4 times fewer bytes than FP16; W8A8 moves only 2 times fewer and its faster integer math does not matter when the GPU is waiting for memory. W8A8 wins at large batches and in long prefills, where the GEMMs become compute-bound.
Several projections read the same LayerNorm output. How does that constrain SmoothQuant?advanced
The factor \(s\) is folded into the norm, so all consumers of that norm output see the same \(\hat{X}\) and must all be multiplied by the same \(s\). So \(m^w_j\) is taken as the maximum over the stacked Q, K and V weight rows (or up and gate for the FFN). The down projection in the FFN reads the output of the activation function, not a norm, so smoothing it needs the factor folded into the up projection’s output columns, which works for ReLU-like or gated paths but needs care. [Likely, as implementations differ]

Exercises

  1. A token has values 0.9, −2.1, 1.4 and an outlier of 60. Quantize it to INT8 with one per-token scale. What are the integers for the three normal values, and their relative errors?
  2. Using (108.4), how many INT8 levels does a channel with \(m_j = 3.5\) get if the tensor maximum is 60? How many with INT4?
  3. Compute the SmoothQuant factor for a channel with \(m^x = 50\), \(m^w = 0.04\) at \(\alpha = 0.5\) and \(\alpha = 0.8\), and the resulting maxima on both sides.
  4. In LLM.int8(), a 4,096-wide model has 7 outlier channels. What fraction of the multiply-adds run in FP16? Why can the layer still be slower than FP16?
  5. Compute the effective bits (108.9) of INT4 weights with group size 64 and 128, and of an Atom-style layer with \(C = 5{,}120\), \(k = 128\), \(g = 128\).
  6. Show that OmniQuant’s shift (108.12) gives exactly the same output. Then explain why a shift helps an always-positive channel under symmetric quantization.
  7. Why does Atom quantize the outlier channels to INT8 instead of keeping them in FP16, if FP16 is more accurate?
Answerscheck
1. \(\Delta = 60/127 = 0.4724\). \(0.9/0.4724 = 1.905 \to 2 \to 0.945\) (error 5.0%); \(-2.1/0.4724 = -4.445 \to -4 \to -1.890\) (10.0%); \(1.4/0.4724 = 2.963 \to 3 \to 1.417\) (1.2%). 2. \(2 \times 127 \times 3.5/60 = 14.8\) levels in INT8; \(2 \times 7 \times 3.5/60 = 0.82\), so in INT4 the channel rounds to 0 or \(\pm 1\) step, and most of its values become 0. 3. \(\alpha = 0.5\): \(s = \sqrt{50}/\sqrt{0.04} = 7.071/0.2 = 35.36\); both maxima become \(\sqrt{50 \times 0.04} = 1.414\). \(\alpha = 0.8\): \(s = 50^{0.8}/0.04^{0.2} = 22.87/0.5253 = 43.53\); activations \(50/43.53 = 1.149\), weights \(0.04 \times 43.53 = 1.741\). 4. \(7/4096 = 0.17\%\). Overheads: finding the outlier columns needs a pass over \(X\), gathering them breaks memory layout, there are two GEMM launches and an add, and quantizing \(X\) and dequantizing the output costs memory traffic; for medium-size matrices these dominate (Table 5 of the paper: 0.86× at 6.7B). 5. \(4 + 16/64 = 4.25\) and \(4 + 16/128 = 4.125\). Atom-style: \((4 \times 4992 + 8 \times 128)/5120 + 16/128 = 4.1 + 0.125 = 4.225\) bits. 6. \([(X - \delta) \oslash s][s \odot W] = (X - \delta) W = XW - \delta W\), and the bias adds back \(\delta W\), so the output is \(XW + B\). An always-positive channel with values in \([a, b]\), \(a \gt 0\), uses only the positive half of a symmetric grid with scale \(b/Q\); after subtracting \(\delta = (a + b)/2\) it spans \([-(b - a)/2, (b - a)/2]\), and the step becomes \((b - a)/(2Q)\), less than half as large. 7. An FP16 part would need a separate FP16 GEMM, and mixing FP16 and INT4 GEMMs is the same slow path as LLM.int8(). INT8 outliers run on integer tensor cores in the same fused kernel, and the ablation shows INT8 outliers cost only 0.05 perplexity (11.34 to 11.39).

Key takeaways

  • Weight–activation quantization is needed to use integer tensor cores; it pays off when GEMMs are compute-bound (prefill, large batches), not in batch-1 decoding.
  • LLM activations have outliers in a few fixed channels, present in almost every token. They crush the resolution of per-tensor and per-token scales (108.4), and a per-input-channel scale does not fit an integer GEMM (108.3).
  • LLM.int8() runs channels with \(|x| \ge 6\) in FP16 and the rest in vector-wise INT8 (108.5): no accuracy loss up to 175B, half the memory, but often slower than FP16.
  • SmoothQuant divides activation channels by \(s_j = m_x^\alpha / m_w^{1-\alpha}\) and multiplies weight rows by the same factor (108.6)–(108.7), folded offline into the norm. W8A8 with static per-tensor scales then keeps OPT-175B within 0.1 points of FP16, with up to 1.56× speed-up.
  • ZeroQuant uses group-wise weights, dynamic per-token activations and fused kernels, plus layer-by-layer distillation (108.10); finer scales alone do not fix channel outliers.
  • OmniQuant learns clipping (108.11) and scale-and-shift transformations (108.12) by minimizing block output error; Atom reaches W4A4 with 128 INT8 outlier channels, reordering and groups of 128 (about 4.25 bits, 108.13).

Key formulas

\[ \bar{x} = \operatorname{clamp}\!\big(\lfloor x/\Delta \rceil, -Q, Q\big), \quad Y_{to} \approx \Delta^x_t \Delta^w_o \textstyle\sum_j \bar{X}_{tj} \bar{W}_{jo}, \quad \text{levels}_j \approx 2Q\, m_j / m \]
\[ Y \approx X_{:,\mathcal{O}} W_{\mathcal{O},:} + \operatorname{diag}(\Delta^x) \bar{X}_{:,\bar{\mathcal{O}}} \bar{W}_{\bar{\mathcal{O}},:} \operatorname{diag}(\Delta^w), \qquad Y = (X \operatorname{diag}(s)^{-1})(\operatorname{diag}(s) W), \quad s_j = \frac{m_{x,j}^{\alpha}}{m_{w,j}^{1-\alpha}} \]
\[ h = \frac{\gamma \max W - \beta \min W}{2^N - 1}, \qquad Y = [(X - \delta) \oslash s][s \odot W] + [B + \delta W], \qquad b_{\text{eff}} = \frac{4(C - k) + 8k}{C} + \frac{16}{g} \]

References for this session

  1. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  2. Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  3. Yao et al. (2022). ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  4. Shao et al. (2023). OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
  5. Zhao et al. (2023). Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
  6. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  7. Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  8. Yao et al. (2022). ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
  9. Shao et al. (2023). OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
  10. Zhao et al. (2023). Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
  11. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  12. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  13. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  14. Jacob et al. (2017). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
  15. Wei et al. (2022). Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models
  16. Wei et al. (2023). Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling
  17. Yuan et al. (2023). RPTQ: Reorder-based Post-training Quantization for Large Language Models
  18. Dettmers et al. (code). bitsandbytes: k-bit quantization for PyTorch (LLM.int8() implementation)
  19. Ashkboos et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs