Weight–activation quantization: LLM.int8(), SmoothQuant, ZeroQuant, OmniQuant
Weight-only quantization makes the weights small, but the matrix multiply still runs in 16-bit. To use INT8 or INT4 tensor cores, the activations must be quantized too, and that is hard because a few channels carry huge outliers. This session shows four answers: keep the outlier channels in FP16 (LLM.int8()), move the difficulty into the weights with a per-channel scale (SmoothQuant), use finer scales per token and per group (ZeroQuant), and learn the clipping and the scales (OmniQuant), up to Atom's 4-bit mixed-precision recipe.
What you will learn
- Explain why activation outliers sit in a few fixed channels, and why that breaks per-tensor and per-token INT8 but not per-channel scales
- Write the integer GEMM with per-token and per-output-channel scales, and say which scales an INT8 kernel can and cannot handle
- Apply LLM.int8(): threshold the outlier channels, run them in FP16, and know when it is slower than FP16
- Derive SmoothQuant's per-channel factor s, fold it into the previous layer, and choose the migration strength α
- Describe ZeroQuant's group-wise weights, dynamic per-token activations, kernel fusion and layer-by-layer distillation
- Describe OmniQuant's learnable weight clipping and learnable equivalent transformation, and Atom's 4-bit recipe (128 INT8 outlier channels, groups of 128)
- Read the papers' accuracy and speed tables and pick a method for a given model, batch size and GPU
Why this matters
Weight-only quantization (GPTQ, AWQ) makes a model small, but the matrix multiply still runs in 16-bit: the weights are turned back into FP16 just before they are used. To run the multiply itself on fast integer units (INT8 or INT4 tensor cores), the activations must be quantized too. Activations of large language models are hard to quantize, because a few channels carry values that are 20 to 100 times larger than the rest. This session is about the four standard answers to that problem.
Session 104 showed when this matters: weight-only quantization helps when decoding is limited by memory traffic (small batch), and weight–activation quantization (written W8A8, W4A4, …: weight bits, activation bits) helps when the multiply is limited by compute (prefill, large batches, serving many users). An A100 does 312 TFLOPS in FP16, 624 TOPS in INT8 and 1,248 TOPS in INT4 on its tensor cores. [Certain, as quoted in the Atom paper] Those factors of 2 and 4 are only available if both inputs of the multiply are integers.
- Outliers live in a few channels. We look at the data first: where the large values are, and why the usual scales fail (Figure 1).
- LLM.int8() keeps the few outlier channels in FP16 and runs more than 99.9% of the values in INT8 (Figures 2 and 3).
- SmoothQuant divides each activation channel by a factor and multiplies the matching weight row by the same factor. The output does not change, but both sides become easy to quantize (Figures 4 and 5).
- ZeroQuant uses finer scales (groups for weights, one scale per token for activations), fuses them into kernels, and adds a cheap layer-by-layer distillation.
- OmniQuant learns the clipping range and the smoothing factors by gradient descent (Figure 6); Atom combines mixed precision, groups and reordering to reach 4-bit weights and activations (Figure 7).
In the efficient-inference survey this is the weight–activation part of post-training quantization (its section 5.2.1). [Certain] Typical interview questions: “Why are activations harder to quantize than weights?”, “Why can’t you just use a per-channel scale for activations?”, “Derive the SmoothQuant scale”, “When is LLM.int8() slower than FP16?”, “What does W4A4 need to work?”. They all come back to one fact: outliers sit in fixed channels, and a fast integer GEMM can only take scales on its outer dimensions.
Outliers live in a few channels
The quantizer and the integer GEMM
We use the symmetric quantizer from session 103. A group of numbers shares one scale \(\Delta\) (a step size). Each number becomes a small integer:
- Read the symbols
- \(\bar{x}\)
- the stored integer; we read it back as \(\Delta \bar{x}\)
- \(\lfloor \cdot \rceil\)
- round to the nearest integer
- \(b\)
- the number of bits: \(Q = 127\) for INT8, \(Q = 7\) for INT4
- \(\max |x|\)
- the largest magnitude in the group that shares the scale (this rule is called absmax)
The rounding error of one number is at most \(\Delta / 2\). So the error of every number in a group is set by the largest number in that group. That is the whole problem of this session.
A linear layer computes \(Y = X W\) with activations \(X \in \R^{T \times C}\) (\(T\) tokens, \(C\) input channels) and weights \(W \in \R^{C \times O}\). Which groups can share a scale? Give each row of \(X\) (one token) its own scale \(\Delta^x_t\) and each column of \(W\) (one output channel) its own scale \(\Delta^w_o\). Then
- Read the symbols
- \(\Delta^x_t\)
- the scale of token \(t\) (per-token scale); one scale for all of \(X\) is per-tensor
- \(\Delta^w_o\)
- the scale of output channel \(o\) (per-channel weight scale)
- \(\sum_j \bar{X}_{tj} \bar{W}_{jo}\)
- a pure integer dot product, accumulated in INT32 on the tensor cores
Both scales sit outside the sum, so the GEMM runs on integers and the scales are applied once per output, in the epilogue (the last step of the kernel that writes the result). LLM.int8() calls this vector-wise quantization. [Certain]
Now try a scale per input channel \(j\) of \(X\), \(\Delta^x_j\). It would sit inside the sum: \(\sum_j \Delta^x_j \bar{X}_{tj} \bar{W}_{jo}\). The kernel would have to multiply by a different float inside its inner loop, and the integer GEMM is gone.
So the rule is: scales may depend on the token or on the output channel, not on the input channel. Keep this in mind; every method below works around it.
What the activations look like
Dettmers et al. measured the hidden states that enter the linear layers of OPT-style models and found emergent outlier features. [Certain, from the LLM.int8() paper]
- Most values are small: the non-outlier feature dimensions usually stay in about \([-3.5, 3.5]\). The outliers are 3 to 20 times larger than that.
- They sit in fixed feature dimensions (columns of \(X\)): in a 6.7B model about 150,000 outlier values per 2,048-token sequence, but in only 6 hidden dimensions. Up to 13B parameters, at most 7 dimensions.
- Around 6.7B parameters there is a phase shift: the same outlier dimensions appear in all layers and in about 75% of all tokens. Smaller models have them in some layers only.
- They are important: setting those few dimensions to zero drops the mean top-1 softmax probability from about 40% to about 20% and raises validation perplexity by 600 to 1000%. Removing 7 random dimensions changes perplexity by about 0.1%.
- They are mostly one-sided: in most models an outlier dimension is always positive or always negative.
The SmoothQuant paper adds two more observations: outliers are about 100 times larger than most activation values, and weights are flat and easy to quantize; INT8 and even INT4 weights were already known to keep accuracy. [Certain] Why the outliers exist is not fully settled; later work links them to attention heads that learn to “do nothing” and to the residual stream needing a few huge directions. [Speculation, as far as one cause is concerned]
What do outliers do to the quantizer? With one scale for the whole tensor, \(\Delta = m / Q\) with \(m\) the global maximum. A normal channel \(j\) with maximum \(m_j\) only uses a few of the integer levels:
- Read the symbols
- \(m_j\)
- the largest magnitude in input channel \(j\) (over the tokens of a calibration set)
- \(m\)
- the largest magnitude in the whole quantization group (here the whole tensor)
- \(\text{levels}_j\)
- the number of distinct integers channel \(j\) can reach; INT8 has \(2Q + 1 = 255\)
With \(m_j = 1.2\) and \(m = 45\), INT8 gives channel \(j\) only \(2 \times 127 \times 1.2 / 45 \approx 6.8\) levels: the channel is quantized with less than 3 bits. The SmoothQuant paper reports 2 to 3 effective levels for non-outlier channels of real models. [Certain] A per-token scale does not help, because the outlier channel is large in almost every token, so every row still has an outlier in it. Only a per-input-channel scale would fix it, and (108.3) says the INT8 GEMM cannot use one.
How to read Figure 1
Start with 0 outlier channels: all three scales give under 1% error in INT8. Now move to 4 outlier channels of size 40: the dark columns appear, and the per-tensor and per-token errors jump to about 9%, ten times higher, while the per-channel error stays at 0.5%. The readout shows why: the median normal channel now gets only about 12 of the 255 INT8 levels (108.4). Per-token does not help, because every token contains the outliers. Switch to INT4: with outliers, per-tensor and per-token errors are about 98%, which means almost every normal value is rounded to zero. Per-channel would work, but its scale sits on the inner dimension of the GEMM (108.3).
LLM.int8(): keep the outlier channels in FP16
The idea
If at most 7 of thousands of channels are the problem, take them out. LLM.int8() (Dettmers, Lewis, Belkada, Zettlemoyer, 2022) splits the multiply by input channel: the outlier channels go through an FP16 GEMM, and the rest through an INT8 GEMM with vector-wise scales (108.2). [Certain]
- Read the symbols
- \(\mathcal{O}\)
- the set of outlier input channels: any channel with at least one value of magnitude \(\alpha\) or more in the current input
- \(\bar{\mathcal{O}}\)
- all other channels; they are quantized to INT8 with per-token and per-output-channel scales
- \(X_{:,\mathcal{O}}, W_{\mathcal{O},:}\)
- the outlier columns of \(X\) and the matching rows of \(W\), kept in FP16
- \(\alpha\)
- the threshold; the paper uses \(\alpha = 6.0\)
Because both halves split the sum over \(j\), their outputs simply add: no approximation comes from the split itself. The outlier set is found at run time from the current input, so no calibration data is needed. The weights are stored in INT8; the FP16 rows for the outlier channels are dequantized on the fly. [Certain for the method; the storage detail follows the bitsandbytes implementation, Likely]
Two tokens, four channels, two outputs. Channel 3 is an outlier channel:
\(X = \begin{pmatrix} 0.5 & -1.2 & 45 & 0.8 \\ 1.1 & 0.3 & -38 & -0.6 \end{pmatrix}\), \(\ W = \begin{pmatrix} 0.2 & -0.1 \\ 0.4 & 0.3 \\ 0.05 & -0.02 \\ -0.3 & 0.25 \end{pmatrix}\), exact \(Y = \begin{pmatrix} 1.63 & -1.11 \\ -1.38 & 0.59 \end{pmatrix}\).
Vector-wise INT8 only. Token 1 has \(\Delta^x_1 = 45/127 = 0.3543\). Its normal values become \(0.5 \to 1 \to 0.354\), \(-1.2 \to -3 \to -1.063\), \(0.8 \to 2 \to 0.709\): errors of up to 29%. The result is \(\begin{pmatrix} 1.701 & -1.027 \\ -1.379 & 0.539 \end{pmatrix}\), a relative error of 4.85%.
LLM.int8(). Channel 3 has \(\max |x| = 45 \ge 6\), so it goes to FP16. The other three channels now have \(\Delta^x_1 = 1.2/127\) and \(\Delta^x_2 = 1.1/127\), so they are quantized finely. The result is \(\begin{pmatrix} 1.629 & -1.109 \\ -1.382 & 0.592 \end{pmatrix}\), a relative error of 0.13%: 37 times smaller, with 3 of 4 channels still in INT8.
- input: \(X\) (\(2 \times 4\)), \(W\) (\(4 \times 2\)), threshold \(\alpha\), \(Q = 2^{b-1} - 1\)
- for each channel \(j\): put \(j\) in \(\mathcal{O}\) if \(\max_t |X_{tj}| \ge \alpha\) // (108.5)
- \(Y_{16} \leftarrow X_{:,\mathcal{O}} W_{\mathcal{O},:}\) // FP16 GEMM
- for each token \(t\): \(\Delta^x_t \leftarrow \max_{j \notin \mathcal{O}} |X_{tj}| / Q\), \(\bar{X}_{tj} \leftarrow \lfloor X_{tj} / \Delta^x_t \rceil\) // (108.1)
- \(\Delta^w_o \leftarrow \max_{j \notin \mathcal{O}} |W_{jo}| / Q\), \(\bar{W}_{jo} \leftarrow \lfloor W_{jo} / \Delta^w_o \rceil\) // per output column
- for each \((t, o)\): \(A_{to} \leftarrow \sum_{j \notin \mathcal{O}} \bar{X}_{tj} \bar{W}_{jo}\) // integers only
- \(Y_{to} \leftarrow (Y_{16})_{to} + \Delta^x_t \Delta^w_o A_{to}\) // epilogue, (108.2)
How to read Figure 2
Press Play, or Step through by hand. Steps 2 to 5 scan the four channels: only channel 3 has \(\max |x| = 45 \ge 6\), so it turns orange. Step 6 runs the FP16 product of that one column: \(Y_{16} = (2.25, -0.9; -1.9, 0.76)\). Then each token gets a scale from its normal channels only: \(\Delta^x_1 = 1.2/127 = 0.00945\), so the stored integers of token 1 are 53, −127 and 85. The integer dot products are exact integers, for example \(A_{11} = 53 \cdot 63 + (-127) \cdot 127 + 85 \cdot (-95) = -20865\). The last step adds the two parts: error 0.13% against 4.85% for plain vector-wise INT8. Switch to INT4: the decomposition gives 1.92%, plain INT4 gives 71%. Move \(\alpha\) to 1: three channels go to FP16 and the result is exact, but most of the work is now in the slow path. Move it to 50: no channel is an outlier and you get the plain vector-wise result.
What the threshold does
A low \(\alpha\) sends many channels to FP16: very accurate, but the FP16 part grows and the speed-up shrinks. A high \(\alpha\) keeps everything in INT8 and the outliers come back. Because the gap between normal values (at most about 3.5) and outliers (often 20 to 60) is wide, any \(\alpha\) in a wide middle range finds the same few channels. Figure 3 shows this.
How to read Figure 3
At the paper’s \(\alpha = 6\) the 4 outlier channels turn orange and go to FP16; 93.75% of the channels stay in INT8, and the output error falls from 6.78% (vector-wise) to 0.60%, about 11 times lower. Move \(\alpha\) anywhere from 4 to 30: nothing changes, because no channel lies in that range. Below 4 the normal channels start to leave INT8 (at \(\alpha = 2\), 30 channels are FP16: better accuracy, but half the work is now in the slow path). Above 40, the outliers return to INT8 and the error is back to 6.78%. In a real 4,096-wide model, 6 or 7 FP16 channels are 0.15 to 0.17% of the channels.
What it achieved, and what it costs
On C4 perplexity, plain INT8 methods broke down as models grew, while LLM.int8() matched the 32-bit model at every size. [Certain, Table 1 of the paper]
| C4 perplexity (lower is better) | 125M | 1.3B | 2.7B | 6.7B | 13B |
|---|---|---|---|---|---|
| 32-bit float | 25.65 | 15.91 | 14.43 | 13.30 | 12.45 |
| INT8 absmax, per-tensor | 87.76 | 16.55 | 15.11 | 14.59 | 19.08 |
| INT8 absmax, vector-wise | 35.84 | 16.82 | 14.98 | 14.13 | 16.48 |
| INT8 zero-point, vector-wise | 25.72 | 15.94 | 14.36 | 13.38 | 13.47 |
| LLM.int8() (vector-wise + decomposition) | 25.83 | 15.93 | 14.44 | 13.24 | 12.45 |
Asymmetric (zero-point) quantization does better than absmax because outliers are one-sided: an always-positive channel wastes half of a symmetric range. It still fails at 13B.
The price is speed. The method was built to save memory (a 175B model in INT8 needs about 175 GB instead of 350 GB, so OPT-175B or BLOOM-176B fits on one server of consumer GPUs). Its kernels are slower than FP16 for small matrices. Measured speed of the first feed-forward layer of GPT-3-sized models, relative to FP16 [Certain, Table 5 of the paper]:
| Model size (model dimension) | 2.7B (2,560) | 6.7B (4,096) | 13B (5,140) | 175B (12,288) |
|---|---|---|---|---|
| INT8 GEMM alone, no quantization overhead | 1.63× | 1.67× | 2.13× | 2.29× |
| Vector-wise INT8 (with quantize and dequantize) | 0.94× | 1.18× | 1.59× | 2.00× |
| LLM.int8() (vector-wise + decomposition) | 0.64× | 0.86× | 1.22× | 1.81× |
Numbers below 1 are slow-downs. The overhead comes from finding the outlier columns, gathering them, running two GEMMs and adding the results. End to end, BLOOM-176B in INT8 was “slightly slower but close” to 16-bit in the paper’s Hugging Face benchmark. [Certain] So LLM.int8() is a memory method: use it to fit a model, not to make it faster. It is the 8-bit mode of the bitsandbytes library (load_in_8bit=True in Hugging Face Transformers). [Certain]
SmoothQuant: move the difficulty into the weights
The idea: an exact rescaling
LLM.int8() needs two GEMMs because the scale for the outlier channels cannot go inside one INT8 GEMM (108.3). SmoothQuant (Xiao, Lin, Seznec, Wu, Demouth, Han, 2022) takes the per-input-channel scale and moves it out of the kernel: it divides input channel \(j\) of \(X\) by a factor \(s_j\) and multiplies row \(j\) of \(W\) by the same factor. [Certain]
This is exact in real numbers. After the change, the outlier channel of \(\hat{X}\) is \(s_j\) times smaller, and row \(j\) of \(\hat{W}\) is \(s_j\) times larger. Weights are easy to quantize, so they can absorb some of the difficulty. Both \(\hat{X}\) and \(\hat{W}\) then go through a normal W8A8 GEMM with scales on the outer dimensions only.
The division by \(s\) is free at run time. \(X\) is the output of a LayerNorm (or RMSNorm), whose last step multiplies each channel by a learned weight \(\gamma_j\). Replace \(\gamma_j\) by \(\gamma_j / s_j\) once, offline, and the norm produces \(\hat{X}\) directly. The weight \(\hat{W}\) is also computed once and then quantized. Inputs that come from a residual add need an extra scaling on that branch. [Certain]
Choosing the factor
If we choose \(s_j = \max |X_j|\), all activation channels end with maximum 1: perfect for the activations, but the outlier is now in the weights, and the weight error is large. If we choose \(s_j = 1 / \max |W_j|\), the weights are perfect and the activations are untouched. SmoothQuant interpolates between the two with a migration strength \(\alpha\):
- Read the symbols
- \(s_j\)
- the smoothing factor of input channel \(j\); \(\hat{X}_{:j} = X_{:j} / s_j\), \(\hat{W}_{j:} = s_j W_{j:}\)
- \(\max_t |X_{tj}|\)
- the largest activation in channel \(j\), measured on calibration data (512 random sentences from the Pile in the paper)
- \(\max_o |W_{jo}|\)
- the largest weight in row \(j\) (the row that multiplies channel \(j\))
- \(\alpha\)
- how much of the difficulty moves to the weights: 0 moves none to the weights, 1 moves all
With \(\alpha = 0.5\) the two maxima become equal:
Each channel then splits its “difficulty” evenly between the two sides. The paper uses \(\alpha = 0.5\) for all OPT and BLOOM models and 0.75 for GLM-130B, whose activations are harder. For Llama-2 and Mistral models with per-token activations it reports 0.8 to 0.9. [Certain] \(\alpha\) is chosen by a small grid search on held-out data.
Channel A is an outlier channel: \(\max |X_A| = 64\), \(\max |W_A| = 0.25\). Channel B is normal: \(\max |X_B| = 2\), \(\max |W_B| = 0.5\).
\(\alpha = 0.5\): \(s_A = \sqrt{64} / \sqrt{0.25} = 8 / 0.5 = 16\). Then \(\max |\hat{X}_A| = 64/16 = 4\) and \(\max |\hat{W}_A| = 0.25 \times 16 = 4\). For B: \(s_B = \sqrt{2}/\sqrt{0.5} = 2\), giving 1 and 1.
Per-tensor INT8 activations before smoothing: the global maximum is 64, so channel B gets \(127 \times 2 / 64 \approx 3.97\) steps on each side of zero. After smoothing the maximum is 4, so B gets \(127 \times 1 / 4 = 31.75\) steps: 8 times finer, 3 more bits of resolution.
\(\alpha = 0.75\): \(s_A = 64^{0.75} / 0.25^{0.25} = 22.63 / 0.7071 = 32\); \(\hat{X}_A\) has maximum 2 and \(\hat{W}_A\) has maximum 8. The activations are easier still; the weight row of channel A is now 16 times larger than channel B’s (8 against 0.5), and if the weights use one scale per output column, the other rows lose resolution. That is the trade-off \(\alpha\) controls.
- \(m^x_j \leftarrow \max_t |X_{tj}|\) for every channel // calibration
- \(m^w_j \leftarrow \max_o |W_{jo}|\) for every row
- for \(j = 1, \dots, 8\) do
- \(s_j \leftarrow (m^x_j)^\alpha / (m^w_j)^{1-\alpha}\) // (108.7)
- \(\hat{X}_{:j} \leftarrow X_{:j} / s_j\), \(\hat{W}_{j:} \leftarrow s_j W_{j:}\) // (108.6), exact
- \(\Delta^x \leftarrow \max |\hat{X}| / Q\), \(\Delta^w \leftarrow \max |\hat{W}| / Q\) // (108.1)
- \(Y \approx \Delta^x \Delta^w \sum_j \bar{X}_{tj} \bar{W}_{jo}\); error against \(XW\) // (108.2)
How to read Figure 4
Press Play. Steps 1 and 2 measure \(m^x\) and \(m^w\): the activation maxima range from 0.59 to 31.7, the weight maxima from 0.05 to 0.65. Then each channel is smoothed in turn; the live box shows (108.7) with the numbers. Channel 3 at \(\alpha = 0.5\): \(s_3 = \sqrt{31.7} / \sqrt{0.05} = 25.18\), so its activations shrink from 31.7 to 1.26 and its weight row grows from 0.05 to 1.26 (108.8). Watch the error points: smoothing the first normal channels changes little (it even rises a bit), but the error drops from 12.8% to 6.2% when channel 3 is smoothed, and to 0.88% after channel 6. The last two normal channels move it a little, to 1.37% with all 8 channels smoothed, against 9.31% without smoothing. The product \(\hat{X}\hat{W}\) equals \(XW\) at every step; only the quantization error changes. Move \(\alpha\) during the animation: \(\alpha = 0\) gives flat weights, \(\alpha = 1\) flat activations. On this small layer the best W8A8 per-tensor point is \(\alpha = 0.65\) (0.79%). Switch to W4A4: 128% without smoothing, 24% at \(\alpha = 0.5\).
How to read Figure 5
Start at W8A8, per-tensor, \(\alpha = 0.5\): both rows of bars line up (108.8) at a common maximum of 0.90, and the output error is 1.10%, against 7.81% without smoothing (dashed line). Move \(\alpha\) to 0: the activations keep their outliers relative to the others (error 1.38%); move it to 1: all activations become flat, but the weights inherit the outliers (1.53%). The curve is a shallow valley: the best point here is \(\alpha = 0.4\). Switch to W4A4: smoothing still helps (95% error without, 20% with), but 20% is far from usable. SmoothQuant alone is an 8-bit method; Figure 7 and the next session show what 4 bits need.
Three levels of speed: O1, O2, O3
After smoothing, the paper offers three settings that trade accuracy for speed. [Certain, Table 2 of the paper] Weights always use one scale per tensor.
| Setting | Weights | Activations | What it costs at run time |
|---|---|---|---|
| SmoothQuant-O1 | per-tensor | per-token, dynamic | compute a max for each token |
| SmoothQuant-O2 | per-tensor | per-tensor, dynamic | compute one max per tensor |
| SmoothQuant-O3 | per-tensor | per-tensor, static | nothing: the scale is fixed from calibration |
Static means the activation scale is computed once on calibration data and frozen; dynamic means it is computed from the actual input at run time. O3 is the fastest because the kernel needs no reduction over the input before it starts.
Results
On OPT-175B, zero-shot accuracy averaged over 7 tasks, and WikiText perplexity [Certain, Table 3 of the paper]:
| OPT-175B | Average accuracy | WikiText perplexity |
|---|---|---|
| FP16 | 66.9% | 10.99 |
| W8A8, naive per-tensor | 35.5% | 93,080 |
| ZeroQuant | 35.8% | 84,648 |
| LLM.int8() | 66.7% | 11.10 |
| SmoothQuant-O1 | 66.5% | 11.11 |
| SmoothQuant-O3 | 66.8% | 11.17 |
Naive W8A8 destroys the model; SmoothQuant even in its fastest setting stays within 0.1 points of FP16. On Llama-2-7B with per-token activations and per-channel weights, WikiText-2 perplexity went from 5.474 to 5.515 (\(\alpha = 0.85\)); on Llama-2-70B from 3.320 to 3.359 (\(\alpha = 0.9\)). [Certain] Speed: up to 1.51× faster and 1.96× less memory in PyTorch, up to 1.56× faster with half the memory in FasterTransformer, OPT-175B served on half the GPUs, and a 530B model on one 8-GPU node. [Certain, as reported] SmoothQuant’s W8A8 path is supported in production engines such as TensorRT-LLM and vLLM. [Likely]
ZeroQuant: finer scales and kernel fusion
ZeroQuant (Yao et al., Microsoft, 2022) came a few months before SmoothQuant and took a different route: keep the data as it is, but use finer scales, and make those scales cheap with custom kernels. [Certain] It has three parts.
- Group-wise weights. The weight matrix is split into groups that each get their own scale (rows of the matrix grouped so that they fit the tensor-core tiles of the A100).
- Token-wise activations, dynamic. One scale per token, computed at run time, exactly the per-token scale of (108.2). It needs no calibration of activation ranges. Computing it naively costs extra memory traffic, so ZeroQuant fuses the quantization into the previous operator (for example the LayerNorm) and the dequantization into the GEMM epilogue, which scales the INT32 result by both scales before writing FP16.
- Layer-by-layer knowledge distillation (LKD). For lower bits (INT4 weights in the feed-forward layers), each quantized layer is trained to copy the original layer’s output, one layer at a time, using the original model’s outputs of the previous layers as inputs.
Group scales on the weights are fine for an integer GEMM when each group spans a block of the inner dimension: the kernel multiplies a block in integers, scales that partial sum in float, and accumulates. This costs more than (108.2), but much less than one scale per element. With \(g\) values per group and a 16-bit scale per group, storage per weight is
so INT4 with \(g = 128\) costs 4.125 bits per weight. The LKD loss for layer \(k\) is
- Read the symbols
- \(L_k, \hat{L}_k\)
- the original layer \(k\) and its quantized copy, the only part that is trained
- \(L_{k-1} \cdots L_1(X)\)
- the input of layer \(k\), computed with the original earlier layers
- \(\operatorname{MSE}\)
- mean squared error; the paper notes KL divergence can be used too
Only one layer and its optimizer states are in memory at a time, no labels are needed, and the data does not have to be the original training data. [Certain] This idea, “minimize the output error of one block”, reappears in GPTQ, AWQ’s scale search and OmniQuant.
Reported results [Certain, as reported in the ZeroQuant paper]: INT8 weights and activations with up to 5.19× (BERT-base) and 4.16× (GPT-3-style 350M) speed-up over FP16 on A100; INT4 feed-forward weights with INT8 attention weights and INT8 activations, with LKD, for 3× less memory; GPT-J-6B 3.67× faster; GPT-NeoX-20B served on 1 GPU instead of 2 with latency from 65 ms to 25 ms. But per-token scales do not fix channel outliers: in the SmoothQuant paper’s test, ZeroQuant’s W8A8 failed on OPT-175B (35.8% accuracy, table above). ZeroQuant works on models without strong emergent outliers. [Likely, as a reading of the two papers together]
OmniQuant: learn the clipping and the scales
SmoothQuant’s \(s\) comes from a formula with one hand-tuned \(\alpha\). The clipping range of the weights is usually the full min–max. OmniQuant (Shao et al., 2023) makes both learnable and trains them, block by block, to minimize the output error of each transformer block (the same objective as (108.10), applied to a whole block). [Certain] The model weights themselves stay frozen; only a few quantization parameters are trained, so it keeps the cost of post-training quantization.
Learnable weight clipping (LWC)
For each output channel, OmniQuant learns how much of the range to keep. With \(N\) bits and asymmetric (zero-point) quantization:
- Read the symbols
- \(\gamma, \beta \in [0, 1]\)
- clipping strengths for the upper and the lower bound; stored as sigmoids of free parameters, so they stay in \((0, 1)\)
- \(h\)
- the step size; \(\gamma = \beta = 1\) gives ordinary min–max quantization
- \(z\)
- the zero point: the integer that represents the real number 0
- \(W_q\)
- the stored integers; read back as \((W_q - z) h\)
Clipping trades two errors. A smaller range gives a smaller step, so normal weights are rounded more finely, but the few largest weights are cut off. The gradient through the rounding is passed with the straight-through estimator (treat \(\lfloor \cdot \rceil\) as the identity in the backward pass; see session 111).
How to read Figure 6
At 1.000 (min–max) the grid spans all weights, and the step is wide. In 4-bit, lowering the clipping to about 0.905 reduces the error by 12% (ratio 1.13): the largest weight is clipped, but all the others are rounded more finely. In 3-bit, the gain is larger: the best clipping is about 0.67, and the error is 1.55 times lower than min–max. Press “Learn the clipping”: gradient descent finds 0.898 (4-bit) or 0.711 (3-bit), close to the grid optimum. Real OmniQuant minimizes the block output error, not the weight error, so it also takes into account which weights meet large activations.
Learnable equivalent transformation (LET)
For the activations, OmniQuant generalizes SmoothQuant with a learnable scale \(s\) and a learnable shift \(\delta\) per channel (the shift idea comes from Outlier Suppression+):
- Read the symbols
- \(\delta, s \in \R^{1 \times C}\)
- per-channel shift and scale, trained by gradient descent
- \(\oslash, \odot\)
- element-wise division of each column of \(X - \delta\) by \(s_j\), and multiplication of each row \(j\) of \(W\) by \(s_j\)
- \(B + \delta W\)
- the new bias; it absorbs the shift exactly
The shift helps one-sided outliers: an always-positive channel is moved to be centred around zero, so a symmetric quantizer stops wasting half its range. OmniQuant also applies a scale between queries and keys inside attention (\(Q K^\top = (Q \oslash s)(s \odot K)^\top\)), which helps when the KV cache is quantized. [Certain]
Recipe and results [Certain, from the paper]: 128 random 2,048-token segments of WikiText-2 as calibration data, 20 epochs, learning rate 5e-3 for LWC and 1e-2 for LET, on a single A100; 1 to 16 hours for LLaMA-2 7B to 70B. WikiText-2 perplexity of LLaMA-1-7B (FP16 5.68): W6A6 SmoothQuant 6.03, OmniQuant 5.96; W4A4 SmoothQuant 25.25, OmniQuant 11.26. At W4A4, learning the parameters halves the perplexity, but 11.26 is still twice the FP16 value. 4-bit activations need more than smoothing.
Atom: 4-bit weights and activations
Atom (Zhao et al., 2023) asks what it takes to make W4A4 accurate and fast for serving, where large batches make the GEMMs compute-bound. It combines the ideas above into one system. [Certain]
- Mixed precision with reordering. 128 outlier channels, chosen offline as the channels with the largest sum of squares on 128 calibration sentences, are kept in INT8; all other channels use INT4. Irregular mixed precision is slow (that was LLM.int8()’s problem), so Atom reorders channels: outlier channels are moved to the end of the activation (by a reorder fused into the previous operator, under 0.5% of run time) and the weight rows are permuted the same way, once, offline. Now the INT4 part and the INT8 part are both dense blocks.
- Group quantization with groups of 128 for both weights and activations, with the group dequantization fused into the GEMM pipeline.
- Dynamic per-group activation scales, clipping (factors 0.9 for activations and 0.85 for weights, found by grid search), GPTQ for the INT4 weights, and an INT4 KV cache.
The storage per value with \(C\) channels, \(k\) INT8 outlier channels and one 16-bit scale per group of \(g\):
for Llama-7B (\(C = 4{,}096\)); the paper reports 4.251 effective bits. [Certain for the reported value; our formula is a simplification]
The ablation on Llama-7B (WikiText-2 perplexity, FP16 = 5.68) shows what each step buys [Certain, Table 3 of the Atom paper]:
| Step | Perplexity |
|---|---|
| W4A4 round-to-nearest | 2,315.52 |
| + keep 128 outlier channels in FP16 | 11.34 |
| + quantize those outliers to INT8 | 11.39 |
| + group size 128 | 6.22 |
| + clipping | 6.13 |
| + GPTQ for the weights | 6.04 |
| + INT4 KV cache | 6.16 |
Two lessons. First, outliers in INT8 are almost as good as FP16 (+0.05), which is what makes the dense mixed-precision kernel possible. Second, at 4 bits, outlier handling alone is not enough (11.39); the fine group scales close most of the remaining gap. In serving, Atom reports up to 7.73× the throughput of FP16 and 2.53× that of INT8 at the same latency target, with a 1.4% average zero-shot accuracy drop and 0.3 perplexity increase on Llama-65B. [Certain, as reported] These speeds need INT4 tensor cores (A100 and RTX-class GPUs); NVIDIA’s Hopper H100 dropped INT4 tensor-core support, which pushed later 4-bit work toward FP8 and FP4 formats. [Likely]
How to read Figure 7
Step 1 is the W8A8 vector-wise reference: 6.78% error. Step 2 drops to 4 bits with the same per-token and per-channel scales: the error is 94%; almost every normal activation rounds to zero, because each token’s scale is set by its outliers. Step 3 reorders the 4 channels with the largest sum of squares to the end and keeps them in INT8: 11.8%. Step 4 splits the other 60 channels into 4 groups of 15 with their own scales for activations and weights: 9.5%, now below the INT8 error without outlier handling (step 1). The order of the steps and the size of each jump follow the Atom ablation table; the numbers are from this toy layer, not from Llama.
Putting the methods side by side
| Method | Weights | Activations | How it handles outliers | Needs data? | Main benefit |
|---|---|---|---|---|---|
| LLM.int8() | INT8 per-channel | INT8 per-token, dynamic | FP16 GEMM for channels with \(|x| \ge 6\) | no | memory (2× smaller), no accuracy loss |
| SmoothQuant | INT8 (per-tensor or per-channel) | INT8 per-tensor static (O3) to per-token | per-channel factor \(s\) moved into the weights | calibration (512 sentences) | speed (up to 1.56×) and memory |
| ZeroQuant | INT8 or INT4, group-wise | INT8 per-token, dynamic | finer scales only; LKD for INT4 | optional (LKD) | speed via fused kernels |
| OmniQuant | INT4 to INT6, learned clipping | INT4 to INT6 per-token | learned scale and shift (LET) | 128 segments, 1–16 GPU hours | best PTQ accuracy at W6A6 in 2023 |
| Atom | INT4 groups of 128 (GPTQ) | INT4 groups of 128 + 128 INT8 channels | reorder + mixed precision + clipping | 128 sentences | serving throughput (up to 7.73× FP16) |
All quality numbers above come from the papers themselves, with different models and tasks; compare methods only inside one table. [Certain that these are the reported numbers] The next session, session 109, takes the other road: instead of handling outliers channel by channel, it rotates the activations so that no channel is an outlier. That is how QuaRot reaches 4-bit weights, activations and KV cache without mixed precision.
Pseudo-code
- input: activations \(X\) (FP16, \(T \times C\)); INT8 weights \(\bar{W}\) with column scales \(\Delta^w\); threshold \(\alpha = 6\)
- \(\mathcal{O} \leftarrow \{ j : \max_t |X_{tj}| \ge \alpha \}\) // found per input, no calibration
- \(Y_{16} \leftarrow X_{:,\mathcal{O}} \cdot \operatorname{dequant}(\bar{W}_{\mathcal{O},:})\) // FP16 GEMM, few columns
- \(\Delta^x_t \leftarrow \max_{j \notin \mathcal{O}} |X_{tj}| / 127\) for every token \(t\); \(\bar{X} \leftarrow \lfloor X_{:,\bar{\mathcal{O}}} / \Delta^x \rceil\)
- \(A \leftarrow \bar{X}\, \bar{W}_{\bar{\mathcal{O}},:}\) // INT8 GEMM, INT32 accumulate
- return \(Y_{16} + \operatorname{diag}(\Delta^x)\, A\, \operatorname{diag}(\Delta^w)\) // formula (108.5)
- input: model, calibration texts (for example 512 sentences), migration strength \(\alpha\)
- run the model on the texts; for each linear layer record \(m^x_j = \max_t |X_{tj}|\) over all tokens
- for each norm followed by linear layers (QKV after the attention norm, up/gate after the FFN norm) do
- \(m^w_j \leftarrow \max_o |W_{jo}|\) over all the linear layers that read this norm // they share one s
- \(s_j \leftarrow (m^x_j)^\alpha / (m^w_j)^{1-\alpha}\) // formula (108.7)
- norm weight \(\gamma_j \leftarrow \gamma_j / s_j\) (and bias \(\leftarrow\) bias \(/ s_j\)); each \(W_{j:} \leftarrow s_j W_{j:}\) // formula (108.6), exact
- quantize all weights to INT8; for O3, run calibration again to fix static activation scales
- grid-search \(\alpha \in \{0.4, 0.45, \dots, 0.9\}\) on held-out perplexity and keep the best
The ideas of this session in PyTorch, on a synthetic layer with 4 outlier channels. The script is deterministic and runs on a CPU in a few seconds. Integers are kept in float tensors, which is exact here because the integer products are small. The printed values are at the end.
import torch torch.manual_seed(0) T, C, O = 64, 256, 128 X = torch.randn(T, C) * (0.6 + 0.4 * torch.rand(C)) # 256 normal channels out = torch.tensor([3, 50, 120, 200]) # 4 outlier channels X[:, out] = torch.tensor([-1., 1., -1., -1.]) * 40 * (0.7 + 0.3 * torch.rand(T, 4)) W = 0.05 * torch.randn(C, O) W[out] *= 0.1 # small weights read the outliers Y = X @ W def q_absmax(x, bits=8, dim=None): """Symmetric absmax quantization. dim=None: one scale; dim=1: per row; dim=0: per column.""" qmax = 2 ** (bits - 1) - 1 m = x.abs().amax() if dim is None else x.abs().amax(dim=dim, keepdim=True) d = m.clamp(min=1e-12) / qmax return (x / d).round().clamp(-qmax, qmax), d # integers (kept as float), scales def rel(a, b): return ((a - b).norm() / b.norm()).item() def wxax(X, W, act="tensor", wgt="tensor", bits=8): xq, dx = q_absmax(X, bits, None if act == "tensor" else 1) # per-token = one scale per row wq, dw = q_absmax(W, bits, None if wgt == "tensor" else 0) # per-channel = one per output column return (xq @ wq) * dx * dw # integer GEMM, then scale the outer dims (108.2) def llm_int8(X, W, alpha=6.0): o = X.abs().amax(0) >= alpha # outlier channels (108.5) return X[:, o] @ W[o] + wxax(X[:, ~o], W[~o], "token", "channel"), int(o.sum()) def smooth(X, W, a=0.5): s = X.abs().amax(0).pow(a) / W.abs().amax(1).pow(1 - a) # (108.7) return X / s, W * s[:, None] # (108.6): same product def group_fake_quant(W, bits=4, g=64): """Group-wise weights: one scale per output column and per group of g input rows.""" Wg = W.reshape(C // g, g, O) q, d = q_absmax(Wg, bits, 1) return (q * d).reshape(C, O) print("1) per-tensor W8A8 :", round(rel(wxax(X, W), Y), 4)) print(" per-token W8A8 :", round(rel(wxax(X, W, "token", "channel"), Y), 4)) Yd, n = llm_int8(X, W) print("2) LLM.int8() :", round(rel(Yd, Y), 4), "with", n, "FP16 channels") for a in [0.25, 0.5, 0.75]: Xs, Ws = smooth(X, W, a) print(f"3) SmoothQuant a={a}: per-tensor", round(rel(wxax(Xs, Ws), Y), 4), " per-token", round(rel(wxax(Xs, Ws, "token"), Y), 4)) # 4) 4-bit weights and activations: the Atom recipe in miniature def a4_token(X, bits=4): q, d = q_absmax(X, bits, 1); return q * d def a4_group(X, bits=4, g=64): q, d = q_absmax(X.reshape(T, C // g, g), bits, 2); return (q * d).reshape(T, C) print("4) W4A4 per-token/per-channel:", round(rel(wxax(X, W, "token", "channel", 4), Y), 4)) keep = X.abs().amax(0).topk(8).indices # 8 outlier channels (Atom: 128) rest = torch.ones(C, dtype=torch.bool); rest[keep] = False Xo, Wo = wxax(X[:, keep], W[keep], "token", "channel", 8), None Y_mix = Xo + wxax(X[:, rest], W[rest], "token", "channel", 4) print(" + outliers in INT8 :", round(rel(Y_mix, Y), 4)) Xr, Wr = X[:, rest], W[rest] Cr = Xr.shape[1] def grp(M, bits, g, dim): shp = M.shape if dim == 1: # activations: groups along the channel axis of each token q, d = q_absmax(M.reshape(shp[0], -1, g), bits, 2) else: # weights: groups along the input axis of each output column q, d = q_absmax(M.reshape(-1, g, shp[1]), bits, 1) return (q * d).reshape(shp) Y_grp = Xo + grp(Xr, 4, 31, 1) @ grp(Wr, 4, 31, 0) # 248 = 8 groups of 31 print(" + group quantization :", round(rel(Y_grp, Y), 4)) # 5) OmniQuant-style learnable weight clipping (one strength per output column), 4-bit def lwc_quant(W, gamma, beta, bits=4): L = 2 ** bits - 1 hi, lo = torch.sigmoid(gamma) * W.amax(0), torch.sigmoid(beta) * W.amin(0) h = (hi - lo) / L # (108.11) z = (-lo / h).round() ste = lambda v: (v.round() - v).detach() + v # straight-through round q = torch.clamp(ste(W / h) + z, 0, L) return (q - z) * h gamma = torch.full((O,), 4.0, requires_grad=True) # sigmoid(4) = 0.982 beta = torch.full((O,), 4.0, requires_grad=True) opt = torch.optim.AdamW([gamma, beta], lr=5e-2, weight_decay=0.0) with torch.no_grad(): print("5) W4 MinMax output error :", round(rel(X @ lwc_quant(W, gamma + 99, beta + 99), Y), 4)) for step in range(300): loss = ((X @ lwc_quant(W, gamma, beta) - Y) ** 2).mean() # output error, not weight error opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): print(" W4 learned clipping :", round(rel(X @ lwc_quant(W, gamma, beta), Y), 4), " mean clip strength", round(torch.sigmoid(gamma).mean().item(), 3)) # Printed (PyTorch 2.8, CPU): # 1) per-tensor W8A8 : 0.1045 # per-token W8A8 : 0.0939 # 2) LLM.int8() : 0.0089 with 4 FP16 channels # 3) SmoothQuant a=0.25: per-tensor 0.0115 per-token 0.0107 # 3) SmoothQuant a=0.5: per-tensor 0.0114 per-token 0.0109 # 3) SmoothQuant a=0.75: per-tensor 0.0118 per-token 0.0115 # 4) W4A4 per-token/per-channel: 0.9843 # + outliers in INT8 : 0.159 # + group quantization : 0.1206 # 5) W4 MinMax output error : 0.3888 # W4 learned clipping : 0.2565 mean clip strength 0.665
Where this is used
| Application | Method and why | Link |
|---|---|---|
| Loading a model too big for your GPU | LLM.int8() through bitsandbytes halves weight memory with no calibration; accept the slower GEMMs. | session 58 |
| High-throughput chat serving | W8A8 (SmoothQuant-style) uses INT8 tensor cores in prefill and in large decode batches, where GEMMs are compute-bound. | session 135 |
| Long-prompt prefill: RAG, long documents | Prefill is compute-bound even at batch 1, so W8A8 speeds up time to first token; weight-only quantization does not. | session 104 |
| Video understanding: video QA, Video-RAG, key-moment detection | A video adds thousands of visual tokens per request; prefill over them is a large compute-bound GEMM where W8A8 or W4A4 pays off. Vision encoders have activation outliers too, so smoothing is applied there as well. [Likely] | session 112 |
| Video retrieval, classification and fingerprinting at scale | Embedding millions of clips is pure batched throughput: integer GEMMs with static (O3-style) scales give the lowest cost per clip. | session 140 |
| Edge and mobile inference | NPUs often run integer arithmetic only; static per-tensor activation scales after smoothing match what such hardware supports. [Likely] | session 140 |
| 4-bit serving on INT4-capable GPUs | Atom-style mixed precision with groups and reordering, or rotations (QuaRot) to remove outliers first. | session 109 |
Common mistakes
- Expecting LLM.int8() to be faster. It is a memory method. For 6.7B-sized layers its GEMM ran at 0.86× of FP16 in the paper; only the 13B and 175B shapes were faster.
- Using a per-input-channel activation scale in an INT8 GEMM. It sits inside the dot product (108.3). Either migrate it into the weights (SmoothQuant) or split the GEMM (LLM.int8()).
- Thinking per-token scales fix outliers. Outlier channels are large in almost every token, so each token’s scale is still set by them (Figure 1: 8.6% error per-token against 9.2% per-tensor).
- Forgetting that several linear layers share one smoothing factor. Q, K and V all read the same norm output, so they must use the same \(s\), computed from the maximum over all three weight matrices. A different \(s\) per projection cannot be folded into one norm.
- Calibrating on the wrong data. Static (O3) scales are frozen from calibration; if the deployed traffic (code, another language, long contexts) has larger activations, values clip. Use dynamic per-token scales when traffic is unknown.
- Pushing \(\alpha\) to 1. All activation channels become flat, but the outliers move into the weights (Figure 5, right end of the curve).
- Mixed precision without reordering. Gathering scattered outlier channels at run time makes the kernel irregular and slow; Atom reorders once offline and fuses the activation reorder.
- Comparing W4A4 results across papers. Some papers keep some layers or the KV cache in higher precision, or use groups; check exactly what was quantized before comparing perplexities.
Interview questions
Why are activations harder to quantize than weights in LLMs?basic
Why can you use per-token activation scales and per-channel weight scales in an INT8 GEMM, but not per-channel activation scales?core
Explain LLM.int8(). How does it choose the outliers and what does it cost?basic
Derive the SmoothQuant factor and show that the output is unchanged.core
How do you pick \(\alpha\) for SmoothQuant, and what happens at the extremes?core
What are SmoothQuant O1, O2 and O3?basic
ZeroQuant reports 5× speed-ups but fails on OPT-175B. Why?advanced
What does OmniQuant learn, and how is it trained?advanced
Why does clipping weights ever help, if it throws away information?core
What does Atom do to make W4A4 accurate and fast?advanced
You serve an 8B model at batch 1 with short prompts. Should you use W8A8 or W4A16?core
Several projections read the same LayerNorm output. How does that constrain SmoothQuant?advanced
Exercises
- A token has values 0.9, −2.1, 1.4 and an outlier of 60. Quantize it to INT8 with one per-token scale. What are the integers for the three normal values, and their relative errors?
- Using (108.4), how many INT8 levels does a channel with \(m_j = 3.5\) get if the tensor maximum is 60? How many with INT4?
- Compute the SmoothQuant factor for a channel with \(m^x = 50\), \(m^w = 0.04\) at \(\alpha = 0.5\) and \(\alpha = 0.8\), and the resulting maxima on both sides.
- In LLM.int8(), a 4,096-wide model has 7 outlier channels. What fraction of the multiply-adds run in FP16? Why can the layer still be slower than FP16?
- Compute the effective bits (108.9) of INT4 weights with group size 64 and 128, and of an Atom-style layer with \(C = 5{,}120\), \(k = 128\), \(g = 128\).
- Show that OmniQuant’s shift (108.12) gives exactly the same output. Then explain why a shift helps an always-positive channel under symmetric quantization.
- Why does Atom quantize the outlier channels to INT8 instead of keeping them in FP16, if FP16 is more accurate?
Answerscheck
Key takeaways
- Weight–activation quantization is needed to use integer tensor cores; it pays off when GEMMs are compute-bound (prefill, large batches), not in batch-1 decoding.
- LLM activations have outliers in a few fixed channels, present in almost every token. They crush the resolution of per-tensor and per-token scales (108.4), and a per-input-channel scale does not fit an integer GEMM (108.3).
- LLM.int8() runs channels with \(|x| \ge 6\) in FP16 and the rest in vector-wise INT8 (108.5): no accuracy loss up to 175B, half the memory, but often slower than FP16.
- SmoothQuant divides activation channels by \(s_j = m_x^\alpha / m_w^{1-\alpha}\) and multiplies weight rows by the same factor (108.6)–(108.7), folded offline into the norm. W8A8 with static per-tensor scales then keeps OPT-175B within 0.1 points of FP16, with up to 1.56× speed-up.
- ZeroQuant uses group-wise weights, dynamic per-token activations and fused kernels, plus layer-by-layer distillation (108.10); finer scales alone do not fix channel outliers.
- OmniQuant learns clipping (108.11) and scale-and-shift transformations (108.12) by minimizing block output error; Atom reaches W4A4 with 128 INT8 outlier channels, reordering and groups of 128 (about 4.25 bits, 108.13).
Key formulas
References for this session
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Yao et al. (2022). ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
- Shao et al. (2023). OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
- Zhao et al. (2023). Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Yao et al. (2022). ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
- Shao et al. (2023). OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
- Zhao et al. (2023). Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Jacob et al. (2017). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
- Wei et al. (2022). Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models
- Wei et al. (2023). Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling
- Yuan et al. (2023). RPTQ: Reorder-based Post-training Quantization for Large Language Models
- Dettmers et al. (code). bitsandbytes: k-bit quantization for PyTorch (LLM.int8() implementation)
- Ashkboos et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs