LLM & VLM courseBlog
103Track 5: LLM inference and serving · session 26 of 65

Quantization math: scale, zero point, granularity and error

Quantization stores numbers with fewer bits, for example 4-bit integers instead of 16-bit floats. This session builds the math behind every quantization method in this track: the scale and the zero point, symmetric and asymmetric grids, which numbers share one scale (per-tensor, per-channel, per-group, per-token), the two kinds of error (rounding and clipping), and why LLMs, with their huge outliers, are harder to quantize than older networks.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Session 37 (numbers on GPUs) and Session 81 (inference metrics); basic matrix multiplication

What you will learn

  • The affine map x ≈ s(q − z): how the scale s and the zero point z are computed, and why z is an integer
  • Symmetric vs asymmetric quantization, and why weights are usually symmetric
  • How an integer matrix multiply handles scales and zero points, and which scales can be pulled out of the sum
  • Granularity: per-tensor, per-channel, per-group (for example 128) and per-token, with their accuracy and storage costs
  • Rounding error vs clipping error, the 6 dB-per-bit rule, and three ways to choose the range: min-max, percentile, MSE search
  • Why LLM activations are hard: a few outlier channels (up to 20× larger, from about 6.7B parameters) waste most of the codes

Why this matters

Llama 3 8B has about 8.03 billion weights. In BF16 (2 bytes each) that is 16.1 GB. In 4-bit integers it is about 4 GB, plus a small amount for scales. The model then fits on a phone-class or a 12 GB gaming GPU, and every decode step reads four times fewer bytes from memory.

The last sessions made attention and the feed-forward layers cheaper by changing the architecture (session 97, session 95, session 100). Quantization keeps the architecture and changes the numbers: it stores weights (and sometimes activations and the KV cache) with fewer bits. In the efficient-inference survey it is the main part of “model compression” (its Section 5.2). [Certain, survey Section 5.2.1] Decoding is limited by memory bandwidth (session 81), so fewer bytes per weight means more tokens per second.

Every quantization method in this track (GPTQ, AWQ, SmoothQuant, QuaRot, KIVI, QLoRA) is built on one small piece of math: a scale, a zero point, a choice of which numbers share them, and a choice of range. This session is that piece. The next sessions only change how these four things are chosen.

The survey states the core problem in one sentence: compared with smaller models, the weights and activations of LLMs “often exhibit more outliers and have a wider distribution range”. [Certain, survey Section 5.2.1] By the end of this session you will be able to show, with numbers, why one outlier can turn an 8-bit format into a 3-bit one.

The idea: a ruler with few marks

A float can store almost any value. An integer with \(b\) bits can store only \(2^b\) different values: 256 for INT8, 16 for INT4. Quantization picks a set of \(2^b\) real values (the grid), and stores, for each number, the index (the code) of the nearest grid point. To use the number, we turn the code back into a real value. That step is dequantization.

In this session the grid is uniform: the grid points are equally spaced. The distance between two neighbours is the scale (or step size) \(s\). Non-uniform grids (NF4, FP4, look-up tables) come in session 105.

Uniform affine quantization

The standard map, used in TensorFlow Lite, PyTorch, TensorRT and almost every LLM method, comes from Jacob et al. (2018). [Certain] A real value \(x\) becomes an integer code \(q\):

\[ q = \operatorname{clamp}\!\Big( \operatorname{round}\!\Big(\frac{x}{s}\Big) + z,\; q_{\min},\; q_{\max} \Big) \tag{103.1} \]

and the code becomes a real value again:

\[ \hat{x} = s\,(q - z) \tag{103.2} \]
Read the symbols
\(x, \hat{x}\)
the real value, and its dequantized approximation
\(q\)
the integer code that is stored: 4 or 8 bits
\(s\)
the scale: a positive real number, the distance between two neighbouring grid points
\(z\)
the zero point: an integer, the code that means real 0
\(q_{\min}, q_{\max}\)
the smallest and largest code: \(0\) and \(2^b - 1\) for unsigned codes, \(-2^{b-1}\) and \(2^{b-1}-1\) for signed codes
\(\operatorname{clamp}(v, a, c)\)
\(\min(\max(v, a), c)\): values outside the range are pushed to the nearest end

“Affine” means “linear plus a shift”: \(\hat x\) is a linear function of \(q\), shifted by \(z\). Two numbers, \(s\) and \(z\), describe the whole grid. They are the quantization parameters, and they are stored next to the codes in 16 or 32 bits.

Choosing s and z from a range: asymmetric quantization

Suppose we decide that the values we care about lie in a range \([\alpha, \beta]\), with \(\alpha \le 0 \le \beta\). The simplest choice is \(\alpha = \min x\) and \(\beta = \max x\) (min-max). We spread the \(2^b\) codes over this range:

\[ s = \frac{\beta - \alpha}{2^b - 1}, \qquad z = \operatorname{round}\!\Big(\!-\frac{\alpha}{s}\Big), \qquad q \in \{0, 1, \dots, 2^b - 1\} \tag{103.3} \]

This is asymmetric quantization: the range does not have to be centered on 0. Why must \(z\) be an integer? So that real 0 is exactly on the grid: \(\hat x = s(z - z) = 0\). Zero is everywhere in networks (padding, ReLU outputs, masked values), and Jacob et al. require that it be represented with no error. [Certain]

Worked example: five numbers in 4 bits, asymmetric

Take \(x = (-1.2,\ -0.3,\ 0.0,\ 0.4,\ 2.5)\) and \(b = 4\), so codes 0 to 15. Min-max gives \(\alpha = -1.2\), \(\beta = 2.5\).

Scale: \(s = 3.7 / 15 = 0.2467\). Zero point: \(-\alpha/s = 4.865\), so \(z = 5\).

Codes: \(x/s + z = (0.14,\ 3.78,\ 5,\ 6.62,\ 15.14)\). Rounding and clamping to \([0, 15]\) gives \(q = (0, 4, 5, 7, 15)\).

Dequantized: \(\hat x = 0.2467\,(q - 5) = (-1.233,\ -0.247,\ 0.000,\ 0.493,\ 2.467)\). The errors \(x - \hat x\) are \((0.033,\ -0.053,\ 0,\ -0.093,\ 0.033)\). All are below \(s/2 = 0.123\), and 0 is exact.

The same numbers in INT8 (codes 0 to 255): \(s = 3.7/255 = 0.01451\), \(z = 83\), \(q = (0, 62, 83, 111, 255)\), largest error \(0.0063\). Four more bits make the step 17 times smaller (\(255/15 = 17\)).

Note one detail: because \(z\) was rounded, the grid covers \([-1.233, 2.467]\), not exactly \([-1.2, 2.5]\). The end value 2.5 is now slightly outside and is clipped to 2.467. This is normal and the error is less than one step.

Figure 1 runs this recipe step by step, with the formulas filled in by the real numbers at every step. The default input is the worked example above.

4
    Figure 1. Formulas (103.1) to (103.4) run live on a few numbers. Blue dots are the inputs \(x_i\). After step 3 the integer grid appears: one tick per code \(q\), drawn at its real value \(s(q - z)\); the green tick is the zero point. Each value is then rounded to its code (orange diamond, \(\hat x_i\)); the red segment is the error \(x_i - \hat x_i\). The table fills in as the steps run.
    How to read Figure 1

    Press Play, or Step one line at a time. The box on the left is the program; the highlighted line is the one running, and the panel on the right shows its formula with the numbers put in. With the defaults you get \(s = 0.2467\), \(z = 5\) and codes \((0, 4, 5, 7, 15)\), as in the worked example; the mean squared error is 0.00276. Switch to Symmetric: \(z = 0\), \(s = 0.3571\), and the MSE rises to 0.00433. Choose “One outlier” (seven small values and 4.1): in symmetric INT4 the step is 0.586 and five of the seven small values become code 0. Raise the bits to 8 and every error shrinks below \(s/2\) again.

    Symmetric quantization

    If we force \(z = 0\), the grid is centered on 0. We use signed codes and usually drop the most negative one, so the grid is the same on both sides (restricted range: \(-127 \dots 127\) for INT8, \(-7 \dots 7\) for INT4):

    \[ s = \frac{\max_i |x_i|}{2^{b-1} - 1}, \qquad z = 0, \qquad q \in \{-(2^{b-1} - 1), \dots, 2^{b-1} - 1\} \tag{103.4} \]

    For the same five numbers in 4 bits: \(s = 2.5/7 = 0.357\), \(q = (-3, -1, 0, 1, 7)\), \(\hat x = (-1.071, -0.357, 0, 0.357, 2.5)\), largest error 0.129. That is worse than asymmetric (0.093), because the codes \(-7 \dots -4\) cover \([-2.5, -1.43]\), where there is no data. Four of the 15 codes are wasted.

    So why is symmetric so common? Because it makes the integer matrix multiply cheaper, as the next section shows. Weights of trained networks are roughly symmetric around 0, so for weights little is lost. Activations after ReLU or GELU, or after softmax, are skewed, and asymmetric helps there. [Likely]

    Figure 2 lets you move one value freely. Move \(x\) outside the shaded band to see clipping, and switch to symmetric to see the wasted codes.

    Figure 2. Formulas (103.1) to (103.4) on a number line. Each tick is one code; the number above it is the code \(q\), and the tick is drawn at its real value \(s(q - z)\). The red dot is \(x\), the orange diamond is \(\hat x\).
    How to read Figure 2

    Start with the defaults: they are the worked example (\(x = 0.4\) gives \(q = 7\) and \(\hat x = 0.493\)). Raise the bits: the ticks get denser and the error shrinks by about half per bit. Move \(x\) past 2.5: the diamond stops at the last tick and the error grows without limit. That is clipping. Switch to symmetric: the green zero-point tick jumps to code 0 in the middle, and the readout counts the codes on the negative side that no value in \([\alpha, \beta]\) uses.

    Integer arithmetic: where the scales go

    Quantization saves memory just by storing codes. To also save compute, the matrix multiply itself must run on integers (on recent GPUs, INT8 tensor cores have about twice the peak throughput of FP16 ones [Likely]). Take one output of a linear layer, a dot product over \(K\) inputs, with the activation \(x\) and the weight \(w\) both quantized (formula 103.2):

    \[ y = \sum_{k=1}^{K} s_x (q^x_k - z_x)\; s_w (q^w_k - z_w) = s_x s_w \Big[ \sum_k q^x_k q^w_k \;-\; z_w \sum_k q^x_k \;-\; z_x \sum_k q^w_k \;+\; K z_x z_w \Big] \tag{103.5} \]
    Read the symbols
    \(s_x, z_x\)
    scale and zero point of the activations
    \(s_w, z_w\)
    scale and zero point of the weights
    \(\sum_k q^x_k q^w_k\)
    the expensive part: an integer dot product, accumulated in INT32

    Look at the four terms inside the bracket. The first is the real work. The third and fourth depend only on the weights and can be computed once, offline. The second, \(z_w \sum_k q^x_k\), depends on the input, so it costs extra work for every input. That is why weights are almost always quantized symmetrically (\(z_w = 0\)): the second term disappears. Activations can stay asymmetric at a small cost. [Certain for the algebra; the common practice is described in Jacob et al. and Nagel et al.]

    After the integer sum, the INT32 result must be turned into the next layer’s INT8 codes. With output scale \(s_y\), this is one multiplication by \(M = s_x s_w / s_y\), a real number. Jacob et al. found \(M\) is usually between 0 and 1 and store it as a fixed-point integer times a power of 2, so the whole layer runs without floating point. [Likely]

    Which scales can be pulled out of the sum?

    Formula (103.5) has one \(s_x\) and one \(s_w\) for the whole dot product. A whole matrix multiply \(Y = XW\) (\(X\) is tokens × input channels, \(W\) is input × output channels) can still have many scales, as long as each one is constant along the summed index \(k\):

    \[ Y_{ij} = \sum_k s^x_i\, q^x_{ik}\; s^w_j\, q^w_{kj} = s^x_i\, s^w_j \sum_k q^x_{ik}\, q^w_{kj} \tag{103.6} \]

    Here \(s^x_i\) is one scale per token (row of \(X\)) and \(s^w_j\) is one scale per output channel (column of \(W\)). Both come out of the sum, so the integer GEMM runs unchanged and each output is rescaled once at the end. A scale per input channel \(k\) would sit inside the sum and break the integer GEMM. This one formula explains the standard choices for W8A8: per-channel weights, per-token activations. It also explains why activation outliers, which live in input channels, are so painful (see below).

    Granularity: which numbers share a scale

    The granularity is the size of the block that shares one \((s, z)\). Smaller blocks fit the local range better, so the error is lower, but more scales must be stored and applied.

    Figure 3. Four granularities. Cells with the same color and shade share one scale. Per-group splits each output row into pieces of \(g\) input values (\(g = 4\) in the drawing, 128 in practice).
    • Per-tensor: one scale for the whole matrix. Almost free to store. One large value anywhere sets the step for everything.
    • Per-channel (weights): one scale per output channel. Rows of a weight matrix often differ in size by several times, so this is a large gain. It fits formula (103.6), so it costs nothing in the integer GEMM. Standard for INT8 weights since Jacob et al. and Krishnamoorthi. [Likely]
    • Per-group (also “group-wise” or “block-wise”): each output row is cut into groups of \(g\) consecutive input values, each with its own scale (and zero point if asymmetric). The standard for 4-bit LLM weights, usually \(g = 128\): GPTQ and AWQ report results as “INT4-g128”. [Likely] ZeroQuant combined group-wise weights with token-wise activations. [Certain, survey Section 5.2.1] llama.cpp uses blocks of 32. [Likely]
    • Per-token (activations): one scale per row of \(X\). Computed on the fly, because activations change with every input (dynamic quantization), unlike weight scales, which are fixed offline (static).

    Per-group scales vary along the summed index \(k\), so they do not come out of the sum in (103.6). Kernels for 4-bit weights (W4A16) therefore dequantize each group to FP16 inside the kernel and multiply in FP16. That is fine for decoding, where the cost is reading memory, not math (session 104).

    Figure 4 runs symmetric quantization block by block on a small 4 × 8 weight matrix. Row 2 holds one outlier weight, 3.70; every other weight is below 1.2 in size. For each block the program finds the largest size, sets the scale from it, and rounds the block. Watch where the outlier’s large step spreads.

    1. split \(W\) into blocks // tensor: 1, channel: 1 per row, group: \(g\) values of a row
    2. for each block \(B\):
    3. \(m \leftarrow \max_{w \in B} |w|\)
    4. \(s_B \leftarrow m/(2^{b-1} - 1)\) // (103.4)
    5. for each \(w \in B\): \(q \leftarrow \operatorname{clamp}(\operatorname{round}(w/s_B),\ -q_{\max},\ q_{\max})\) // (103.1)
    6. \(\hat w \leftarrow s_B\, q;\ \ \sum e^2 \leftarrow \sum e^2 + (w - \hat w)^2\) // (103.2)
    7. \(\text{SQNR} \leftarrow 10 \log_{10}\big(\sum w^2 / \sum e^2\big)\)
    Figure 4. Granularity, computed live on a 4 × 8 matrix (rows = output channels). In each cell: the weight \(w\) on top and, once its block is done, its code \(q\) below. Thick outline: the block being processed; orange ring: its largest weight, which sets the scale. Red shading: the error \(|w - \hat w|\), on a fixed color scale (the largest per-tensor error at the chosen bits). Bottom: SQNR for each granularity; the bar of the chosen one grows as its blocks are done.
    How to read Figure 4

    Press Play with the defaults (per-tensor, INT4). There is one block, its maximum is the outlier, so \(s = 3.70/7 = 0.5286\). Most weights are smaller than one step: 9 of the 32 become code 0, and the whole of row 0 gets only codes −1 and 0. The SQNR is 14.16 dB. Per-channel gives each row its own scale (row 0 gets \(s = 0.0443\)) and reaches 19.22 dB, but row 2 is still ruled by the outlier. Per-group 4 confines the outlier to its own group of four: 23.46 dB. Per-group 2: 26.15 dB. Untick the outlier and per-tensor INT4 jumps to 21.17 dB. SQNR is defined in the next section; higher is better. The scale cost on this toy is large (16 bits per block: 0.5, 2, 4 and 8 bits per weight); on a real 4096-wide layer it is small, as the next part shows.

    The storage cost of scales

    Each group stores a scale (often FP16) and, if asymmetric, a zero point. The effective bits per weight are

    \[ b_{\text{eff}} = b + \frac{b_s + b_z}{g} \tag{103.7} \]
    Read the symbols
    \(b\)
    bits per code (for example 4)
    \(b_s, b_z\)
    bits for the scale and the zero point of one group (\(b_z = 0\) if symmetric)
    \(g\)
    number of weights that share them
    Worked example: what group size 128 costs

    INT4, \(g = 128\), FP16 scale, no zero point: \(4 + 16/128 = 4.125\) bits. With an FP16 zero point too: \(4 + 32/128 = 4.25\) bits. With a 4-bit zero point (as many GPTQ packers store it): \(4 + 20/128 = 4.156\) bits. Blocks of 32 with an FP16 scale: \(4 + 16/32 = 4.5\) bits.

    For the 8.03B weights of Llama 3 8B at 4.125 bits: \(8.03 \times 10^9 \times 4.125 / 8 = 4.14\) GB, against 16.06 GB in BF16 (3.9× smaller). In practice the embedding and output layers are often kept in higher precision, so real files are somewhat larger. [Likely]

    Per-channel scales are almost free on a real layer: one FP16 scale per row of 4096 inputs is \(16/4096 = 0.004\) bits per weight.

    Figure 5 shows the trade on a bigger weight matrix: 16 output rows × 64 inputs. The rows have very different sizes (like real weight matrices), and four single weights are 5 times larger than their row (weight outliers).

    Figure 5. Symmetric quantization of a 16 × 64 weight matrix (seeded, deterministic). Top: the size of each weight. Bottom: the size of each error \(|W - \hat W|\), on a fixed color scale (the largest per-tensor error at the chosen bits). Black lines mark blocks that share one scale.
    How to read Figure 5

    At INT4 per-tensor, the bottom map is dark everywhere and the SQNR (signal-to-quantization-noise ratio, explained in the next section; higher is better) is 11.82 dB. Per-channel gives each row its own step: small rows become almost error-free and the SQNR rises to 19.19 dB. Per-group 16 goes to 21.25 dB, and the four outliers now spoil only their own group of 16, not a whole row. But read the storage line: on this tiny matrix, per-channel already costs 0.25 bits per weight and group 8 costs 2 bits. On real layers (4,096 inputs, groups of 128) the costs are 0.004 and 0.125 bits.

    Two kinds of error: rounding and clipping

    Every quantized value is wrong in one of two ways.

    • Rounding error: \(x\) is inside the range and moves to the nearest grid point. The error is at most \(s/2\). If \(x\) is spread smoothly over many steps, the error is close to uniform on \([-s/2, s/2]\), with mean square \(s^2/12\).
    • Clipping error: \(x\) is outside the range and is clamped to the end \(c\). The error is \(|x| - c\), which can be huge.

    For a symmetric grid clipped at \(\pm c\), with \(s = c / (2^{b-1} - 1)\), the mean squared error is the sum of the two parts:

    \[ \operatorname{MSE}(c) \approx \underbrace{\frac{s^2}{12}\, P(|x| \le c)}_{\text{rounding: grows with } c} \;+\; \underbrace{\E\big[(|x| - c)^2 \,;\, |x| \gt c\big]}_{\text{clipping: shrinks with } c} \tag{103.8} \]

    A small \(c\) gives a fine grid but clips many values; a large \(c\) clips nothing but makes the step large. The best \(c\) is in between, and with fewer bits it moves lower (each step matters more). Banner et al. computed the best \(c\) in closed form for Gaussian and Laplace data. [Certain that they did; the values below are my own computation]

    The 6 dB per bit rule

    Engineers measure quality with the signal-to-quantization-noise ratio, \(\text{SQNR} = 10 \log_{10}\big(\E[x^2] / \operatorname{MSE}\big)\) in decibels (dB). Higher is better; +10 dB means 10 times less noise power. If there is no clipping, \(s \approx 2c / 2^b\), so \(\operatorname{MSE} \approx c^2 / (3 \cdot 4^b)\), and for data with standard deviation \(\sigma\):

    \[ \text{SQNR} \approx 6.02\, b \;+\; 4.77 \;-\; 20 \log_{10}\frac{c}{\sigma} \quad \text{dB} \tag{103.9} \]

    Two lessons. Each extra bit buys 6.02 dB (\(20 \log_{10} 2\)), a factor 4 in noise power. And a range \(c\) that is 2 times wider than needed costs 6.02 dB, exactly one bit. A range 20 times wider costs 26 dB, more than four bits. Outliers make \(c / \sigma\) large, and formula (103.9) shows the bill.

    Worked example: Gaussian data in 4 bits

    For unit Gaussian data and symmetric INT4 (codes \(-7 \dots 7\)), I computed (103.8) exactly: the best clip is \(c = 2.47\sigma\), with SQNR 18.90 dB. In Figure 6 below, the 4,096 Gaussian samples have \(\max|x| = 3.58\sigma\); min-max gives 16.55 dB, and the best \(c\) on the slider grid is 71% of the max, \(2.54\sigma\), with 18.87 dB. The MSE search in the PyTorch code below, on a million samples, also finds 18.90 dB.

    Check formula (103.9) at INT8, min-max, \(c = 3.58\sigma\): \(48.16 + 4.77 - 11.08 = 41.86\) dB. The figure measures 41.87 dB.

    Choosing the range

    Choosing \(c\) (or \(\alpha, \beta\)) is called calibration. For weights we see all values; for activations we run a few hundred example inputs (the calibration set) and record statistics. Three common rules:

    1. Min-max: \(c = \max|x|\). No clipping at all, but one outlier sets the step for everyone. Good at 8 bits with no outliers.
    2. Percentile: \(c\) = the 99.9th (or 99.99th) percentile of \(|x|\). Clips the rare largest values. Cheap and robust, but the right percentile depends on the bits and the data.
    3. MSE search: try many \(c\) and keep the one with the smallest error:
    \[ c^\star = \argmin_{c} \; \sum_i \big( x_i - Q_c(x_i) \big)^2 \tag{103.10} \]

    where \(Q_c\) quantizes and dequantizes with clip \(c\). A grid of 50 to 100 values of \(c\) is enough. Variants minimize the error of the layer output \(\lVert XW - X\hat W \rVert\) instead of the weights, which leads to GPTQ and AWQ (session 106, session 107). OmniQuant even learns the clipping range with gradient descent. [Likely]

    rounding MSEclipping MSEtotal MSEbest \(c\)
    Figure 6. Formula (103.8) measured on 4,096 seeded samples with unit variance, symmetric quantization. Top: the data and the clip lines \(\pm c\). Bottom: the two error parts and their sum as \(c\) moves from 2% to 100% of \(\max|x|\).
    How to read Figure 6

    The red rounding curve rises with \(c\) (bigger steps); the orange clipping curve falls to zero at min-max. The total has a valley; the green dot marks its bottom. At INT4 Gaussian, min-max gives 16.55 dB and the best clip 18.87 dB. At INT8 Gaussian the best is min-max itself: with 255 codes, steps are tiny and no clipping pays. Now pick “Gaussian + 0.5% outliers” at INT4: min-max falls to 5.81 dB, the 99.9th percentile gives 8.98 dB, and the best clip (41% of max) 10.95 dB, still far below the clean Gaussian. Clipping helps, but it cannot fully fix outliers. Laplace data sits in between.

    Algorithm 1: quantize a tensor with a chosen range (asymmetric or symmetric)
    1. input: tensor \(x\), bits \(b\), mode, range rule (min-max, percentile \(p\), or MSE)
    2. if rule = min-max: \(\alpha \leftarrow \min(\min x, 0)\), \(\beta \leftarrow \max(\max x, 0)\)
    3. if rule = percentile: \(\alpha, \beta \leftarrow\) the \(p\) and \(1 - p\) quantiles of \(x\) // clip the rare extremes
    4. if mode = symmetric:
    5. \(s \leftarrow \max(|\alpha|, |\beta|) / (2^{b-1} - 1)\); \(z \leftarrow 0\); \(q_{\min}, q_{\max} \leftarrow -(2^{b-1}-1),\ 2^{b-1}-1\) // (103.4)
    6. else:
    7. \(s \leftarrow (\beta - \alpha)/(2^b - 1)\); \(z \leftarrow \operatorname{round}(-\alpha/s)\); \(q_{\min}, q_{\max} \leftarrow 0,\ 2^b - 1\) // (103.3); \(z\) is an integer
    8. \(q \leftarrow \operatorname{clamp}(\operatorname{round}(x/s) + z,\ q_{\min},\ q_{\max})\) // (103.1), store \(q\) in \(b\) bits
    9. return \(q, s, z\) // later: \(\hat x = s(q - z)\), formula (103.2)
    Algorithm 2: per-group INT4 weights with an MSE-searched clip (round-to-nearest, “RTN”)
    1. input: weight \(W\) of shape (out, in), bits \(b = 4\), group size \(g = 128\), clip grid \(\{0.50, 0.51, \dots, 1.00\}\)
    2. reshape \(W\) to shape \((\text{out} \cdot \text{in} / g,\ g)\) // one row per group
    3. for each group row \(\mathbf{w}\):
    4. best \(\leftarrow \infty\)
    5. for each \(a\) in the clip grid:
    6. \(s \leftarrow a \cdot \max|\mathbf{w}| / 7\); \(\hat{\mathbf{w}} \leftarrow s \cdot \operatorname{clamp}(\operatorname{round}(\mathbf{w}/s), -7, 7)\)
    7. if \(\lVert \mathbf{w} - \hat{\mathbf{w}} \rVert^2 \lt\) best: best \(\leftarrow\) that error; keep \(s\) // (103.10)
    8. store the 4-bit codes and one FP16 scale for this group // 4.125 bits per weight, (103.7)
    9. return codes \((\text{out}, \text{in})\) and scales \((\text{out}, \text{in} / g)\)

    Algorithm 2 is the baseline that every method in this track is compared with. It is called round-to-nearest (RTN), because each weight is simply rounded on its own. GPTQ and AWQ keep the same storage format and only choose the codes and scales more cleverly.

    Why LLMs are hard: outliers

    For CNNs such as ResNet, INT8 with per-channel weights is nearly lossless, and the methods above were enough. [Likely] Large language models broke this. Dettmers et al. (LLM.int8()) measured the hidden states of transformers and found emergent outlier features:

    • Features with magnitudes up to 20 times larger than in other dimensions first appear in about 25% of the layers. At around 6.7B parameters there is a “phase shift”: all layers and 75% of the sequence positions are affected. [Certain, LLM.int8() Section 1]
    • They are very systematic: at 6.7B, about 150,000 outliers per sequence, but in only 6 feature dimensions across the whole transformer. [Certain]
    • They matter: setting these dimensions to zero degrades validation perplexity by 600 to 1000%, while removing the same number of random features changes it by about 0.1%. [Certain] So we cannot just clip them.

    SmoothQuant summarizes the practical result: “weights are easy to quantize while activations are not”. [Certain, SmoothQuant abstract] Weights are roughly bell-shaped with a few mild outliers; activations have a few channels that are huge in every token.

    Now combine this with formula (103.6). The outliers live in a few input channels \(k\) of \(X\). A per-token scale is set by the largest value in the token, which is the outlier channel, so every other channel of that token gets a coarse grid. A per-channel scale for \(X\) would fix it, but it sits inside the sum and breaks the integer GEMM. Figure 7 shows one token.

    Figure 7. One token’s activation vector: 255 channels of seeded Gaussian values (largest about 3.0) and one outlier channel. Symmetric per-token quantization: one scale for the whole token, set by its largest value.
    How to read Figure 7

    With the outlier at 60, INT8 has step 0.472, and the 255 normal channels use only 12 of the 255 codes: about 3.6 bits of real resolution, SQNR 17.11 dB. Drag the outlier down to 1 and the same channels use 129 codes and reach 43.31 dB. Switch to INT4 with the outlier at 60: the step is 8.57, every normal value rounds to 0, and the SQNR is 0 dB, so the token carries no information except the outlier. Tick the checkbox: the outlier goes to a separate 16-bit path and the normal channels are back at full quality. That is the mixed-precision decomposition of LLM.int8(), covered in session 108.

    There are three families of fixes, and each has its own session:

    • Keep outliers apart: run the few outlier channels in FP16 (LLM.int8()), or keep outlier weights in high precision (SpQR, OWQ).
    • Move the difficulty: divide each activation channel by a factor and multiply the matching weight row by it, so the output is unchanged (SmoothQuant, AWQ). The activations become flat and the weights absorb a little extra range, which weights handle well.
    • Rotate: multiply by an orthogonal matrix that spreads one outlier over all channels (QuaRot, SpinQuant; session 109).

    The KV cache has its own pattern: keys have outlier channels, values do not. So KIVI quantizes keys per channel and values per token. [Certain, survey Section 5.2.1] Details are in session 110.

    Code: quantization from scratch in PyTorch

    This code implements formulas (103.1) to (103.4) with any granularity, and reproduces the numbers on this page. It runs on a CPU in a few seconds and is deterministic.

    import torch
    torch.manual_seed(0)
    
    def fake_quant(x, bits=8, symmetric=True, dim=None, group=None, clip=1.0):
        """Quantize then dequantize x. dim: axis that keeps its own scale
        (None = per-tensor). group: split the last axis into groups of this size."""
        shape = x.shape
        if group is not None:                       # per-group: one scale per `group` values
            x = x.reshape(-1, group); dim = 0
        red = [d for d in range(x.ndim) if d != dim] if dim is not None else list(range(x.ndim))
        if symmetric:                               # formula (103.4)
            qmax = 2 ** (bits - 1) - 1
            s = clip * x.abs().amax(dim=red, keepdim=True) / qmax
            q = torch.clamp(torch.round(x / s), -qmax, qmax)
            xh = s * q
        else:                                       # formula (103.3), codes 0 .. 2^b - 1
            lo = clip * x.amin(dim=red, keepdim=True).clamp(max=0)
            hi = clip * x.amax(dim=red, keepdim=True).clamp(min=0)
            s = (hi - lo) / (2 ** bits - 1)
            z = torch.round(-lo / s)
            q = torch.clamp(torch.round(x / s) + z, 0, 2 ** bits - 1)
            xh = s * (q - z)                        # formula (103.2)
        return xh.reshape(shape)
    
    def sqnr_db(x, xh):
        return (10 * torch.log10(x.pow(2).mean() / (x - xh).pow(2).mean())).item()
    
    # 1) the worked example: asymmetric vs symmetric, 4 bits
    x = torch.tensor([-1.2, -0.3, 0.0, 0.4, 2.5])
    print(fake_quant(x, 4, symmetric=False))   # [-1.2333, -0.2467, 0.0000, 0.4933, 2.4667]
    print(fake_quant(x, 4, symmetric=True))    # [-1.0714, -0.3571, 0.0000, 0.3571, 2.5000]
    
    # 2) granularity on a weight matrix whose rows have very different sizes
    W = torch.randn(1024, 4096) * torch.exp(torch.empty(1024, 1).uniform_(-1.5, 1.0))
    for name, kw in [("per-tensor", {}), ("per-channel", {"dim": 0}), ("per-group 128", {"group": 128})]:
        print(f"INT4 {name:14s} SQNR {sqnr_db(W, fake_quant(W, 4, **kw)):5.2f} dB")
    # INT4 per-tensor     SQNR  8.22 dB
    # INT4 per-channel    SQNR 16.05 dB
    # INT4 per-group 128  SQNR 18.61 dB
    
    # 3) choose the clipping range by MSE (formula 103.10) for 4-bit Gaussian data
    g = torch.randn(1_000_000)
    best = min((((g - fake_quant(g, 4, clip=a)) ** 2).mean().item(), a)
               for a in [i / 100 for i in range(20, 101)])
    print(f"min-max SQNR {sqnr_db(g, fake_quant(g, 4)):.2f} dB, best clip {best[1]:.2f} x max, "
          f"SQNR {sqnr_db(g, fake_quant(g, 4, clip=best[1])):.2f} dB")
    # min-max SQNR 13.60 dB, best clip 0.49 x max, SQNR 18.90 dB
    
    # 4) one outlier channel in the activations (16 tokens x 512 channels)
    X = torch.randn(16, 512); X[:, 37] = 60.0
    keep = torch.arange(512) != 37                 # the 511 normal channels
    for name, kw in [("per-tensor", {}), ("per-token", {"dim": 0}), ("per-channel", {"dim": 1})]:
        Xh = fake_quant(X, 8, **kw)
        print(f"INT8 {name:11s} SQNR on normal channels {sqnr_db(X[:, keep], Xh[:, keep]):5.2f} dB")
    Xh = fake_quant(X[:, keep], 8, dim=0)          # outlier channel kept in 16-bit
    print(f"INT8 per-token, outlier channel in FP16  {sqnr_db(X[:, keep], Xh):5.2f} dB")
    # INT8 per-tensor  SQNR on normal channels 17.33 dB
    # INT8 per-token   SQNR on normal channels 17.33 dB   (no help: every token has the outlier)
    # INT8 per-channel SQNR on normal channels 46.65 dB   (fixes it, but not GEMM-friendly)
    # INT8 per-token, outlier channel in FP16  42.80 dB

    Read the results together. On a 4096-wide matrix, per-channel scales gain 7.8 dB over per-tensor (more than one bit) and groups of 128 add 2.6 dB more. The MSE-chosen clip gains 5.3 dB over min-max at 4 bits on a million Gaussian samples (where \(\max|x|\) is about \(5\sigma\), so min-max wastes a lot). And the last block is the whole outlier story in four lines: per-token does nothing against a channel outlier, per-channel would fix it but does not fit the GEMM, and splitting off the one channel recovers almost everything.

    Where this is used

    SettingTypical schemeWhy
    LLM weights for local or batch-1 serving (GPTQ, AWQ, llama.cpp)INT4 symmetric or asymmetric, per-group 128 (llama.cpp: blocks of 32), activations stay FP16 (W4A16)Decode is memory-bound; about 4.1 to 4.5 bits per weight, formula (103.7) [Likely for the llama.cpp block size]
    High-throughput LLM serving (W8A8)INT8 weights per-channel symmetric, activations per-token (dynamic), with SmoothQuantBoth fit formula (103.6), so INT8 tensor cores run the GEMM
    KV cache at long contextKeys per-channel, values per-token, 2 to 4 bits (KIVI)Keys have outlier channels; values do not [Certain, survey 5.2.1]
    QLoRA fine-tuning4-bit NF4 base weights, blocks of 64 with a scale per block, scales quantized againBase weights frozen; only memory matters (session 58) [Likely for the block size]
    Mobile vision models (classification, detection)INT8, per-channel weights, asymmetric per-tensor activations, integer-only kernelsThe original setting of Jacob et al.: CPUs and NPUs without fast floats
    Vision encoders of VLMs and video models (action recognition, video classification)W8A8 or W4A16 on the ViT; per-token activationsA video has many frames, so the encoder runs many times; ViT activations also show outlier channels [Likely]
    Video retrieval, Video-RAG and fingerprinting indexesINT8 scalar quantization of embedding vectors, a scale (and offset) per dimension4× smaller index than FP32 with little recall loss; similarity can run on integers [Likely]

    Common mistakes

    • Mixing up the denominators. Asymmetric uses all \(2^b\) codes: \(s = (\beta - \alpha)/(2^b - 1)\). Symmetric restricted range uses \(s = \max|x| / (2^{b-1} - 1)\). Using the wrong one shifts every value.
    • A real-valued zero point. If \(z\) is not rounded to an integer, real 0 is no longer exact, and padding or masked values become small non-zero numbers.
    • Per-channel on the wrong axis. For nn.Linear, the weight is (out, in): one scale per row (output channel), not per column. A scale per input channel cannot leave the sum in (103.6).
    • Forgetting the cost of scales. “4-bit” models with group 128 and FP16 scale and zero are really 4.25 bits; with small groups the overhead becomes large (group 32 with scale and zero: 5 bits).
    • Calibrating on the wrong data. Activation ranges from random noise or from a different domain (code vs chat, images vs video frames) give wrong clips. Use a few hundred real, representative samples.
    • Clipping activation outliers away. In LLMs the outlier channels carry real information: zeroing them raises perplexity by 600 to 1000% (LLM.int8()). Clip weights carefully; do not simply clip activation outliers.
    • Judging only by average error. A low MSE on weights does not guarantee a good model. Measure the layer output error and, in the end, perplexity and task accuracy (session 40).
    • Division by zero. An all-zero group gives \(s = 0\). Real code adds a tiny minimum, for example s.clamp(min=1e-8).

    Interview questions

    Write the quantize and dequantize formulas. What are the scale and the zero point?basic
    \(q = \operatorname{clamp}(\operatorname{round}(x/s) + z, q_{\min}, q_{\max})\) and \(\hat x = s(q - z)\). The scale \(s\) is the spacing between grid points (range divided by number of steps). The zero point \(z\) is the integer code that represents real 0. For asymmetric min-max: \(s = (\max - \min)/(2^b - 1)\), \(z = \operatorname{round}(-\min/s)\). For symmetric: \(z = 0\), \(s = \max|x| / (2^{b-1}-1)\).
    Why is the zero point an integer?basic
    So that real 0 maps to a code exactly: \(\hat x = s(z - z) = 0\). Zero appears constantly (zero padding, ReLU outputs, masks), and an exact zero avoids a systematic bias. It also keeps all the arithmetic in (103.5) on integers.
    Symmetric or asymmetric: which for weights, which for activations, and why?core
    Weights: symmetric, because they are roughly centered on 0 and because a weight zero point adds the input-dependent term \(z_w \sum_k q^x_k\) to every dot product (formula 103.5). Activations: asymmetric is useful when they are skewed (after ReLU or GELU, after softmax), because symmetric would waste the codes on the empty side; its extra term depends only on the weights and can be precomputed. Many LLM W8A8 recipes still use symmetric for both, for simplicity.
    Why per-channel scales for weights but per-token scales for activations?core
    In \(Y_{ij} = \sum_k X_{ik} W_{kj}\), a scale can be pulled out of the sum only if it does not depend on \(k\). One scale per output channel \(j\) of \(W\) and one per token \(i\) of \(X\) both qualify, so \(Y_{ij} = s^x_i s^w_j \sum_k q^x_{ik} q^w_{kj}\) and the integer GEMM is unchanged. A scale per input channel \(k\) (which is what activation outliers would need) varies inside the sum and breaks the integer GEMM.
    What does group size 128 mean, and what does it cost?core
    Each row of the weight matrix is cut into pieces of 128 consecutive input weights, and each piece gets its own scale (and zero point if asymmetric). Cost in storage: \(16/128 = 0.125\) extra bits per weight for an FP16 scale, 0.25 with an FP16 zero point too, so INT4-g128 is about 4.13 to 4.25 bits. Cost in compute: the scale varies along the summed dimension, so kernels must dequantize inside the GEMM (fine for memory-bound decode). Gain: an outlier only hurts its own group, and the step fits the local range.
    Explain rounding error vs clipping error and how you pick the range.core
    Inside the range, each value moves to the nearest grid point: error at most \(s/2\), mean square about \(s^2/12\). Outside, values are clamped: error \(|x| - c\), unbounded. A wider range removes clipping but makes \(s\) larger. The best range balances the two (formula 103.8). Rules: min-max (no clipping, fine at 8 bits without outliers), percentile (cheap), MSE search over a grid of \(c\) (best for low bits), or minimize the layer output error on calibration data.
    Derive the “6 dB per bit” rule. What does a range twice too wide cost?advanced
    Without clipping, \(s \approx 2c/2^b\), so the noise power is \(s^2/12 = c^2/(3 \cdot 4^b)\). SQNR \(= 10\log_{10}(\sigma^2 \cdot 3 \cdot 4^b / c^2) = 6.02b + 4.77 - 20\log_{10}(c/\sigma)\) dB. One more bit: +6.02 dB. A range twice as wide: \(-20\log_{10} 2 = -6.02\) dB, exactly one bit lost. An outlier 20× above the normal range costs about 26 dB, more than four bits.
    Why are LLMs harder to quantize than CNNs or small transformers?core
    Activations of large transformers have systematic outlier features: per LLM.int8(), magnitudes up to 20× the other dimensions, spreading to all layers at about 6.7B parameters, concentrated in about 6 hidden dimensions. With a per-tensor or per-token scale, that one channel sets the step, so all other channels get only a few codes (formula 103.9). And they cannot be clipped: zeroing them raises perplexity by 600–1000%. Weights are much better behaved, hence weight-only quantization is easy and W8A8 needs tricks such as SmoothQuant.
    Does per-token quantization fix activation outliers?advanced
    No, not the channel-wise kind. The outlier is in the same channel in every token, so every token’s maximum is the outlier, and every token gets the same coarse step. In the code above, per-token gives 17.33 dB on the normal channels, the same as per-tensor; per-channel would give 46.65 dB but is not GEMM-friendly. Fixes: split off the outlier channels in FP16 (LLM.int8()), migrate the scale into the weights (SmoothQuant), or rotate (QuaRot).
    Walk through an INT8 linear layer with zero points. What runs in integers and what does not?advanced
    Expand \(\sum_k s_x(q^x_k - z_x)\,s_w(q^w_k - z_w)\) into \(s_x s_w[\sum q^x q^w - z_w\sum q^x - z_x\sum q^w + K z_x z_w]\). The main sum runs on INT8 inputs with INT32 accumulation. \(\sum q^w\) and \(K z_x z_w\) are precomputed per output channel; \(z_w \sum q^x\) needs a per-input row sum (avoided with \(z_w = 0\)). The bias is stored as INT32 with scale \(s_x s_w\). Finally the INT32 result is multiplied by \(M = s_x s_w/s_y\) (as fixed point) and rounded to the next layer’s codes.
    How much memory do the weights of an 8B model need in BF16, INT8 and INT4-g128?basic
    With 8.03B weights: BF16 \(8.03\times10^9 \times 2\) B = 16.06 GB; INT8 per-channel ≈ 8.03 GB (scales are negligible); INT4-g128 with FP16 scales \(8.03\times10^9 \times 4.125/8\) ≈ 4.14 GB. Real checkpoints are a bit larger if embeddings and the output head stay in 16 bits. Add the KV cache and activations on top.

    Exercises

    1. Quantize \(x = (-0.5, 0.1, 0.9, 3.0)\) to unsigned INT4 with asymmetric min-max. Give \(s\), \(z\), the codes and \(\hat x\).
    2. Same numbers, symmetric INT4 (codes \(-7 \dots 7\)). How many codes are never used by any value in \([-0.5, 3.0]\)?
    3. A weight matrix is quantized to INT3 with group size 64, an FP16 scale and an FP16 zero point. What are the effective bits per weight? What group size gives exactly 3.25 bits with a scale only?
    4. Using formula (103.9), estimate the SQNR of INT8 min-max on data with \(\sigma = 1\) and one outlier at 60. Compare with Figure 7.
    5. Why can a per-group scale not be pulled out of the integer GEMM, while a per-output-channel scale can? Write the sum.
    6. In Figure 6, at INT4 with “Gaussian + 0.5% outliers”, which rule wins: min-max, the 99.9th percentile or the best MSE? By how many dB?
    Answerscheck
    1. \(s = 3.5/15 = 0.2333\); \(-\alpha/s = 2.14\), so \(z = 2\). \(x/s + z = (-0.14, 2.43, 5.86, 14.86)\), so \(q = (0, 2, 6, 15)\) and \(\hat x = 0.2333(q - 2) = (-0.467, 0, 0.933, 3.033)\). Note that 0.1 became 0 and 3.0 became 3.033 (the grid ends at 3.033, because \(z\) was rounded). 2. \(s = 3/7 = 0.4286\). The codes \(-7 \dots -2\) are at \(-3.0 \dots -0.857\), all below \(-0.5 - s/2 = -0.714\): 6 of 15 codes are unused. 3. \(3 + 32/64 = 3.5\) bits. For \(3 + 16/g = 3.25\): \(g = 64\). 4. \(c/\sigma = 60\): \(48.16 + 4.77 - 35.56 = 17.37\) dB. Figure 7 measures 17.11 dB (the formula assumes the values fill the range evenly; here they are Gaussian). 5. Per output channel: \(Y_{ij} = \sum_k s^x_i q^x_{ik} s^w_j q^w_{kj} = s^x_i s^w_j \sum_k q^x_{ik} q^w_{kj}\), the scales do not depend on \(k\). Per group: \(s^w_{j, \lfloor k/g \rfloor}\) changes with \(k\), so \(Y_{ij} = s^x_i \sum_{\text{groups } G} s^w_{jG} \sum_{k \in G} q^x_{ik} q^w_{kj}\): one partial sum per group, each rescaled; kernels dequantize instead. 6. Best MSE (clip at 41% of max): 10.95 dB, against 8.98 dB for the 99.9th percentile (+1.97 dB) and 5.81 dB for min-max (+5.14 dB).

    Key takeaways

    • Uniform affine quantization stores a code \(q\) and rebuilds \(\hat x = s(q - z)\) (103.1, 103.2). The integer zero point makes real 0 exact.
    • Asymmetric (103.3) fits skewed data; symmetric (103.4) wastes codes on skewed data but removes the weight zero-point term in the integer GEMM (103.5), so weights are usually symmetric.
    • Scales that do not depend on the summed index come out of the GEMM (103.6): per-channel weights and per-token activations are free; per-group weights need dequantization inside the kernel and cost \((b_s + b_z)/g\) bits (103.7), 0.125 bits for an FP16 scale at \(g = 128\).
    • Error = rounding (grows with the range) + clipping (shrinks with it) (103.8). Each bit is worth 6.02 dB, and a range twice too wide wastes one bit (103.9). Choose the range by min-max, percentile or MSE search (103.10).
    • LLM activations have a few huge, systematic outlier channels (up to 20×, from about 6.7B parameters). With one scale per token they waste most codes, so W8A8 and W4A4 need the methods of the next sessions.

    Key formulas

    \[ q = \operatorname{clamp}\!\big(\operatorname{round}(x/s) + z,\, q_{\min},\, q_{\max}\big), \quad \hat x = s(q - z), \quad s_{\text{asym}} = \frac{\beta - \alpha}{2^b - 1}, \quad s_{\text{sym}} = \frac{\max|x|}{2^{b-1} - 1} \]
    \[ Y_{ij} = s^x_i s^w_j \sum_k q^x_{ik} q^w_{kj}, \quad b_{\text{eff}} = b + \frac{b_s + b_z}{g}, \quad \text{SQNR} \approx 6.02b + 4.77 - 20\log_{10}\frac{c}{\sigma} \]

    References for this session

    1. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
    2. Jacob et al. (2017). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
    3. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
    4. Jacob et al. (2017). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
    5. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
    6. Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
    7. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
    8. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
    9. Yao et al. (2022). ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
    10. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
    11. Nagel et al. (2021). A White Paper on Neural Network Quantization
    12. Banner et al. (2019). Post-training 4-bit quantization of convolution networks for rapid-deployment