Weight PTQ 1: from OBQ to GPTQ
Rounding every weight to the nearest 4-bit or 3-bit value is simple, but at 3 bits it breaks large models. GPTQ quantizes a layer one column at a time and, after each column, changes the weights that are not yet quantized so that the layer output stays almost the same. This session builds GPTQ step by step from its parent method OBQ, with the math, the speed tricks and working code.
What you will learn
- Why post-training quantization of LLM weights is solved one layer at a time, with the objective ‖WX − ŴX‖²
- What the Hessian H = 2XXᵀ is, why it depends only on the calibration inputs, and how it tells which weights matter
- The OBQ step: quantize one weight, then move the remaining weights to cancel its error (and how to derive it)
- Why OBQ costs O(d_row·d_col³) and how a fixed column order brings GPTQ to O(max(d_row·d_col², d_col³))
- Lazy batch updates, the Cholesky form of the inverse Hessian, and dampening
- Practical GPTQ: calibration data, groups, act-order, kernels, and what the paper actually measured
Why this matters
When an LLM generates text one token at a time, each step must read every weight from GPU memory. The math per weight is tiny, so the speed is set by how many bytes we read. Store the weights in 4 bits instead of 16, and we read 4 times fewer bytes. The hard part is doing this without making the model worse.
The simplest method is round-to-nearest (RTN): put each weight on the nearest point of the quantization grid from session 103. RTN works well at 8 bits. At 4 bits it starts to hurt, and at 3 bits it can destroy a model. The GPTQ paper (Frantar et al., 2022) shows this on OPT-175B: WikiText2 perplexity is 8.34 in FP16, 10.54 with 4-bit RTN, and about 7,300 with 3-bit RTN. GPTQ gets 8.37 at 4 bits and 8.68 at 3 bits, on the same grid. [Certain] It quantized the 175B model in about 4 hours on one A100 GPU, using only 128 short pieces of random web text. [Certain]
The difference is one idea. RTN rounds every weight on its own. GPTQ rounds the weights of a layer one column at a time, and after each column it changes the weights that are not yet rounded so that their change cancels the error just made. The layer output stays close to the original, even though many weights are rounded "the wrong way".
GPTQ is one of the most used methods to make 4-bit checkpoints of open models; GPTQ models load in Hugging Face Transformers and in serving engines such as vLLM. [Likely] This is the first of two sessions on weight-only post-training quantization (PTQ: quantize after training, with no retraining). The next one, session 107, adds AWQ, outliers and mixed precision.
The layer-wise problem: keep WX close to ŴX
Quantizing all weights of a large model together is a huge problem with no good solution. So PTQ methods cut it into small problems: one linear layer at a time. Take a linear layer with weight matrix \(W \in \R^{d_{row} \times d_{col}}\). It maps an input vector of size \(d_{col}\) to an output of size \(d_{row}\). Run a small set of calibration texts through the model and collect the inputs of this layer as the columns of a matrix \(X \in \R^{d_{col} \times n}\) (\(n\) = number of tokens). We want quantized weights \(\hat W\) whose outputs on these inputs are as close as possible to the original outputs:
- Read the symbols
- \(W\)
- the original weights of the layer (FP16), one row per output channel
- \(\hat W\)
- the quantized weights; every entry must be a point on the fixed quantization grid of its row
- \(X\)
- calibration inputs of this layer, one column per token
- \(\lVert \cdot \rVert_F^2\)
- sum of the squares of all entries (Frobenius norm squared)
Note what this objective cares about: not the weights, but the outputs. A weight that only meets small inputs can be rounded badly with little harm. Two weights whose inputs move together can make errors that cancel.
The objective splits into one independent problem per row. Write \(\mathbf{w}\) for one row of \(W\), \(\hat{\mathbf{w}}\) for its quantized version and \(\delta = \hat{\mathbf{w}} - \mathbf{w}\) for the change (all row vectors). Then
\(H\) is the Hessian (matrix of second derivatives) of the row error with respect to the weights. Because the error is exactly quadratic, formula (106.2) is not an approximation. Two facts make \(H\) very useful:
- \(H\) depends only on the inputs \(X\), not on the weights. All \(d_{row}\) rows of the layer share the same \(H\).
- \(H\) is small: \(d_{col} \times d_{col}\). For a layer with 4,096 inputs it is a 4,096 × 4,096 matrix, built once from the calibration data.
The Hessian tells which weights matter
Write formula (106.2) out entry by entry:
The two parts say two different things:
- Diagonal \(H_{jj}\): twice the energy of input channel \(j\). An input channel whose activations are 6 times larger has a 36 times larger \(H_{jj}\), so the same rounding error on its weight costs 36 times more. This is why channels with large activations ("outlier channels") are a big topic in LLM quantization (session 107).
- Off-diagonal \(H_{jk}\): how much inputs \(j\) and \(k\) move together. If they are positively correlated and we round \(w_j\) up, we can cancel most of the damage by moving \(w_k\) down. RTN never uses this. GPTQ is built on it.
If \(H\) were diagonal (all inputs uncorrelated), the best rounding would be RTN, because the error would be a sum of independent terms. Real LLM activations are strongly correlated, so there is a lot to gain.
One row with two weights \(\mathbf{w} = (0.6,\ 0.6)\). The grid is the integers. The two inputs have equal energy and correlation 0.9: \(XX^\top = \begin{pmatrix} 1 & 0.9 \\ 0.9 & 1 \end{pmatrix}\).
RTN rounds both up to \((1, 1)\). Then \(\delta = (0.4, 0.4)\) and \(E = 0.16 + 0.16 + 2 \cdot 0.9 \cdot 0.16 = 0.608\). Both errors push the output the same way.
Round one, then compensate. Round \(w_1\) to 1 (error \(+0.4\)). Formula (106.5) below says: move \(w_2\) by \(-0.9 \times 0.4 = -0.36\), to \(0.24\). Now round \(w_2\) to 0. Result \((1, 0)\), \(\delta = (0.4, -0.6)\), \(E = 0.16 + 0.36 - 2 \cdot 0.9 \cdot 0.24 = 0.088\). That is 7 times smaller than RTN, and a search over all grid points shows it is the best possible. Note that \(w_2 = 0.6\) was rounded to 0, the "wrong" way. Try it in Figure 1.
How to read Figure 1
With \(\rho = 0\) the ellipses are circles, the compensation step is zero, and both methods give the same point. Raise \(|\rho|\): the ellipses become long and thin, and the nearest grid point in plain distance (RTN) is often not the nearest in output error. The orange path follows the long axis of the ellipse and usually lands on the best grid point. It is not always the best: the method is greedy, one weight at a time.
OBS and OBQ: quantize one weight, fix the rest
The compensation step comes from a 1993 pruning method, Optimal Brain Surgeon (OBS) (Hassibi and Stork). OBS removes one weight from a network (sets it to 0) and changes all other weights to cancel the damage, using the inverse Hessian. Optimal Brain Quantization (OBQ) (Frantar and Alistarh, 2022, in the "Optimal Brain Compression" paper) uses the same math, but instead of "set the weight to 0" the constraint is "set the weight to its grid value".
Let \(F\) be the set of weights in a row that are still in full precision, and \(H_F\) the Hessian restricted to them. We force weight \(q\) to \(\operatorname{quant}(w_q)\) (its nearest grid point) and choose the change \(\delta_F\) of the free weights that keeps the error (106.2) as small as possible. This is a quadratic minimized under one linear constraint; a Lagrange multiplier gives the exact answer:
- Read the symbols
- \(\operatorname{quant}(w_q)\)
- weight \(q\) rounded to the nearest grid point
- \(H_F^{-1}\)
- the inverse of the Hessian of the weights still free (\(F\) includes \(q\))
- \((H_F^{-1})_{:,q}\)
- column \(q\) of that inverse: the direction in which all free weights move
- \(\delta_F\)
- the change of all free weights; its entry \(q\) moves \(w_q\) exactly onto the grid
- \(\Delta E_q\)
- the increase of the output error caused by this step, after the best compensation
Check it on the worked example: \(H^{-1} = (2XX^\top)^{-1}\) has \([H^{-1}]_{21} / [H^{-1}]_{11} = -0.9\), so \(\delta_2 = -(0.6 - 1)(-0.9) = -0.36\). And \(\Delta E = 0.4^2 / (2 \cdot 2.632) = 0.0304\): this is the error just before \(w_2\) is rounded.
OBQ is greedy: at each step it picks the weight whose quantization costs least,
then applies (106.5), removes \(q\) from \(F\), and repeats until the row is done. After removing \(q\), we need the inverse of the smaller Hessian. We do not invert again: one step of Gaussian elimination on the old inverse gives it.
Here \((\cdot)_{-q}\) means "drop row and column \(q\)". This costs \(O(d_{col}^2)\) per weight instead of \(O(d_{col}^3)\) for a new inverse.
- input: row \(\mathbf{w}\), inverse Hessian \(H^{-1} = (2XX^\top)^{-1}\), grid
- \(F \leftarrow\) all \(d_{col}\) weights
- while \(F\) is not empty:
- pick \(q\) in \(F\) with the smallest \((\operatorname{quant}(w_q) - w_q)^2 / [H_F^{-1}]_{qq}\) // formula (106.6)
- \(\mathbf{w}_F \leftarrow \mathbf{w}_F + \delta_F\) // formula (106.5); now \(w_q = \operatorname{quant}(w_q)\)
- \(H_F^{-1} \leftarrow\) Gaussian elimination of row and column \(q\) // formula (106.7), \(O(d_{col}^2)\)
- \(F \leftarrow F \setminus \{q\}\)
- return \(\mathbf{w}\) (now all on the grid)
Watch Algorithm 1 run in Figure 2. It is one row of 5 weights on the integer grid. The inputs have correlation \(\rho^{|j-k|}\) between channels \(j\) and \(k\), so \(XX^\top_{jk} = \rho^{|j-k|}\) and \(H = 2XX^\top\). Every number is computed in your browser: the scores of (106.6), the move \(\delta_F\) of (106.5), and the elimination (106.7) of the inverse Hessian. Each step adds exactly \(\Delta E_q\) to the error, so the sum of the \(\Delta E_q\) is the final error \(E = \tfrac12 \delta H \delta^\top\). The figure checks this at every step.
- input: row \(\mathbf{w}\), \(H^{-1} = (2XX^\top)^{-1}\), grid
- \(F \leftarrow\) all \(d_{col}\) weights
- while \(F\) is not empty:
- pick \(q\): smallest \((\operatorname{quant}(w_q) - w_q)^2 / [H_F^{-1}]_{qq}\) // (106.6)
- \(\mathbf{w}_F \leftarrow \mathbf{w}_F + \delta_F\) // (106.5)
- \(H_F^{-1} \leftarrow\) eliminate row and column \(q\) // (106.7)
- \(F \leftarrow F \setminus \{q\}\)
- return \(\mathbf{w}\)
How to read Figure 2
Press Step. At “pick” the bars show the score of every free weight; with the greedy order the shortest bar wins. At “update” the chosen weight jumps onto the grid and the other free weights move by \(\delta_F\) (green cells, with the change written under the value). In the first step only the direct neighbors of \(q\) move: for these inputs \(H^{-1}\) is nonzero only on and next to its diagonal. Each elimination creates new nonzero entries, so later steps move more weights. At “eliminate” row and column \(q\) leave the inverse and the other entries change. Row A with \(\rho = 0.9\) ends at error 0.2353 against 0.4751 for RTN, which is also the best grid point. Set \(\rho = 0\): \(H^{-1}\) is diagonal, nothing moves, and the result is RTN. Now take row B with \(\rho = 0.9\): greedy ends at 0.2480, the same as RTN, while the fixed left-to-right order finds the best point, 0.1949. With \(\rho = 0.5\) it is the other way round (greedy 0.4788, fixed 0.6600, RTN 0.5538). Both orders are greedy one-weight-at-a-time methods, and neither is always best.
OBQ is accurate, but the cost is a problem. Each row needs \(d_{col}\) steps, each step updates a \(d_{col} \times d_{col}\) inverse, and each row has its own order, so its own sequence of inverses. That is \(O(d_{col}^3)\) per row and \(O(d_{row} \cdot d_{col}^3)\) per layer. The GPTQ paper reports that OBQ quantizes ResNet-50 (25M parameters) in about 1 hour on one GPU; models with billions of parameters are out of reach. [Certain]
GPTQ step 1: one fixed order for all rows
The first GPTQ insight is empirical: the greedy order of (106.6) helps only a little over any fixed order, especially on large layers. The paper's explanation: weights with large individual errors get quantized late, when only a few free weights remain to compensate, so the gain of the clever order is partly lost. [Certain, as the paper's finding] You can see a tiny version of this in Figure 2: on row B the fixed left-to-right order beats the greedy order at \(\rho = 0.9\) and loses at \(\rho = 0.5\).
So GPTQ quantizes column by column, left to right, in the same order for every row. Now the free set \(F\) is the same for all rows at every step. And \(H_F\) depends only on \(X\), not on the weights. So all rows share one sequence of inverse Hessians, and formula (106.7) runs only \(d_{col}\) times per layer (once per column), not \(d_{row} \cdot d_{col}\) times. The update (106.5) of all rows at once becomes one outer product.
The gain is a factor of \(\min\{d_{row}, d_{col}\}\). For a 4,096 × 4,096 layer that is a factor of 4,096: about \(2.8 \times 10^{14}\) basic operations become about \(6.9 \times 10^{10}\) (constant factors ignored). Figure 5 lets you change the shape.
GPTQ step 2: lazy batch updates
A direct implementation of step 1 is still slow on a GPU. After each column it updates the whole remaining weight matrix and the whole inverse with an outer product: only a few FLOPs per number read from memory. The GPU waits for memory, not for compute (the memory-bound problem from session 82).
The fix: the rounding decision for column \(j\) only depends on the updates that have reached column \(j\). Updates to columns far to the right can wait. So GPTQ works on a block of \(B = 128\) columns at a time. Inside the block it does the column-by-column work on a small slice. When the block is done, it applies all the waiting updates to the rest of the matrix in one matrix multiplication, which a GPU runs efficiently. With \(Q\) the set of columns in the block and \(\mathbf{e}\) their errors:
This is the multi-weight version of (106.5). The number of FLOPs does not change, and the result is exactly the same as column by column (the code below checks it). Only the memory traffic falls. The paper reports an order of magnitude speed-up from this trick on very large models. [Certain]
GPTQ step 3: the Cholesky form, and dampening
Applying (106.7) thousands of times collects rounding errors in floating point. The paper observed that for models above a few billion parameters, \(H_F^{-1}\) becomes indefinite (it gets negative eigenvalues) in at least a few layers. Then (106.5) pushes the weights in wrong directions and the layer is ruined. [Certain]
Two fixes work together. First, dampening: add a small constant to the diagonal before inverting. GPTQ uses 1% of the average diagonal value:
This also helps when some input channels are always zero in the calibration data, which would make \(H\) singular (not invertible).
Second, the Cholesky trick. When we quantize column \(j\), the update needs only one row of the current inverse, row \(j\), from the diagonal to the right. With the fixed order, these rows are exactly what a Cholesky decomposition computes. Write \(H^{-1} = U^\top U\) with \(U\) upper triangular. Then for every step \(j\) and every \(k \ge j\):
where \(F_j\) is the set of columns \(j, j+1, \dots\) still free at step \(j\). So we compute \(U\) once, at the start, with a fast and stable library routine, and never apply (106.7) at all. In the algorithm below, the scaled error is \(e = (w_j - q_j) / U_{jj}\) and the update of the rest of the row is \(-e \cdot U_{j,j:}\). We checked (106.11) numerically on random matrices.
- input: \(W \in \R^{d_{row} \times d_{col}}\), calibration inputs \(X\), block size \(B\), grid per row
- \(H \leftarrow 2XX^\top + \lambda I\) // formulas (106.3), (106.10)
- \(U \leftarrow \operatorname{Cholesky}(H^{-1})\), upper triangular // formula (106.11), computed once
- for \(i = 0, B, 2B, \dots\): // one block of B columns
- \(E \leftarrow 0^{d_{row} \times B}\)
- for \(j = i, \dots, i + B - 1\):
- \(Q_{:,j} \leftarrow \operatorname{quant}(W_{:,j})\) // round the whole column
- \(E_{:,j-i} \leftarrow (W_{:,j} - Q_{:,j}) / U_{jj}\) // scaled error, one number per row
- \(W_{:,j:i+B} \leftarrow W_{:,j:i+B} - E_{:,j-i}\, U_{j,\,j:i+B}\) // compensate inside the block
- \(W_{:,i+B:} \leftarrow W_{:,i+B:} - E\, U_{i:i+B,\; i+B:}\) // lazy update, one matmul, formula (106.9)
- return \(Q\)
Now watch Algorithm 2 run in Figure 3, on a layer of 3 rows and 6 columns. The calibration inputs are 32 tokens with correlation \(\rho\) between neighboring input channels (\(x_j = \rho\, x_{j-1} + \sqrt{1-\rho^2}\, z_j\), fixed seed). The figure computes \(H = 2XX^\top/n\) (dividing by the number of tokens \(n\) only rescales \(H\) and does not change the result), adds the dampening (106.10), and takes the Cholesky factor \(U\) of \(H^{-1}\) (106.11). Each row has its own min-max grid with \(2^b\) levels. Then it runs the loop, one line at a time.
- \(H \leftarrow 2XX^\top/n + \lambda I\) // (106.3), (106.10)
- \(U \leftarrow \operatorname{Cholesky}(H^{-1})\), upper // (106.11)
- for each block of \(B\) columns \(i, \dots, e-1\):
- for \(j = i, \dots, e-1\):
- \(Q_{:,j} \leftarrow \operatorname{quant}(W_{:,j})\)
- \(E_{:,j} \leftarrow (W_{:,j} - Q_{:,j}) / U_{jj}\)
- \(W_{:,k} \leftarrow W_{:,k} - E_{:,j}\, U_{jk}\), \(j \lt k \lt e\)
- \(W_{:,e:} \leftarrow W_{:,e:} - E\, U_{i:e,\, e:}\) // lazy (106.9)
- return \(Q\)
How to read Figure 3
Press Step. At “quant” the blue column is rounded and the GPTQ error bar jumps up: the rounding error of the column is not yet compensated. At “error” the formula divides the rounding error by \(U_{jj}\). At “update” the free columns of the block absorb it (green cells) and the error bar drops back. With \(B = 3\), columns 4 to 6 stay dashed during the first block: their share of the correction waits in \(E\) and arrives in one step at the lazy update. Change \(B\): the final weights and errors are exactly the same for \(B = 1, 2, 3, 6\); only the order of the memory work changes. With the defaults (2 bits, \(\rho = 0.9\)) GPTQ ends at 0.0264 against 0.0839 for RTN, 3.2 times lower; with 3 bits it is 0.0020 against 0.0132. Set \(\rho = 0\): with only 32 tokens the inputs are still a little correlated (sample correlations up to 0.26), the corrections are small, and in this example GPTQ picks exactly the RTN grid points at 2, 3 and 4 bits.
What GPTQ buys, and when
How much better than RTN is GPTQ? It depends on the bit width and on how correlated the inputs are. Figure 4 runs both on a layer of 32 × 64 weights with 256 calibration tokens. You control the correlation between neighboring input channels and the size of a few "outlier" input channels (every 16th channel is scaled up).
How to read Figure 4
Each extra bit cuts the RTN error by about 4 (the grid step halves, the squared error falls by 4). GPTQ follows the same slope but lower. At \(\rho = 0\) GPTQ is only about 10% better: there is little to compensate with. At \(\rho = 0.9\) it is about 6 times better. Make the outlier channels large and turn on act-order: quantizing those columns first, while many free columns can still absorb their error, lowers the error further.
The same pattern holds on real models. In the GPTQ paper's tables (WikiText2 perplexity, lower is better): [Certain]
| Model | FP16 | RTN 4-bit | GPTQ 4-bit | RTN 3-bit | GPTQ 3-bit |
|---|---|---|---|---|---|
| OPT-1.3B | 14.63 | 48.17 | 15.47 | 1.3e4 | 20.97 |
| OPT-13B | 10.13 | 11.32 | 10.31 | 3.4e3 | 11.61 |
| OPT-175B | 8.34 | 10.54 | 8.37 | 7.3e3 | 8.68 |
| BLOOM-176B | 8.11 | 8.37 | 8.21 | 571 | 8.64 |
Two things to notice. Larger models are generally easier to quantize. And 4-bit RTN on OPT-175B (10.54) is worse than the full-precision OPT-13B (10.13), a model 10 times smaller, while GPTQ loses only 0.03. [Certain] On small vision models, where the slow methods can still run, GPTQ is close to OBQ and other accurate PTQ methods: on ResNet-50 at 4 bits, 75.71% top-1 for GPTQ versus 75.72% for OBQ, from 76.13% in full precision. [Certain]
Running GPTQ on a real model
Calibration and the order of layers
The paper's calibration set is 128 random segments of 2,048 tokens from the C4 web dataset: about 262,000 tokens, with nothing task-specific. [Certain] The model is processed one transformer block at a time: load the block on the GPU, run the calibration inputs through it, collect \(H\) for each of its linear layers, quantize them, then run the inputs through the quantized block to get the inputs of the next block. So each layer is fitted to the inputs it will really see, errors of earlier layers included. This is why one 80 GB GPU is enough even for a 175B model. [Certain]
- input: model, calibration batches (e.g. 128 × 2,048 tokens), bits, group size
- \(Z \leftarrow\) hidden states after the embedding layer, for all calibration tokens
- for each transformer block:
- run \(Z\) through the block with hooks; for every linear layer add \(2XX^\top\) of its inputs to its \(H\)
- for each linear layer of the block: \(\hat W \leftarrow \operatorname{GPTQ}(W, H)\) // Algorithm 2
- \(Z \leftarrow\) output of the quantized block on \(Z\) // next block sees real, quantized inputs
- save integer weights, scales and zero points; embeddings and output layer usually stay FP16
Groups
The basic grid has one scale and one zero point per row. With groups, every \(g\) consecutive weights of a row get their own scale and zero point (session 103). GPTQ fits well with groups: the grid of a group is chosen when the algorithm reaches its first column, from the already updated weights. The cost is extra bits. With an FP16 scale and a \(b\)-bit zero point per group, the overhead is \((16 + b)/g\) bits per weight: \(19/128 \approx 0.15\) for 3-bit with \(g = 128\). On OPT-175B, 3-bit GPTQ with \(g = 128\) reaches 8.45 perplexity (FP16: 8.34). [Certain] 4-bit with \(g = 128\) is the most common setting for published GPTQ checkpoints. [Likely]
In FP16: \(70 \times 10^9 \times 2\) bytes = 140 GB, two 80 GB GPUs at least. At 4 bits with \(g = 128\) (FP16 scale + 4-bit zero point): \(4 + 20/128 = 4.156\) bits per weight, so \(70 \times 10^9 \times 4.156 / 8 = 36.4\) GB, plus the FP16 embeddings and output layer. It fits on one 80 GB GPU, with room for the KV cache. The GPTQ paper did the same for OPT-175B at 3 bits: about 63 GB with FP16 embeddings, run on one A100 80 GB instead of five. [Certain]
Act-order
The fixed order does not have to be left to right. The GPTQ code has an option, usually called act-order (desc_act), that sorts the columns by \(H_{jj}\), largest first: the weights that meet the largest activations are quantized while many free weights can still compensate. It often improves accuracy, as in Figure 4. [Likely] With groups, it means a group's columns are no longer next to each other in memory, which needs an extra index at inference time and can slow some kernels; "static groups" (fix the groups before reordering) is a common workaround. [Likely]
Kernels: where the speed comes from
GPTQ only changes the stored weights. At inference, a mixed-precision kernel reads the 3-bit or 4-bit integers, turns them back into FP16 inside the GPU's fast on-chip memory, and multiplies with FP16 activations. During decoding (batch size 1, matrix-vector products) the time is mostly reading weights, so fewer bytes means more speed. The paper measured, for 3-bit OPT-175B, 71 ms per token on one A100 versus 230 ms for FP16 on five A100s (3.24×), and 130 ms on two A6000s versus 589 ms on eight (4.53×). [Certain] For large batches (prefill, busy servers) the matmuls become compute-bound and the gain shrinks; kernels such as Marlin were built to keep 4-bit fast at moderate batch sizes. [Likely] See session 104 for this trade-off.
GPTQ in PyTorch
The code builds a toy layer with correlated inputs and a few outlier channels, then compares RTN, GPTQ (Algorithm 2) and GPTQ with act-order. It also checks that the fast GPTQ (Cholesky, blocks) gives exactly the same weights as the slow version that applies formulas (106.5) and (106.7) column by column. It runs on CPU in a few seconds; we ran it with PyTorch 2.8.
import torch torch.manual_seed(0) torch.set_default_dtype(torch.float64) # 1) a toy layer: W is (d_row, d_col), X holds n calibration inputs as columns d_row, d_col, n, rho = 64, 128, 512, 0.8 Z = torch.randn(d_col, n) X = torch.empty(d_col, n); X[0] = Z[0] for j in range(1, d_col): # neighbouring input channels are correlated X[j] = rho * X[j - 1] + (1 - rho**2) ** 0.5 * Z[j] X[::16] *= 6.0 # a few "outlier" input channels W = torch.randn(d_row, d_col) def make_grid(W, bits): # per-row asymmetric min-max grid lo, hi = W.min(1).values, W.max(1).values scale = (hi - lo) / (2**bits - 1) zero = torch.round(-lo / scale) return scale, zero, 2**bits - 1 def quant(col, scale, zero, qmax): # round one column to its row's grid q = torch.clamp(torch.round(col / scale) + zero, 0, qmax) return scale * (q - zero) def rel_err(W, Q, X): # ||WX - QX||^2 / ||WX||^2 return (((W - Q) @ X) ** 2).sum() / ((W @ X) ** 2).sum() def rtn(W, bits): s, z, m = make_grid(W, bits) return torch.stack([quant(W[:, j], s, z, m) for j in range(W.shape[1])], 1) def gptq(W, X, bits, block=32, damp=0.01, act_order=False): W = W.clone(); H = 2 * X @ X.T # formula (106.3) s, z, m = make_grid(W, bits) perm = torch.argsort(torch.diag(H), descending=True) if act_order else torch.arange(W.shape[1]) W, H = W[:, perm], H[perm][:, perm] H += damp * torch.diag(H).mean() * torch.eye(H.shape[0]) # dampening, (106.10) U = torch.linalg.cholesky(torch.cholesky_inverse(torch.linalg.cholesky(H)), upper=True) # (106.11) Q = torch.zeros_like(W) for i in range(0, W.shape[1], block): e = min(i + block, W.shape[1]); Err = torch.zeros(W.shape[0], e - i) for j in range(i, e): Q[:, j] = quant(W[:, j], s, z, m) Err[:, j - i] = (W[:, j] - Q[:, j]) / U[j, j] W[:, j:e] -= Err[:, j - i : j - i + 1] * U[j, j:e] # update inside the block W[:, e:] -= Err @ U[i:e, e:] # lazy update of the rest return Q[:, torch.argsort(perm)] def obq_fixed_order(W, X, bits, damp=0.01): # formulas (106.5) and (106.7), no tricks W = W.clone(); H = 2 * X @ X.T H += damp * torch.diag(H).mean() * torch.eye(H.shape[0]) s, z, m = make_grid(W, bits); Hinv = torch.linalg.inv(H); Q = torch.zeros_like(W) for q in range(W.shape[1]): Q[:, q] = quant(W[:, q], s, z, m) W -= torch.outer((W[:, q] - Q[:, q]) / Hinv[q, q], Hinv[:, q]) Hinv -= torch.outer(Hinv[:, q], Hinv[q, :]) / Hinv[q, q] # remove weight q return Q for bits in (3, 4): print(bits, "bits RTN %.5f GPTQ %.5f GPTQ+act-order %.5f" % ( rel_err(W, rtn(W, bits), X), rel_err(W, gptq(W, X, bits), X), rel_err(W, gptq(W, X, bits, act_order=True), X))) Q1, Q2 = gptq(W, X, 4), obq_fixed_order(W, X, 4) print("same result as plain OBQ in fixed order:", torch.equal(Q1, Q2), (Q1 - Q2).abs().max().item()) # Output (PyTorch 2.8, CPU): # 3 bits RTN 0.05040 GPTQ 0.01562 GPTQ+act-order 0.01175 # 4 bits RTN 0.01000 GPTQ 0.00316 GPTQ+act-order 0.00243 # same result as plain OBQ in fixed order: True 0.0
In practice you do not write this yourself. Libraries such as GPTQModel (the successor of AutoGPTQ) and the Hugging Face GPTQConfig run Algorithm 3 on a full model, and vLLM and other engines load the result with fast kernels. [Likely]
How much compute? Figure 5
How to read Figure 5
The ratio OBQ / GPTQ is always \(\min\{d_{row}, d_{col}\}\). For square LLM layers it is in the thousands. GPTQ's \(d_{col}^3\) term (the Cholesky) dominates only when the layer has more inputs than outputs. The readout also counts how many times the whole remaining matrix is updated: \(d_{col}\) times without lazy batching, \(d_{col}/128\) times with it.
The measured runtimes match this picture: on one A100, GPTQ took 20.9 minutes for OPT-13B, 4.2 hours for OPT-175B, and 3.8 hours for BLOOM-176B. [Certain] The paper notes that ZeroQuant's layer-wise distillation needed about 3 hours for a 1.3B model on the same hardware. [Certain]
Where this is used
| Setting | Typical use of GPTQ | Why |
|---|---|---|
| Serving open LLMs on fewer GPUs | 4-bit, \(g = 128\) checkpoints of 7B to 70B models loaded in vLLM or Transformers [Likely] | About 3.5 to 4 times less weight memory; faster decoding, which is memory-bound |
| Local and edge inference | 4-bit or 3-bit weights on one consumer GPU | A 70B model drops from 140 GB to about 36 GB of weights |
| VLMs (vision-language models) | GPTQ on the language model; the vision encoder often stays in FP16 [Likely] | The language model holds most of the parameters (session 197) |
| Video captioning and Video-RAG on a single GPU | A 4-bit VLM answers questions over retrieved frames or clips | Frees memory for the long KV cache that many frame tokens need |
| Key-moment detection or action recognition with an LLM head | Quantize the LLM that turns frame features into labels or alerts; keep a calibration set of in-domain prompts | Lower latency per decision; calibration on matching data protects the domain |
| Pruning (SparseGPT) | The same Hessian, column order and Cholesky machinery, with "set to 0" instead of "round" (session 114) | OBS-style compensation works for any weight constraint |
Common mistakes
- Too little or wrong calibration data. \(H\) is estimated from the calibration tokens. Too few tokens give a noisy or singular \(H\); text very different from the real use (for example only English prose for a code model) fits the layers to the wrong inputs.
- No dampening. Without \(\lambda I\), dead input channels make \(H\) singular and the Cholesky fails or the weights blow up. 1% of the mean diagonal is the usual default; raise it if a layer fails.
- Using full-precision inputs for later blocks. Each block should be calibrated on the outputs of the already quantized blocks, as in Algorithm 3. Otherwise errors of earlier layers are not taken into account.
- Thinking GPTQ fixes outliers. GPTQ reduces the effect of rounding, but one huge weight still stretches the grid of its row or group. Smaller groups, AWQ-style scaling or rotations handle that (session 107, session 109).
- Forgetting the overhead bits. "4-bit" with \(g = 128\) is about 4.16 bits per weight; \(g = 32\) is about 4.6. Compare methods at equal real size.
- Measuring only perplexity on one dataset. Check a few downstream tasks too, especially long-context and reasoning tasks, which can be hurt more than WikiText2 shows. [Likely]
- Expecting a big prefill speed-up. Weight-only quantization mainly speeds up memory-bound decoding. At large batch sizes the gain can disappear, depending on the kernel.
- Act-order with groups on a kernel that does not support it. The model may load but run slowly, or the reordering may be ignored. Check what your serving engine supports. [Likely]
Interview questions
What objective does GPTQ minimize, and why layer by layer?basic
What is the Hessian in GPTQ, and why does it not depend on the weights?core
Derive the OBQ / OBS update for one weight.advanced
Why is GPTQ so much faster than OBQ?core
Lazy batching does not reduce FLOPs. Why does it help?core
Why does GPTQ need a Cholesky decomposition and dampening?advanced
When does GPTQ beat RTN by a lot, and when not?core
How would you choose the calibration data, and what if it does not match deployment?core
What are group size and act-order, and how do they interact?advanced
Weight-only 4-bit GPTQ: why does it speed up decoding more than prefill?core
How much memory does a 70B model need at 4 bits with group size 128?basic
Exercises
- In the worked example (\(\mathbf{w} = (0.6, 0.6)\), integer grid), set the correlation to 0. What do RTN and the compensated method give, and what is the error?
- Same weights, correlation \(-0.9\). Compute the RTN error and the result of "round \(w_1\), compensate, round \(w_2\)".
- Show from formula (106.5) that GPTQ equals RTN when \(H\) is diagonal.
- A layer has \(d_{row} = 11{,}008\) and \(d_{col} = 4{,}096\). By what factor is GPTQ cheaper than OBQ (leading order)? Which term of GPTQ's cost dominates?
- Compute the overhead bits per weight for 4-bit with \(g = 64\), FP16 scale and 4-bit zero point. What is the total size of a 7B model's linear weights?
- The mean of the diagonal of \(H\) is 50. What \(\lambda\) does GPTQ add? Why is dampening a small bias but a big help?
Answerscheck
Key takeaways
- Weight PTQ is solved layer by layer: keep \(WX\) close to \(\hat W X\) on calibration inputs (106.1). The error of each row is exactly \(\tfrac12 \delta H \delta^\top\) with \(H = 2XX^\top\) (106.2, 106.3).
- \(H\) depends only on the inputs. Its diagonal says which weights matter; its off-diagonal says which weights can cancel each other's errors (106.4).
- OBQ quantizes one weight and moves the free weights by \(-\frac{w_q - \operatorname{quant}(w_q)}{[H^{-1}]_{qq}}(H^{-1})_{:,q}\) (106.5), greedily (106.6), updating the inverse by Gaussian elimination (106.7). Accurate but \(O(d_{row} d_{col}^3)\).
- GPTQ: one fixed column order for all rows (106.8), lazy block updates of 128 columns (106.9), dampening (106.10) and the Cholesky factor of \(H^{-1}\) computed once (106.11).
- Result: 175B models at 3 to 4 bits in about 4 GPU hours, near-FP16 perplexity at 4 bits, and 3 to 4.5 times faster decoding with the right kernels.
Key formulas
References for this session
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Frantar & Alistarh (2022). Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Hassibi & Stork (1993). Second order derivatives for network pruning: Optimal Brain Surgeon
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Yao et al. (2022). ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
- Frantar & Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
- Frantar, Castro, Chen, Hoefler, Alistarh (2024). MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models
- Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration