Rotations against outliers: QuIP, QuIP#, QuaRot, SpinQuant
A few activation channels in an LLM are a hundred times larger than the rest, and they ruin 4-bit quantization. Multiply the activations by an orthogonal matrix and the weights by its transpose: the output stays exactly the same, but the outliers are spread over all channels. This session builds that trick from the math up and follows it through QuIP, QuIP#, QuaRot and SpinQuant.
What you will learn
- Explain why the quantization error of a vector grows with its max-to-RMS ratio, and why one outlier channel wipes out the small channels at 4 bits
- Prove computational invariance: rotating activations by Q and weights by Q transpose leaves the output unchanged, also through RMSNorm once its scale is folded
- Build a Hadamard matrix, run the fast Walsh–Hadamard transform in n log n steps, and say why random signs make the spreading work for any input
- Describe QuIP and QuIP#: incoherence of weights and Hessians, the proxy loss that rotations do not change, and the E8 lattice codebook
- Place every QuaRot rotation in a Llama block: which ones merge into the weights and which ones must run online
- Explain why SpinQuant learns its rotations, how a Cayley step keeps a matrix orthogonal, and what it costs
- Quote the reported results of the four papers correctly and pick a method for a given deployment
Why this matters
At 8 bits, an LLM can be quantized with almost no loss. At 4 bits for weights and activations it usually breaks. The reason is a handful of outlier channels: a few coordinates of the hidden vector that are tens or hundreds of times larger than the others. This session shows a simple and exact cure. Multiply the activations by an orthogonal matrix \(Q\) and the next weight matrix by \(Q^{\top}\). The model computes the same function, but each outlier is now spread over all channels, and 4-bit quantization works.
Session 108 met the outliers and handled them in two ways: LLM.int8() keeps the outlier channels in 16 bits, and SmoothQuant moves part of their size from the activations into the weights with a per-channel scale. Both are diagonal fixes: they treat each channel on its own. A rotation is a full matrix. It mixes all channels, so one large channel becomes many medium ones. New in this session:
- Why the error of round-to-nearest grows with the max-to-RMS ratio of a vector, and what an outlier does to the small channels (Figure 1).
- Computational invariance: the proof that \(xW = (xQ)(Q^{\top}W)\) also holds through RMSNorm, so a whole model can be rotated without retraining (Figure 2 checks the one-layer case number by number).
- The Hadamard matrix and the fast Walsh–Hadamard transform, which applies a rotation in \(n \log_2 n\) additions (Figure 3).
- QuIP and QuIP#: rotations for 2-bit weights, with a proof that the GPTQ-style proxy loss does not change.
- QuaRot: where each rotation goes in a Llama block, and which ones cost time at inference (Figure 4).
- SpinQuant: why some rotations are better than others (Figure 5) and how to learn one while keeping it orthogonal.
This is part of the post-training quantization section (5.2.1) of the efficient-inference survey Track 5. [Certain] Rotations are now a standard part of 4-bit inference: Meta shipped its quantized Llama 3.2 1B and 3B models with SpinQuant as one of two methods. [Certain] In interviews you hear: “Why does rotating a model not change its output?”, “Why a Hadamard matrix and not any orthogonal matrix?”, “Which rotations can be merged into weights and which cannot?”, “How is SpinQuant different from QuaRot?”. All of them come down to the two facts in the title of the next section.
The problem: one outlier sets the step for everyone
Round-to-nearest with one scale per token
Session 103 defined symmetric quantization. For activations we use one scale per token (per row of the activation matrix), because it can be computed on the fly. For a vector \(\mathbf{x} \in \R^n\) and \(b\) bits:
- Read the symbols
- \(\mathbf{x}\)
- one token's activation vector, the input of a linear layer, with \(n\) channels
- \(\hat{\mathbf{x}}\)
- its quantized version: every entry is an integer times \(\Delta\)
- \(\Delta\)
- the step size: the distance between two neighbouring quantization levels
- \(q_{\max}\)
- the largest integer used: 7 for 4 bits, 3 for 3 bits, 127 for 8 bits
The step \(\Delta\) is set by the largest entry. Every entry is rounded to the nearest multiple of \(\Delta\), so each one gets an error of at most \(\Delta / 2\). If we model the rounding error as uniform on \([-\Delta/2, \Delta/2]\) (a standard approximation), its variance is \(\Delta^2 / 12\) per entry. Divide the total by \(\|\mathbf{x}\|^2 = n \cdot \operatorname{RMS}(\mathbf{x})^2\):
- Read the symbols
- \(\operatorname{RMS}(\mathbf{x})\)
- root mean square: \(\sqrt{\tfrac{1}{n}\sum_i x_i^2}\), the typical size of an entry
- \(\max_i |x_i| / \operatorname{RMS}(\mathbf{x})\)
- the max-to-RMS ratio (also called the crest factor). It is 1 when all entries have the same size and \(\sqrt{n}\) when one entry holds all the energy
So the relative error does not depend on the bits alone. It depends on the bits and on the shape of the vector. A vector with a flat profile is easy; a vector with one spike is hard. LLM activations have spikes: a few fixed channels (the same channels for almost every token) are much larger than the rest. [Certain, observed by LLM.int8() and SmoothQuant]
There is a second, worse effect. When \(\Delta\) is larger than twice a small entry, that entry rounds to 0. Its information is gone, not just noisy. At 4 bits an outlier 20 times larger than the typical entry already gives \(\Delta \approx 20/7\) times the typical entry, so most channels become 0. Figure 1 shows this.
The fix in one line: rotate
Take any orthogonal matrix \(Q\) (\(Q Q^{\top} = I\)). For a linear layer \(\mathbf{y} = \mathbf{x} W\):
- Read the symbols
- \(W\)
- the weight matrix, \(n \times m\); we use row vectors, so \(\mathbf{x}\) is \(1 \times n\)
- \(Q\)
- an \(n \times n\) orthogonal matrix: it keeps lengths, \(\|\mathbf{x} Q\| = \|\mathbf{x}\|\)
- \(\mathbf{x} Q,\ Q^{\top} W\)
- the rotated activation and the rotated weight; we quantize these instead of \(\mathbf{x}\) and \(W\)
Without quantization the output is exactly the same. With quantization we now quantize \(\mathbf{x}Q\), and we can choose \(Q\) so that \(\mathbf{x}Q\) has no spikes. The weight \(Q^{\top}W\) is computed once, offline. This is the whole idea of the session. The rest is: which \(Q\), how to apply it cheaply, and how to fit it into a real transformer.
Why does a rotation spread a spike? Look at the extreme case: all energy in channel \(i\), \(\mathbf{x} = a\,\mathbf{e}_i\). Then \(\mathbf{x}Q\) is \(a\) times row \(i\) of \(Q\). If every entry of \(Q\) has size \(1/\sqrt{n}\) (this is the Hadamard matrix of the next section):
The worst possible vector becomes the best possible one. For \(n = 4096\), an outlier of size 100 becomes 4,096 entries of size \(100/64 = 1.5625\). A real activation is a spike plus many normal entries; the rotation turns it into normal entries plus a small flat offset, which quantizes well.
How to read Figure 1
Start with “No rotation”, 4 bits, outlier 20. The step is 2.857, larger than twice every normal value, so all 15 normal channels round to 0: their error is 100%. Switch to “Randomized Hadamard”. The bars are now all of similar height (max-to-RMS falls from 3.95 to 1.27), the step falls to 0.918, the total error falls from 15.5% to 4.6% and the normal channels keep most of their information (25.1% error instead of 100%). Move the outlier to 1: there is no outlier, and the rotation gives no gain (8.0% against 7.7% total error). Move it to 50 with 4 bits: rotation still cuts the total error (6.3% to 4.3%), but the normal channels now have 67.5% error. With only 16 channels the spike is too big to hide; the readout shows why the rotation works better in real models with thousands of channels.
The last observation in the box is important. A rotation does not create information. It spreads the quantization error evenly over all channels, so every channel gets an error of about \(\Delta_{\text{rot}}/\sqrt{12}\). In Figure 1 the error after rotating is the same size in every channel, and the small channels get a large relative error when the outlier dominates the norm. In a real model, \(n\) is 4,096 or more, so the outlier is divided by \(\sqrt{n} = 64\), not by 4, and the normal channels survive much better.
Take \(\mathbf{x} = (12, 1, -2, 1)\) and \(b = 3\), so \(q_{\max} = 3\). Here \(\|\mathbf{x}\|^2 = 150\) and \(\operatorname{RMS} = \sqrt{150/4} = 6.124\). The max-to-RMS ratio is \(12 / 6.124 = 1.96\).
Without rotation. \(\Delta = 12/3 = 4\). Divide: \((3, 0.25, -0.5, 0.25)\); round: \((3, 0, 0, 0)\). So \(\hat{\mathbf{x}} = (12, 0, 0, 0)\). The three small channels are gone. Squared error \(= 0 + 1 + 4 + 1 = 6\), relative error \(\sqrt{6/150} = 0.200\).
With the 4 × 4 Hadamard matrix \(H_4\) (rows \((1,1,1,1), (1,-1,1,-1), (1,1,-1,-1), (1,-1,-1,1)\), divided by 2): \(\mathbf{z} = \mathbf{x} H_4 = (6, 4, 7, 7)\). Its max-to-RMS ratio is \(7/6.124 = 1.14\). Now \(\Delta = 7/3 = 2.333\); \(\mathbf{z}/\Delta = (2.571, 1.714, 3, 3)\) rounds to \((3, 2, 3, 3)\), so \(\hat{\mathbf{z}} = (7, 4.667, 7, 7)\). Rotate back with \(H_4^{\top}\): \((12.833, 1.167, -1.167, 1.167)\). The errors are \((0.833, 0.167, 0.833, 0.167)\), squared error \(1.444\), relative error \(\sqrt{1.444/150} = 0.098\).
The rotation halved the error, and the small channels now have the right sign and roughly the right size instead of being 0.
The worked example only looked at the activation. Figure 2 sends the same vector through a small linear layer \(W\) (4 inputs, 3 outputs) and computes both sides of (109.3) one dot product at a time. First the original layer \(\mathbf{y} = \mathbf{x}W\). Then the rotated path: \(\mathbf{z} = \mathbf{x}Q\), the folded weight \(W' = Q^{\top}W\), and \(\mathbf{y}' = \mathbf{z}W'\). Every \(y'_j\) comes out equal to \(y_j\). Only in the last steps do we quantize the activation (the weights stay exact here, so that we see the effect of the activation alone), and there the two paths separate.
- input: activation \(\mathbf{x}\) (1 × 4), weight \(W\) (4 × 3), orthogonal \(Q\) (4 × 4)
- for \(j = 1..3\): \(y_j \leftarrow \sum_i x_i W_{ij}\) // the original layer
- for \(j = 1..4\): \(z_j \leftarrow \sum_i x_i Q_{ij}\) // rotate the activation: \(\mathbf{z} = \mathbf{x}Q\)
- \(W' \leftarrow Q^{\top} W\) // offline, once per layer
- for \(j = 1..3\): \(y'_j \leftarrow \sum_i z_i W'_{ij}\) // equals \(y_j\), by (109.3)
- \(\hat{\mathbf{x}} \leftarrow \operatorname{quant}_b(\mathbf{x})\); \(\ \hat{\mathbf{y}} \leftarrow \hat{\mathbf{x}} W\) // (109.1), no rotation
- \(\hat{\mathbf{z}} \leftarrow \operatorname{quant}_b(\mathbf{z})\); \(\ \hat{\mathbf{y}}' \leftarrow \hat{\mathbf{z}} W'\) // (109.1), rotated
- compare \(\|\hat{\mathbf{y}} - \mathbf{y}\| / \|\mathbf{y}\|\) with \(\|\hat{\mathbf{y}}' - \mathbf{y}\| / \|\mathbf{y}\|\)
How to read Figure 2
Press Play, or Step through it. The highlighted line of the algorithm box is the one that runs, and the box next to it shows the same sum with the real numbers. The cells used in the sum get a thick outline. Steps 2 to 4 give \(\mathbf{y} = (8.25, -14, 1.5)\). Steps 5 to 8 give \(\mathbf{z} = (6, 4, 7, 7)\) for \(Q = H_4\): the outlier 12 is now spread over all four entries. Steps 10 to 12 give \(\mathbf{y}'\), equal to \(\mathbf{y}\) entry by entry. With 3 bits, quantizing \(\mathbf{x}\) erases channels 2 to 4 and the output error is 20.6%; quantizing \(\mathbf{z}\) gives 7.0%. With \(Q = D H_4\) the rotated error is 6.6%. With \(Q = I\) both paths are the same. Switch to 4 bits with \(Q = H_4\): \(\mathbf{z}/\Delta\) is exactly \((6, 4, 7, 7)\), so the rotated error is 0, against 6.3% without rotation. That is a lucky integer case; the next figure uses less tidy numbers.
Hadamard matrices and the fast transform
Why Hadamard
Any orthogonal \(Q\) keeps the output exact. We want three more things: every entry of \(Q\) should be small (\(\pm 1/\sqrt{n}\) is the smallest possible, because each row has unit length), applying \(Q\) at runtime should be cheap, and it should work for every input, not only for one-hot spikes. A Hadamard matrix has entries \(\pm 1\), rows that are orthogonal to each other, and (scaled by \(1/\sqrt{n}\)) it is orthogonal. The Sylvester construction builds one for every power of 2:
- Read the symbols
- \(H_n\)
- the \(n \times n\) normalized Hadamard matrix; every entry is \(\pm 1/\sqrt{n}\) and \(H_n H_n^{\top} = I\)
- block form
- the recursion is what makes the fast transform possible: one level of the recursion is one stage of sums and differences
Because of the block form, \(\mathbf{x} H_n\) can be computed in \(\log_2 n\) stages. Each stage takes pairs of entries \((u, v)\) and replaces them by \((u + v, u - v)\). That is the fast Walsh–Hadamard transform (FWHT): \(n \log_2 n\) additions and one scaling, instead of \(n^2\) multiply-adds for a dense matrix. For \(n = 4096\): 49,152 additions against 16,777,216 multiply-adds, about 341 times fewer. No weights are stored, because the matrix is fixed.
Figure 3 runs the whole pipeline on 8 channels, one butterfly per step: optional random signs, the three stages, the scaling, then 4-bit quantization of the rotated vector and the rotation back. At the end it quantizes the same \(\mathbf{x}\) without any rotation and compares the two errors channel by channel.
- input: \(\mathbf{x} \in \R^n\) with \(n = 8\), signs \(\mathbf{s} \in \{\pm 1\}^n\), bits \(b\)
- \(\mathbf{y} \leftarrow \mathbf{x} \odot \mathbf{s}\) // the \(D\) in \(Q = DH\)
- for \(h = 1, 2, 4\): // one stage per level of (109.5)
- for each pair \((j, j + h)\) with \(j \bmod 2h \lt h\):
- \(u, v \leftarrow y_j, y_{j+h}\); \(\ y_j \leftarrow u + v\); \(\ y_{j+h} \leftarrow u - v\) // one butterfly
- \(\mathbf{z} \leftarrow \mathbf{y} / \sqrt{n}\) // now \(\mathbf{z} = \mathbf{x} D H_8\)
- \(\Delta \leftarrow \max_j |z_j| / q_{\max}\); \(\ \hat{\mathbf{z}} \leftarrow \Delta \cdot \operatorname{round}(\mathbf{z}/\Delta)\) // (109.1)
- \(\hat{\mathbf{x}} \leftarrow \operatorname{FWHT}(\hat{\mathbf{z}}) / \sqrt{n} \odot \mathbf{s}\) // rotate back: \((DH)^{\top} = HD\)
- compare with \(\operatorname{quant}_b(\mathbf{x})\), quantized without rotation
How to read Figure 3
Press Play or Step. Step 1 shows the input: one value of 10 and seven values near 1 (max-to-RMS 2.77). Each step of a stage runs one butterfly, and the box on the right shows \(u\), \(v\), \(u + v\) and \(u - v\) with the real numbers. Watch the orange cells: the outlier reaches 2 cells after stage 1, 4 after stage 2 and all 8 after stage 3. Dividing by \(\sqrt{8} = 2.828\) gives values between 2.33 and 4.67 (max-to-RMS 1.29), and the length of the vector is unchanged. The whole transform used \(8 \times 3 = 24\) additions. With 4 bits the error after rotating back is 6.5%, against 12.1% for plain quantization, where three small channels round to 0. Turn on the random signs: the rotated error falls to 3.1%. Set the outlier to 30 with 3 bits and no signs: now the rotation is worse (11.4% against 7.1%). With only 8 channels a big spike stays big after spreading (\(30/\sqrt{8} = 10.6\) in every channel) and the small values are lost in the step; real layers have thousands of channels.
Random signs: a guarantee for every input
A fixed \(H\) has one weakness: if the input happens to be a row of \(H\), the output is a spike again (because \(H\) maps its own rows to one-hot vectors). The fix is to flip the sign of each channel at random first. With a diagonal matrix \(D\) of random \(\pm 1\):
- Read the symbols
- \(D\)
- diagonal matrix with independent random signs \(\pm 1\); \(DH_n\) is the randomized Hadamard transform (RHT)
- \(\delta\)
- the failure probability we accept, for example 0.01
- \(\|\mathbf{x}\| / \sqrt{n}\)
- the RMS of \(\mathbf{x}\) (and of \(\mathbf{x}Q\)); so the bound says max-to-RMS is at most \(\sqrt{2 \ln(2n/\delta)}\)
Why it holds: entry \(j\) of \(\mathbf{x}DH\) is \(\sum_i \pm x_i / \sqrt{n}\) with independent random signs. Hoeffding’s inequality bounds each entry, and a union bound over the \(n\) entries gives (109.6). For \(n = 4096\) and \(\delta = 0.01\) the bound is \(0.0815\,\|\mathbf{x}\|\), while a pure spike has max \(\|\mathbf{x}\|\). The max-to-RMS ratio is at most 5.2 for any input, against 64 for the worst unrotated input. QuIP# proves this kind of bound (it calls it incoherence, next section), and QuaRot uses the same randomized Hadamard. [Certain]
Sizes that are not powers of 2
Real layer sizes are not always powers of 2. Llama 2 7B has an MLP width of 11,008; Llama 3 8B has 14,336. [Certain] Hadamard matrices also exist for many sizes that are multiples of 4 (for example 12, 20, 28, 172). QuaRot and QuIP# combine them with the Kronecker product: [Certain]
The \(H_{2^k}\) part runs as an FWHT; the small \(H_m\) part is a dense \(m \times m\) multiply. Note that \(11{,}008 = 2^8 \times 43\) does not work: a Hadamard matrix of size 43 cannot exist, because sizes above 2 must be multiples of 4. Which factorization each code base uses is an implementation detail. [Likely]
Rotating a whole transformer: computational invariance
The residual stream
Formula (109.3) is for one layer. In a transformer, the same hidden vector \(\mathbf{x}\) travels along the residual stream (session 22) and is read by many layers. QuaRot uses an idea from SliceGPT, called computational invariance: rotate the residual stream itself. [Certain] Multiply the embedding matrix by \(Q\), so every token enters as \(\mathbf{x}Q\). Every layer that reads the stream gets \(Q^{\top}\) on its input side (\(W_q, W_k, W_v, W_{\text{up}}, W_{\text{gate}}\) and the LM head). Every layer that writes to the stream gets \(Q\) on its output side (\(W_o\), \(W_{\text{down}}\)). Then the stream carries \(\mathbf{x}Q\) everywhere, and each residual addition \(\mathbf{x}Q + \mathbf{f}Q = (\mathbf{x} + \mathbf{f})Q\) stays consistent.
RMSNorm
One thing stands between the layers: RMSNorm, \(\operatorname{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\operatorname{RMS}(\mathbf{x})} \odot \alpha\) (here \(\alpha \in \R^n\) is a vector of learned scales, applied element by element). The division commutes with \(Q\), because a rotation keeps the length. The element-wise scale does not commute, so we first fold it into the next weight matrix:
- Read the symbols
- \(\alpha\)
- the learned per-channel scale of RMSNorm; after folding, the norm has no scale (all ones)
- \(W'\)
- the new weight of the layer after the norm: scale first, rotation second
Check: \(\frac{\mathbf{x}Q}{\operatorname{RMS}(\mathbf{x}Q)} W' = \frac{\mathbf{x}}{\operatorname{RMS}(\mathbf{x})} Q Q^{\top} \operatorname{diag}(\alpha) W = \operatorname{RMSNorm}(\mathbf{x})\, W\). The PyTorch script below confirms this numerically. For LayerNorm models, the mean subtraction is also a linear map, and SliceGPT shows how to turn LayerNorm into RMSNorm first. [Certain] After this, the whole network computes exactly the same function in full precision, but every matrix that reads the residual stream now sees \(\mathbf{x}Q\) without outliers.
- input: model weights; hidden size \(d\); a random sign seed
- \(Q \leftarrow D H_d\) // randomized Hadamard, (109.6); SpinQuant would use a learned \(R_1\) here
- for each RMSNorm: fold \(\alpha\) into the matrices that read its output, set \(\alpha \leftarrow \mathbf{1}\) // (109.8)
- \(E \leftarrow E Q\) // embedding rows enter the stream rotated
- for each layer: \(W_q, W_k, W_v, W_{\text{up}}, W_{\text{gate}} \leftarrow Q^{\top} W\); \(\ W_o, W_{\text{down}} \leftarrow W Q\)
- \(W_{\text{head}} \leftarrow Q^{\top} W_{\text{head}}\) // the LM head reads the final stream
- check on a few prompts that the logits match the original model to float tolerance
- quantize the rotated weights (RTN or GPTQ) // the full-precision model is unchanged; only quantization improves
QuIP and QuIP#: rotations for 2-bit weights
The proxy loss does not see the rotation
The rotation idea first appeared for weight-only quantization, in QuIP (Chee et al., 2023). Session 106 showed that GPTQ quantizes a layer by minimizing a proxy loss: the output error on calibration inputs. With \(H = \E[\mathbf{x}^{\top}\mathbf{x}]\), the \(n \times n\) second-moment matrix of the inputs (the “Hessian” of the proxy), the loss and its behaviour under rotations are:
- Read the symbols
- \(\hat{W}\)
- the quantized weight; \(\ell\) measures how much the layer output changes
- \(H\)
- the input second-moment matrix, estimated from calibration data, as in GPTQ
- \(U, V\)
- random orthogonal matrices on the output side and the input side
- \(\tilde{W}, \tilde{H}\)
- the rotated problem: quantize \(\tilde{W}\) with the rotated inputs \(\mathbf{x}V\), then undo \(U\) on the output
If \(\hat{\tilde{W}} - \tilde{W} = V^{\top}(\hat{W} - W)U\), the loss is the same: \(\operatorname{tr}(U^{\top}(\hat{W} - W)^{\top} V V^{\top} H V V^{\top}(\hat{W} - W) U) = \ell(\hat{W})\) because \(VV^{\top} = I\) and the trace is unchanged by \(U\). So we may solve the rotated problem instead, and choose \(U, V\) to make it easy. QuIP calls a problem easy when it is incoherent: [Certain]
- Read the symbols
- \(\mu\)
- the incoherence parameter; small \(\mu\) means no entry is much larger than average
- \(\|W\|_F / \sqrt{mn}\)
- the RMS of the entries of \(W\); the first condition is “max-to-RMS of \(W\) is at most \(\mu\)”, the matrix version of the ratio in (109.2)
- eigenvector condition
- the directions in which the inputs vary most are spread over all coordinates, so no single weight is much more important than the others
The first condition makes the weights easy to round with a small grid. The second makes rounding errors in different coordinates matter about equally. QuIP pairs incoherence processing with an adaptive rounding method (LDLQ) and proves that it is equivalent to OPTQ, the method behind GPTQ. It was the first method to give usable LLMs at 2 bits per weight. [Certain] Its \(U\) and \(V\) were Kronecker products of two smaller random orthogonal matrices, to make them cheap to apply. [Certain]
QuIP#: Hadamard plus a lattice codebook
QuIP# (Tseng et al., 2024) makes three changes: [Certain]
- It replaces the Kronecker product with the randomized Hadamard transform \(DH\), which is faster and has better incoherence bounds (with probability \(1 - \delta\), \(\mu_H = \sqrt{2 \ln(2n^2/\delta)}\) for the Hessian).
- After the RHT the weights look like samples from a round, ball-shaped Gaussian. Rounding each weight to a 1-D grid wastes space on the corners of a cube. QuIP# instead rounds groups of 8 weights to the nearest point of a codebook built from the E8 lattice, a very dense packing of points in 8 dimensions. Its E8P codebook has \(2^{16}\) points for 8 weights, so 2 bits per weight, and decoding needs only a 256-entry table (about 1 KiB) plus sign and shift bits.
- It fine-tunes the remaining full-precision parameters after quantization.
Reported result: Llama 2 70B at 2 bits per weight reaches WikiText-2 perplexity 3.91 against 3.12 in FP16 (context 4,096). [Certain] QuIP and QuIP# are weight-only: they make the model small and decoding fast (memory-bound), but the matrix multiplies still run in 16 bits. For 4-bit activations we need QuaRot.
QuaRot: 4-bit weights, activations and KV cache
Where the rotations go
QuaRot (Ashkboos et al., 2024) quantizes every linear layer’s input to 4 bits, and the KV cache too. [Certain] It needs the input of every matrix multiply to be free of outliers. The residual-stream rotation of Algorithm 1 covers the inputs of \(W_q, W_k, W_v, W_{\text{up}}, W_{\text{gate}}\). Two inputs are not on the residual stream:
- The input of \(W_{\text{down}}\) is the SwiGLU output, \(\operatorname{SiLU}(\mathbf{x}W_{\text{gate}}) \odot (\mathbf{x}W_{\text{up}})\). The element-wise product does not commute with a rotation, so no rotation can be folded through it. QuaRot applies an online Hadamard to this vector at runtime and folds \(H^{\top}\) into \(W_{\text{down}}\).
- The input of \(W_o\) is the attention output. A Hadamard per head can be folded into \(W_v\) and \(W_o\) (attention mixes tokens, not channels, so it commutes with a per-head rotation). QuaRot also adds an online Hadamard across heads before \(W_o\).
For the KV cache, keys pass through RoPE, which rotates pairs of channels by a position-dependent angle. A fixed rotation cannot be folded into \(W_k\) through RoPE, so QuaRot applies an online Hadamard to queries and keys after RoPE. Since \(\mathbf{q} H (\mathbf{k} H)^{\top} = \mathbf{q}\mathbf{k}^{\top}\), the attention scores do not change, and the rotated keys are stored in 4 bits. [Certain] Values are already rotated per head through \(W_v\). Figure 4 shows all of them.
How to read Figure 4
Read from top to bottom: the residual stream is the vertical line on the left and carries \(\mathbf{x}R_1\). The top row (attention) reads from it and adds its output back; then the bottom row (MLP) does the same. Click \(R_1\): it is merged into seven kinds of matrices and costs nothing at runtime. Click \(R_4\): it must run online, because SwiGLU’s element-wise product sits between \(W_{\text{up}}\) and \(W_{\text{down}}\). Click \(R_3\): it sits after RoPE and makes the 4-bit KV cache possible. Every blue box receives a rotated input and runs as a 4-bit matrix multiply.
The rest of the recipe and the results
After rotating, QuaRot quantizes weights with GPTQ (per output channel, symmetric), activations per token (symmetric, clipping ratio 0.9) and the KV cache asymmetrically in groups of 128 (clipping ratio 0.95). [Certain] The online Hadamards are run in 16 bits; everything else is a 4-bit matrix multiply. The cost of the online parts is small: for the 11,008-wide MLP of Llama 2 7B, \(H_{64} \otimes H_{172}\) is 66,048 additions plus 1,893,376 multiply-adds per token, about 4.2% of the 45,088,768 multiply-adds of \(W_{\text{down}}\) itself.
Reported results: [Certain]
| Model (WikiText-2 perplexity) | FP16 | QuaRot W4A4KV4 |
|---|---|---|
| Llama 2 7B | 5.47 | 6.10 |
| Llama 2 70B | 3.32 | 3.79 |
For Llama 2 70B the paper also reports 99% of the zero-shot accuracy kept, and that plain round-to-nearest (no calibration data at all) gives lossless 6-bit and 8-bit models once rotated. [Certain] It reports up to 3.33× faster prefill and up to 3.89× less memory for Llama 2 70B with its 4-bit kernels; the exact batch sizes and the memory figure differ between parts of the paper, so treat these as “about 3×”. [Likely]
- input: stream \(\mathbf{h}\) (already rotated by \(Q\)); 4-bit weights with folded rotations and norm scales
- \(\mathbf{a} \leftarrow \mathbf{h} / \operatorname{RMS}(\mathbf{h})\); \(\ \hat{\mathbf{a}} \leftarrow\) quantize per token to INT4 // (109.1); no outliers thanks to \(Q\)
- \(\mathbf{q}, \mathbf{k}, \mathbf{v} \leftarrow \hat{\mathbf{a}} W_{q}, \hat{\mathbf{a}} W_{k}, \hat{\mathbf{a}} W_{v}\) // INT4 GEMMs
- \(\mathbf{q}, \mathbf{k} \leftarrow \operatorname{RoPE}(\mathbf{q}), \operatorname{RoPE}(\mathbf{k})\); \(\ \mathbf{q}, \mathbf{k} \leftarrow \mathbf{q} H, \mathbf{k} H\) per head // online, scores unchanged
- append INT4 \(\mathbf{k}, \mathbf{v}\) to the cache (asymmetric, groups of 128); attention in 16 bits
- \(\mathbf{o} \leftarrow\) online Hadamard across heads, quantize to INT4, \(\ \mathbf{h} \leftarrow \mathbf{h} + \hat{\mathbf{o}} W_o\)
- \(\mathbf{a} \leftarrow \mathbf{h} / \operatorname{RMS}(\mathbf{h})\), quantize; \(\ \mathbf{g} \leftarrow \operatorname{SiLU}(\hat{\mathbf{a}} W_{\text{gate}}) \odot (\hat{\mathbf{a}} W_{\text{up}})\)
- \(\mathbf{g} \leftarrow \operatorname{FWHT}(\mathbf{g})\), quantize to INT4; \(\ \mathbf{h} \leftarrow \mathbf{h} + \hat{\mathbf{g}} W_{\text{down}}\) // \(H^{\top}\) is folded into \(W_{\text{down}}\)
SpinQuant: learn the rotation
Not all rotations are equal
The randomized Hadamard comes with a guarantee, but the guarantee is only a bound. Different random signs give different models. SpinQuant (Liu et al., 2024) tried 100 random rotations on Llama 2 7B with 4-bit weights and activations and found up to 13 points of difference in average zero-shot accuracy between the best and the worst. [Certain] If the choice of rotation matters that much, it is worth optimizing. Figure 5 shows the same effect in two dimensions.
How to read Figure 5
At \(\theta = 0\) (no rotation) the 3-bit error is 10.28%. Press “Sweep \(\theta\)” to animate the rotation from 0° to 90°: watch the points cross grid lines as the cloud turns, and the blue dot trace the error curve. Move the slider: the error goes up and down, between 7.23% (\(\theta = 71.5^\circ\)) and 12.05% (\(\theta = 43^\circ\)). Press “Random rotation” a few times: some random angles are good, some are worse than doing nothing. Press “Find the best angle”: this is what SpinQuant does in thousands of dimensions with gradient steps instead of a search. Switch to 4 bits: the same curve shape, with errors between 1.35% and 2.30%. The shape is ragged because rounding is not smooth; this is why learning a rotation needs a straight-through estimator.
The objective and the Cayley step
SpinQuant keeps the same structure as Figure 4 and makes the mergeable rotations learnable: \(R_1\) on the residual stream and \(R_2\) per head between \(W_v\) and \(W_o\). The online ones, \(R_3\) and \(R_4\), stay fixed Hadamards because they must be fast. [Certain] The loss is the ordinary task loss (cross-entropy) of the network with quantized activations:
- Read the symbols
- \(\mathcal{L}_Q\)
- cross-entropy of the network whose activations (and KV) are quantized after rotation; the weights \(W\) stay frozen in 16 bits during this step
- \(X\)
- the calibration set: 800 samples from WikiText-2 in the paper
- \(\mathcal{M}\)
- the set of orthogonal matrices (the Stiefel manifold); the rotation must stay orthogonal, or the invariance (109.3) breaks
Rounding has zero gradient almost everywhere, so the gradient passes through the quantizer with the straight-through estimator (session 111). A plain gradient step \(R - \eta G\) would leave the set \(\mathcal{M}\). SpinQuant uses Cayley SGD, which moves along the set: [Certain] build a skew-symmetric matrix from the gradient \(G = \partial \mathcal{L}/\partial R\) and apply its Cayley transform:
- Read the symbols
- \(Y\)
- skew-symmetric (\(Y^{\top} = -Y\)); for small \(\eta\) the step moves \(R\) by \(\eta Y R = \eta (R G^{\top} R - G)\), the negative gradient projected onto the directions that keep \(R\) orthogonal
- \(\eta\)
- learning rate; SpinQuant starts at 1.5 and decays linearly to 0 over 100 iterations [Certain]
- Cayley transform
- for any skew-symmetric \(Y\), \((I - aY)^{-1}(I + aY)\) is orthogonal, so \(R\) stays orthogonal at every step (exercise 5)
This is the standard form of a Cayley step; the paper uses an efficient iterative version of it. [Likely] After the rotations are learned, SpinQuant merges them into the weights and quantizes the weights with GPTQ. [Certain] The learning is cheap: about 25 minutes for Llama 2 7B and about 3.5 hours for Llama 2 70B in the paper. [Certain] For Llama 2 7B, \(R_1\) is \(4096 \times 4096\) and there are 32 per-layer \(R_2\) of \(128 \times 128\): 17,301,504 parameters, 0.26% of the model, and all of them disappear into the weights afterwards.
Reported results: with 4-bit weights, activations and KV cache, SpinQuant brings the zero-shot gap to full precision on Llama 2 7B down to 2.9 points; it beats LLM-QAT by 19.1 and SmoothQuant by 25.0 points, and on Llama 3 8B it shrinks QuaRot’s gap by up to 45.1% (relative). [Certain]
Runnable code
The script is deterministic (fixed seed) and runs on a CPU in a few seconds. Part 1 checks the Hadamard matrix and the FWHT. Part 2 checks computational invariance through RMSNorm. Part 3 measures the INT4 error with and without the rotation. Part 4 learns a rotation from the identity with Cayley steps (109.12). It uses a smooth proxy loss, the kurtosis of each token (large when a vector has spikes), instead of SpinQuant’s cross-entropy with a straight-through estimator, so the loop is short and stable. The printed values are in the comments.
import torch torch.manual_seed(0) def hadamard(n): # Sylvester construction, formula (109.5); n a power of 2 H = torch.ones(1, 1) while H.shape[0] < n: H = torch.cat([torch.cat([H, H], 1), torch.cat([H, -H], 1)], 0) return H / n ** 0.5 # orthonormal: H @ H.T = I def fwht(x): # fast Walsh-Hadamard transform on the last dim, O(n log n) n, h, x = x.shape[-1], 1, x.clone() while h < n: x = x.view(*x.shape[:-1], n // (2 * h), 2, h) a, b = x[..., 0, :], x[..., 1, :] x = torch.stack([a + b, a - b], -2).view(*x.shape[:-3], n) h *= 2 return x / n ** 0.5 def quant(x, bits=4): # symmetric per-token round-to-nearest, formula (109.1) qm = 2 ** (bits - 1) - 1 d = x.abs().amax(-1, keepdim=True) / qm return (x / d).round().clamp(-qm, qm) * d # 1) the Hadamard matrix and the fast transform agree n, m, T = 256, 64, 32 H = hadamard(n) x = torch.randn(T, n) print(torch.allclose(H @ H.T, torch.eye(n), atol=1e-5), torch.allclose(fwht(x), x @ H, atol=1e-4)) # True True # 2) computational invariance through RMSNorm, formulas (109.3) and (109.8) x[:, 7] *= 40; x[:, 100] *= 25 # two outlier channels, as in real LLMs alpha = torch.rand(n) + 0.5 # RMSNorm scale W = torch.randn(n, m) / n ** 0.5 # the next linear layer rms = lambda v: v / v.pow(2).mean(-1, keepdim=True).sqrt() y = (rms(x) * alpha) @ W # original block D = torch.diag(torch.randint(0, 2, (n,)).float() * 2 - 1) Q = D @ H # randomized Hadamard, formula (109.6) W_rot = Q.T @ (alpha[:, None] * W) # fold alpha, then absorb the rotation y_rot = rms(x @ Q) @ W_rot # the residual stream now carries x Q print(torch.allclose(y, y_rot, atol=1e-4)) # True # 3) INT4 activations: the rotation removes the outliers a, a_r = rms(x), rms(x @ Q) peak = lambda v: (v.abs().amax(-1) / v.pow(2).mean(-1).sqrt()).mean().item() err = lambda out: ((out - y).norm() / y.norm()).item() print("max/RMS %.1f -> %.1f" % (peak(a), peak(a_r))) print("INT4 error %.3f -> %.3f" % (err(quant(a) @ (alpha[:, None] * W)), err(quant(a_r) @ W_rot))) # max/RMS 12.4 -> 2.2 # INT4 error 0.264 -> 0.079 # 4) learn a rotation with Cayley steps, formula (109.12); smooth proxy loss = kurtosis k, I = 64, torch.eye(64) z0 = torch.randn(512, k); z0[:, 3] *= 30; z0[:, 40] *= 12 R, lr = I.clone(), 0.02 for step in range(301): R.requires_grad_(True) z = z0 @ R L = (z.pow(4).mean(-1) / z.pow(2).mean(-1).pow(2)).mean() G, = torch.autograd.grad(L, R) with torch.no_grad(): Y = R @ G.T - G @ R.T # skew-symmetric direction R = torch.linalg.solve(I - lr / 2 * Y, (I + lr / 2 * Y) @ R) if step % 150 == 0: print(step, "kurtosis %.2f, max/RMS %.1f, orthogonal %s" % (L.item(), peak(z.detach()), torch.allclose(R @ R.T, I, atol=1e-3))) print("Hadamard: kurtosis %.2f, max/RMS %.1f" % (((z0 @ hadamard(k)).pow(4).mean(-1) / (z0 @ hadamard(k)).pow(2).mean(-1).pow(2)).mean().item(), peak(z0 @ hadamard(k)))) # 0 kurtosis 37.09, max/RMS 6.6, orthogonal True # 150 kurtosis 1.81, max/RMS 1.9, orthogonal True # 300 kurtosis 1.77, max/RMS 1.9, orthogonal True # Hadamard: kurtosis 1.85, max/RMS 1.9
Part 3 is the session in two numbers: the same layer, the same 4 bits, and the output error falls from 26.4% to 7.9% only because of an exact rotation. Part 4 starts from “no rotation” (max-to-RMS 6.6) and finds a rotation as flat as the Hadamard one (1.9), with \(R\) orthogonal at every step.
Choosing a method
| Method | What is quantized | Rotation | Runtime cost | Needs data |
|---|---|---|---|---|
| QuIP | weights only (down to 2 bits) | random orthogonal (Kronecker of two) | rotations on input and output of each layer | yes (Hessian) |
| QuIP# | weights only, E8 lattice codebook | randomized Hadamard | FWHT per layer plus codebook lookup | yes, plus fine-tuning |
| QuaRot | weights, activations, KV cache (4 bits) | randomized Hadamard, merged plus online | a few FWHTs per layer | only for GPTQ; RTN works at 6 to 8 bits |
| SpinQuant | weights, activations, KV cache (4 bits) | learned \(R_1, R_2\), Hadamard \(R_3, R_4\) | same as QuaRot | yes: 800 samples to learn, then GPTQ |
A simple rule: for weight-only compression to 2 or 3 bits, use the QuIP# family. For 4-bit weights and activations on hardware with INT4 or FP4 matrix units, use QuaRot, and SpinQuant if you can spend an hour of calibration for better accuracy. At 8 bits, SmoothQuant is usually enough. [Likely]
Where this is used
| Application | Why rotations matter | Link |
|---|---|---|
| On-device LLMs | Meta shipped quantized Llama 3.2 1B and 3B with SpinQuant as one of two methods, for phone CPUs. [Certain] | session 140 |
| 4-bit prefill-heavy serving | Prefill is compute-bound, so only 4-bit activations (QuaRot, SpinQuant) make the matrix units faster; weight-only methods do not. | session 104 |
| Long-context and many-user decoding | The rotation after RoPE (\(R_3\)) makes a 4-bit KV cache accurate, which fits about 3.8 times more tokens than 16 bits (4 bits plus one 16-bit scale and zero point per group of 128). | session 110 |
| Very large models on few GPUs | QuIP# at 2 bits per weight stores a 70B model in about 17.5 GB of weights (70B × 2 bits), plus small codebook and scale overheads. | session 106 |
| Video-language models and Video-RAG | A long video gives hundreds of thousands of visual tokens; prefill on them is compute-bound, and their KV cache dominates memory. W4A4KV4 cuts both. Vision encoders also show outlier channels. [Likely] | session 197 |
| Video retrieval and fingerprinting at scale | Embedding models run over every frame of a large archive; 4-bit activations raise the frames per second per GPU. Rotation keeps the embeddings close to the 16-bit ones. [Speculation] | session 197 |
| FP4 and microscaling formats | Block formats share one scale per 16 or 32 values, the same max-sets-the-step problem; Hadamard rotations are also used with them. [Likely] | session 105 |
Common mistakes
- Forgetting to fold the RMSNorm scale first. \(\operatorname{diag}(\alpha)\) does not commute with \(Q\). Rotating without folding changes the model output; the logits check in Algorithm 1 catches it.
- Using a non-orthogonal or unnormalized Hadamard. The \(\pm 1\) matrix must be divided by \(\sqrt{n}\). Without it, every rotated weight is scaled by \(\sqrt{n}\) and the outputs are wrong by that factor (or by \(n\) if both sides are unnormalized).
- Trying to merge the rotation before \(W_{\text{down}}\) into \(W_{\text{up}}\). SwiGLU’s element-wise product and SiLU do not commute with a rotation. That rotation must be online.
- Rotating keys before RoPE. RoPE does not commute with a general rotation, so the scores change. Rotate queries and keys after RoPE, with the same matrix.
- Expecting a rotation to help a vector without outliers. If max-to-RMS is already small, rotating gives nothing (Figure 1 with outlier 1); the gain comes only from flattening spikes.
- Using a fixed Hadamard without random signs. The bound (109.6) needs the random \(D\); a structured input can still produce a spike after a fixed \(H\).
- Reporting a QuaRot or SpinQuant number without the settings. Results depend on RTN versus GPTQ, the clipping ratios, group sizes and whether the KV cache is quantized. Compare only like with like.
- Learning the rotation with a plain gradient step. \(R - \eta G\) is not orthogonal, and invariance breaks. Use a Cayley step (109.12) or re-orthogonalize (QR) after each step.
Interview questions
Why does multiplying activations by an orthogonal matrix not change a linear layer’s output?basic
Why do outliers hurt per-token INT4 quantization so much?basic
How does a rotation help, in one sentence, and what is its limit?basic
Why Hadamard matrices rather than random orthogonal matrices?core
Walk through how a whole Llama model is rotated without changing its output.core
Which rotations in QuaRot or SpinQuant must run online, and why?core
What is incoherence in QuIP, and why does it make weight quantization easier?advanced
What does QuIP# add on top of QuIP?advanced
How is SpinQuant different from QuaRot, and what does it cost?core
How do you take a gradient step on a rotation matrix and keep it orthogonal?advanced
Is rotation compatible with SmoothQuant or GPTQ? Would you use both?advanced
Exercises
- Compute \(\mathbf{x}H_4\) for \(\mathbf{x} = (8, 0, 0, 0)\). What are the max-to-RMS ratios before and after?
- Use (109.2) to predict the relative error (not squared) of 4-bit quantization for a vector whose max-to-RMS ratio is 12.4, and for 2.2. Compare the ratio of the two predictions with the INT4 errors printed in part 3 of the code.
- For \(n = 4096\), how many additions does an FWHT need, and how many multiply-adds does a dense \(n \times n\) rotation need? What is the ratio?
- Explain why \(11{,}008 = 2^8 \times 43\) cannot be used for a Kronecker Hadamard, and give a factorization that works.
- Prove that for skew-symmetric \(Y\) and any real \(a\), the Cayley matrix \(C = (I - aY)^{-1}(I + aY)\) is orthogonal.
- How many parameters does SpinQuant learn for Llama 2 7B (hidden size 4,096, 32 layers, head size 128)? What fraction of 6.74B is that? Why is the runtime cost zero?
- Show that rotating queries and keys of one head by the same orthogonal \(H\) after RoPE leaves every attention score unchanged. Why can you not do the same with two different matrices?
Answerscheck
Key takeaways
- Per-token quantization error grows with the max-to-RMS ratio (109.2); one outlier sets the step for all channels and erases the small ones.
- \(\mathbf{x}W = (\mathbf{x}Q)(Q^{\top}W)\) for any orthogonal \(Q\) (109.3): the model is unchanged, but the quantized inputs are flat.
- A Hadamard matrix turns a spike into a flat vector (109.4); random signs make this hold for any input with high probability (109.6), and the FWHT applies it in \(n\log_2 n\) additions.
- Computational invariance rotates the whole residual stream once RMSNorm scales are folded (109.8); only the rotations before \(W_{\text{down}}\) and after RoPE must run online.
- QuIP and QuIP# use rotations for incoherent weights and Hessians (109.9, 109.10) and reach 2-bit weights; QuIP# adds the RHT and an E8 lattice codebook.
- QuaRot gives W4A4KV4 with random Hadamards: Llama 2 70B goes from 3.32 to 3.79 WikiText-2 perplexity.
- SpinQuant learns \(R_1, R_2\) with Cayley SGD (109.12) on 800 samples; random rotations differ by up to 13 points, and learning closes much of the gap.
Key formulas
References for this session
- Chee et al. (2023). QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- Tseng et al. (2024). QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
- Ashkboos et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- Liu et al. (2024). SpinQuant: LLM Quantization with Learned Rotations
- Chee et al. (2023). QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- Tseng et al. (2024). QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
- Ashkboos et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- Liu et al. (2024). SpinQuant: LLM Quantization with Learned Rotations
- Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Ashkboos et al. (2024). SliceGPT: Compress Large Language Models by Deleting Rows and Columns
- Liu et al. (2023). LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Li, Fuxin, Todorovic (2020). Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
- Meta AI (2024). Introducing quantized Llama models with increased speed and a reduced memory footprint