LLM & VLM courseBlog
109Track 5: LLM inference and serving · session 32 of 65

Rotations against outliers: QuIP, QuIP#, QuaRot, SpinQuant

A few activation channels in an LLM are a hundred times larger than the rest, and they ruin 4-bit quantization. Multiply the activations by an orthogonal matrix and the weights by its transpose: the output stays exactly the same, but the outliers are spread over all channels. This session builds that trick from the math up and follows it through QuIP, QuIP#, QuaRot and SpinQuant.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 103, 106 and 108 (quantization math, GPTQ, SmoothQuant); session 22 (RMSNorm and the residual stream)

What you will learn

  • Explain why the quantization error of a vector grows with its max-to-RMS ratio, and why one outlier channel wipes out the small channels at 4 bits
  • Prove computational invariance: rotating activations by Q and weights by Q transpose leaves the output unchanged, also through RMSNorm once its scale is folded
  • Build a Hadamard matrix, run the fast Walsh–Hadamard transform in n log n steps, and say why random signs make the spreading work for any input
  • Describe QuIP and QuIP#: incoherence of weights and Hessians, the proxy loss that rotations do not change, and the E8 lattice codebook
  • Place every QuaRot rotation in a Llama block: which ones merge into the weights and which ones must run online
  • Explain why SpinQuant learns its rotations, how a Cayley step keeps a matrix orthogonal, and what it costs
  • Quote the reported results of the four papers correctly and pick a method for a given deployment

Why this matters

At 8 bits, an LLM can be quantized with almost no loss. At 4 bits for weights and activations it usually breaks. The reason is a handful of outlier channels: a few coordinates of the hidden vector that are tens or hundreds of times larger than the others. This session shows a simple and exact cure. Multiply the activations by an orthogonal matrix \(Q\) and the next weight matrix by \(Q^{\top}\). The model computes the same function, but each outlier is now spread over all channels, and 4-bit quantization works.

Session 108 met the outliers and handled them in two ways: LLM.int8() keeps the outlier channels in 16 bits, and SmoothQuant moves part of their size from the activations into the weights with a per-channel scale. Both are diagonal fixes: they treat each channel on its own. A rotation is a full matrix. It mixes all channels, so one large channel becomes many medium ones. New in this session:

  • Why the error of round-to-nearest grows with the max-to-RMS ratio of a vector, and what an outlier does to the small channels (Figure 1).
  • Computational invariance: the proof that \(xW = (xQ)(Q^{\top}W)\) also holds through RMSNorm, so a whole model can be rotated without retraining (Figure 2 checks the one-layer case number by number).
  • The Hadamard matrix and the fast Walsh–Hadamard transform, which applies a rotation in \(n \log_2 n\) additions (Figure 3).
  • QuIP and QuIP#: rotations for 2-bit weights, with a proof that the GPTQ-style proxy loss does not change.
  • QuaRot: where each rotation goes in a Llama block, and which ones cost time at inference (Figure 4).
  • SpinQuant: why some rotations are better than others (Figure 5) and how to learn one while keeping it orthogonal.

This is part of the post-training quantization section (5.2.1) of the efficient-inference survey Track 5. [Certain] Rotations are now a standard part of 4-bit inference: Meta shipped its quantized Llama 3.2 1B and 3B models with SpinQuant as one of two methods. [Certain] In interviews you hear: “Why does rotating a model not change its output?”, “Why a Hadamard matrix and not any orthogonal matrix?”, “Which rotations can be merged into weights and which cannot?”, “How is SpinQuant different from QuaRot?”. All of them come down to the two facts in the title of the next section.

The problem: one outlier sets the step for everyone

Round-to-nearest with one scale per token

Session 103 defined symmetric quantization. For activations we use one scale per token (per row of the activation matrix), because it can be computed on the fly. For a vector \(\mathbf{x} \in \R^n\) and \(b\) bits:

\[ \hat{\mathbf{x}} = \Delta \cdot \operatorname{clamp}\!\left(\operatorname{round}\!\left(\frac{\mathbf{x}}{\Delta}\right), -q_{\max}, q_{\max}\right), \qquad \Delta = \frac{\max_i |x_i|}{q_{\max}}, \quad q_{\max} = 2^{b-1} - 1 \tag{109.1} \]
Read the symbols
\(\mathbf{x}\)
one token's activation vector, the input of a linear layer, with \(n\) channels
\(\hat{\mathbf{x}}\)
its quantized version: every entry is an integer times \(\Delta\)
\(\Delta\)
the step size: the distance between two neighbouring quantization levels
\(q_{\max}\)
the largest integer used: 7 for 4 bits, 3 for 3 bits, 127 for 8 bits

The step \(\Delta\) is set by the largest entry. Every entry is rounded to the nearest multiple of \(\Delta\), so each one gets an error of at most \(\Delta / 2\). If we model the rounding error as uniform on \([-\Delta/2, \Delta/2]\) (a standard approximation), its variance is \(\Delta^2 / 12\) per entry. Divide the total by \(\|\mathbf{x}\|^2 = n \cdot \operatorname{RMS}(\mathbf{x})^2\):

\[ \frac{\E\,\|\hat{\mathbf{x}} - \mathbf{x}\|^2}{\|\mathbf{x}\|^2} \approx \frac{n \Delta^2 / 12}{n \operatorname{RMS}(\mathbf{x})^2} = \frac{1}{12\, q_{\max}^2} \left(\frac{\max_i |x_i|}{\operatorname{RMS}(\mathbf{x})}\right)^{2} \tag{109.2} \]
Read the symbols
\(\operatorname{RMS}(\mathbf{x})\)
root mean square: \(\sqrt{\tfrac{1}{n}\sum_i x_i^2}\), the typical size of an entry
\(\max_i |x_i| / \operatorname{RMS}(\mathbf{x})\)
the max-to-RMS ratio (also called the crest factor). It is 1 when all entries have the same size and \(\sqrt{n}\) when one entry holds all the energy

So the relative error does not depend on the bits alone. It depends on the bits and on the shape of the vector. A vector with a flat profile is easy; a vector with one spike is hard. LLM activations have spikes: a few fixed channels (the same channels for almost every token) are much larger than the rest. [Certain, observed by LLM.int8() and SmoothQuant]

There is a second, worse effect. When \(\Delta\) is larger than twice a small entry, that entry rounds to 0. Its information is gone, not just noisy. At 4 bits an outlier 20 times larger than the typical entry already gives \(\Delta \approx 20/7\) times the typical entry, so most channels become 0. Figure 1 shows this.

The fix in one line: rotate

Take any orthogonal matrix \(Q\) (\(Q Q^{\top} = I\)). For a linear layer \(\mathbf{y} = \mathbf{x} W\):

\[ \mathbf{y} = \mathbf{x} W = \mathbf{x} Q Q^{\top} W = (\mathbf{x} Q)\,(Q^{\top} W) \tag{109.3} \]
Read the symbols
\(W\)
the weight matrix, \(n \times m\); we use row vectors, so \(\mathbf{x}\) is \(1 \times n\)
\(Q\)
an \(n \times n\) orthogonal matrix: it keeps lengths, \(\|\mathbf{x} Q\| = \|\mathbf{x}\|\)
\(\mathbf{x} Q,\ Q^{\top} W\)
the rotated activation and the rotated weight; we quantize these instead of \(\mathbf{x}\) and \(W\)

Without quantization the output is exactly the same. With quantization we now quantize \(\mathbf{x}Q\), and we can choose \(Q\) so that \(\mathbf{x}Q\) has no spikes. The weight \(Q^{\top}W\) is computed once, offline. This is the whole idea of the session. The rest is: which \(Q\), how to apply it cheaply, and how to fit it into a real transformer.

Why does a rotation spread a spike? Look at the extreme case: all energy in channel \(i\), \(\mathbf{x} = a\,\mathbf{e}_i\). Then \(\mathbf{x}Q\) is \(a\) times row \(i\) of \(Q\). If every entry of \(Q\) has size \(1/\sqrt{n}\) (this is the Hadamard matrix of the next section):

\[ |(\mathbf{x} Q)_j| = \frac{|a|}{\sqrt{n}} \ \text{ for all } j, \qquad \frac{\max_j |x_j|}{\operatorname{RMS}(\mathbf{x})} = \sqrt{n} \ \longrightarrow\ \frac{\max_j |(\mathbf{x}Q)_j|}{\operatorname{RMS}(\mathbf{x}Q)} = 1 \tag{109.4} \]

The worst possible vector becomes the best possible one. For \(n = 4096\), an outlier of size 100 becomes 4,096 entries of size \(100/64 = 1.5625\). A real activation is a spike plus many normal entries; the rotation turns it into normal entries plus a small flat offset, which quantizes well.

value that is quantizedoutlier channelquantization levels
Figure 1. One token with 16 channels: 15 normal values and one outlier in channel 5. The bars are the values that get quantized, either \(\mathbf{x}\) itself or \(\mathbf{x}DH\) (random signs, then a Hadamard matrix). The thin lines are the quantization levels of formula (109.1). The readout gives the max-to-RMS ratio, the step \(\Delta\), and the error after quantizing and rotating back, for the whole vector and for the 15 normal channels only.
How to read Figure 1

Start with “No rotation”, 4 bits, outlier 20. The step is 2.857, larger than twice every normal value, so all 15 normal channels round to 0: their error is 100%. Switch to “Randomized Hadamard”. The bars are now all of similar height (max-to-RMS falls from 3.95 to 1.27), the step falls to 0.918, the total error falls from 15.5% to 4.6% and the normal channels keep most of their information (25.1% error instead of 100%). Move the outlier to 1: there is no outlier, and the rotation gives no gain (8.0% against 7.7% total error). Move it to 50 with 4 bits: rotation still cuts the total error (6.3% to 4.3%), but the normal channels now have 67.5% error. With only 16 channels the spike is too big to hide; the readout shows why the rotation works better in real models with thousands of channels.

The last observation in the box is important. A rotation does not create information. It spreads the quantization error evenly over all channels, so every channel gets an error of about \(\Delta_{\text{rot}}/\sqrt{12}\). In Figure 1 the error after rotating is the same size in every channel, and the small channels get a large relative error when the outlier dominates the norm. In a real model, \(n\) is 4,096 or more, so the outlier is divided by \(\sqrt{n} = 64\), not by 4, and the normal channels survive much better.

Worked example: 3-bit quantization of a 4-channel vector

Take \(\mathbf{x} = (12, 1, -2, 1)\) and \(b = 3\), so \(q_{\max} = 3\). Here \(\|\mathbf{x}\|^2 = 150\) and \(\operatorname{RMS} = \sqrt{150/4} = 6.124\). The max-to-RMS ratio is \(12 / 6.124 = 1.96\).

Without rotation. \(\Delta = 12/3 = 4\). Divide: \((3, 0.25, -0.5, 0.25)\); round: \((3, 0, 0, 0)\). So \(\hat{\mathbf{x}} = (12, 0, 0, 0)\). The three small channels are gone. Squared error \(= 0 + 1 + 4 + 1 = 6\), relative error \(\sqrt{6/150} = 0.200\).

With the 4 × 4 Hadamard matrix \(H_4\) (rows \((1,1,1,1), (1,-1,1,-1), (1,1,-1,-1), (1,-1,-1,1)\), divided by 2): \(\mathbf{z} = \mathbf{x} H_4 = (6, 4, 7, 7)\). Its max-to-RMS ratio is \(7/6.124 = 1.14\). Now \(\Delta = 7/3 = 2.333\); \(\mathbf{z}/\Delta = (2.571, 1.714, 3, 3)\) rounds to \((3, 2, 3, 3)\), so \(\hat{\mathbf{z}} = (7, 4.667, 7, 7)\). Rotate back with \(H_4^{\top}\): \((12.833, 1.167, -1.167, 1.167)\). The errors are \((0.833, 0.167, 0.833, 0.167)\), squared error \(1.444\), relative error \(\sqrt{1.444/150} = 0.098\).

The rotation halved the error, and the small channels now have the right sign and roughly the right size instead of being 0.

The worked example only looked at the activation. Figure 2 sends the same vector through a small linear layer \(W\) (4 inputs, 3 outputs) and computes both sides of (109.3) one dot product at a time. First the original layer \(\mathbf{y} = \mathbf{x}W\). Then the rotated path: \(\mathbf{z} = \mathbf{x}Q\), the folded weight \(W' = Q^{\top}W\), and \(\mathbf{y}' = \mathbf{z}W'\). Every \(y'_j\) comes out equal to \(y_j\). Only in the last steps do we quantize the activation (the weights stay exact here, so that we see the effect of the activation alone), and there the two paths separate.

positive entrynegative entryentries used in this stepoutput error, no rotationoutput error, rotated
  1. input: activation \(\mathbf{x}\) (1 × 4), weight \(W\) (4 × 3), orthogonal \(Q\) (4 × 4)
  2. for \(j = 1..3\): \(y_j \leftarrow \sum_i x_i W_{ij}\) // the original layer
  3. for \(j = 1..4\): \(z_j \leftarrow \sum_i x_i Q_{ij}\) // rotate the activation: \(\mathbf{z} = \mathbf{x}Q\)
  4. \(W' \leftarrow Q^{\top} W\) // offline, once per layer
  5. for \(j = 1..3\): \(y'_j \leftarrow \sum_i z_i W'_{ij}\) // equals \(y_j\), by (109.3)
  6. \(\hat{\mathbf{x}} \leftarrow \operatorname{quant}_b(\mathbf{x})\); \(\ \hat{\mathbf{y}} \leftarrow \hat{\mathbf{x}} W\) // (109.1), no rotation
  7. \(\hat{\mathbf{z}} \leftarrow \operatorname{quant}_b(\mathbf{z})\); \(\ \hat{\mathbf{y}}' \leftarrow \hat{\mathbf{z}} W'\) // (109.1), rotated
  8. compare \(\|\hat{\mathbf{y}} - \mathbf{y}\| / \|\mathbf{y}\|\) with \(\|\hat{\mathbf{y}}' - \mathbf{y}\| / \|\mathbf{y}\|\)
Figure 2. Formula (109.3) computed in the browser on the worked example, \(\mathbf{x} = (x_1, 1, -2, 1)\) with \(x_1 = 12\), and a fixed 4 × 3 weight \(W\). Left panel (top on a phone): the original layer. Right panel: the rotated layer, with \(\mathbf{z} = \mathbf{x}Q\) and \(W' = Q^{\top}W\). Below them: the rotation \(Q\) and the error of each output after the activation is quantized with one scale per token, formula (109.1). \(D = \operatorname{diag}(1, -1, 1, 1)\) flips the sign of channel 2.
How to read Figure 2

Press Play, or Step through it. The highlighted line of the algorithm box is the one that runs, and the box next to it shows the same sum with the real numbers. The cells used in the sum get a thick outline. Steps 2 to 4 give \(\mathbf{y} = (8.25, -14, 1.5)\). Steps 5 to 8 give \(\mathbf{z} = (6, 4, 7, 7)\) for \(Q = H_4\): the outlier 12 is now spread over all four entries. Steps 10 to 12 give \(\mathbf{y}'\), equal to \(\mathbf{y}\) entry by entry. With 3 bits, quantizing \(\mathbf{x}\) erases channels 2 to 4 and the output error is 20.6%; quantizing \(\mathbf{z}\) gives 7.0%. With \(Q = D H_4\) the rotated error is 6.6%. With \(Q = I\) both paths are the same. Switch to 4 bits with \(Q = H_4\): \(\mathbf{z}/\Delta\) is exactly \((6, 4, 7, 7)\), so the rotated error is 0, against 6.3% without rotation. That is a lucky integer case; the next figure uses less tidy numbers.

Hadamard matrices and the fast transform

Why Hadamard

Any orthogonal \(Q\) keeps the output exact. We want three more things: every entry of \(Q\) should be small (\(\pm 1/\sqrt{n}\) is the smallest possible, because each row has unit length), applying \(Q\) at runtime should be cheap, and it should work for every input, not only for one-hot spikes. A Hadamard matrix has entries \(\pm 1\), rows that are orthogonal to each other, and (scaled by \(1/\sqrt{n}\)) it is orthogonal. The Sylvester construction builds one for every power of 2:

\[ H_1 = [1], \qquad H_{2n} = \frac{1}{\sqrt{2}} \begin{bmatrix} H_n & H_n \\ H_n & -H_n \end{bmatrix} \tag{109.5} \]
Read the symbols
\(H_n\)
the \(n \times n\) normalized Hadamard matrix; every entry is \(\pm 1/\sqrt{n}\) and \(H_n H_n^{\top} = I\)
block form
the recursion is what makes the fast transform possible: one level of the recursion is one stage of sums and differences

Because of the block form, \(\mathbf{x} H_n\) can be computed in \(\log_2 n\) stages. Each stage takes pairs of entries \((u, v)\) and replaces them by \((u + v, u - v)\). That is the fast Walsh–Hadamard transform (FWHT): \(n \log_2 n\) additions and one scaling, instead of \(n^2\) multiply-adds for a dense matrix. For \(n = 4096\): 49,152 additions against 16,777,216 multiply-adds, about 341 times fewer. No weights are stored, because the matrix is fixed.

Figure 3 runs the whole pipeline on 8 channels, one butterfly per step: optional random signs, the three stages, the scaling, then 4-bit quantization of the rotated vector and the rotation back. At the end it quantizes the same \(\mathbf{x}\) without any rotation and compares the two errors channel by channel.

contains part of the outlier \(x_0\)sum \(u + v\) (solid)difference \(u - v\) (dashed)error, no rotationerror, rotated
  1. input: \(\mathbf{x} \in \R^n\) with \(n = 8\), signs \(\mathbf{s} \in \{\pm 1\}^n\), bits \(b\)
  2. \(\mathbf{y} \leftarrow \mathbf{x} \odot \mathbf{s}\) // the \(D\) in \(Q = DH\)
  3. for \(h = 1, 2, 4\): // one stage per level of (109.5)
  4. for each pair \((j, j + h)\) with \(j \bmod 2h \lt h\):
  5. \(u, v \leftarrow y_j, y_{j+h}\); \(\ y_j \leftarrow u + v\); \(\ y_{j+h} \leftarrow u - v\) // one butterfly
  6. \(\mathbf{z} \leftarrow \mathbf{y} / \sqrt{n}\) // now \(\mathbf{z} = \mathbf{x} D H_8\)
  7. \(\Delta \leftarrow \max_j |z_j| / q_{\max}\); \(\ \hat{\mathbf{z}} \leftarrow \Delta \cdot \operatorname{round}(\mathbf{z}/\Delta)\) // (109.1)
  8. \(\hat{\mathbf{x}} \leftarrow \operatorname{FWHT}(\hat{\mathbf{z}}) / \sqrt{n} \odot \mathbf{s}\) // rotate back: \((DH)^{\top} = HD\)
  9. compare with \(\operatorname{quant}_b(\mathbf{x})\), quantized without rotation
Figure 3. The randomized Hadamard transform of an 8-channel vector with an outlier in channel 0, computed in the browser with the fast transform. Each stage pairs entries that are \(h = 1, 2, 4\) apart and writes their sum and difference; orange cells already contain a part of the outlier. After the three stages, one division by \(\sqrt{8}\) gives \(\mathbf{z} = \mathbf{x}DH_8\). The last three columns quantize \(\mathbf{z}\) with formula (109.1), rotate it back, and compare each channel’s error with plain quantization of \(\mathbf{x}\). Under each column: its max-to-RMS ratio. The random signs \(D\) are explained in the next subsection.
How to read Figure 3

Press Play or Step. Step 1 shows the input: one value of 10 and seven values near 1 (max-to-RMS 2.77). Each step of a stage runs one butterfly, and the box on the right shows \(u\), \(v\), \(u + v\) and \(u - v\) with the real numbers. Watch the orange cells: the outlier reaches 2 cells after stage 1, 4 after stage 2 and all 8 after stage 3. Dividing by \(\sqrt{8} = 2.828\) gives values between 2.33 and 4.67 (max-to-RMS 1.29), and the length of the vector is unchanged. The whole transform used \(8 \times 3 = 24\) additions. With 4 bits the error after rotating back is 6.5%, against 12.1% for plain quantization, where three small channels round to 0. Turn on the random signs: the rotated error falls to 3.1%. Set the outlier to 30 with 3 bits and no signs: now the rotation is worse (11.4% against 7.1%). With only 8 channels a big spike stays big after spreading (\(30/\sqrt{8} = 10.6\) in every channel) and the small values are lost in the step; real layers have thousands of channels.

Random signs: a guarantee for every input

A fixed \(H\) has one weakness: if the input happens to be a row of \(H\), the output is a spike again (because \(H\) maps its own rows to one-hot vectors). The fix is to flip the sign of each channel at random first. With a diagonal matrix \(D\) of random \(\pm 1\):

\[ Q = D H_n, \qquad \Pr\!\left[\max_j |(\mathbf{x} Q)_j| \gt \|\mathbf{x}\| \sqrt{\frac{2 \ln(2n/\delta)}{n}}\right] \le \delta \tag{109.6} \]
Read the symbols
\(D\)
diagonal matrix with independent random signs \(\pm 1\); \(DH_n\) is the randomized Hadamard transform (RHT)
\(\delta\)
the failure probability we accept, for example 0.01
\(\|\mathbf{x}\| / \sqrt{n}\)
the RMS of \(\mathbf{x}\) (and of \(\mathbf{x}Q\)); so the bound says max-to-RMS is at most \(\sqrt{2 \ln(2n/\delta)}\)

Why it holds: entry \(j\) of \(\mathbf{x}DH\) is \(\sum_i \pm x_i / \sqrt{n}\) with independent random signs. Hoeffding’s inequality bounds each entry, and a union bound over the \(n\) entries gives (109.6). For \(n = 4096\) and \(\delta = 0.01\) the bound is \(0.0815\,\|\mathbf{x}\|\), while a pure spike has max \(\|\mathbf{x}\|\). The max-to-RMS ratio is at most 5.2 for any input, against 64 for the worst unrotated input. QuIP# proves this kind of bound (it calls it incoherence, next section), and QuaRot uses the same randomized Hadamard. [Certain]

Sizes that are not powers of 2

Real layer sizes are not always powers of 2. Llama 2 7B has an MLP width of 11,008; Llama 3 8B has 14,336. [Certain] Hadamard matrices also exist for many sizes that are multiples of 4 (for example 12, 20, 28, 172). QuaRot and QuIP# combine them with the Kronecker product: [Certain]

\[ H_{2^k m} = H_{2^k} \otimes H_m, \qquad 11{,}008 = 64 \times 172, \qquad 14{,}336 = 512 \times 28 \tag{109.7} \]

The \(H_{2^k}\) part runs as an FWHT; the small \(H_m\) part is a dense \(m \times m\) multiply. Note that \(11{,}008 = 2^8 \times 43\) does not work: a Hadamard matrix of size 43 cannot exist, because sizes above 2 must be multiples of 4. Which factorization each code base uses is an implementation detail. [Likely]

Rotating a whole transformer: computational invariance

The residual stream

Formula (109.3) is for one layer. In a transformer, the same hidden vector \(\mathbf{x}\) travels along the residual stream (session 22) and is read by many layers. QuaRot uses an idea from SliceGPT, called computational invariance: rotate the residual stream itself. [Certain] Multiply the embedding matrix by \(Q\), so every token enters as \(\mathbf{x}Q\). Every layer that reads the stream gets \(Q^{\top}\) on its input side (\(W_q, W_k, W_v, W_{\text{up}}, W_{\text{gate}}\) and the LM head). Every layer that writes to the stream gets \(Q\) on its output side (\(W_o\), \(W_{\text{down}}\)). Then the stream carries \(\mathbf{x}Q\) everywhere, and each residual addition \(\mathbf{x}Q + \mathbf{f}Q = (\mathbf{x} + \mathbf{f})Q\) stays consistent.

RMSNorm

One thing stands between the layers: RMSNorm, \(\operatorname{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\operatorname{RMS}(\mathbf{x})} \odot \alpha\) (here \(\alpha \in \R^n\) is a vector of learned scales, applied element by element). The division commutes with \(Q\), because a rotation keeps the length. The element-wise scale does not commute, so we first fold it into the next weight matrix:

\[ \frac{\mathbf{x} Q}{\operatorname{RMS}(\mathbf{x} Q)} = \frac{\mathbf{x}}{\operatorname{RMS}(\mathbf{x})}\, Q, \qquad W' = Q^{\top} \operatorname{diag}(\alpha)\, W \tag{109.8} \]
Read the symbols
\(\alpha\)
the learned per-channel scale of RMSNorm; after folding, the norm has no scale (all ones)
\(W'\)
the new weight of the layer after the norm: scale first, rotation second

Check: \(\frac{\mathbf{x}Q}{\operatorname{RMS}(\mathbf{x}Q)} W' = \frac{\mathbf{x}}{\operatorname{RMS}(\mathbf{x})} Q Q^{\top} \operatorname{diag}(\alpha) W = \operatorname{RMSNorm}(\mathbf{x})\, W\). The PyTorch script below confirms this numerically. For LayerNorm models, the mean subtraction is also a linear map, and SliceGPT shows how to turn LayerNorm into RMSNorm first. [Certain] After this, the whole network computes exactly the same function in full precision, but every matrix that reads the residual stream now sees \(\mathbf{x}Q\) without outliers.

Algorithm 1: rotate a Llama model offline (computational invariance)
  1. input: model weights; hidden size \(d\); a random sign seed
  2. \(Q \leftarrow D H_d\) // randomized Hadamard, (109.6); SpinQuant would use a learned \(R_1\) here
  3. for each RMSNorm: fold \(\alpha\) into the matrices that read its output, set \(\alpha \leftarrow \mathbf{1}\) // (109.8)
  4. \(E \leftarrow E Q\) // embedding rows enter the stream rotated
  5. for each layer: \(W_q, W_k, W_v, W_{\text{up}}, W_{\text{gate}} \leftarrow Q^{\top} W\); \(\ W_o, W_{\text{down}} \leftarrow W Q\)
  6. \(W_{\text{head}} \leftarrow Q^{\top} W_{\text{head}}\) // the LM head reads the final stream
  7. check on a few prompts that the logits match the original model to float tolerance
  8. quantize the rotated weights (RTN or GPTQ) // the full-precision model is unchanged; only quantization improves

QuIP and QuIP#: rotations for 2-bit weights

The proxy loss does not see the rotation

The rotation idea first appeared for weight-only quantization, in QuIP (Chee et al., 2023). Session 106 showed that GPTQ quantizes a layer by minimizing a proxy loss: the output error on calibration inputs. With \(H = \E[\mathbf{x}^{\top}\mathbf{x}]\), the \(n \times n\) second-moment matrix of the inputs (the “Hessian” of the proxy), the loss and its behaviour under rotations are:

\[ \ell(\hat{W}) = \E\,\|\mathbf{x}(\hat{W} - W)\|^2 = \operatorname{tr}\!\left((\hat{W} - W)^{\top} H (\hat{W} - W)\right), \qquad \tilde{W} = V^{\top} W U, \quad \tilde{H} = V^{\top} H V \tag{109.9} \]
Read the symbols
\(\hat{W}\)
the quantized weight; \(\ell\) measures how much the layer output changes
\(H\)
the input second-moment matrix, estimated from calibration data, as in GPTQ
\(U, V\)
random orthogonal matrices on the output side and the input side
\(\tilde{W}, \tilde{H}\)
the rotated problem: quantize \(\tilde{W}\) with the rotated inputs \(\mathbf{x}V\), then undo \(U\) on the output

If \(\hat{\tilde{W}} - \tilde{W} = V^{\top}(\hat{W} - W)U\), the loss is the same: \(\operatorname{tr}(U^{\top}(\hat{W} - W)^{\top} V V^{\top} H V V^{\top}(\hat{W} - W) U) = \ell(\hat{W})\) because \(VV^{\top} = I\) and the trace is unchanged by \(U\). So we may solve the rotated problem instead, and choose \(U, V\) to make it easy. QuIP calls a problem easy when it is incoherent: [Certain]

\[ \max_{ij} |W_{ij}| \le \mu \frac{\|W\|_F}{\sqrt{mn}}, \qquad \max_{ij} |(\text{eigenvectors of } H)_{ij}| \le \frac{\mu}{\sqrt{n}} \tag{109.10} \]
Read the symbols
\(\mu\)
the incoherence parameter; small \(\mu\) means no entry is much larger than average
\(\|W\|_F / \sqrt{mn}\)
the RMS of the entries of \(W\); the first condition is “max-to-RMS of \(W\) is at most \(\mu\)”, the matrix version of the ratio in (109.2)
eigenvector condition
the directions in which the inputs vary most are spread over all coordinates, so no single weight is much more important than the others

The first condition makes the weights easy to round with a small grid. The second makes rounding errors in different coordinates matter about equally. QuIP pairs incoherence processing with an adaptive rounding method (LDLQ) and proves that it is equivalent to OPTQ, the method behind GPTQ. It was the first method to give usable LLMs at 2 bits per weight. [Certain] Its \(U\) and \(V\) were Kronecker products of two smaller random orthogonal matrices, to make them cheap to apply. [Certain]

QuIP#: Hadamard plus a lattice codebook

QuIP# (Tseng et al., 2024) makes three changes: [Certain]

  • It replaces the Kronecker product with the randomized Hadamard transform \(DH\), which is faster and has better incoherence bounds (with probability \(1 - \delta\), \(\mu_H = \sqrt{2 \ln(2n^2/\delta)}\) for the Hessian).
  • After the RHT the weights look like samples from a round, ball-shaped Gaussian. Rounding each weight to a 1-D grid wastes space on the corners of a cube. QuIP# instead rounds groups of 8 weights to the nearest point of a codebook built from the E8 lattice, a very dense packing of points in 8 dimensions. Its E8P codebook has \(2^{16}\) points for 8 weights, so 2 bits per weight, and decoding needs only a 256-entry table (about 1 KiB) plus sign and shift bits.
  • It fine-tunes the remaining full-precision parameters after quantization.

Reported result: Llama 2 70B at 2 bits per weight reaches WikiText-2 perplexity 3.91 against 3.12 in FP16 (context 4,096). [Certain] QuIP and QuIP# are weight-only: they make the model small and decoding fast (memory-bound), but the matrix multiplies still run in 16 bits. For 4-bit activations we need QuaRot.

QuaRot: 4-bit weights, activations and KV cache

Where the rotations go

QuaRot (Ashkboos et al., 2024) quantizes every linear layer’s input to 4 bits, and the KV cache too. [Certain] It needs the input of every matrix multiply to be free of outliers. The residual-stream rotation of Algorithm 1 covers the inputs of \(W_q, W_k, W_v, W_{\text{up}}, W_{\text{gate}}\). Two inputs are not on the residual stream:

  • The input of \(W_{\text{down}}\) is the SwiGLU output, \(\operatorname{SiLU}(\mathbf{x}W_{\text{gate}}) \odot (\mathbf{x}W_{\text{up}})\). The element-wise product does not commute with a rotation, so no rotation can be folded through it. QuaRot applies an online Hadamard to this vector at runtime and folds \(H^{\top}\) into \(W_{\text{down}}\).
  • The input of \(W_o\) is the attention output. A Hadamard per head can be folded into \(W_v\) and \(W_o\) (attention mixes tokens, not channels, so it commutes with a per-head rotation). QuaRot also adds an online Hadamard across heads before \(W_o\).

For the KV cache, keys pass through RoPE, which rotates pairs of channels by a position-dependent angle. A fixed rotation cannot be folded into \(W_k\) through RoPE, so QuaRot applies an online Hadamard to queries and keys after RoPE. Since \(\mathbf{q} H (\mathbf{k} H)^{\top} = \mathbf{q}\mathbf{k}^{\top}\), the attention scores do not change, and the rotated keys are stored in 4 bits. [Certain] Values are already rotated per head through \(W_v\). Figure 4 shows all of them.

rotation merged into weights (free at inference)online Hadamard (runs at inference)4-bit matrix multiply
Figure 4. One Llama block after rotation, with SpinQuant’s names \(R_1\) to \(R_4\) (QuaRot uses fixed randomized Hadamards in the same places). Merged rotations are folded into weights offline; online ones run at inference (see the legend). Click any box or rotation to see what it does, what it costs and whether it can be learned.
How to read Figure 4

Read from top to bottom: the residual stream is the vertical line on the left and carries \(\mathbf{x}R_1\). The top row (attention) reads from it and adds its output back; then the bottom row (MLP) does the same. Click \(R_1\): it is merged into seven kinds of matrices and costs nothing at runtime. Click \(R_4\): it must run online, because SwiGLU’s element-wise product sits between \(W_{\text{up}}\) and \(W_{\text{down}}\). Click \(R_3\): it sits after RoPE and makes the 4-bit KV cache possible. Every blue box receives a rotated input and runs as a 4-bit matrix multiply.

The rest of the recipe and the results

After rotating, QuaRot quantizes weights with GPTQ (per output channel, symmetric), activations per token (symmetric, clipping ratio 0.9) and the KV cache asymmetrically in groups of 128 (clipping ratio 0.95). [Certain] The online Hadamards are run in 16 bits; everything else is a 4-bit matrix multiply. The cost of the online parts is small: for the 11,008-wide MLP of Llama 2 7B, \(H_{64} \otimes H_{172}\) is 66,048 additions plus 1,893,376 multiply-adds per token, about 4.2% of the 45,088,768 multiply-adds of \(W_{\text{down}}\) itself.

Reported results: [Certain]

Model (WikiText-2 perplexity)FP16QuaRot W4A4KV4
Llama 2 7B5.476.10
Llama 2 70B3.323.79

For Llama 2 70B the paper also reports 99% of the zero-shot accuracy kept, and that plain round-to-nearest (no calibration data at all) gives lossless 6-bit and 8-bit models once rotated. [Certain] It reports up to 3.33× faster prefill and up to 3.89× less memory for Llama 2 70B with its 4-bit kernels; the exact batch sizes and the memory figure differ between parts of the paper, so treat these as “about 3×”. [Likely]

Algorithm 2: one QuaRot decoder layer at inference (W4A4KV4)
  1. input: stream \(\mathbf{h}\) (already rotated by \(Q\)); 4-bit weights with folded rotations and norm scales
  2. \(\mathbf{a} \leftarrow \mathbf{h} / \operatorname{RMS}(\mathbf{h})\); \(\ \hat{\mathbf{a}} \leftarrow\) quantize per token to INT4 // (109.1); no outliers thanks to \(Q\)
  3. \(\mathbf{q}, \mathbf{k}, \mathbf{v} \leftarrow \hat{\mathbf{a}} W_{q}, \hat{\mathbf{a}} W_{k}, \hat{\mathbf{a}} W_{v}\) // INT4 GEMMs
  4. \(\mathbf{q}, \mathbf{k} \leftarrow \operatorname{RoPE}(\mathbf{q}), \operatorname{RoPE}(\mathbf{k})\); \(\ \mathbf{q}, \mathbf{k} \leftarrow \mathbf{q} H, \mathbf{k} H\) per head // online, scores unchanged
  5. append INT4 \(\mathbf{k}, \mathbf{v}\) to the cache (asymmetric, groups of 128); attention in 16 bits
  6. \(\mathbf{o} \leftarrow\) online Hadamard across heads, quantize to INT4, \(\ \mathbf{h} \leftarrow \mathbf{h} + \hat{\mathbf{o}} W_o\)
  7. \(\mathbf{a} \leftarrow \mathbf{h} / \operatorname{RMS}(\mathbf{h})\), quantize; \(\ \mathbf{g} \leftarrow \operatorname{SiLU}(\hat{\mathbf{a}} W_{\text{gate}}) \odot (\hat{\mathbf{a}} W_{\text{up}})\)
  8. \(\mathbf{g} \leftarrow \operatorname{FWHT}(\mathbf{g})\), quantize to INT4; \(\ \mathbf{h} \leftarrow \mathbf{h} + \hat{\mathbf{g}} W_{\text{down}}\) // \(H^{\top}\) is folded into \(W_{\text{down}}\)

SpinQuant: learn the rotation

Not all rotations are equal

The randomized Hadamard comes with a guarantee, but the guarantee is only a bound. Different random signs give different models. SpinQuant (Liu et al., 2024) tried 100 random rotations on Llama 2 7B with 4-bit weights and activations and found up to 13 points of difference in average zero-shot accuracy between the best and the worst. [Certain] If the choice of rotation matters that much, it is worth optimizing. Figure 5 shows the same effect in two dimensions.

Figure 5. Left: 300 two-dimensional points (an elongated cloud, tilted by 25°) after rotation by \(\theta\), with the quantization grid of one shared scale (per tensor, set by the largest coordinate). Right: the relative squared error \(\|\hat{X} - X\|^2 / \|X\|^2\) for every angle from 0° to 90° (the curve repeats every 90°). The dot is the current angle. “Random rotation” draws an angle from a seeded generator; “Find the best angle” searches the curve.
How to read Figure 5

At \(\theta = 0\) (no rotation) the 3-bit error is 10.28%. Press “Sweep \(\theta\)” to animate the rotation from 0° to 90°: watch the points cross grid lines as the cloud turns, and the blue dot trace the error curve. Move the slider: the error goes up and down, between 7.23% (\(\theta = 71.5^\circ\)) and 12.05% (\(\theta = 43^\circ\)). Press “Random rotation” a few times: some random angles are good, some are worse than doing nothing. Press “Find the best angle”: this is what SpinQuant does in thousands of dimensions with gradient steps instead of a search. Switch to 4 bits: the same curve shape, with errors between 1.35% and 2.30%. The shape is ragged because rounding is not smooth; this is why learning a rotation needs a straight-through estimator.

The objective and the Cayley step

SpinQuant keeps the same structure as Figure 4 and makes the mergeable rotations learnable: \(R_1\) on the residual stream and \(R_2\) per head between \(W_v\) and \(W_o\). The online ones, \(R_3\) and \(R_4\), stay fixed Hadamards because they must be fast. [Certain] The loss is the ordinary task loss (cross-entropy) of the network with quantized activations:

\[ \min_{R_1, R_2 \in \mathcal{M}} \ \mathcal{L}_Q\big(R_1, R_2 \mid W, X\big), \qquad \mathcal{M} = \{R : R^{\top} R = I\} \tag{109.11} \]
Read the symbols
\(\mathcal{L}_Q\)
cross-entropy of the network whose activations (and KV) are quantized after rotation; the weights \(W\) stay frozen in 16 bits during this step
\(X\)
the calibration set: 800 samples from WikiText-2 in the paper
\(\mathcal{M}\)
the set of orthogonal matrices (the Stiefel manifold); the rotation must stay orthogonal, or the invariance (109.3) breaks

Rounding has zero gradient almost everywhere, so the gradient passes through the quantizer with the straight-through estimator (session 111). A plain gradient step \(R - \eta G\) would leave the set \(\mathcal{M}\). SpinQuant uses Cayley SGD, which moves along the set: [Certain] build a skew-symmetric matrix from the gradient \(G = \partial \mathcal{L}/\partial R\) and apply its Cayley transform:

\[ Y = R G^{\top} - G R^{\top}, \qquad R \leftarrow \left(I - \tfrac{\eta}{2} Y\right)^{-1} \left(I + \tfrac{\eta}{2} Y\right) R \tag{109.12} \]
Read the symbols
\(Y\)
skew-symmetric (\(Y^{\top} = -Y\)); for small \(\eta\) the step moves \(R\) by \(\eta Y R = \eta (R G^{\top} R - G)\), the negative gradient projected onto the directions that keep \(R\) orthogonal
\(\eta\)
learning rate; SpinQuant starts at 1.5 and decays linearly to 0 over 100 iterations [Certain]
Cayley transform
for any skew-symmetric \(Y\), \((I - aY)^{-1}(I + aY)\) is orthogonal, so \(R\) stays orthogonal at every step (exercise 5)

This is the standard form of a Cayley step; the paper uses an efficient iterative version of it. [Likely] After the rotations are learned, SpinQuant merges them into the weights and quantizes the weights with GPTQ. [Certain] The learning is cheap: about 25 minutes for Llama 2 7B and about 3.5 hours for Llama 2 70B in the paper. [Certain] For Llama 2 7B, \(R_1\) is \(4096 \times 4096\) and there are 32 per-layer \(R_2\) of \(128 \times 128\): 17,301,504 parameters, 0.26% of the model, and all of them disappear into the weights afterwards.

Reported results: with 4-bit weights, activations and KV cache, SpinQuant brings the zero-shot gap to full precision on Llama 2 7B down to 2.9 points; it beats LLM-QAT by 19.1 and SmoothQuant by 25.0 points, and on Llama 3 8B it shrinks QuaRot’s gap by up to 45.1% (relative). [Certain]

Runnable code

The script is deterministic (fixed seed) and runs on a CPU in a few seconds. Part 1 checks the Hadamard matrix and the FWHT. Part 2 checks computational invariance through RMSNorm. Part 3 measures the INT4 error with and without the rotation. Part 4 learns a rotation from the identity with Cayley steps (109.12). It uses a smooth proxy loss, the kurtosis of each token (large when a vector has spikes), instead of SpinQuant’s cross-entropy with a straight-through estimator, so the loop is short and stable. The printed values are in the comments.

import torch
torch.manual_seed(0)

def hadamard(n):                       # Sylvester construction, formula (109.5); n a power of 2
    H = torch.ones(1, 1)
    while H.shape[0] < n:
        H = torch.cat([torch.cat([H, H], 1), torch.cat([H, -H], 1)], 0)
    return H / n ** 0.5                # orthonormal: H @ H.T = I

def fwht(x):                           # fast Walsh-Hadamard transform on the last dim, O(n log n)
    n, h, x = x.shape[-1], 1, x.clone()
    while h < n:
        x = x.view(*x.shape[:-1], n // (2 * h), 2, h)
        a, b = x[..., 0, :], x[..., 1, :]
        x = torch.stack([a + b, a - b], -2).view(*x.shape[:-3], n)
        h *= 2
    return x / n ** 0.5

def quant(x, bits=4):                  # symmetric per-token round-to-nearest, formula (109.1)
    qm = 2 ** (bits - 1) - 1
    d = x.abs().amax(-1, keepdim=True) / qm
    return (x / d).round().clamp(-qm, qm) * d

# 1) the Hadamard matrix and the fast transform agree
n, m, T = 256, 64, 32
H = hadamard(n)
x = torch.randn(T, n)
print(torch.allclose(H @ H.T, torch.eye(n), atol=1e-5), torch.allclose(fwht(x), x @ H, atol=1e-4))
# True True

# 2) computational invariance through RMSNorm, formulas (109.3) and (109.8)
x[:, 7] *= 40; x[:, 100] *= 25         # two outlier channels, as in real LLMs
alpha = torch.rand(n) + 0.5            # RMSNorm scale
W = torch.randn(n, m) / n ** 0.5       # the next linear layer
rms = lambda v: v / v.pow(2).mean(-1, keepdim=True).sqrt()
y = (rms(x) * alpha) @ W               # original block
D = torch.diag(torch.randint(0, 2, (n,)).float() * 2 - 1)
Q = D @ H                              # randomized Hadamard, formula (109.6)
W_rot = Q.T @ (alpha[:, None] * W)     # fold alpha, then absorb the rotation
y_rot = rms(x @ Q) @ W_rot             # the residual stream now carries x Q
print(torch.allclose(y, y_rot, atol=1e-4))
# True

# 3) INT4 activations: the rotation removes the outliers
a, a_r = rms(x), rms(x @ Q)
peak = lambda v: (v.abs().amax(-1) / v.pow(2).mean(-1).sqrt()).mean().item()
err = lambda out: ((out - y).norm() / y.norm()).item()
print("max/RMS %.1f -> %.1f" % (peak(a), peak(a_r)))
print("INT4 error %.3f -> %.3f" % (err(quant(a) @ (alpha[:, None] * W)), err(quant(a_r) @ W_rot)))
# max/RMS 12.4 -> 2.2
# INT4 error 0.264 -> 0.079

# 4) learn a rotation with Cayley steps, formula (109.12); smooth proxy loss = kurtosis
k, I = 64, torch.eye(64)
z0 = torch.randn(512, k); z0[:, 3] *= 30; z0[:, 40] *= 12
R, lr = I.clone(), 0.02
for step in range(301):
    R.requires_grad_(True)
    z = z0 @ R
    L = (z.pow(4).mean(-1) / z.pow(2).mean(-1).pow(2)).mean()
    G, = torch.autograd.grad(L, R)
    with torch.no_grad():
        Y = R @ G.T - G @ R.T          # skew-symmetric direction
        R = torch.linalg.solve(I - lr / 2 * Y, (I + lr / 2 * Y) @ R)
    if step % 150 == 0:
        print(step, "kurtosis %.2f, max/RMS %.1f, orthogonal %s" % (L.item(), peak(z.detach()), torch.allclose(R @ R.T, I, atol=1e-3)))
print("Hadamard: kurtosis %.2f, max/RMS %.1f" % (((z0 @ hadamard(k)).pow(4).mean(-1) / (z0 @ hadamard(k)).pow(2).mean(-1).pow(2)).mean().item(), peak(z0 @ hadamard(k))))
# 0 kurtosis 37.09, max/RMS 6.6, orthogonal True
# 150 kurtosis 1.81, max/RMS 1.9, orthogonal True
# 300 kurtosis 1.77, max/RMS 1.9, orthogonal True
# Hadamard: kurtosis 1.85, max/RMS 1.9

Part 3 is the session in two numbers: the same layer, the same 4 bits, and the output error falls from 26.4% to 7.9% only because of an exact rotation. Part 4 starts from “no rotation” (max-to-RMS 6.6) and finds a rotation as flat as the Hadamard one (1.9), with \(R\) orthogonal at every step.

Choosing a method

MethodWhat is quantizedRotationRuntime costNeeds data
QuIPweights only (down to 2 bits)random orthogonal (Kronecker of two)rotations on input and output of each layeryes (Hessian)
QuIP#weights only, E8 lattice codebookrandomized HadamardFWHT per layer plus codebook lookupyes, plus fine-tuning
QuaRotweights, activations, KV cache (4 bits)randomized Hadamard, merged plus onlinea few FWHTs per layeronly for GPTQ; RTN works at 6 to 8 bits
SpinQuantweights, activations, KV cache (4 bits)learned \(R_1, R_2\), Hadamard \(R_3, R_4\)same as QuaRotyes: 800 samples to learn, then GPTQ

A simple rule: for weight-only compression to 2 or 3 bits, use the QuIP# family. For 4-bit weights and activations on hardware with INT4 or FP4 matrix units, use QuaRot, and SpinQuant if you can spend an hour of calibration for better accuracy. At 8 bits, SmoothQuant is usually enough. [Likely]

Where this is used

ApplicationWhy rotations matterLink
On-device LLMsMeta shipped quantized Llama 3.2 1B and 3B with SpinQuant as one of two methods, for phone CPUs. [Certain]session 140
4-bit prefill-heavy servingPrefill is compute-bound, so only 4-bit activations (QuaRot, SpinQuant) make the matrix units faster; weight-only methods do not.session 104
Long-context and many-user decodingThe rotation after RoPE (\(R_3\)) makes a 4-bit KV cache accurate, which fits about 3.8 times more tokens than 16 bits (4 bits plus one 16-bit scale and zero point per group of 128).session 110
Very large models on few GPUsQuIP# at 2 bits per weight stores a 70B model in about 17.5 GB of weights (70B × 2 bits), plus small codebook and scale overheads.session 106
Video-language models and Video-RAGA long video gives hundreds of thousands of visual tokens; prefill on them is compute-bound, and their KV cache dominates memory. W4A4KV4 cuts both. Vision encoders also show outlier channels. [Likely]session 197
Video retrieval and fingerprinting at scaleEmbedding models run over every frame of a large archive; 4-bit activations raise the frames per second per GPU. Rotation keeps the embeddings close to the 16-bit ones. [Speculation]session 197
FP4 and microscaling formatsBlock formats share one scale per 16 or 32 values, the same max-sets-the-step problem; Hadamard rotations are also used with them. [Likely]session 105

Common mistakes

  • Forgetting to fold the RMSNorm scale first. \(\operatorname{diag}(\alpha)\) does not commute with \(Q\). Rotating without folding changes the model output; the logits check in Algorithm 1 catches it.
  • Using a non-orthogonal or unnormalized Hadamard. The \(\pm 1\) matrix must be divided by \(\sqrt{n}\). Without it, every rotated weight is scaled by \(\sqrt{n}\) and the outputs are wrong by that factor (or by \(n\) if both sides are unnormalized).
  • Trying to merge the rotation before \(W_{\text{down}}\) into \(W_{\text{up}}\). SwiGLU’s element-wise product and SiLU do not commute with a rotation. That rotation must be online.
  • Rotating keys before RoPE. RoPE does not commute with a general rotation, so the scores change. Rotate queries and keys after RoPE, with the same matrix.
  • Expecting a rotation to help a vector without outliers. If max-to-RMS is already small, rotating gives nothing (Figure 1 with outlier 1); the gain comes only from flattening spikes.
  • Using a fixed Hadamard without random signs. The bound (109.6) needs the random \(D\); a structured input can still produce a spike after a fixed \(H\).
  • Reporting a QuaRot or SpinQuant number without the settings. Results depend on RTN versus GPTQ, the clipping ratios, group sizes and whether the KV cache is quantized. Compare only like with like.
  • Learning the rotation with a plain gradient step. \(R - \eta G\) is not orthogonal, and invariance breaks. Use a Cayley step (109.12) or re-orthogonalize (QR) after each step.

Interview questions

Why does multiplying activations by an orthogonal matrix not change a linear layer’s output?basic
Because \(QQ^{\top} = I\): \(\mathbf{x}W = \mathbf{x}QQ^{\top}W = (\mathbf{x}Q)(Q^{\top}W)\) (109.3). We store \(Q^{\top}W\) as the new weight, computed once offline, and feed \(\mathbf{x}Q\). In full precision the result is identical up to floating-point error. Only after quantization does it differ, and it is better because \(\mathbf{x}Q\) has no outliers.
Why do outliers hurt per-token INT4 quantization so much?basic
The step \(\Delta = \max|x_i| / 7\) is set by the largest entry. The relative error grows with the square of the max-to-RMS ratio (109.2). Worse, any entry smaller than \(\Delta/2\) rounds to 0: with an outlier 20 times the typical size, most channels are erased (Figure 1: 100% error on the normal channels). At 8 bits \(\Delta\) is 18 times smaller, which is why INT8 survives.
How does a rotation help, in one sentence, and what is its limit?basic
It spreads one large channel over all \(n\) channels: a spike \(a\mathbf{e}_i\) becomes \(n\) entries of size \(a/\sqrt{n}\) (109.4), so max-to-RMS falls from \(\sqrt{n}\) to 1. Its limit: it spreads the quantization error evenly too, so it does not create resolution; if the outlier holds almost all the energy and \(n\) is small, the small components still get a large relative error.
Why Hadamard matrices rather than random orthogonal matrices?core
Three reasons. (1) All entries are \(\pm 1/\sqrt{n}\), the flattest possible, which is ideal for spreading spikes. (2) They can be applied in \(n\log_2 n\) additions with the FWHT and need no stored matrix, which matters for the online rotations. (3) With random signs (\(DH\)) they still give the incoherence guarantee (109.6). A dense random orthogonal matrix spreads equally well but costs \(n^2\) to apply and \(n^2\) numbers to store. SpinQuant shows a learned rotation can beat random ones, but only for the rotations that are merged into weights.
Walk through how a whole Llama model is rotated without changing its output.core
Fold every RMSNorm scale into the following matrices (109.8). Multiply the embedding by \(Q\). Every matrix that reads the stream (\(W_q, W_k, W_v, W_{\text{gate}}, W_{\text{up}}\), LM head) becomes \(Q^{\top}W\); every matrix that writes to it (\(W_o, W_{\text{down}}\)) becomes \(WQ\). Residual additions stay consistent, and RMSNorm without a scale commutes with \(Q\) because it only divides by the length. This is SliceGPT’s computational invariance, used by QuaRot and SpinQuant.
Which rotations in QuaRot or SpinQuant must run online, and why?core
Two kinds. Before \(W_{\text{down}}\) (\(R_4\)): its input comes from SiLU and an element-wise product, which do not commute with rotations, so nothing upstream can absorb it. On queries and keys after RoPE (\(R_3\)): RoPE is a position-dependent rotation, so a fixed rotation cannot be folded into \(W_k\). Since \(\mathbf{q}H(\mathbf{k}H)^{\top} = \mathbf{q}\mathbf{k}^{\top}\), rotating both after RoPE keeps the scores and makes the 4-bit KV cache work. QuaRot also adds an online Hadamard across heads before \(W_o\). These are FWHTs and cost a few percent of a layer.
What is incoherence in QuIP, and why does it make weight quantization easier?advanced
A weight matrix is \(\mu\)-incoherent if no entry exceeds \(\mu\) times the RMS of its entries; a Hessian is incoherent if its eigenvectors have no entry above \(\mu/\sqrt{n}\) (109.10). The first means a small grid covers all weights; the second means the important input directions are spread over all coordinates, so rounding errors in single weights matter about equally and adaptive rounding (LDLQ/OPTQ) works well. Random orthogonal \(U, V\) make both hold with high probability, and the proxy loss \(\operatorname{tr}(\Delta W^{\top} H \Delta W)\) is unchanged by them (109.9).
What does QuIP# add on top of QuIP?advanced
The randomized Hadamard transform instead of Kronecker products of random orthogonal matrices (faster, better bounds); vector quantization of 8 weights at a time with the E8P lattice codebook (\(2^{16}\) points, 2 bits per weight, decoded from a 256-entry table), which fits the round Gaussian shape of incoherent weights better than a per-weight grid; and fine-tuning after quantization. It reports WikiText-2 perplexity 3.91 for Llama 2 70B at 2 bits (FP16 3.12, context 4,096). It is weight-only.
How is SpinQuant different from QuaRot, and what does it cost?core
The structure is the same; SpinQuant learns the two mergeable rotations (\(R_1\) on the stream, \(R_2\) per head) instead of using random Hadamards, because random rotations differ by up to 13 zero-shot points. It minimizes the cross-entropy of the activation-quantized network on 800 WikiText-2 samples with Cayley SGD, 100 iterations, about 25 minutes for 7B and 3.5 hours for 70B; then it merges the rotations and runs GPTQ. Inference cost is identical to QuaRot, because learned rotations disappear into the weights and the online ones stay Hadamard.
How do you take a gradient step on a rotation matrix and keep it orthogonal?advanced
Project the gradient onto the tangent space with a skew-symmetric \(Y = RG^{\top} - GR^{\top}\), then use the Cayley transform \(R \leftarrow (I - \tfrac{\eta}{2}Y)^{-1}(I + \tfrac{\eta}{2}Y)R\) (109.12). For skew-symmetric \(Y\) the Cayley factor is orthogonal, so \(R\) stays orthogonal exactly (up to float error). Alternatives are a matrix exponential of \(Y\), or a plain step followed by QR or a polar decomposition. The quantizer’s gradient is passed through with the straight-through estimator.
Is rotation compatible with SmoothQuant or GPTQ? Would you use both?advanced
With GPTQ, yes: QuaRot and SpinQuant both rotate first and then quantize the rotated weights with GPTQ, and QuIP’s rounding is OPTQ itself. With SmoothQuant the two do the same job in different ways: SmoothQuant moves outlier size into the weights with a diagonal scale, rotation spreads it with a full matrix. After a rotation the activations are already flat, so a smoothing scale adds little. A diagonal scale applied before the rotation is still possible (it just becomes part of the folded weights). [Likely]

Exercises

  1. Compute \(\mathbf{x}H_4\) for \(\mathbf{x} = (8, 0, 0, 0)\). What are the max-to-RMS ratios before and after?
  2. Use (109.2) to predict the relative error (not squared) of 4-bit quantization for a vector whose max-to-RMS ratio is 12.4, and for 2.2. Compare the ratio of the two predictions with the INT4 errors printed in part 3 of the code.
  3. For \(n = 4096\), how many additions does an FWHT need, and how many multiply-adds does a dense \(n \times n\) rotation need? What is the ratio?
  4. Explain why \(11{,}008 = 2^8 \times 43\) cannot be used for a Kronecker Hadamard, and give a factorization that works.
  5. Prove that for skew-symmetric \(Y\) and any real \(a\), the Cayley matrix \(C = (I - aY)^{-1}(I + aY)\) is orthogonal.
  6. How many parameters does SpinQuant learn for Llama 2 7B (hidden size 4,096, 32 layers, head size 128)? What fraction of 6.74B is that? Why is the runtime cost zero?
  7. Show that rotating queries and keys of one head by the same orthogonal \(H\) after RoPE leaves every attention score unchanged. Why can you not do the same with two different matrices?
Answerscheck
1. Every entry of row 1 of \(H_4\) is \(1/2\), so \(\mathbf{x}H_4 = (4, 4, 4, 4)\). Before: RMS \(= \sqrt{64/4} = 4\), max 8, ratio 2 (\(= \sqrt{4}\), the maximum possible). After: ratio 1, the minimum. 2. With \(q_{\max} = 7\), relative error \(\approx \text{ratio} / (7\sqrt{12})\): \(12.4 / 24.25 = 0.51\) and \(2.2 / 24.25 = 0.091\). The ratio of predictions is \(12.4/2.2 = 5.6\); the printed output errors are 0.264 and 0.079, a ratio of 3.3. The formula uses averages (the mean of max-to-RMS over tokens), measures the input rather than the output, and the model of uniform rounding error is rough, so it gives the right direction and order of size, not the exact number. 3. \(4096 \times 12 = 49{,}152\) additions against \(4096^2 = 16{,}777{,}216\) multiply-adds: about 341 times fewer. 4. A Hadamard matrix of size \(m \gt 2\) must have \(m\) divisible by 4, and 43 is not. \(64 \times 172\) works (172 = 4 × 43, and a Hadamard matrix of size 172 is known). 5. \(Y^{\top} = -Y\), so \((I + aY)^{\top} = I - aY\) and \((I - aY)^{\top} = I + aY\). Also \(I + aY\) and \(I - aY\) commute (both are polynomials in \(Y\)), and \(I - aY\) is invertible because the eigenvalues of \(Y\) are purely imaginary. Then \(C^{\top}C = (I - aY)(I + aY)^{-1}(I - aY)^{-1}(I + aY) = (I - aY)(I - aY)^{-1}(I + aY)^{-1}(I + aY) = I\). 6. \(4096^2 + 32 \times 128^2 = 16{,}777{,}216 + 524{,}288 = 17{,}301{,}504\), about 0.26% of 6.74B. After training they are multiplied into the weights, so the deployed model has the same shapes and the same number of operations as before. 7. Score \(= \mathbf{q}H(\mathbf{k}H)^{\top} = \mathbf{q}HH^{\top}\mathbf{k}^{\top} = \mathbf{q}\mathbf{k}^{\top}\). With two matrices \(A \ne B\) you get \(\mathbf{q}AB^{\top}\mathbf{k}^{\top}\), and \(AB^{\top} \ne I\) in general, so the scores change.

Key takeaways

  • Per-token quantization error grows with the max-to-RMS ratio (109.2); one outlier sets the step for all channels and erases the small ones.
  • \(\mathbf{x}W = (\mathbf{x}Q)(Q^{\top}W)\) for any orthogonal \(Q\) (109.3): the model is unchanged, but the quantized inputs are flat.
  • A Hadamard matrix turns a spike into a flat vector (109.4); random signs make this hold for any input with high probability (109.6), and the FWHT applies it in \(n\log_2 n\) additions.
  • Computational invariance rotates the whole residual stream once RMSNorm scales are folded (109.8); only the rotations before \(W_{\text{down}}\) and after RoPE must run online.
  • QuIP and QuIP# use rotations for incoherent weights and Hessians (109.9, 109.10) and reach 2-bit weights; QuIP# adds the RHT and an E8 lattice codebook.
  • QuaRot gives W4A4KV4 with random Hadamards: Llama 2 70B goes from 3.32 to 3.79 WikiText-2 perplexity.
  • SpinQuant learns \(R_1, R_2\) with Cayley SGD (109.12) on 800 samples; random rotations differ by up to 13 points, and learning closes much of the gap.

Key formulas

\[ \frac{\|\hat{\mathbf{x}} - \mathbf{x}\|^2}{\|\mathbf{x}\|^2} \approx \frac{1}{12 q_{\max}^2}\left(\frac{\max_i |x_i|}{\operatorname{RMS}(\mathbf{x})}\right)^2, \qquad \mathbf{x}W = (\mathbf{x}Q)(Q^{\top}W) \]
\[ H_{2n} = \tfrac{1}{\sqrt{2}}\begin{bmatrix} H_n & H_n \\ H_n & -H_n \end{bmatrix}, \qquad Q = DH_n, \qquad W' = Q^{\top}\operatorname{diag}(\alpha)W \]
\[ \ell(\hat{W}) = \operatorname{tr}\big((\hat{W} - W)^{\top} H (\hat{W} - W)\big), \qquad R \leftarrow \left(I - \tfrac{\eta}{2}Y\right)^{-1}\left(I + \tfrac{\eta}{2}Y\right)R \]

References for this session

  1. Chee et al. (2023). QuIP: 2-Bit Quantization of Large Language Models With Guarantees
  2. Tseng et al. (2024). QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
  3. Ashkboos et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
  4. Liu et al. (2024). SpinQuant: LLM Quantization with Learned Rotations
  5. Chee et al. (2023). QuIP: 2-Bit Quantization of Large Language Models With Guarantees
  6. Tseng et al. (2024). QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
  7. Ashkboos et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
  8. Liu et al. (2024). SpinQuant: LLM Quantization with Learned Rotations
  9. Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  10. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  11. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  12. Ashkboos et al. (2024). SliceGPT: Compress Large Language Models by Deleting Rows and Columns
  13. Liu et al. (2023). LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
  14. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  15. Li, Fuxin, Todorovic (2020). Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
  16. Meta AI (2024). Introducing quantized Llama models with increased speed and a reduced memory footprint