LLM & VLM courseBlog
110Track 5: LLM inference and serving · session 33 of 65

KV-cache quantization: KIVI and KVQuant

At long context or big batch, the KV cache can take more GPU memory than the model weights. This session shows how to store keys and values in 2 to 4 bits without breaking the model: keys per channel and values per token (KIVI), and non-uniform levels, pre-RoPE keys and a few kept outliers (KVQuant). It ends with the test that matters: long-context accuracy, not only perplexity.

Time: 2.5 to 3.5 hours (reading 2 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 80 (the KV cache), 103 (quantization math) and 14 (RoPE)

What you will learn

  • When the KV cache is bigger than the weights, and how many bytes a low-bit cache saves
  • Why keys have outlier channels, and why keys must be quantized per channel but values per token
  • How KIVI streams a 2-bit cache with a small full-precision window of recent tokens, and what that window costs in bits
  • Why KVQuant quantizes keys before RoPE, and how it adds non-uniform levels, sparse outliers and a full-precision first token
  • How per-channel and per-token scales fold into the attention matrix products
  • How to evaluate a quantized cache: perplexity hides failures that passkey, LongBench and RULER show

Why this matters

A model of 8 billion parameters in bfloat16 takes 16 GB. Its KV cache for one user with 128k tokens of context also takes about 16 GB. For 32 users with 4k tokens each it takes 17 GB. At long context or big batch, the cache, not the model, fills the GPU. Storing keys and values in 2 to 4 bits instead of 16 is the most direct way to fit more context and more users.

Session 80 gave the size of the cache: \(s = 2 L\, n_{kv}\, d_h\, b\) bytes per token. Only the last factor, the bytes per number, is left after the attention design is fixed. The quantization sessions so far (session 103, session 107, session 108, session 109) made weights and activations small. The KV cache is a third kind of tensor. It is an activation (it depends on the input), but it is stored for a long time and read at every decode step, like a weight. So it has its own problems and its own methods.

The efficient-inference survey lists four methods in this area: FlexGen and Atom store the cache in INT4, KVQuant uses non-uniform levels found offline on calibration data, and KIVI uses a tuning-free 2-bit scheme with keys per channel and values per token. [Certain, survey Section 5.2.1] We study the last two in depth:

  • KIVI (Liu et al., ICML 2024): look at the numbers in the cache, see that keys and values have different shapes of outliers, and quantize them along different axes. Store recent tokens in full precision. No calibration, no training.
  • KVQuant (Hooper et al., 2024): keys per channel too, but quantized before RoPE; non-uniform levels chosen with a sensitivity weight; the 1% largest values kept apart in full precision; the first token kept in full precision.

Both papers report big gains. KIVI reports 2.6× less peak memory for Llama-2-7B (weights included), up to 4× larger batches and 2.35× to 3.47× more throughput on one A100. KVQuant reports less than 0.1 perplexity increase at 3 bits, and, from its memory calculation, LLaMA-7B with up to 1 million tokens of context on one A100-80GB. [Certain that the papers report these] The last part of this session shows why such numbers are not enough, and what to measure instead.

Interview questions on this topic: “Why are keys quantized per channel and values per token?”, “Why not use GPTQ for the KV cache?”, “Why quantize keys before RoPE?”, “Your 2-bit cache has the same perplexity; can you ship it?”. By the end you can answer each with a formula and a number.

When the cache is bigger than the weights

Take the weight memory \(M_W = N_{\text{params}} \cdot b_W\) and the cache memory from session 80. With a cache of \(\bar b\) bits per number, the cache is

\[ M_{\text{KV}} = 2 L\, n_{kv}\, d_h \cdot \frac{\bar b}{8} \cdot \sum_{i=1}^{B} T_i, \qquad M_{\text{KV}} = M_W \ \text{ when } \ B\,T = \frac{M_W}{s} \ \ (\text{equal lengths}) \tag{110.1} \]
Read the symbols
\(\bar b\)
the average number of bits per stored key or value number, including the cost of scales, zero points and anything kept in full precision. 16 for bfloat16.
\(s\)
bytes per token at the current precision, \(2 L\, n_{kv}\, d_h \bar b / 8\)
\(B, T_i\)
number of sequences in the batch and the length of sequence \(i\)
\(B\,T\)
the total number of cached tokens on the GPU. The cache equals the weights when this total reaches \(M_W / s\).
Worked example: the crossover point in bfloat16

Llama 2 7B (MHA, 32 layers, 32 KV heads, \(d_h = 128\)): \(s = 524{,}288\) bytes. Weights: 13.48 GB. The cache passes the weights at \(13.48 \times 10^9 / 524{,}288 \approx 25{,}700\) cached tokens, for example 7 users at 4k tokens.

Llama 3 8B (GQA, 8 KV heads): \(s = 131{,}072\) bytes, weights 16.06 GB. Crossover at about 122,500 tokens: one user at its full 128k context, or 32 users at 4,096 tokens (\(32 \times 4096 \times 131{,}072 = 17.18\) GB).

Llama 3 70B: \(s = 327{,}680\) bytes, weights 141.1 GB. 16 users at 32k tokens need 171.8 GB of cache, more than the weights.

So two settings make the cache the main cost: long context (RAG over long documents, agents, video) and big batches (any busy server). GQA already cut the cache of Llama 3 by 4× compared with MHA. Going from 16 to about 3 bits cuts it by about 5× more, and it needs no retraining.

Memory is not the only gain. Decoding is memory-bound: each step reads the weights once and every user’s cache once (session 80, formula 80.9). A cache that is 5× smaller is read 5× faster, if the kernel can read the low-bit codes directly. KVQuant reports custom kernels with up to about 1.7× speedup over the fp16 matrix-vector products. [Certain that the paper reports this] With a small cache, the server also fits a bigger batch, which spreads the weight reads over more users.

Quantization of a cache in one formula

Session 103 built uniform quantization. KIVI uses the asymmetric (min–max) version, with a zero point equal to the minimum of the group. For a group of numbers \(X\) and \(b\) bits:

\[ z = \min X, \quad s = \frac{\max X - \min X}{2^{b} - 1}, \quad c = \operatorname{round}\!\left(\frac{x - z}{s}\right) \in \{0, \dots, 2^{b}-1\}, \quad \hat x = c\, s + z \tag{110.2} \]
Read the symbols
group \(X\)
the set of numbers that share one scale and one zero point. Which numbers form a group is the whole design question of this session.
\(z\)
zero point: here the smallest value of the group, stored in 16 bits
\(s\)
scale: the step between two levels, stored in 16 bits
\(c\)
the code, a \(b\)-bit integer. Only the codes are stored per number.
\(\hat x\)
the value the attention kernel will use. The error is at most \(s/2\) per number.

Two facts follow at once. First, the error of every number in a group is set by the range of the group: one huge value makes the step \(s\) large for all the others. Second, each group pays 32 extra bits for its scale and zero point. With groups of \(G\) numbers the average cost is

\[ \bar b_{\text{group}} = b + \frac{32}{G}, \qquad \text{e.g. } b = 2,\ G = 32: \ \bar b = 3 \text{ bits} \tag{110.3} \]

So “2-bit KIVI” with \(G = 32\) really stores 3 bits per number, before counting the full-precision part. Keep this in mind whenever you compare methods by their bit width.

Keys have outlier channels, values do not

What the cache looks like

Write the key cache of one head as a matrix \(K \in \R^{T \times d_h}\): one row per token, one column per channel (one of the \(d_h\) numbers of a key). The value cache \(V\) has the same shape. KIVI plotted these matrices for Llama, Falcon and Mistral and found two different patterns. [Certain, KIVI paper]

  • Keys: a few fixed channels have very large values for almost every token. Inside one channel, the values over tokens are similar. LLM.int8() (session 108) found the same kind of fixed outlier dimensions in hidden states.
  • Values: no clear outlier channels. The size of the values changes more from token to token than from channel to channel.

Now choose the groups of (110.2). There are two natural choices for a matrix of shape (tokens × channels):

\[ \text{per-token: } X_j = \{ K_{j,1}, \dots, K_{j,d_h} \}, \qquad \text{per-channel: } X_i = \{ K_{1,i}, \dots, K_{G,i} \} \tag{110.4} \]
Read the symbols
\(K_{j,i}\)
channel \(i\) of the key of token \(j\)
per-token group
one row: all channels of one token share a scale. This is how activations are usually quantized, and how a new token arrives.
per-channel group
one column, cut into pieces of \(G\) tokens: one channel over \(G\) tokens shares a scale

With per-token groups, every row of \(K\) contains the outlier channels, so every row has a huge range and a huge step \(s\). All the normal channels fall onto one or two levels. With per-channel groups, the outlier channel gets its own large scale, and the normal channels keep fine steps. The error stays inside the outlier channel. [Certain, KIVI paper]

Worked example: four keys, 2 bits

Four tokens, four channels; channel 3 is an outlier channel:

\[ K = \begin{bmatrix} 0.2 & -0.4 & 9.0 & 0.1 \\ -0.3 & 0.5 & 10.0 & -0.2 \\ 0.4 & 0.1 & 8.6 & 0.3 \\ -0.1 & -0.3 & 9.4 & 0.0 \end{bmatrix} \]

Per-token, row 1: \(z = -0.4\), \(s = 9.4 / 3 = 3.133\). The codes are \((0, 0, 3, 0)\): the three small channels all become \(-0.4\). Every row looks the same: codes \((0, 0, 3, 0)\), so the small channels lose all their information. The largest error in the small channels is 0.8.

Per-channel over the 4 tokens: the scales are \(0.233, 0.300, 0.467, 0.167\). The largest error in the small channels is 0.1; in the outlier channel it is 0.133.

Now a query \(\mathbf{q} = (1, 1, 0, 1)\) that reads only the small channels. The true scores \(K\mathbf{q}\) are \((-0.1, 0.0, 0.8, -0.4)\). Per-token gives \((-1.2, -0.9, 0.3, -0.9)\): wrong order, wrong sizes. Per-channel gives \((-0.1, 0.0, 0.9, -0.5)\), almost exact. [Certain, computed in Python]

Figure 1 runs formula (110.2) on this matrix, one group at a time. For each group it finds the zero point and the step, rounds every number to a code, and shows on a number line where each number lands. The last step multiplies the result by the query.

  1. for each group \(X\) // a row (per token) or a column (per channel)
  2. \(z \leftarrow \min X\), \(\;s \leftarrow (\max X - \min X)/(2^b - 1)\)
  3. \(c \leftarrow \operatorname{round}\big((x - z)/s\big)\) for every \(x \in X\)
  4. \(\hat x \leftarrow c\, s + z\), error \(|\hat x - x| \le s/2\)
  5. scores \(\hat K \mathbf q\) against \(K \mathbf q\), with \(\mathbf q = (1, 1, 0, 1)\)
Figure 1. Formula (110.2) computed in the browser on the 4 × 4 key matrix of the worked example (channel 3 is the outlier channel). Left: the exact keys, with a frame on the current group. Middle: the quantized keys \(\hat K\), filled in as each group is done; the small number is the code \(c\). Right: the error \(|\hat x - x|\) of every number (darker red is larger). Bottom: the current group on a number line, with its \(2^b\) levels (ticks) and an arrow from each exact value (orange) to the level it is rounded to (blue). The last step compares the scores \(\hat K\mathbf q\) with the exact scores \(K\mathbf q\).
How to read Figure 1

Press Play or Step. The highlighted lines of the algorithm box are the ones that run in this step, and the box next to it shows the same formulas with the real numbers. With rows as groups at 2 bits, row 1 has \(z = -0.4\) and \(s = 9.4/3 = 3.133\): on the number line, the three small numbers all snap to the lowest level, and the codes are \((0, 0, 3, 0)\) in every row. The largest error in the small channels is 0.80 and the scores end at \((-1.2, -0.9, 0.3, -0.9)\) against the true \((-0.1, 0.0, 0.8, -0.4)\). Switch to columns: each column gets its own small step (0.233, 0.300, 0.467, 0.167), the errors fall to at most 0.13, and the scores are \((-0.1, 0.0, 0.9, -0.5)\). At 4 bits with rows, the small channels still have errors up to 0.30, worse than 2 bits with columns (0.10).

KIVI measured the same effect on Llama-2-13B, averaged over layers and heads: per-token key quantization gives a relative attention-score error of 47.00, per-channel gives 9.60, about 5× smaller. [Certain, KIVI Table 2]

Why values are different

Values enter attention in a different way. The output of one head is a weighted sum of value rows:

\[ \mathbf{o} = \sum_{j=1}^{T} p_j\, \hat{\mathbf{v}}_j = \mathbf{o}_{\text{exact}} + \sum_{j=1}^{T} p_j\, (\hat{\mathbf{v}}_j - \mathbf{v}_j) \tag{110.5} \]
Read the symbols
\(p_j\)
attention probability of token \(j\); most are near zero, a few are large (KIVI reports an average attention sparsity of 84.3%)
\(\hat{\mathbf{v}}_j - \mathbf{v}_j\)
the quantization error of the value row of token \(j\)

The output error is a mix of row errors, weighted by \(p_j\). What matters is that the few tokens with large \(p_j\) have accurate value rows. Per-token quantization gives each token its own scale, fitted to that token’s own range, so “the error of one token does not affect the others” (KIVI’s words). Per-channel quantization lets a few large tokens set the scale of a whole column, and an important token with small values then gets a coarse grid. [Likely, our reading of the KIVI argument] KIVI’s measurement: the relative output error is 3.55 for per-token values and 49.89 for per-channel values, about 14× larger. [Certain, KIVI Table 2]

The task results agree. On Llama-2-7B with 2-bit fake quantization (quantize then dequantize, no real kernel), GSM8K accuracy is 13.50 in 16 bits, 5.76 with keys per channel and values per token, 0.83 with both per token, and 0.00 with both per channel. [Certain, KIVI paper] The only working 2-bit combination is keys per channel, values per token. The full KIVI method (below) brings GSM8K back to 12.74.

Figure 2 lets you try all four combinations on a small cache with two outlier key channels.

Figure 2. A toy cache of 12 tokens × 8 channels for one head. Keys have two outlier channels (channel 2 near +5, channel 7 near −4); values have two tokens with large values (tokens 4 and 10). The maps show the absolute error \(|\hat x - x|\) of every number after quantization with formula (110.2); darker means larger error, and the frames show which numbers share a scale. The bars at the bottom compare the attention probabilities with exact keys (outline) and with quantized keys (filled).
How to read Figure 2

Start at 2 bits with both per token. The key map is dark everywhere except the outlier channels: each row’s range is about 10, so the normal channels lose their values, and the attention bars move far from their outlines. The readout shows a key error of 0.368 and an attention-score error of 1.419. Switch keys to per channel: the key error falls to 0.070, the score error to 0.299, and the bars almost match. Now switch values to per channel: the value error rises from 0.221 to 0.350, because tokens 4 and 10 set the range of every column. The output error is smallest with keys per channel and values per token (0.331 at 2 bits). At 4 bits all choices are closer together: granularity matters most when bits are few.

The same groups fit the matrix products

There is also a reason from the kernel side. A decode step computes scores \(\mathbf{q} \cdot \hat{\mathbf{k}}_j\) (a sum over channels \(i\)) and an output \(\sum_j p_j \hat{\mathbf{v}}_j\) (a sum over tokens \(j\)). Put \(\hat K_{j,i} = s_i c_{j,i} + z_i\) (per-channel keys) and \(\hat{\mathbf{v}}_j = s_j \mathbf{c}_j + z_j \mathbf{1}\) (per-token values, one group per row for simplicity):

\[ \mathbf{q} \cdot \hat{\mathbf{k}}_j = \sum_{i} (q_i s_i)\, c_{j,i} + \sum_{i} q_i z_i, \qquad \sum_j p_j \hat{\mathbf{v}}_j = \sum_j (p_j s_j)\, \mathbf{c}_j + \Big(\sum_j p_j z_j\Big) \mathbf{1} \tag{110.6} \]
Read the symbols
\(c_{j,i}, \mathbf{c}_j\)
the stored integer codes
\(q_i s_i\)
the per-channel key scales can be multiplied into the query once per step, and \(\sum_i q_i z_i\) is one number shared by all tokens
\(p_j s_j\)
the per-token value scales can be multiplied into the attention probabilities

In both products the scale runs along the axis that is summed. So the scales can be moved out of the inner loop, and the kernel can multiply directly with the integer codes. [Certain, algebra] Grouping keys per token would put the scale on the other axis, which is just as easy for the score product; the problem with per-token keys is accuracy, not speed. Real kernels with groups of \(G\) need one such correction per group. [Likely]

KIVI: a streaming 2-bit cache

The streaming problem

Per-channel keys create a practical problem. A per-channel group covers \(G\) tokens of one channel. During decoding, tokens arrive one at a time: the group of the newest tokens is not complete yet, so its range is unknown. Values have no such problem: a per-token group is complete as soon as the token arrives.

KIVI’s answer is a small residual: the most recent tokens stay in full precision. [Certain, KIVI paper]

  • Keys: new keys go into a full-precision buffer. When the buffer holds \(R\) tokens, all of them are quantized per channel (in groups of \(G\) tokens), appended to the quantized key cache, and the buffer is emptied.
  • Values: new values go into a full-precision queue of length \(R\). When the queue is longer than \(R\), the oldest value leaves it and is quantized per token (in groups of \(G\) channels).
  • Attention: each step attends to the quantized part and the full-precision part and joins the two results. In prefill, attention uses the exact keys and values; only the stored cache is quantized.

The paper uses \(G = 32\) and \(R = 128\) in its main experiments. [Certain] The residual has a second benefit: the most recent tokens often get much of the attention, and they are exact. In an ablation on Llama-2-13B GSM8K, \(R = 32, 64, 96, 128\) gave 20.62, 19.86, 20.55 and 20.77 (no clear trend in this range), while \(G = 32, 64, 128\) gave 20.77, 21.00 and 17.29: groups that are too large hurt. [Certain, KIVI ablation tables]

Algorithm 1: one decode step with a KIVI cache (one layer, one KV head)
  1. state: quantized keys \(\hat K\) (per channel), full-precision key buffer \(K_r\) (\(\lt R\) rows); quantized values \(\hat V\) (per token), full-precision value queue \(V_r\) (\(\le R\) rows)
  2. input: the new token’s \(\mathbf{q}, \mathbf{k}, \mathbf{v}\)
  3. append \(\mathbf{k}\) to \(K_r\); append \(\mathbf{v}\) to \(V_r\)
  4. if \(K_r\) has \(R\) rows: \(\hat K \leftarrow [\hat K;\ \text{quant}_{\text{channel}}(K_r, G)]\); \(K_r \leftarrow \varnothing\) // groups of \(G\) tokens per channel
  5. if \(V_r\) has \(R+1\) rows: pop the oldest row \(\mathbf{v}_{\text{old}}\); \(\hat V \leftarrow [\hat V;\ \text{quant}_{\text{token}}(\mathbf{v}_{\text{old}}, G)]\) // groups of \(G\) channels
  6. scores \(\leftarrow [\,\hat K \mathbf{q};\ K_r \mathbf{q}\,] / \sqrt{d_h}\) // low-bit kernel + normal kernel, formula (110.6)
  7. \(\mathbf{p} \leftarrow \softmax(\text{scores})\) over all cached tokens
  8. return \(\mathbf{o} = \mathbf{p}_{\text{old}}^{\top} \hat V + \mathbf{p}_{\text{new}}^{\top} V_r\) // split \(\mathbf{p}\) by where each value lives

Note that the key and value boundaries are not at the same token: keys are quantized in blocks of \(R\), values one by one. The softmax runs over all tokens at once; only the matrix products are split. Figure 3 runs this algorithm on a toy cache with real numbers, one line at a time, and tracks the error and the memory as tokens arrive.

Residual
key, quantized per channelvalue, quantized per tokenfull precision (residual)KIVI output errornaive: all per token, no residual
  1. prefill \(P\) tokens with exact \(K, V\); store: first \(\lfloor P/R \rfloor R\) keys per channel, all but the last \(R\) values per token
  2. new token: append \(\mathbf k\) to \(K_r\), \(\mathbf v\) to \(V_r\)
  3. if \(|K_r| = R\): quantize \(K_r\) per channel, append to \(\hat K\), \(K_r \leftarrow \varnothing\)
  4. if \(|V_r| = R + 1\): pop the oldest \(\mathbf v\), quantize it per token, append to \(\hat V\)
  5. \(\mathbf p \leftarrow \softmax\big([\hat K \mathbf q;\ K_r \mathbf q] / \sqrt{d_h}\big)\)
  6. \(\mathbf o \leftarrow \mathbf p_{\text{old}}^{\top} \hat V + \mathbf p_{\text{new}}^{\top} V_r\); compare with exact attention
Figure 3. Algorithm 1 run in the browser on a toy head with \(d_h = 4\) channels and 16 tokens (prefill of \(P = 5\), then 11 decode steps). Key channel 2 is an outlier channel near +6, and value token 9 is large. Each column is a token; each cell shows the number the kernel will read (the dequantized value if the cell is quantized). Key groups are the \(R\) tokens of one channel (\(G = R\) here), and a value group is one whole token. Each decode token takes three steps: append, quantize, attend. The bars show the attention probabilities with exact keys (outline) and with the KIVI cache (filled). The error chart compares the output with exact attention, for KIVI and for a naive cache that quantizes every key and value per token at once, with no residual. Memory counts \(b + 1\) bits per quantized number (as if the groups had \(G = 32\) numbers, formula 110.3) and 16 bits per residual number.
How to read Figure 3

The first step is the prefill: with \(R = 4\), keys 1–4 are quantized as one group per channel and key 5 waits in the buffer; value 1 is quantized and values 2–5 sit in the queue. Press Step: a new token is appended (line 2), then lines 3 and 4 run or are skipped, then the attention (lines 5 and 6). The live box shows each formula with its numbers: the zero point and step of every key channel when a block of keys is quantized, and the codes of the value that leaves the queue. Watch the full-precision keys go 1, 2, 3, 0, 1, 2, 3, 0 while the value queue always holds 4. At 2 bits the KIVI error stays below the naive error at every decode step (mean 0.138 against 0.359); at the end it is 0.296 against 0.466, with 4.63 bits per number against 3. Try \(R = 8\): the mean error falls to 0.086, but the cache costs 6.25 bits at 16 tokens. In a real model the residual is fixed (\(R = 128\)) and the context is long, so this cost shrinks to about 3.05 bits at 32k, formula (110.7).

What the residual costs

The full-precision tokens cost 16 bits each. Count the worst case, \(R\) residual tokens for both keys and values, at a context of \(T\) tokens:

\[ \bar b(T) = \frac{(T - R)\,\big(b + 32/G\big) + 16\,R}{T} \qquad (T \gt R) \tag{110.7} \]
Read the symbols
\(b + 32/G\)
bits per quantized number, codes plus scale and zero point, formula (110.3)
\(16 R\)
the residual in 16 bits
\(T\)
current context length. The residual is fixed, so its share shrinks as \(T\) grows.
Worked example: bits and bytes of KIVI-2 (\(b = 2, G = 32, R = 128\))

At \(T = 4096\): \(\bar b = (3968 \times 3 + 2048)/4096 = 3.41\) bits. At \(T = 32{,}768\): 3.05 bits, the exact number KVQuant’s comparison table gives for KIVI at 32k context. At \(T = 131{,}072\): 3.01 bits.

Llama 3 8B, one sequence of 131,072 tokens: 17.18 GB in bfloat16, 8.59 GB in FP8, 5.38 GB with KIVI-4 (5.01 bits), 3.24 GB with KIVI-2. That is 5.3× smaller than bfloat16, not 8×. [Certain, computed with (110.1) and (110.7)]

On an 80 GB GPU with \(u = 0.9\) and bfloat16 weights, the room after the weights (55.9 GB) holds about 427,000 tokens in bfloat16 and about 2.28 million tokens with KIVI-2.

KIVI’s own results, with this setting: on LongBench the average for Llama-2-7B is 44.52 in 16 bits, 44.59 for KIVI-4 and 44.27 for KIVI-2; for Mistral-7B 46.58, 46.56 and 45.85. [Certain, KIVI tables] On short tasks the 2-bit method is close to 16 bits. We will see later that a harder long-context test tells a different story.

KVQuant: four ideas for 3 bits and below

KVQuant starts from the same observation (keys per channel, values per token) and asks how to go below 4 bits without losing accuracy. On LLaMA-7B, Wikitext-2 perplexity in fp16 is 5.68. Plain uniform 3-bit with per-token groups gives 10.87; switching keys to per channel gives 7.05. [Certain, KVQuant paper] Then it adds four more ideas.

1. Quantize keys before RoPE

RoPE (session 14) rotates each pair of key channels \((a, b)\) by an angle that grows with the position \(t\). With the pairing \((i, i + d_h/2)\) used in Llama:

\[ \begin{bmatrix} a'_t \\ b'_t \end{bmatrix} = \begin{bmatrix} \cos t\theta_i & -\sin t\theta_i \\ \sin t\theta_i & \cos t\theta_i \end{bmatrix} \begin{bmatrix} a_t \\ b_t \end{bmatrix}, \qquad \theta_i = 10000^{-2i/d_h} \tag{110.8} \]
Read the symbols
\(a_t, b_t\)
channels \(i\) and \(i + d_h/2\) of the key of token \(t\) before RoPE (the output of \(W_K\))
\(a'_t, b'_t\)
the same channels after RoPE, which is what most caches store
\(\theta_i\)
the rotation speed of pair \(i\): 1 radian per token for \(i = 0\), about \(10^{-4}\) for the last pair

Suppose channel \(a\) is an outlier channel: always near 5, while \(b\) is small. Before RoPE, the column of \(a\) is almost constant: a perfect per-channel group. After RoPE, \(a'_t = a_t \cos t\theta_i - b_t \sin t\theta_i\) swings between about \(-5\) and \(+5\) as the position changes, and the outlier also leaks into \(b'\). Both columns now have a range of about 10, and per-channel quantization loses its advantage. [Certain, from (110.8)]

KVQuant therefore stores keys before RoPE, \(\hat{\mathbf{k}}_j = \text{quant}(W_K \mathbf{h}_j)\), and applies the rotation after dequantization, inside the attention kernel:

\[ \text{score}_{t,j} = \mathbf{q}_t^{\top} R_{\Theta, j}\, \hat{\mathbf{k}}_j, \qquad \hat{\mathbf{k}}_j = \text{dequant}\big(\text{quant}(W_K \mathbf{h}_j)\big) \tag{110.9} \]

Here \(R_{\Theta, j}\) is the block-diagonal RoPE rotation at position \(j\) and \(\mathbf{q}_t\) is the query, already rotated to its own position. The cost: the kernel must compute \(\cos j\theta_i\) and \(\sin j\theta_i\) for every cached token at every step. KVQuant fuses this into its kernel. [Certain, KVQuant paper] The gain on LLaMA-7B at 3 bits: perplexity 7.05 → 6.23. [Certain] The session 80 session noted that StreamingLLM also stores keys before rotation, for a different reason: it re-assigns positions inside the cache.

Figure 4. One pair of key channels over 64 positions, in a head of size \(d_h = 128\). Channel \(a\) (orange) is an outlier channel near 5; channel \(b\) (green) is small. Left: before RoPE. Right: after RoPE with formula (110.8) for pair \(i\). The thin lines are the \(2^b\) levels of a per-channel quantizer for channel \(a\) over these 64 tokens. The readout compares the relative error of quantizing after RoPE with quantizing before RoPE and rotating afterwards.
How to read Figure 4

At pair 8 (\(\theta = 0.316\), about 3 turns in 64 tokens) the rotated channel \(a'\) swings from −5 to +5, so its quantizer levels spread over a range of 11.7 instead of 2.5. At 3 bits the error after RoPE is 0.126, about 5× the 0.024 of the pre-RoPE path. Move the slider right. Slow pairs (\(i \ge 40\)) barely rotate over 64 tokens, so the two errors become equal. In a real model with thousands of tokens even slow pairs turn a lot: pair 63 turns 0.6 times over 32k tokens. How much pre-RoPE helps depends on which pairs hold the outliers.

2. Non-uniform levels, weighted by sensitivity

Uniform levels are equally spaced. But keys and values are not uniformly spread: most numbers are near the center. Session 105 showed NF4, whose levels follow a normal distribution. KVQuant goes one step further and learns the levels per layer, offline, from calibration data. Each channel (for keys) or token (for values) is first normalized to \([-1, 1]\); then one set of \(2^b\) levels per layer is chosen by solving

\[ Q^{*} = \argmin_{Q} \sum_{i=1}^{N} \mathcal{F}_{ii} \big( A_i - Q(A_i) \big)^2, \qquad \mathcal{F}_{ii} \approx \mathbb{E}\big[\, g_i^2 \,\big] \tag{110.10} \]
Read the symbols
\(A_i\)
one key or value number from the calibration set (after normalization)
\(Q(A_i)\)
the nearest of the \(2^b\) levels
\(g_i\)
the gradient of the loss with respect to \(A_i\)
\(\mathcal{F}_{ii}\)
the diagonal of the Fisher information: how much the loss reacts to an error in \(A_i\). Numbers with a large \(\mathcal{F}_{ii}\) get levels close to them.

This is a weighted k-means in one dimension: the levels are the cluster centers. The paper calibrates on 16 samples of 2k tokens from Wikitext-2; at run time the stored levels are only rescaled per channel or per token. These formats are called nuq2, nuq3 and nuq4. [Certain, KVQuant paper] On LLaMA-7B, nuq3 improves on uniform 3-bit by about 0.29 perplexity. [Certain that the paper reports this]

Algorithm 2: KVQuant offline calibration and online quantization
  1. offline, once per model: run the fp16 model on calibration text (16 × 2k tokens); keep the pre-RoPE keys and the values of every layer, and the gradients of the loss with respect to them
  2.   for every layer: normalize each key channel and each value token to \([-1, 1]\); ignore the first token // it stays fp16 (idea 4)
  3.   choose the \(2^b\) levels by Fisher-weighted k-means, formula (110.10)
  4.   for keys: store per-channel outlier thresholds (the 1% tails) // keys: thresholds known offline
  5. online, for each new token: key \(\mathbf{k} \leftarrow W_K \mathbf{h}\), before RoPE
  6.   split into dense + sparse with the per-vector thresholds, formula (110.11); values: compute the token’s thresholds now
  7.   normalize the dense part with the channel (key) or token (value) scale; store the nearest nuq level for each number
  8.   store the sparse outliers in fp16 with their positions (a compressed sparse format)
  9. attention: dequantize, add the sparse part, apply RoPE to keys on the fly, formula (110.9)

3. Dense-and-sparse: keep the 1% largest numbers

A few very large numbers still stretch the range of each group. KVQuant removes them: each key channel and each value token is split into a dense part, quantized with nuq, and a sparse part with the largest values, stored exactly:

\[ \mathbf{x} = \mathbf{x}_{\text{dense}} + \mathbf{x}_{\text{sparse}}, \qquad (\mathbf{x}_{\text{sparse}})_i = x_i \cdot \mathbb{1}\big[\, x_i \lt \tau_{\text{lo}} \ \text{or}\ x_i \gt \tau_{\text{hi}} \,\big] \tag{110.11} \]
Read the symbols
\(\tau_{\text{lo}}, \tau_{\text{hi}}\)
thresholds per vector (per channel for keys, per token for values), set so that about 1% of the numbers fall outside them
\(\mathbf{x}_{\text{sparse}}\)
stored in fp16 in a sparse format; the dense part now has a much smaller range

For keys the thresholds come from calibration. For values they must be found online for every new token; KVQuant runs this top-k search on the CPU in parallel with the GPU. [Certain, KVQuant paper] At 3 bits on LLaMA-7B, 1% outliers improve perplexity from 5.94 to 5.75; one threshold per matrix instead of per vector gives only 5.85. [Certain] The sparse part is not free: with 1% outliers the reported average is 4.33 bits for nuq4, 3.33 for nuq3 and 2.33 for nuq2, a third of a bit for outliers and scales. [Certain for the numbers]

4. Keep the first token in full precision

The first token of a sequence often gets a large share of the attention in every head, the attention sink seen in StreamingLLM (session 116). An error in its key or value is multiplied by a large \(p_j\) in (110.5). KVQuant keeps this one token in fp16 and leaves it out of calibration. At 2 bits on LLaMA-7B this alone takes perplexity from 8.47 to 7.23. [Certain, KVQuant paper] It costs nothing measurable: one token out of thousands.

Method (LLaMA-7B, Wikitext-2, fp16 = 5.68)4-bit3-bit2-bit
uniform int, per token5.9810.8711,779
NF (normal float), per token5.877.333,210
Atom5.776.1737.37
FlexGen5.735.9311.09
KVQuant (nuq)5.725.877.23
KVQuant, 1% outliers5.695.756.01

[Certain, KVQuant Table 1; all baselines as run by the KVQuant authors] By perplexity, 3-bit KVQuant with 1% outliers is 0.07 above fp16, and even 2 bits is only 0.33 above. Is a 2-bit cache then ready to ship? The next section says no.

Measure long-context accuracy, not only perplexity

Perplexity averages the loss over all tokens of a text. Most tokens can be predicted from the last few hundred tokens, and those are exactly the tokens that a KIVI residual keeps in full precision, or that get most of the attention anyway. A cache that ruins the memory of token 3,000 in a 30,000-token context can still have almost the same perplexity. The tasks that use the far context are the ones that break.

So test with tasks that need the far context:

  • Passkey retrieval: a random number is hidden in a long filler text; the model must repeat it. The score is the fraction of correct answers.
  • LongBench (Bai et al., 2023): the 16 English tasks in KVQuant’s table cover long-document QA, multi-document QA, summarization, few-shot learning, code, and synthetic retrieval and counting.
  • RULER (Hsieh et al., 2024): synthetic tasks at a chosen length: needle-in-a-haystack with one or many keys and values, multi-hop variable tracking, word counting and QA. Many models score far below their claimed context length on it. [Certain, RULER paper]

KVQuant ran all three on LLaMA-2-7B-32K (a 32k-context version of Llama 2 7B), with KIVI-2 (\(G = 32\), \(R = 128\)) as a baseline. [Certain, KVQuant paper] Figure 5 shows the task scores.

Figure 5. Long-context results on LLaMA-2-7B-32K as reported in the KVQuant paper (Tables 2, 3 and 4). Bars are grouped by task. The average bit widths are the paper’s estimates (for KIVI they include the fp16 residual at the benchmark’s context length). All runs are by the KVQuant authors, so KIVI is a baseline here, not tuned by its own authors.
How to read Figure 5

RULER is the default. KVQuant at 3.33 bits stays close to fp16 on most tasks (average 53.65 against 56.40). KIVI-2 at 3.05 bits drops on the single-needle tasks (76 against 100 on Niah1) and on multi-key retrieval (MKey2: 11.4 against 68.2); its average is 39.78. KVQuant at 2.33 bits, which had a perplexity only 0.33 above fp16, falls to 36.54. Click “change vs fp16” to see the losses as negative bars. Switch to Passkey: KIVI finds the passkey only 68–76% of the time at every length, KVQuant almost always. On LongBench the gaps are smaller (averages 31.96, 31.21 and 30.04): it mixes many tasks that a short local context can partly solve.

Put the numbers side by side. KVQuant 2-bit with 1% outliers: perplexity +0.33 (6.01 against 5.68), RULER −19.86 points (36.54 against 56.40), a 35% relative loss. KVQuant 3-bit: perplexity +0.07, RULER −2.75 points (−4.9%). KIVI-2: LongBench −1.92 points, RULER −16.62 points (−29.5%). [Certain for the arithmetic on the reported numbers] The perplexity ranking and the RULER ranking agree in direction, but perplexity hides the size of the failure.

A good evaluation plan for a quantized cache:

  1. Perplexity on long documents, as a quick sanity check (catches broken kernels).
  2. Needle-style retrieval (passkey, RULER needles) at several lengths up to the target context, and several needle depths.
  3. Multi-key and multi-value retrieval and variable tracking (RULER): these break first.
  4. Your real long tasks (LongBench-style QA and summarization, or your RAG set), with the same prompt format as production.
  5. Short reasoning tasks with long outputs (GSM8K): the residual covers the prompt only at the start, and errors in the generated chain add up.

Plan the memory

Figure 6 combines everything: weights + KV cache must fit into \(u \cdot M_{\text{GPU}}\), with the average bits \(\bar b\) of each format. For KIVI it uses formula (110.7) at the chosen context; for KVQuant the reported 4.33, 3.33 and 2.33 bits; for FP8 8 bits (one scale per tensor, as in vLLM’s --kv-cache-dtype fp8).

Figure 6. Memory of weights plus KV cache, formula (110.1), against the usable budget \(0.9 \cdot M_{\text{GPU}}\) (dashed line). The light bar behind shows the bfloat16 cache for comparison. Parameter counts come from the model config files; 4-bit weights ignore their small scale overhead. The readout gives the average bits, the cache size, and the longest context that fits at this batch size.
How to read Figure 6

The default is Llama 3 8B, one user at 128k tokens: weights 16.06 GB plus a 17.18 GB bfloat16 cache. Pick KIVI-2: the cache shrinks to 3.24 GB at 3.01 bits. Now raise the batch to 16: bfloat16 needs 275 GB and does not fit; KIVI-2 needs 52 GB and fits. Move the context down to 4,096: KIVI-2’s average rises to 3.41 bits because the 128-token residual is a bigger share. Pick Llama 3 70B on 2 × 80 GB: with 16-bit weights there is almost no room left for any cache; switch to 4-bit weights and a low-bit cache to serve long contexts.

A complete example in PyTorch

The code builds a synthetic cache of one head (512 tokens, \(d_h = 128\)) with the two patterns of this session: three fixed outlier channels in the keys, and a few large tokens in the values. It compares the four granularity choices at 2 and 4 bits, runs a KIVI-style streaming cache, measures pre-RoPE against post-RoPE key quantization, and tests dense-and-sparse values. It uses a fixed seed and only CPU operations, so it is deterministic.

import torch
torch.manual_seed(0)
T, d = 512, 128                                   # tokens, head size

# Synthetic cache with the patterns of section 3:
K = torch.randn(T, d)
K[:, [5, 70, 100]] += torch.tensor([9.0, -7.0, 6.0])   # keys: 3 fixed outlier channels
V = torch.randn(T, d)
V[torch.randperm(T)[:8]] *= 6                      # values: a few large tokens, no outlier channels
q = torch.randn(d)

def quant(X, bits, dim, G):
    """Min-max (asymmetric) round-to-nearest, formula (110.2).
    dim = the axis the statistics run along; groups of G elements on that axis."""
    Xt = X.movedim(dim, -1)
    Xg = Xt.reshape(*Xt.shape[:-1], -1, G)          # (..., n_groups, G)
    mn = Xg.amin(-1, keepdim=True); mx = Xg.amax(-1, keepdim=True)
    s = (mx - mn).clamp(min=1e-6) / (2 ** bits - 1)
    Xq = torch.round((Xg - mn) / s) * s + mn        # quantize, then dequantize
    return Xq.reshape(Xt.shape).movedim(-1, dim)

def attn(q, K, V):
    p = torch.softmax(K @ q / d ** 0.5, dim=0)
    return p @ V
rel = lambda a, b: ((a - b).norm() / b.norm()).item()
o = attn(q, K, V)

# 1) granularity: per-token = stats over channels (dim=1); per-channel = over tokens (dim=0)
for kn, kd in [("K per-token  ", 1), ("K per-channel", 0)]:
    for vn, vd in [("V per-token  ", 1), ("V per-channel", 0)]:
        e2 = rel(attn(q, quant(K, 2, kd, 32), quant(V, 2, vd, 32)), o)
        e4 = rel(attn(q, quant(K, 4, kd, 32), quant(V, 4, vd, 32)), o)
        print(f"{kn} {vn} output error: 2-bit {e2:.3f}, 4-bit {e4:.3f}")

# 2) a KIVI-style streaming cache: quantized part + full-precision residual
class KIVICache:
    def __init__(s, bits=2, G=32, R=128):
        s.bits, s.G, s.R = bits, G, R
        s.Kq, s.Kr, s.Vq, s.Vr = [], [], [], []
    def append(s, k, v):
        s.Kr.append(k); s.Vr.append(v)
        if len(s.Kr) == s.R:                        # keys: wait for R tokens, then
            s.Kq.append(quant(torch.stack(s.Kr), s.bits, 0, s.G))   # per-channel
            s.Kr = []
        if len(s.Vr) > s.R:                         # values: oldest token leaves the
            s.Vq.append(quant(s.Vr.pop(0)[None], s.bits, 1, s.G))  # window, per-token
    def get(s):
        K_ = torch.cat(s.Kq + ([torch.stack(s.Kr)] if s.Kr else []))
        V_ = torch.cat(s.Vq + [torch.stack(s.Vr)])
        return K_, V_
    def avg_bits(s):
        n = sum(x.shape[0] for x in s.Kq) + sum(x.shape[0] for x in s.Vq)
        r = len(s.Kr) + len(s.Vr)
        return (n * (s.bits + 32 / s.G) + r * 16) / (n + r)

c = KIVICache()
for t in range(T - 12):                             # stream 500 tokens one by one
    c.append(K[t], V[t])
Kc, Vc = c.get()
print("K: quantized", sum(x.shape[0] for x in c.Kq), "residual", len(c.Kr),
      "| V: quantized", len(c.Vq), "residual", len(c.Vr))
print(f"KIVI-2 output error {rel(attn(q, Kc, Vc), attn(q, K[:500], V[:500])):.3f}, "
      f"average bits {c.avg_bits():.2f}")

# 3) pre-RoPE vs post-RoPE keys: rotate pairs (i, i + d/2) by angle pos * theta_i
pos = torch.arange(T, dtype=torch.float32)[:, None]
theta = 10000 ** (-torch.arange(d // 2, dtype=torch.float32) / (d // 2))
cos, sin = (pos * theta).cos(), (pos * theta).sin()
def rope(X):
    a, b = X[:, : d // 2], X[:, d // 2 :]
    return torch.cat([a * cos - b * sin, a * sin + b * cos], dim=1)
Kr = rope(K)
post = rel(quant(Kr, 3, 0, 32), Kr)                 # quantize the rotated keys
pre = rel(rope(quant(K, 3, 0, 32)), Kr)             # quantize first, rotate after
print(f"3-bit per-channel key error: post-RoPE {post:.3f}, pre-RoPE {pre:.3f}")

# 4) dense-and-sparse: keep the 1% largest |values| in full precision
def dense_sparse(X, bits, dim, G, frac=0.01):
    thr = X.abs().flatten().kthvalue(int((1 - frac) * X.numel())).values
    mask = X.abs() > thr                            # the sparse outliers
    Xd = quant(torch.where(mask, torch.zeros_like(X), X), bits, dim, G)
    return torch.where(mask, X, Xd), mask.float().mean().item()
Vds, f = dense_sparse(V, 2, 1, 32)
print(f"2-bit V per-token error {rel(quant(V, 2, 1, 32), V):.3f}, "
      f"with {100 * f:.1f}% outliers kept {rel(Vds, V):.3f}")

Output (PyTorch 2.8, CPU):

K per-token   V per-token   output error: 2-bit 1.205, 4-bit 0.343
K per-token   V per-channel output error: 2-bit 1.385, 4-bit 0.350
K per-channel V per-token   output error: 2-bit 0.592, 4-bit 0.115
K per-channel V per-channel output error: 2-bit 0.758, 4-bit 0.117
K: quantized 384 residual 116 | V: quantized 372 residual 128
KIVI-2 output error 0.563, average bits 6.17
3-bit per-channel key error: post-RoPE 0.138, pre-RoPE 0.111
2-bit V per-token error 0.394, with 1.0% outliers kept 0.315

Read the results. (1) Keys per channel halve the output error at 2 bits and cut it 3× at 4 bits; the value choice matters less here, but per-token is better at 2 bits. (2) The streaming cache holds 384 keys in 3 quantized blocks of 128 and 116 in the buffer, while values have 372 quantized and exactly 128 in the queue: the two boundaries differ, as in Figure 3. At only 500 tokens the residual dominates the cost (6.17 bits); at 32k it would be about 3.05. (3) Pre-RoPE quantization lowers the key error from 0.138 to 0.111. (4) Keeping 1% of values exact lowers their error by 20%. The errors are large because Gaussian toy data at 2 bits is a hard case; real caches have more structure. [Certain for the printed numbers, run twice with identical output]

Where this is used

SettingWhy the cache must shrinkTypical choice
High-throughput chat servingBatch size is limited by \(\sum_i T_i \cdot s\), formula (110.1)FP8 cache (vLLM, TensorRT-LLM) as a safe first step; 4-bit with care [Likely]
Long-document QA, RAG over long contextsOne prompt of 32k–128k tokens can need as much memory as the weightsKIVI-4 or KVQuant-3/4; test with RULER at the real length
Agents with long tool tracesContext grows over many steps and must stay exact enough for retrieval4-bit with a residual; check multi-key retrieval
Single-GPU or edge long contextNo second GPU; weights already 4-bitKVQuant nuq2/nuq3 with outliers and kernels
Offloading to CPU memory (FlexGen)The cache moves over PCIe; fewer bytes means faster transfer4-bit group-wise cache [Certain, survey]
Video QA, Video-RAG, key-moment detectionThousands of frame tokens per video enter the cache (session 80: 64 frames × 576 tokens)Combine token pooling with a low-bit cache; test needle-in-the-video tasks
Streaming video and live captioningUnbounded input; old frames are rarely attended but must not be corruptedSliding window + attention-sink tokens in fp16 + low-bit older frames [Speculation]
VLM image QA with many imagesEach image adds hundreds of visual tokensSame rules; check that visual tokens have outlier channels like text [Likely]

Common mistakes

  • Quantizing keys per token because that is how activations are usually done. The outlier channels then destroy every key (Figure 2).
  • Quantizing values per channel by symmetry with keys. KIVI’s 2-bit test with both per channel gives 0.00 on GSM8K.
  • Quantizing keys after RoPE when the kernel can rotate on the fly. RoPE spreads the outlier channel over its pair (Figure 4).
  • Reporting the code width as the bit width. “2-bit” KIVI with \(G = 32\) is 3 bits plus the residual; KVQuant 2-bit with 1% outliers is 2.33 bits. Compare methods at their average bits (110.7).
  • Trusting perplexity alone. KVQuant 2-bit-1% is +0.33 perplexity but −19.86 RULER points.
  • Testing at short context. With a 128-token residual, a 200-token test barely touches the quantized part. Test at the real production length.
  • Quantizing the first token (the attention sink). It is one token, and keeping it exact took KVQuant 2-bit perplexity from 8.47 to 7.23.
  • Expecting speed without a kernel. Fake quantization (quantize then dequantize in PyTorch) saves no memory traffic. The gain needs kernels that read the low-bit codes.

Interview questions

When is the KV cache bigger than the weights? Give a number.basic
When the total number of cached tokens passes \(M_W / s\) (110.1). Llama 3 8B in bfloat16: \(16.06 \times 10^9 / 131{,}072 \approx 122{,}500\) tokens, so one user at 128k, or 32 users at 4k (17.2 GB of cache against 16.1 GB of weights). Llama 2 7B with MHA already crosses at about 25,700 tokens. Long context and big batches both push the cache past the weights.
Why are keys quantized per channel and values per token?core
Keys have a few fixed outlier channels that are large for almost every token. Per-token groups contain these outliers, so their step size is huge and the normal channels collapse to one or two levels; per-channel groups give the outlier channel its own scale. KIVI measured 47.00 vs 9.60 relative attention-score error on Llama-2-13B. Values have no outlier channels, and they enter the output as a probability-weighted sum of rows (110.5); per-token groups keep each important token accurate on its own scale. KIVI measured 3.55 vs 49.89 output error. At 2 bits, only keys-per-channel with values-per-token keeps GSM8K usable (5.76 vs 0.00–0.83 for the others; 12.74 with the full KIVI method).
Why does KIVI keep a full-precision residual of recent tokens?core
Per-channel key groups span \(G\) tokens, but decoding produces one token at a time, so the newest group is incomplete and its range unknown. KIVI keeps up to \(R\) keys in fp16 and quantizes them together when \(R\) have arrived. Values are kept in a queue of the last \(R\) tokens for symmetry and accuracy: the most recent tokens often get much attention, and keeping them exact helps. The cost is \(16R\) bits per channel, which is small at long context: with \(R = 128\), \(G = 32\) the average is 3.41 bits at 4k and 3.05 bits at 32k.
A colleague says “2-bit KV cache = 8× smaller than fp16”. Correct them.core
Count the overhead. Each group of \(G\) numbers stores a 16-bit scale and a 16-bit zero point: \(b + 32/G\) = 3 bits for \(b = 2\), \(G = 32\) (110.3). KIVI also keeps up to 128 tokens in fp16, so the average is \(((T-R) \cdot 3 + 16R)/T\) (110.7): 3.41 bits at 4k, 3.01 at 128k. The real ratio is about 5.3×, not 8×. KVQuant’s 2-bit with 1% outliers averages 2.33 bits (6.9×).
Why quantize keys before RoPE? What does it cost?core
RoPE rotates each channel pair \((a, b)\) by \(t\theta_i\) (110.8). If \(a\) is an outlier channel, after rotation both channels swing between about \(\pm|a|\) over positions, so each column has a large range and per-channel quantization loses its advantage. Stored before RoPE, the outlier column is nearly constant and quantizes well. The cost: the attention kernel must dequantize and apply RoPE at position \(j\) to every cached key in every step (110.9), which KVQuant fuses into its kernel. On LLaMA-7B at 3 bits, perplexity went from 7.05 to 6.23.
What is the Fisher-weighted non-uniform data type in KVQuant?advanced
A set of \(2^b\) levels per layer, learned offline. On calibration data (16 × 2k tokens), keys (per channel) and values (per token) are normalized to \([-1,1]\), and the levels are found by 1-D k-means that minimizes \(\sum_i \mathcal{F}_{ii} (A_i - Q(A_i))^2\) (110.10), where \(\mathcal{F}_{ii}\) is the squared gradient of the loss with respect to \(A_i\), averaged over data. Numbers the loss is sensitive to pull levels toward them. At run time only the per-channel or per-token scale is computed; the levels are a lookup table. It improved 3-bit LLaMA-7B by about 0.29 perplexity over uniform levels.
Why is a 2-bit cache with +0.33 perplexity not ready to ship?core
Perplexity averages over all tokens, and most tokens depend on nearby context, which stays accurate (residual, recent tokens, attention sink). Tasks that need one fact from far back are what break. On LLaMA-2-7B-32K, KVQuant 2-bit with 1% outliers has perplexity 6.01 vs 5.68, but RULER at 32k drops from 56.40 to 36.54 (−35% relative), with multi-value retrieval falling from 55.95 to 16.65. Ship only after needle, multi-key and real long-task tests at production length.
Why not use GPTQ or AWQ to quantize the KV cache?basic
They are offline methods for fixed weight matrices: they solve a small optimization per matrix using calibration data, which takes seconds to minutes. The KV cache is new data for every request and every token, quantized at decode time inside the serving loop. It needs a cheap online rule such as round-to-nearest with min–max scales (KIVI). KVQuant moves the expensive part offline (levels, key thresholds) and keeps the online part cheap.
How can a kernel compute attention on quantized keys and values without dequantizing every number?advanced
Expand the products (110.6). With per-channel key scales \(s_i\) and zero points \(z_i\): \(\mathbf{q} \cdot \hat{\mathbf{k}}_j = \sum_i (q_i s_i) c_{j,i} + \sum_i q_i z_i\): multiply the scales into the query once, and the second term is one number for all tokens. With per-token value scales: \(\sum_j p_j \hat{\mathbf{v}}_j = \sum_j (p_j s_j) \mathbf{c}_j + (\sum_j p_j z_j)\mathbf{1}\): fold the scales into the probabilities. With groups, one such correction per group. The inner loop then works on integer codes, and the memory traffic is the low-bit codes, which is where the decode speedup comes from.
Why does KVQuant keep the first token in fp16, and the 1% outliers per vector?advanced
The first token is usually an attention sink: many heads put a large probability on it, so any error in its key or value is amplified in (110.5). Keeping one token exact costs almost nothing and improved 2-bit LLaMA-7B perplexity from 8.47 to 7.23. The outliers: a few very large numbers set the range of a group; removing the top 1% per channel (keys) or per token (values) into a sparse fp16 store shrinks the range of the rest (110.11). Per-vector thresholds work better than one per matrix (5.75 vs 5.85 at 3 bits). The price is about 0.33 extra bits and a CPU top-k for values.
KIVI or KVQuant or FP8 for a new production service?advanced
Start from the need. If the cache is only somewhat too big, an FP8 cache is the safe first step: built into vLLM, simple per-tensor scales, 2× smaller. If you need 4–5× and want no calibration, KIVI-style 4-bit (or 2-bit with long-context tests) with \(G = 32\), \(R = 128\) is easy to integrate. If you need 3 bits or below at very long context, KVQuant’s pieces (pre-RoPE keys, nuq levels, 1% outliers, fp16 sink) give the best accuracy per bit but need calibration and custom kernels. In every case, decide with RULER-style tests at production length, and check the real kernel speed, not fake quantization.

Exercises

  1. Compute the average bits of KIVI-2 with \(G = 64\) and \(R = 128\) at \(T = 8192\). How much smaller than bfloat16 is the cache?
  2. Llama 3 70B (\(s = 327{,}680\) bytes in bfloat16, 141.1 GB of weights in 16 bits, 35.3 GB in 4 bits) serves 16 users at 32,768 tokens with KIVI-2 (\(G = 32\), \(R = 128\)). How big is the cache? Does it fit on 2 × 80 GB with 16-bit weights, or on one 80 GB GPU with 4-bit weights (\(u = 0.9\))?
  3. Quantize the key row \((0.3, -0.2, 7.9, 0.1)\) to 2 bits per token with (110.2). Give \(s\), the codes and \(\hat x\).
  4. Check (110.6) by hand: \(\mathbf{q} = (2, 1)\), per-channel scales \(s = (0.5, 0.1)\), zero points \(z = (-1, 3)\), codes \(c = (3, 2)\).
  5. With \(d_h = 128\) and \(\theta_i = 10000^{-2i/d_h}\), how many full turns does RoPE pair \(i = 0\) make over 32,768 positions? And pair \(i = 63\)?
  6. From Figure 5: compute the relative RULER loss (in % of the fp16 average 56.40) for KVQuant-3bit-1%, KIVI-2 and KVQuant-2bit-1%.
Answerscheck
1. \(b + 32/G = 2.5\); \(\bar b = (8064 \times 2.5 + 2048)/8192 = 2.71\) bits, so \(16 / 2.71 = 5.9\times\) smaller. 2. \(\bar b = (32{,}640 \times 3 + 2048)/32{,}768 = 3.05\) bits. The bfloat16 cache would be \(327{,}680 \times 32{,}768 \times 16 = 171.8\) GB; KIVI-2 makes it \(171.8 \times 3.05/16 = 32.8\) GB. With 16-bit weights: \(141.1 + 32.8 = 173.9 \gt 144\) GB, does not fit. With 4-bit weights on one GPU: \(35.3 + 32.8 = 68.0 \lt 72\) GB, fits. 3. \(z = -0.2\), \(s = 8.1/3 = 2.7\), codes \((0, 0, 3, 0)\), \(\hat x = (-0.2, -0.2, 7.9, -0.2)\): the small channels lose their values (errors 0.5 and 0.3). 4. \(\hat{\mathbf{k}} = (0.5 \cdot 3 - 1,\ 0.1 \cdot 2 + 3) = (0.5, 3.2)\), \(\mathbf{q} \cdot \hat{\mathbf{k}} = 1 + 3.2 = 4.2\). Folded: \((q \odot s) \cdot c = (1, 0.1) \cdot (3, 2) = 3.2\), plus \(\mathbf{q} \cdot z = -2 + 3 = 1\): 4.2. 5. Pair 0: \(\theta = 1\), \(32{,}768 / 2\pi \approx 5{,}215\) turns. Pair 63: \(\theta = 10000^{-126/128} \approx 1.155 \times 10^{-4}\), \(32{,}768 \times 1.155 \times 10^{-4} / 2\pi \approx 0.60\) turns. 6. KVQuant-3bit-1%: \(2.75 / 56.40 = 4.9\%\). KIVI-2: \(16.62 / 56.40 = 29.5\%\). KVQuant-2bit-1%: \(19.86 / 56.40 = 35.2\%\).

Key takeaways

  • The cache passes the weights when the total cached tokens reach \(M_W / s\): about 122k tokens for Llama 3 8B, 26k for Llama 2 7B (110.1). Then the bits per cached number are the main lever.
  • Keys have fixed outlier channels: quantize them per channel. Values have none and enter the output as weighted rows: quantize them per token (KIVI). The other three combinations fail at 2 bits.
  • Count real bits: \(b + 32/G\) per group, plus the fp16 residual (110.3, 110.7). “2-bit” KIVI is about 3.05 bits at 32k.
  • KIVI streams: keys wait in an fp16 buffer of \(R\) tokens and are quantized in blocks; values leave an fp16 queue one by one. No calibration.
  • KVQuant: pre-RoPE keys (rotate after dequantization), Fisher-weighted non-uniform levels, 1% per-vector outliers in fp16, and an fp16 first token. 3.33 bits with +0.07 perplexity on LLaMA-7B.
  • Per-channel key scales fold into the query and per-token value scales into the probabilities, so kernels can work on codes (110.6).
  • Judge a quantized cache by long-context tasks at production length (passkey, RULER, your own long tasks), not by perplexity: +0.33 perplexity was −35% on RULER.

Key formulas

\[ z = \min X, \quad s = \frac{\max X - \min X}{2^b - 1}, \quad \hat x = s \cdot \operatorname{round}\!\Big(\frac{x - z}{s}\Big) + z, \qquad \bar b = b + \frac{32}{G} \]
\[ \bar b(T) = \frac{(T - R)(b + 32/G) + 16 R}{T}, \qquad M_{\text{KV}} = 2 L\, n_{kv}\, d_h\, \frac{\bar b}{8} \sum_i T_i \]
\[ \mathbf{q} \cdot \hat{\mathbf{k}}_j = \sum_i (q_i s_i)\, c_{j,i} + \sum_i q_i z_i, \qquad Q^{*} = \argmin_Q \sum_i \mathcal{F}_{ii} (A_i - Q(A_i))^2 \]

References for this session

  1. Liu et al. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
  2. Hooper et al. (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
  3. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  4. Liu et al. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
  5. Hooper et al. (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
  6. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  7. Sheng et al. (2023). FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  8. Zhao et al. (2023). Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
  9. Ashkboos et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
  10. Hsieh et al. (2024). RULER: What’s the Real Context Size of Your Long-Context Language Models?
  11. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  12. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  13. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  14. Bai et al. (2023). LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
  15. Hugging Face (2025). Transformers documentation: cache strategies (QuantizedCache)