KV-cache quantization: KIVI and KVQuant
At long context or big batch, the KV cache can take more GPU memory than the model weights. This session shows how to store keys and values in 2 to 4 bits without breaking the model: keys per channel and values per token (KIVI), and non-uniform levels, pre-RoPE keys and a few kept outliers (KVQuant). It ends with the test that matters: long-context accuracy, not only perplexity.
What you will learn
- When the KV cache is bigger than the weights, and how many bytes a low-bit cache saves
- Why keys have outlier channels, and why keys must be quantized per channel but values per token
- How KIVI streams a 2-bit cache with a small full-precision window of recent tokens, and what that window costs in bits
- Why KVQuant quantizes keys before RoPE, and how it adds non-uniform levels, sparse outliers and a full-precision first token
- How per-channel and per-token scales fold into the attention matrix products
- How to evaluate a quantized cache: perplexity hides failures that passkey, LongBench and RULER show
Why this matters
A model of 8 billion parameters in bfloat16 takes 16 GB. Its KV cache for one user with 128k tokens of context also takes about 16 GB. For 32 users with 4k tokens each it takes 17 GB. At long context or big batch, the cache, not the model, fills the GPU. Storing keys and values in 2 to 4 bits instead of 16 is the most direct way to fit more context and more users.
Session 80 gave the size of the cache: \(s = 2 L\, n_{kv}\, d_h\, b\) bytes per token. Only the last factor, the bytes per number, is left after the attention design is fixed. The quantization sessions so far (session 103, session 107, session 108, session 109) made weights and activations small. The KV cache is a third kind of tensor. It is an activation (it depends on the input), but it is stored for a long time and read at every decode step, like a weight. So it has its own problems and its own methods.
The efficient-inference survey lists four methods in this area: FlexGen and Atom store the cache in INT4, KVQuant uses non-uniform levels found offline on calibration data, and KIVI uses a tuning-free 2-bit scheme with keys per channel and values per token. [Certain, survey Section 5.2.1] We study the last two in depth:
- KIVI (Liu et al., ICML 2024): look at the numbers in the cache, see that keys and values have different shapes of outliers, and quantize them along different axes. Store recent tokens in full precision. No calibration, no training.
- KVQuant (Hooper et al., 2024): keys per channel too, but quantized before RoPE; non-uniform levels chosen with a sensitivity weight; the 1% largest values kept apart in full precision; the first token kept in full precision.
Both papers report big gains. KIVI reports 2.6× less peak memory for Llama-2-7B (weights included), up to 4× larger batches and 2.35× to 3.47× more throughput on one A100. KVQuant reports less than 0.1 perplexity increase at 3 bits, and, from its memory calculation, LLaMA-7B with up to 1 million tokens of context on one A100-80GB. [Certain that the papers report these] The last part of this session shows why such numbers are not enough, and what to measure instead.
Interview questions on this topic: “Why are keys quantized per channel and values per token?”, “Why not use GPTQ for the KV cache?”, “Why quantize keys before RoPE?”, “Your 2-bit cache has the same perplexity; can you ship it?”. By the end you can answer each with a formula and a number.
When the cache is bigger than the weights
Take the weight memory \(M_W = N_{\text{params}} \cdot b_W\) and the cache memory from session 80. With a cache of \(\bar b\) bits per number, the cache is
- Read the symbols
- \(\bar b\)
- the average number of bits per stored key or value number, including the cost of scales, zero points and anything kept in full precision. 16 for bfloat16.
- \(s\)
- bytes per token at the current precision, \(2 L\, n_{kv}\, d_h \bar b / 8\)
- \(B, T_i\)
- number of sequences in the batch and the length of sequence \(i\)
- \(B\,T\)
- the total number of cached tokens on the GPU. The cache equals the weights when this total reaches \(M_W / s\).
Llama 2 7B (MHA, 32 layers, 32 KV heads, \(d_h = 128\)): \(s = 524{,}288\) bytes. Weights: 13.48 GB. The cache passes the weights at \(13.48 \times 10^9 / 524{,}288 \approx 25{,}700\) cached tokens, for example 7 users at 4k tokens.
Llama 3 8B (GQA, 8 KV heads): \(s = 131{,}072\) bytes, weights 16.06 GB. Crossover at about 122,500 tokens: one user at its full 128k context, or 32 users at 4,096 tokens (\(32 \times 4096 \times 131{,}072 = 17.18\) GB).
Llama 3 70B: \(s = 327{,}680\) bytes, weights 141.1 GB. 16 users at 32k tokens need 171.8 GB of cache, more than the weights.
So two settings make the cache the main cost: long context (RAG over long documents, agents, video) and big batches (any busy server). GQA already cut the cache of Llama 3 by 4× compared with MHA. Going from 16 to about 3 bits cuts it by about 5× more, and it needs no retraining.
Memory is not the only gain. Decoding is memory-bound: each step reads the weights once and every user’s cache once (session 80, formula 80.9). A cache that is 5× smaller is read 5× faster, if the kernel can read the low-bit codes directly. KVQuant reports custom kernels with up to about 1.7× speedup over the fp16 matrix-vector products. [Certain that the paper reports this] With a small cache, the server also fits a bigger batch, which spreads the weight reads over more users.
Quantization of a cache in one formula
Session 103 built uniform quantization. KIVI uses the asymmetric (min–max) version, with a zero point equal to the minimum of the group. For a group of numbers \(X\) and \(b\) bits:
- Read the symbols
- group \(X\)
- the set of numbers that share one scale and one zero point. Which numbers form a group is the whole design question of this session.
- \(z\)
- zero point: here the smallest value of the group, stored in 16 bits
- \(s\)
- scale: the step between two levels, stored in 16 bits
- \(c\)
- the code, a \(b\)-bit integer. Only the codes are stored per number.
- \(\hat x\)
- the value the attention kernel will use. The error is at most \(s/2\) per number.
Two facts follow at once. First, the error of every number in a group is set by the range of the group: one huge value makes the step \(s\) large for all the others. Second, each group pays 32 extra bits for its scale and zero point. With groups of \(G\) numbers the average cost is
So “2-bit KIVI” with \(G = 32\) really stores 3 bits per number, before counting the full-precision part. Keep this in mind whenever you compare methods by their bit width.
Keys have outlier channels, values do not
What the cache looks like
Write the key cache of one head as a matrix \(K \in \R^{T \times d_h}\): one row per token, one column per channel (one of the \(d_h\) numbers of a key). The value cache \(V\) has the same shape. KIVI plotted these matrices for Llama, Falcon and Mistral and found two different patterns. [Certain, KIVI paper]
- Keys: a few fixed channels have very large values for almost every token. Inside one channel, the values over tokens are similar. LLM.int8() (session 108) found the same kind of fixed outlier dimensions in hidden states.
- Values: no clear outlier channels. The size of the values changes more from token to token than from channel to channel.
Now choose the groups of (110.2). There are two natural choices for a matrix of shape (tokens × channels):
- Read the symbols
- \(K_{j,i}\)
- channel \(i\) of the key of token \(j\)
- per-token group
- one row: all channels of one token share a scale. This is how activations are usually quantized, and how a new token arrives.
- per-channel group
- one column, cut into pieces of \(G\) tokens: one channel over \(G\) tokens shares a scale
With per-token groups, every row of \(K\) contains the outlier channels, so every row has a huge range and a huge step \(s\). All the normal channels fall onto one or two levels. With per-channel groups, the outlier channel gets its own large scale, and the normal channels keep fine steps. The error stays inside the outlier channel. [Certain, KIVI paper]
Four tokens, four channels; channel 3 is an outlier channel:
Per-token, row 1: \(z = -0.4\), \(s = 9.4 / 3 = 3.133\). The codes are \((0, 0, 3, 0)\): the three small channels all become \(-0.4\). Every row looks the same: codes \((0, 0, 3, 0)\), so the small channels lose all their information. The largest error in the small channels is 0.8.
Per-channel over the 4 tokens: the scales are \(0.233, 0.300, 0.467, 0.167\). The largest error in the small channels is 0.1; in the outlier channel it is 0.133.
Now a query \(\mathbf{q} = (1, 1, 0, 1)\) that reads only the small channels. The true scores \(K\mathbf{q}\) are \((-0.1, 0.0, 0.8, -0.4)\). Per-token gives \((-1.2, -0.9, 0.3, -0.9)\): wrong order, wrong sizes. Per-channel gives \((-0.1, 0.0, 0.9, -0.5)\), almost exact. [Certain, computed in Python]
Figure 1 runs formula (110.2) on this matrix, one group at a time. For each group it finds the zero point and the step, rounds every number to a code, and shows on a number line where each number lands. The last step multiplies the result by the query.
- for each group \(X\) // a row (per token) or a column (per channel)
- \(z \leftarrow \min X\), \(\;s \leftarrow (\max X - \min X)/(2^b - 1)\)
- \(c \leftarrow \operatorname{round}\big((x - z)/s\big)\) for every \(x \in X\)
- \(\hat x \leftarrow c\, s + z\), error \(|\hat x - x| \le s/2\)
- scores \(\hat K \mathbf q\) against \(K \mathbf q\), with \(\mathbf q = (1, 1, 0, 1)\)
How to read Figure 1
Press Play or Step. The highlighted lines of the algorithm box are the ones that run in this step, and the box next to it shows the same formulas with the real numbers. With rows as groups at 2 bits, row 1 has \(z = -0.4\) and \(s = 9.4/3 = 3.133\): on the number line, the three small numbers all snap to the lowest level, and the codes are \((0, 0, 3, 0)\) in every row. The largest error in the small channels is 0.80 and the scores end at \((-1.2, -0.9, 0.3, -0.9)\) against the true \((-0.1, 0.0, 0.8, -0.4)\). Switch to columns: each column gets its own small step (0.233, 0.300, 0.467, 0.167), the errors fall to at most 0.13, and the scores are \((-0.1, 0.0, 0.9, -0.5)\). At 4 bits with rows, the small channels still have errors up to 0.30, worse than 2 bits with columns (0.10).
KIVI measured the same effect on Llama-2-13B, averaged over layers and heads: per-token key quantization gives a relative attention-score error of 47.00, per-channel gives 9.60, about 5× smaller. [Certain, KIVI Table 2]
Why values are different
Values enter attention in a different way. The output of one head is a weighted sum of value rows:
- Read the symbols
- \(p_j\)
- attention probability of token \(j\); most are near zero, a few are large (KIVI reports an average attention sparsity of 84.3%)
- \(\hat{\mathbf{v}}_j - \mathbf{v}_j\)
- the quantization error of the value row of token \(j\)
The output error is a mix of row errors, weighted by \(p_j\). What matters is that the few tokens with large \(p_j\) have accurate value rows. Per-token quantization gives each token its own scale, fitted to that token’s own range, so “the error of one token does not affect the others” (KIVI’s words). Per-channel quantization lets a few large tokens set the scale of a whole column, and an important token with small values then gets a coarse grid. [Likely, our reading of the KIVI argument] KIVI’s measurement: the relative output error is 3.55 for per-token values and 49.89 for per-channel values, about 14× larger. [Certain, KIVI Table 2]
The task results agree. On Llama-2-7B with 2-bit fake quantization (quantize then dequantize, no real kernel), GSM8K accuracy is 13.50 in 16 bits, 5.76 with keys per channel and values per token, 0.83 with both per token, and 0.00 with both per channel. [Certain, KIVI paper] The only working 2-bit combination is keys per channel, values per token. The full KIVI method (below) brings GSM8K back to 12.74.
Figure 2 lets you try all four combinations on a small cache with two outlier key channels.
How to read Figure 2
Start at 2 bits with both per token. The key map is dark everywhere except the outlier channels: each row’s range is about 10, so the normal channels lose their values, and the attention bars move far from their outlines. The readout shows a key error of 0.368 and an attention-score error of 1.419. Switch keys to per channel: the key error falls to 0.070, the score error to 0.299, and the bars almost match. Now switch values to per channel: the value error rises from 0.221 to 0.350, because tokens 4 and 10 set the range of every column. The output error is smallest with keys per channel and values per token (0.331 at 2 bits). At 4 bits all choices are closer together: granularity matters most when bits are few.
The same groups fit the matrix products
There is also a reason from the kernel side. A decode step computes scores \(\mathbf{q} \cdot \hat{\mathbf{k}}_j\) (a sum over channels \(i\)) and an output \(\sum_j p_j \hat{\mathbf{v}}_j\) (a sum over tokens \(j\)). Put \(\hat K_{j,i} = s_i c_{j,i} + z_i\) (per-channel keys) and \(\hat{\mathbf{v}}_j = s_j \mathbf{c}_j + z_j \mathbf{1}\) (per-token values, one group per row for simplicity):
- Read the symbols
- \(c_{j,i}, \mathbf{c}_j\)
- the stored integer codes
- \(q_i s_i\)
- the per-channel key scales can be multiplied into the query once per step, and \(\sum_i q_i z_i\) is one number shared by all tokens
- \(p_j s_j\)
- the per-token value scales can be multiplied into the attention probabilities
In both products the scale runs along the axis that is summed. So the scales can be moved out of the inner loop, and the kernel can multiply directly with the integer codes. [Certain, algebra] Grouping keys per token would put the scale on the other axis, which is just as easy for the score product; the problem with per-token keys is accuracy, not speed. Real kernels with groups of \(G\) need one such correction per group. [Likely]
KIVI: a streaming 2-bit cache
The streaming problem
Per-channel keys create a practical problem. A per-channel group covers \(G\) tokens of one channel. During decoding, tokens arrive one at a time: the group of the newest tokens is not complete yet, so its range is unknown. Values have no such problem: a per-token group is complete as soon as the token arrives.
KIVI’s answer is a small residual: the most recent tokens stay in full precision. [Certain, KIVI paper]
- Keys: new keys go into a full-precision buffer. When the buffer holds \(R\) tokens, all of them are quantized per channel (in groups of \(G\) tokens), appended to the quantized key cache, and the buffer is emptied.
- Values: new values go into a full-precision queue of length \(R\). When the queue is longer than \(R\), the oldest value leaves it and is quantized per token (in groups of \(G\) channels).
- Attention: each step attends to the quantized part and the full-precision part and joins the two results. In prefill, attention uses the exact keys and values; only the stored cache is quantized.
The paper uses \(G = 32\) and \(R = 128\) in its main experiments. [Certain] The residual has a second benefit: the most recent tokens often get much of the attention, and they are exact. In an ablation on Llama-2-13B GSM8K, \(R = 32, 64, 96, 128\) gave 20.62, 19.86, 20.55 and 20.77 (no clear trend in this range), while \(G = 32, 64, 128\) gave 20.77, 21.00 and 17.29: groups that are too large hurt. [Certain, KIVI ablation tables]
- state: quantized keys \(\hat K\) (per channel), full-precision key buffer \(K_r\) (\(\lt R\) rows); quantized values \(\hat V\) (per token), full-precision value queue \(V_r\) (\(\le R\) rows)
- input: the new token’s \(\mathbf{q}, \mathbf{k}, \mathbf{v}\)
- append \(\mathbf{k}\) to \(K_r\); append \(\mathbf{v}\) to \(V_r\)
- if \(K_r\) has \(R\) rows: \(\hat K \leftarrow [\hat K;\ \text{quant}_{\text{channel}}(K_r, G)]\); \(K_r \leftarrow \varnothing\) // groups of \(G\) tokens per channel
- if \(V_r\) has \(R+1\) rows: pop the oldest row \(\mathbf{v}_{\text{old}}\); \(\hat V \leftarrow [\hat V;\ \text{quant}_{\text{token}}(\mathbf{v}_{\text{old}}, G)]\) // groups of \(G\) channels
- scores \(\leftarrow [\,\hat K \mathbf{q};\ K_r \mathbf{q}\,] / \sqrt{d_h}\) // low-bit kernel + normal kernel, formula (110.6)
- \(\mathbf{p} \leftarrow \softmax(\text{scores})\) over all cached tokens
- return \(\mathbf{o} = \mathbf{p}_{\text{old}}^{\top} \hat V + \mathbf{p}_{\text{new}}^{\top} V_r\) // split \(\mathbf{p}\) by where each value lives
Note that the key and value boundaries are not at the same token: keys are quantized in blocks of \(R\), values one by one. The softmax runs over all tokens at once; only the matrix products are split. Figure 3 runs this algorithm on a toy cache with real numbers, one line at a time, and tracks the error and the memory as tokens arrive.
- prefill \(P\) tokens with exact \(K, V\); store: first \(\lfloor P/R \rfloor R\) keys per channel, all but the last \(R\) values per token
- new token: append \(\mathbf k\) to \(K_r\), \(\mathbf v\) to \(V_r\)
- if \(|K_r| = R\): quantize \(K_r\) per channel, append to \(\hat K\), \(K_r \leftarrow \varnothing\)
- if \(|V_r| = R + 1\): pop the oldest \(\mathbf v\), quantize it per token, append to \(\hat V\)
- \(\mathbf p \leftarrow \softmax\big([\hat K \mathbf q;\ K_r \mathbf q] / \sqrt{d_h}\big)\)
- \(\mathbf o \leftarrow \mathbf p_{\text{old}}^{\top} \hat V + \mathbf p_{\text{new}}^{\top} V_r\); compare with exact attention
How to read Figure 3
The first step is the prefill: with \(R = 4\), keys 1–4 are quantized as one group per channel and key 5 waits in the buffer; value 1 is quantized and values 2–5 sit in the queue. Press Step: a new token is appended (line 2), then lines 3 and 4 run or are skipped, then the attention (lines 5 and 6). The live box shows each formula with its numbers: the zero point and step of every key channel when a block of keys is quantized, and the codes of the value that leaves the queue. Watch the full-precision keys go 1, 2, 3, 0, 1, 2, 3, 0 while the value queue always holds 4. At 2 bits the KIVI error stays below the naive error at every decode step (mean 0.138 against 0.359); at the end it is 0.296 against 0.466, with 4.63 bits per number against 3. Try \(R = 8\): the mean error falls to 0.086, but the cache costs 6.25 bits at 16 tokens. In a real model the residual is fixed (\(R = 128\)) and the context is long, so this cost shrinks to about 3.05 bits at 32k, formula (110.7).
What the residual costs
The full-precision tokens cost 16 bits each. Count the worst case, \(R\) residual tokens for both keys and values, at a context of \(T\) tokens:
- Read the symbols
- \(b + 32/G\)
- bits per quantized number, codes plus scale and zero point, formula (110.3)
- \(16 R\)
- the residual in 16 bits
- \(T\)
- current context length. The residual is fixed, so its share shrinks as \(T\) grows.
At \(T = 4096\): \(\bar b = (3968 \times 3 + 2048)/4096 = 3.41\) bits. At \(T = 32{,}768\): 3.05 bits, the exact number KVQuant’s comparison table gives for KIVI at 32k context. At \(T = 131{,}072\): 3.01 bits.
Llama 3 8B, one sequence of 131,072 tokens: 17.18 GB in bfloat16, 8.59 GB in FP8, 5.38 GB with KIVI-4 (5.01 bits), 3.24 GB with KIVI-2. That is 5.3× smaller than bfloat16, not 8×. [Certain, computed with (110.1) and (110.7)]
On an 80 GB GPU with \(u = 0.9\) and bfloat16 weights, the room after the weights (55.9 GB) holds about 427,000 tokens in bfloat16 and about 2.28 million tokens with KIVI-2.
KIVI’s own results, with this setting: on LongBench the average for Llama-2-7B is 44.52 in 16 bits, 44.59 for KIVI-4 and 44.27 for KIVI-2; for Mistral-7B 46.58, 46.56 and 45.85. [Certain, KIVI tables] On short tasks the 2-bit method is close to 16 bits. We will see later that a harder long-context test tells a different story.
KVQuant: four ideas for 3 bits and below
KVQuant starts from the same observation (keys per channel, values per token) and asks how to go below 4 bits without losing accuracy. On LLaMA-7B, Wikitext-2 perplexity in fp16 is 5.68. Plain uniform 3-bit with per-token groups gives 10.87; switching keys to per channel gives 7.05. [Certain, KVQuant paper] Then it adds four more ideas.
1. Quantize keys before RoPE
RoPE (session 14) rotates each pair of key channels \((a, b)\) by an angle that grows with the position \(t\). With the pairing \((i, i + d_h/2)\) used in Llama:
- Read the symbols
- \(a_t, b_t\)
- channels \(i\) and \(i + d_h/2\) of the key of token \(t\) before RoPE (the output of \(W_K\))
- \(a'_t, b'_t\)
- the same channels after RoPE, which is what most caches store
- \(\theta_i\)
- the rotation speed of pair \(i\): 1 radian per token for \(i = 0\), about \(10^{-4}\) for the last pair
Suppose channel \(a\) is an outlier channel: always near 5, while \(b\) is small. Before RoPE, the column of \(a\) is almost constant: a perfect per-channel group. After RoPE, \(a'_t = a_t \cos t\theta_i - b_t \sin t\theta_i\) swings between about \(-5\) and \(+5\) as the position changes, and the outlier also leaks into \(b'\). Both columns now have a range of about 10, and per-channel quantization loses its advantage. [Certain, from (110.8)]
KVQuant therefore stores keys before RoPE, \(\hat{\mathbf{k}}_j = \text{quant}(W_K \mathbf{h}_j)\), and applies the rotation after dequantization, inside the attention kernel:
Here \(R_{\Theta, j}\) is the block-diagonal RoPE rotation at position \(j\) and \(\mathbf{q}_t\) is the query, already rotated to its own position. The cost: the kernel must compute \(\cos j\theta_i\) and \(\sin j\theta_i\) for every cached token at every step. KVQuant fuses this into its kernel. [Certain, KVQuant paper] The gain on LLaMA-7B at 3 bits: perplexity 7.05 → 6.23. [Certain] The session 80 session noted that StreamingLLM also stores keys before rotation, for a different reason: it re-assigns positions inside the cache.
How to read Figure 4
At pair 8 (\(\theta = 0.316\), about 3 turns in 64 tokens) the rotated channel \(a'\) swings from −5 to +5, so its quantizer levels spread over a range of 11.7 instead of 2.5. At 3 bits the error after RoPE is 0.126, about 5× the 0.024 of the pre-RoPE path. Move the slider right. Slow pairs (\(i \ge 40\)) barely rotate over 64 tokens, so the two errors become equal. In a real model with thousands of tokens even slow pairs turn a lot: pair 63 turns 0.6 times over 32k tokens. How much pre-RoPE helps depends on which pairs hold the outliers.
2. Non-uniform levels, weighted by sensitivity
Uniform levels are equally spaced. But keys and values are not uniformly spread: most numbers are near the center. Session 105 showed NF4, whose levels follow a normal distribution. KVQuant goes one step further and learns the levels per layer, offline, from calibration data. Each channel (for keys) or token (for values) is first normalized to \([-1, 1]\); then one set of \(2^b\) levels per layer is chosen by solving
- Read the symbols
- \(A_i\)
- one key or value number from the calibration set (after normalization)
- \(Q(A_i)\)
- the nearest of the \(2^b\) levels
- \(g_i\)
- the gradient of the loss with respect to \(A_i\)
- \(\mathcal{F}_{ii}\)
- the diagonal of the Fisher information: how much the loss reacts to an error in \(A_i\). Numbers with a large \(\mathcal{F}_{ii}\) get levels close to them.
This is a weighted k-means in one dimension: the levels are the cluster centers. The paper calibrates on 16 samples of 2k tokens from Wikitext-2; at run time the stored levels are only rescaled per channel or per token. These formats are called nuq2, nuq3 and nuq4. [Certain, KVQuant paper] On LLaMA-7B, nuq3 improves on uniform 3-bit by about 0.29 perplexity. [Certain that the paper reports this]
- offline, once per model: run the fp16 model on calibration text (16 × 2k tokens); keep the pre-RoPE keys and the values of every layer, and the gradients of the loss with respect to them
- for every layer: normalize each key channel and each value token to \([-1, 1]\); ignore the first token // it stays fp16 (idea 4)
- choose the \(2^b\) levels by Fisher-weighted k-means, formula (110.10)
- for keys: store per-channel outlier thresholds (the 1% tails) // keys: thresholds known offline
- online, for each new token: key \(\mathbf{k} \leftarrow W_K \mathbf{h}\), before RoPE
- split into dense + sparse with the per-vector thresholds, formula (110.11); values: compute the token’s thresholds now
- normalize the dense part with the channel (key) or token (value) scale; store the nearest nuq level for each number
- store the sparse outliers in fp16 with their positions (a compressed sparse format)
- attention: dequantize, add the sparse part, apply RoPE to keys on the fly, formula (110.9)
3. Dense-and-sparse: keep the 1% largest numbers
A few very large numbers still stretch the range of each group. KVQuant removes them: each key channel and each value token is split into a dense part, quantized with nuq, and a sparse part with the largest values, stored exactly:
- Read the symbols
- \(\tau_{\text{lo}}, \tau_{\text{hi}}\)
- thresholds per vector (per channel for keys, per token for values), set so that about 1% of the numbers fall outside them
- \(\mathbf{x}_{\text{sparse}}\)
- stored in fp16 in a sparse format; the dense part now has a much smaller range
For keys the thresholds come from calibration. For values they must be found online for every new token; KVQuant runs this top-k search on the CPU in parallel with the GPU. [Certain, KVQuant paper] At 3 bits on LLaMA-7B, 1% outliers improve perplexity from 5.94 to 5.75; one threshold per matrix instead of per vector gives only 5.85. [Certain] The sparse part is not free: with 1% outliers the reported average is 4.33 bits for nuq4, 3.33 for nuq3 and 2.33 for nuq2, a third of a bit for outliers and scales. [Certain for the numbers]
4. Keep the first token in full precision
The first token of a sequence often gets a large share of the attention in every head, the attention sink seen in StreamingLLM (session 116). An error in its key or value is multiplied by a large \(p_j\) in (110.5). KVQuant keeps this one token in fp16 and leaves it out of calibration. At 2 bits on LLaMA-7B this alone takes perplexity from 8.47 to 7.23. [Certain, KVQuant paper] It costs nothing measurable: one token out of thousands.
| Method (LLaMA-7B, Wikitext-2, fp16 = 5.68) | 4-bit | 3-bit | 2-bit |
|---|---|---|---|
| uniform int, per token | 5.98 | 10.87 | 11,779 |
| NF (normal float), per token | 5.87 | 7.33 | 3,210 |
| Atom | 5.77 | 6.17 | 37.37 |
| FlexGen | 5.73 | 5.93 | 11.09 |
| KVQuant (nuq) | 5.72 | 5.87 | 7.23 |
| KVQuant, 1% outliers | 5.69 | 5.75 | 6.01 |
[Certain, KVQuant Table 1; all baselines as run by the KVQuant authors] By perplexity, 3-bit KVQuant with 1% outliers is 0.07 above fp16, and even 2 bits is only 0.33 above. Is a 2-bit cache then ready to ship? The next section says no.
Measure long-context accuracy, not only perplexity
Perplexity averages the loss over all tokens of a text. Most tokens can be predicted from the last few hundred tokens, and those are exactly the tokens that a KIVI residual keeps in full precision, or that get most of the attention anyway. A cache that ruins the memory of token 3,000 in a 30,000-token context can still have almost the same perplexity. The tasks that use the far context are the ones that break.
So test with tasks that need the far context:
- Passkey retrieval: a random number is hidden in a long filler text; the model must repeat it. The score is the fraction of correct answers.
- LongBench (Bai et al., 2023): the 16 English tasks in KVQuant’s table cover long-document QA, multi-document QA, summarization, few-shot learning, code, and synthetic retrieval and counting.
- RULER (Hsieh et al., 2024): synthetic tasks at a chosen length: needle-in-a-haystack with one or many keys and values, multi-hop variable tracking, word counting and QA. Many models score far below their claimed context length on it. [Certain, RULER paper]
KVQuant ran all three on LLaMA-2-7B-32K (a 32k-context version of Llama 2 7B), with KIVI-2 (\(G = 32\), \(R = 128\)) as a baseline. [Certain, KVQuant paper] Figure 5 shows the task scores.
How to read Figure 5
RULER is the default. KVQuant at 3.33 bits stays close to fp16 on most tasks (average 53.65 against 56.40). KIVI-2 at 3.05 bits drops on the single-needle tasks (76 against 100 on Niah1) and on multi-key retrieval (MKey2: 11.4 against 68.2); its average is 39.78. KVQuant at 2.33 bits, which had a perplexity only 0.33 above fp16, falls to 36.54. Click “change vs fp16” to see the losses as negative bars. Switch to Passkey: KIVI finds the passkey only 68–76% of the time at every length, KVQuant almost always. On LongBench the gaps are smaller (averages 31.96, 31.21 and 30.04): it mixes many tasks that a short local context can partly solve.
Put the numbers side by side. KVQuant 2-bit with 1% outliers: perplexity +0.33 (6.01 against 5.68), RULER −19.86 points (36.54 against 56.40), a 35% relative loss. KVQuant 3-bit: perplexity +0.07, RULER −2.75 points (−4.9%). KIVI-2: LongBench −1.92 points, RULER −16.62 points (−29.5%). [Certain for the arithmetic on the reported numbers] The perplexity ranking and the RULER ranking agree in direction, but perplexity hides the size of the failure.
A good evaluation plan for a quantized cache:
- Perplexity on long documents, as a quick sanity check (catches broken kernels).
- Needle-style retrieval (passkey, RULER needles) at several lengths up to the target context, and several needle depths.
- Multi-key and multi-value retrieval and variable tracking (RULER): these break first.
- Your real long tasks (LongBench-style QA and summarization, or your RAG set), with the same prompt format as production.
- Short reasoning tasks with long outputs (GSM8K): the residual covers the prompt only at the start, and errors in the generated chain add up.
Plan the memory
Figure 6 combines everything: weights + KV cache must fit into \(u \cdot M_{\text{GPU}}\), with the average bits \(\bar b\) of each format. For KIVI it uses formula (110.7) at the chosen context; for KVQuant the reported 4.33, 3.33 and 2.33 bits; for FP8 8 bits (one scale per tensor, as in vLLM’s --kv-cache-dtype fp8).
How to read Figure 6
The default is Llama 3 8B, one user at 128k tokens: weights 16.06 GB plus a 17.18 GB bfloat16 cache. Pick KIVI-2: the cache shrinks to 3.24 GB at 3.01 bits. Now raise the batch to 16: bfloat16 needs 275 GB and does not fit; KIVI-2 needs 52 GB and fits. Move the context down to 4,096: KIVI-2’s average rises to 3.41 bits because the 128-token residual is a bigger share. Pick Llama 3 70B on 2 × 80 GB: with 16-bit weights there is almost no room left for any cache; switch to 4-bit weights and a low-bit cache to serve long contexts.
A complete example in PyTorch
The code builds a synthetic cache of one head (512 tokens, \(d_h = 128\)) with the two patterns of this session: three fixed outlier channels in the keys, and a few large tokens in the values. It compares the four granularity choices at 2 and 4 bits, runs a KIVI-style streaming cache, measures pre-RoPE against post-RoPE key quantization, and tests dense-and-sparse values. It uses a fixed seed and only CPU operations, so it is deterministic.
import torch torch.manual_seed(0) T, d = 512, 128 # tokens, head size # Synthetic cache with the patterns of section 3: K = torch.randn(T, d) K[:, [5, 70, 100]] += torch.tensor([9.0, -7.0, 6.0]) # keys: 3 fixed outlier channels V = torch.randn(T, d) V[torch.randperm(T)[:8]] *= 6 # values: a few large tokens, no outlier channels q = torch.randn(d) def quant(X, bits, dim, G): """Min-max (asymmetric) round-to-nearest, formula (110.2). dim = the axis the statistics run along; groups of G elements on that axis.""" Xt = X.movedim(dim, -1) Xg = Xt.reshape(*Xt.shape[:-1], -1, G) # (..., n_groups, G) mn = Xg.amin(-1, keepdim=True); mx = Xg.amax(-1, keepdim=True) s = (mx - mn).clamp(min=1e-6) / (2 ** bits - 1) Xq = torch.round((Xg - mn) / s) * s + mn # quantize, then dequantize return Xq.reshape(Xt.shape).movedim(-1, dim) def attn(q, K, V): p = torch.softmax(K @ q / d ** 0.5, dim=0) return p @ V rel = lambda a, b: ((a - b).norm() / b.norm()).item() o = attn(q, K, V) # 1) granularity: per-token = stats over channels (dim=1); per-channel = over tokens (dim=0) for kn, kd in [("K per-token ", 1), ("K per-channel", 0)]: for vn, vd in [("V per-token ", 1), ("V per-channel", 0)]: e2 = rel(attn(q, quant(K, 2, kd, 32), quant(V, 2, vd, 32)), o) e4 = rel(attn(q, quant(K, 4, kd, 32), quant(V, 4, vd, 32)), o) print(f"{kn} {vn} output error: 2-bit {e2:.3f}, 4-bit {e4:.3f}") # 2) a KIVI-style streaming cache: quantized part + full-precision residual class KIVICache: def __init__(s, bits=2, G=32, R=128): s.bits, s.G, s.R = bits, G, R s.Kq, s.Kr, s.Vq, s.Vr = [], [], [], [] def append(s, k, v): s.Kr.append(k); s.Vr.append(v) if len(s.Kr) == s.R: # keys: wait for R tokens, then s.Kq.append(quant(torch.stack(s.Kr), s.bits, 0, s.G)) # per-channel s.Kr = [] if len(s.Vr) > s.R: # values: oldest token leaves the s.Vq.append(quant(s.Vr.pop(0)[None], s.bits, 1, s.G)) # window, per-token def get(s): K_ = torch.cat(s.Kq + ([torch.stack(s.Kr)] if s.Kr else [])) V_ = torch.cat(s.Vq + [torch.stack(s.Vr)]) return K_, V_ def avg_bits(s): n = sum(x.shape[0] for x in s.Kq) + sum(x.shape[0] for x in s.Vq) r = len(s.Kr) + len(s.Vr) return (n * (s.bits + 32 / s.G) + r * 16) / (n + r) c = KIVICache() for t in range(T - 12): # stream 500 tokens one by one c.append(K[t], V[t]) Kc, Vc = c.get() print("K: quantized", sum(x.shape[0] for x in c.Kq), "residual", len(c.Kr), "| V: quantized", len(c.Vq), "residual", len(c.Vr)) print(f"KIVI-2 output error {rel(attn(q, Kc, Vc), attn(q, K[:500], V[:500])):.3f}, " f"average bits {c.avg_bits():.2f}") # 3) pre-RoPE vs post-RoPE keys: rotate pairs (i, i + d/2) by angle pos * theta_i pos = torch.arange(T, dtype=torch.float32)[:, None] theta = 10000 ** (-torch.arange(d // 2, dtype=torch.float32) / (d // 2)) cos, sin = (pos * theta).cos(), (pos * theta).sin() def rope(X): a, b = X[:, : d // 2], X[:, d // 2 :] return torch.cat([a * cos - b * sin, a * sin + b * cos], dim=1) Kr = rope(K) post = rel(quant(Kr, 3, 0, 32), Kr) # quantize the rotated keys pre = rel(rope(quant(K, 3, 0, 32)), Kr) # quantize first, rotate after print(f"3-bit per-channel key error: post-RoPE {post:.3f}, pre-RoPE {pre:.3f}") # 4) dense-and-sparse: keep the 1% largest |values| in full precision def dense_sparse(X, bits, dim, G, frac=0.01): thr = X.abs().flatten().kthvalue(int((1 - frac) * X.numel())).values mask = X.abs() > thr # the sparse outliers Xd = quant(torch.where(mask, torch.zeros_like(X), X), bits, dim, G) return torch.where(mask, X, Xd), mask.float().mean().item() Vds, f = dense_sparse(V, 2, 1, 32) print(f"2-bit V per-token error {rel(quant(V, 2, 1, 32), V):.3f}, " f"with {100 * f:.1f}% outliers kept {rel(Vds, V):.3f}")
Output (PyTorch 2.8, CPU):
K per-token V per-token output error: 2-bit 1.205, 4-bit 0.343 K per-token V per-channel output error: 2-bit 1.385, 4-bit 0.350 K per-channel V per-token output error: 2-bit 0.592, 4-bit 0.115 K per-channel V per-channel output error: 2-bit 0.758, 4-bit 0.117 K: quantized 384 residual 116 | V: quantized 372 residual 128 KIVI-2 output error 0.563, average bits 6.17 3-bit per-channel key error: post-RoPE 0.138, pre-RoPE 0.111 2-bit V per-token error 0.394, with 1.0% outliers kept 0.315
Read the results. (1) Keys per channel halve the output error at 2 bits and cut it 3× at 4 bits; the value choice matters less here, but per-token is better at 2 bits. (2) The streaming cache holds 384 keys in 3 quantized blocks of 128 and 116 in the buffer, while values have 372 quantized and exactly 128 in the queue: the two boundaries differ, as in Figure 3. At only 500 tokens the residual dominates the cost (6.17 bits); at 32k it would be about 3.05. (3) Pre-RoPE quantization lowers the key error from 0.138 to 0.111. (4) Keeping 1% of values exact lowers their error by 20%. The errors are large because Gaussian toy data at 2 bits is a hard case; real caches have more structure. [Certain for the printed numbers, run twice with identical output]
Where this is used
| Setting | Why the cache must shrink | Typical choice |
|---|---|---|
| High-throughput chat serving | Batch size is limited by \(\sum_i T_i \cdot s\), formula (110.1) | FP8 cache (vLLM, TensorRT-LLM) as a safe first step; 4-bit with care [Likely] |
| Long-document QA, RAG over long contexts | One prompt of 32k–128k tokens can need as much memory as the weights | KIVI-4 or KVQuant-3/4; test with RULER at the real length |
| Agents with long tool traces | Context grows over many steps and must stay exact enough for retrieval | 4-bit with a residual; check multi-key retrieval |
| Single-GPU or edge long context | No second GPU; weights already 4-bit | KVQuant nuq2/nuq3 with outliers and kernels |
| Offloading to CPU memory (FlexGen) | The cache moves over PCIe; fewer bytes means faster transfer | 4-bit group-wise cache [Certain, survey] |
| Video QA, Video-RAG, key-moment detection | Thousands of frame tokens per video enter the cache (session 80: 64 frames × 576 tokens) | Combine token pooling with a low-bit cache; test needle-in-the-video tasks |
| Streaming video and live captioning | Unbounded input; old frames are rarely attended but must not be corrupted | Sliding window + attention-sink tokens in fp16 + low-bit older frames [Speculation] |
| VLM image QA with many images | Each image adds hundreds of visual tokens | Same rules; check that visual tokens have outlier channels like text [Likely] |
Common mistakes
- Quantizing keys per token because that is how activations are usually done. The outlier channels then destroy every key (Figure 2).
- Quantizing values per channel by symmetry with keys. KIVI’s 2-bit test with both per channel gives 0.00 on GSM8K.
- Quantizing keys after RoPE when the kernel can rotate on the fly. RoPE spreads the outlier channel over its pair (Figure 4).
- Reporting the code width as the bit width. “2-bit” KIVI with \(G = 32\) is 3 bits plus the residual; KVQuant 2-bit with 1% outliers is 2.33 bits. Compare methods at their average bits (110.7).
- Trusting perplexity alone. KVQuant 2-bit-1% is +0.33 perplexity but −19.86 RULER points.
- Testing at short context. With a 128-token residual, a 200-token test barely touches the quantized part. Test at the real production length.
- Quantizing the first token (the attention sink). It is one token, and keeping it exact took KVQuant 2-bit perplexity from 8.47 to 7.23.
- Expecting speed without a kernel. Fake quantization (quantize then dequantize in PyTorch) saves no memory traffic. The gain needs kernels that read the low-bit codes.
Interview questions
When is the KV cache bigger than the weights? Give a number.basic
Why are keys quantized per channel and values per token?core
Why does KIVI keep a full-precision residual of recent tokens?core
A colleague says “2-bit KV cache = 8× smaller than fp16”. Correct them.core
Why quantize keys before RoPE? What does it cost?core
What is the Fisher-weighted non-uniform data type in KVQuant?advanced
Why is a 2-bit cache with +0.33 perplexity not ready to ship?core
Why not use GPTQ or AWQ to quantize the KV cache?basic
How can a kernel compute attention on quantized keys and values without dequantizing every number?advanced
Why does KVQuant keep the first token in fp16, and the 1% outliers per vector?advanced
KIVI or KVQuant or FP8 for a new production service?advanced
Exercises
- Compute the average bits of KIVI-2 with \(G = 64\) and \(R = 128\) at \(T = 8192\). How much smaller than bfloat16 is the cache?
- Llama 3 70B (\(s = 327{,}680\) bytes in bfloat16, 141.1 GB of weights in 16 bits, 35.3 GB in 4 bits) serves 16 users at 32,768 tokens with KIVI-2 (\(G = 32\), \(R = 128\)). How big is the cache? Does it fit on 2 × 80 GB with 16-bit weights, or on one 80 GB GPU with 4-bit weights (\(u = 0.9\))?
- Quantize the key row \((0.3, -0.2, 7.9, 0.1)\) to 2 bits per token with (110.2). Give \(s\), the codes and \(\hat x\).
- Check (110.6) by hand: \(\mathbf{q} = (2, 1)\), per-channel scales \(s = (0.5, 0.1)\), zero points \(z = (-1, 3)\), codes \(c = (3, 2)\).
- With \(d_h = 128\) and \(\theta_i = 10000^{-2i/d_h}\), how many full turns does RoPE pair \(i = 0\) make over 32,768 positions? And pair \(i = 63\)?
- From Figure 5: compute the relative RULER loss (in % of the fp16 average 56.40) for KVQuant-3bit-1%, KIVI-2 and KVQuant-2bit-1%.
Answerscheck
Key takeaways
- The cache passes the weights when the total cached tokens reach \(M_W / s\): about 122k tokens for Llama 3 8B, 26k for Llama 2 7B (110.1). Then the bits per cached number are the main lever.
- Keys have fixed outlier channels: quantize them per channel. Values have none and enter the output as weighted rows: quantize them per token (KIVI). The other three combinations fail at 2 bits.
- Count real bits: \(b + 32/G\) per group, plus the fp16 residual (110.3, 110.7). “2-bit” KIVI is about 3.05 bits at 32k.
- KIVI streams: keys wait in an fp16 buffer of \(R\) tokens and are quantized in blocks; values leave an fp16 queue one by one. No calibration.
- KVQuant: pre-RoPE keys (rotate after dequantization), Fisher-weighted non-uniform levels, 1% per-vector outliers in fp16, and an fp16 first token. 3.33 bits with +0.07 perplexity on LLaMA-7B.
- Per-channel key scales fold into the query and per-token value scales into the probabilities, so kernels can work on codes (110.6).
- Judge a quantized cache by long-context tasks at production length (passkey, RULER, your own long tasks), not by perplexity: +0.33 perplexity was −35% on RULER.
Key formulas
References for this session
- Liu et al. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- Hooper et al. (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Liu et al. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- Hooper et al. (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Sheng et al. (2023). FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
- Zhao et al. (2023). Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
- Ashkboos et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- Hsieh et al. (2024). RULER: What’s the Real Context Size of Your Long-Context Language Models?
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Bai et al. (2023). LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- Hugging Face (2025). Transformers documentation: cache strategies (QuantizedCache)