LLM & VLM courseBlog
26Track 2: Attention and transformers · session 11 of 15

Attention variants 1: MHA, MQA, GQA, MLA and the KV cache

When a model writes text, it keeps the keys and values of every earlier token in memory: the KV cache. With standard multi-head attention this cache becomes the biggest user of GPU memory for long prompts and large batches. This session shows four ways to build attention (MHA, MQA, GQA and DeepSeek's MLA), how each one shrinks the cache, and what each one costs in quality.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 17, 19 and 25 (scaled dot-product attention, multi-head attention, counting memory)

What you will learn

  • Why generation keeps a KV cache, and the formula for its size in bytes
  • Why decoding is limited by memory traffic, not by arithmetic, and how sharing K and V helps
  • Multi-query attention (MQA): all query heads share one key head and one value head
  • Grouped-query attention (GQA): groups of heads share K and V, and how to convert an MHA checkpoint by mean-pooling
  • Multi-head latent attention (MLA): cache one small latent vector, absorb the up-projections, and keep RoPE in a separate small key
  • How to compute the KV cache of a real model (Llama 2, Llama 3, Mistral, Qwen2.5, Falcon, DeepSeek-V2) from its config file
  • What the papers report about quality: MHA vs GQA vs MQA vs MLA

Why this matters

A language model writes one token at a time. To write token number 4,001 it must attend to the 4,000 tokens before it. It would be far too slow to recompute their keys and values at every step, so the model keeps them in GPU memory. This store is called the KV cache, and for long prompts it is often bigger than the model itself.

Take Llama 2 7B. It uses standard multi-head attention with 32 layers, 32 heads and 128 numbers per head. For each token it stores \(2 \times 32 \times 32 \times 128 = 262{,}144\) numbers, which is 512 KiB in bfloat16 (2 bytes per number). One sequence of 4,096 tokens needs 2 GiB of cache. A batch of 32 such sequences needs 64 GiB, almost all of an 80 GB GPU, before we even load the 13.5 GB of weights. And at every generated token, the GPU must read this whole cache from memory once.

This session is about one idea with four versions: store fewer keys and values per token, and lose as little quality as possible.

  • MHA (multi-head attention): every query head has its own key head and value head. Best quality, biggest cache.
  • MQA (multi-query attention, Shazeer 2019): all query heads share one key head and one value head. The cache shrinks by the number of heads, but quality can drop.
  • GQA (grouped-query attention, Ainslie et al. 2023): heads are split into groups, and each group shares one key and one value head. This is the standard choice today: Llama 2 70B, Llama 3, Mistral 7B and Qwen2.5 all use it. [Certain]
  • MLA (multi-head latent attention, DeepSeek-V2 2024): store one small latent (compressed) vector per token, and rebuild the keys and values of all heads from it. DeepSeek reports a cache close to GQA with 2.25 groups and quality better than MHA. [Certain that the paper reports this]

Interviewers love this topic because it joins model design with hardware: “How big is the KV cache of Llama 3 70B at 128k tokens?”, “Why does GQA make decoding faster if the FLOPs barely change?”, “How do you turn an MHA checkpoint into GQA?”, “Why does MLA need a separate RoPE key?”. By the end of this session you can answer each one with a formula and a number. We build on session 19 and the memory counting of session 25. The full serving story (paging, quantizing and evicting the cache) comes later in session 80 and session 97.

The KV cache and why it is the bottleneck

Recap: multi-head attention

In session 19 each layer takes the hidden vector \(\mathbf{h}_t \in \R^d\) of token \(t\) and makes, for every head \(i = 1, \dots, n_h\), a query, a key and a value of size \(d_h\) (usually \(d_h = d / n_h\)):

\[ \mathbf{q}_{t,i} = W^Q_i \mathbf{h}_t, \quad \mathbf{k}_{t,i} = W^K_i \mathbf{h}_t, \quad \mathbf{v}_{t,i} = W^V_i \mathbf{h}_t, \qquad \mathbf{o}_{t,i} = \sum_{j \le t} \softmax_j\!\left( \frac{\mathbf{q}_{t,i} \cdot \mathbf{k}_{j,i}}{\sqrt{d_h}} \right) \mathbf{v}_{j,i} \tag{26.1} \]
Read the symbols
\(n_h\)
the number of attention heads (query heads)
\(d_h\)
the size of one head’s query, key and value (128 in most large models)
\(W^Q_i, W^K_i, W^V_i\)
learned matrices of shape \(d_h \times d\) for head \(i\)
\(j \le t\)
causal attention: token \(t\) looks only at itself and earlier tokens
\(\mathbf{o}_{t,i}\)
the output of head \(i\); the outputs of all heads are joined and multiplied by \(W^O\)

Why we cache K and V, but not Q

Look at what the new token \(t\) needs. Its own query \(\mathbf{q}_{t,i}\) is used once, right now. But it needs the keys and values of all earlier tokens \(j \lt t\). Those do not change when a new token arrives, because of the causal mask: token \(j\) never sees anything after it. So we compute each key and value once, store them, and at each new step only add one new row. Old queries are never needed again, so we do not keep them.

Generation therefore has two phases. Prefill processes the whole prompt in parallel and fills the cache. Decode then makes one token per step, and each step reads the full cache.

The size formula

Let \(n_{kv}\) be the number of key–value heads (for MHA, \(n_{kv} = n_h\)). Then the cache for a batch is:

\[ M_{\text{KV}} = 2 \cdot L \cdot n_{kv} \cdot d_h \cdot T \cdot B \cdot b \quad \text{bytes} \tag{26.2} \]
Read the symbols
\(2\)
one key and one value
\(L\)
the number of layers; every layer has its own cache
\(n_{kv}\)
the number of key and value heads that are stored. This is the number the variants in this session change.
\(T\)
the number of tokens so far (prompt plus generated text)
\(B\)
the batch size: how many sequences are served at the same time
\(b\)
bytes per number: 2 for bfloat16 or float16, 1 for 8-bit

The cache grows linearly in context length and in batch size. The weights do not grow at all. So for long contexts or many users, the cache wins.

Worked example: Llama 3 8B at 128k tokens

From its config file: \(L = 32\), \(n_h = 32\), \(n_{kv} = 8\), \(d_h = 128\). Per token: \(2 \times 32 \times 8 \times 128 = 65{,}536\) numbers, which is \(131{,}072\) bytes = 128 KiB in bfloat16. For \(T = 131{,}072\) tokens (its full context) and \(B = 1\): \(131{,}072 \times 131{,}072\) bytes \(= 16\) GiB. The weights are about 16 GB, so the cache of a single long conversation is as large as the model.

If the same model used MHA (\(n_{kv} = 32\)), the cache would be four times larger: 512 KiB per token and 64 GiB at 128k tokens.

Decode is memory-bound

Why does a big cache make decoding slow, and not only memory-hungry? Count the work for one decode step in one layer, for one key–value head that is shared by \(r = n_h / n_{kv}\) query heads, with \(T\) cached tokens in bfloat16:

  • Bytes read from memory: the keys and values, \(2 \cdot T \cdot d_h \cdot 2 = 4 T d_h\).
  • FLOPs: each of the \(r\) query heads computes \(T\) dot products with the keys (\(2 T d_h\) FLOPs) and a weighted sum of the values (another \(2 T d_h\)), so \(4 T d_h r\) in total.
\[ \text{arithmetic intensity of decode attention} = \frac{4\,T\,d_h\,r}{4\,T\,d_h} = r = \frac{n_h}{n_{kv}} \ \ \text{FLOPs per byte} \tag{26.3} \]

Arithmetic intensity is the number of FLOPs done for each byte read from memory. An H100 SXM GPU does about 989 TFLOP/s in dense bfloat16 and reads about 3.35 TB/s from its memory, so it needs about 295 FLOPs per byte to keep its compute units busy (the “ridge point” of the roofline model). [Certain for the spec numbers] MHA gives \(r = 1\): the GPU waits for memory almost all the time. The decode time is simply “bytes to read ÷ memory bandwidth”. This is why every byte removed from the cache makes decoding faster, even though the number of FLOPs stays the same. Sharing keys and values among \(r\) heads both shrinks the bytes and raises the intensity by \(r\).

Multi-query attention (MQA)

Shazeer (2019) noticed that during decoding the cost comes from loading keys and values, not from the queries. So he kept many query heads but used a single key head and a single value head, shared by all of them:

\[ \mathbf{k}_t = W^K \mathbf{h}_t, \quad \mathbf{v}_t = W^V \mathbf{h}_t, \qquad \mathbf{o}_{t,i} = \sum_{j \le t} \softmax_j\!\left( \frac{\mathbf{q}_{t,i} \cdot \mathbf{k}_{j}}{\sqrt{d_h}} \right) \mathbf{v}_{j} \tag{26.4} \]

Compare with (26.1): the key and value lost their head index \(i\). Each head still has its own query, so heads can still look at different tokens; but they all read from the same “keys” and get the same “values”. The cache uses \(n_{kv} = 1\), so it shrinks by a factor of \(n_h\) (32 for a 32-head model), and the intensity (26.3) rises to \(r = n_h\).

The cost is quality and stability. With one key–value head, the model has far fewer parameters in \(W^K\) and \(W^V\), and the heads cannot store different kinds of information in their values. The GQA paper reports that MQA can lose quality and that training MQA models can be less stable. [Certain that the paper reports this] PaLM (540B) and Falcon 7B used MQA. [Certain] Falcon 7B, for example, has 71 query heads of size 64 and one key–value head, so its cache is only 8 KiB per token over 32 layers.

A practical problem appears with tensor parallelism, where the heads of one layer are split across several GPUs. Every GPU needs the single key–value head, so it is copied to all of them, and part of the memory saving is lost. The GQA paper points out this waste. [Certain] GQA with 8 key–value heads fits 8 GPUs exactly: one key–value head per GPU.

Grouped-query attention (GQA)

GQA (Ainslie et al., 2023) sits between MHA and MQA. Split the \(n_h\) query heads into \(n_{kv}\) groups of \(r = n_h / n_{kv}\) heads each. Each group shares one key head and one value head:

\[ g(i) = \left\lfloor \frac{i}{r} \right\rfloor, \qquad \mathbf{o}_{t,i} = \sum_{j \le t} \softmax_j\!\left( \frac{\mathbf{q}_{t,i} \cdot \mathbf{k}_{j,g(i)}}{\sqrt{d_h}} \right) \mathbf{v}_{j,g(i)} \tag{26.5} \]
Read the symbols
\(i\)
the query head, counted from 0 to \(n_h - 1\)
\(r\)
the group size: query heads per key–value head
\(g(i)\)
the group (and so the key–value head) that query head \(i\) reads. Heads \(0, \dots, r-1\) use group 0, the next \(r\) use group 1, and so on.

GQA contains both earlier designs: \(n_{kv} = n_h\) is MHA and \(n_{kv} = 1\) is MQA. Llama 3 8B uses 32 query heads and 8 key–value heads, so \(r = 4\). Llama 3 70B uses 64 and 8, so \(r = 8\). [Certain, from the config files]

In code, GQA is MHA with one extra line: repeat each key–value head \(r\) times before the usual attention (repeat_interleave in PyTorch), or, in fast kernels, let \(r\) query heads read the same key–value memory without copying it.

query headkey head (cached)value head (cached)MLA latent (cached)
Figure 1. Who reads which keys and values. A layer with 8 query heads of size \(d_h = 128\). Choose a design; click a query head to highlight the key and value heads it reads. The readout counts the numbers stored in the cache per token for this one layer.
How to read Figure 1

In MHA every query head has its own key and value head: 8 + 8 cached heads, 2,048 numbers per token. In GQA-4, pairs of query heads share a key–value head (1,024 numbers); in GQA-2, groups of four (512); in MQA all 8 share one (256). Click a query head to see its group light up. In MLA nothing per head is stored: a single latent vector (here 512 numbers, \(4 d_h\) as in DeepSeek-V2) plus one shared 64-number RoPE key are cached, and the per-head keys and values (dashed) are rebuilt from the latent. That is 576 numbers, between GQA-2 and GQA-4, yet every head still gets its own key and value.

Figure 1 shows the wiring. Figure 2 runs it. It is Algorithm 1 (below) on a toy layer, one decode step after another, with every number computed in your browser: 4 query heads of size \(d_h = 2\), a hidden size of 4, and 4 tokens that arrive one by one. The MHA weights are fixed (seeded random numbers with one decimal). The GQA-2 and MQA weights are made from them by mean pooling, formula (26.6), with no uptraining.

  1. \(\mathbf{q}_i \leftarrow W^Q_i \mathbf{h}_t\) for every head \(i\)
  2. \(\mathbf{k}_g \leftarrow W^K_g \mathbf{h}_t\), \(\mathbf{v}_g \leftarrow W^V_g \mathbf{h}_t\); append to \(K_g, V_g\) // (26.2)
  3. for \(i = 0\) to \(n_h - 1\):
  4. \(g \leftarrow \lfloor i / r \rfloor\) // (26.5)
  5. \(s_j \leftarrow \mathbf{q}_i \cdot \mathbf{k}_{j,g} / \sqrt{d_h}\) for \(j \le t\)
  6. \(\mathbf{a} \leftarrow \softmax(\mathbf{s})\); \(\mathbf{o}_i \leftarrow \sum_j a_j \mathbf{v}_{j,g}\)
  7. \(\mathbf{y}_t \leftarrow [\mathbf{o}_0; \dots; \mathbf{o}_{n_h-1}]\) // \(W^O = I\) in this toy
query headcached keycached valuehead output
Figure 2. Decoding with MHA, GQA-2 and MQA, step by step: 4 tokens × 7 steps (queries, append to the cache, one step per query head, join the outputs). The bars on the right count the numbers in the cache of this layer for all three designs as tokens arrive; the selected design is outlined.
How to read Figure 2

Press Play, or Step through by hand. At “append” a new row appears in every key head (orange) and value head (green): 4 pairs in MHA, 2 in GQA-2, 1 in MQA, so the bars grow by 16, 8 or 4 numbers per token. In each head step the active query head is outlined, a thick line goes to the key–value head \(g(i) = \lfloor i / r \rfloor\) it reads, and the cache rows are shaded by the attention weight they get. The panel on the right writes formula (26.5) with the live numbers. Watch the counter “read from cache this step”. In MHA every head loads its own pair, so FLOPs per byte stay at 1. In MQA the first head loads the only pair and the other three reuse it, so the ratio climbs to 4 = \(r\) by the end of the step, which is formula (26.3). Switch the design at any time: the step stays and the numbers are recomputed. The last step of each token compares the output with MHA; mean pooling without uptraining changes it a lot here (90.3% for GQA-2 and 133.4% for MQA at token 4), because these random heads are unrelated (\(\rho = 0\) in Figure 3).

Converting an MHA checkpoint: mean pooling and uptraining

Training a large model from scratch is expensive. The GQA paper shows how to turn an existing MHA model into a GQA model cheaply, in two steps:

  1. Mean-pool the key and value projections inside each group. The new key head of group \(g\) is the average of the old key heads of that group:
\[ W^K_{g} = \frac{1}{r} \sum_{i \,:\, g(i) = g} W^K_i, \qquad W^V_{g} = \frac{1}{r} \sum_{i \,:\, g(i) = g} W^V_i \tag{26.6} \]
  1. Uptrain: continue pre-training the converted model for a short time, so that it adapts to the shared heads. The paper uses 5% of the original pre-training steps. [Certain]

The paper compared three ways to start the shared head: the mean of the heads, the first head of the group only, and random weights. Mean pooling worked best, then picking one head, then random. [Certain] Mean pooling keeps the most information from the original heads. But it is not exact: the attention output changes after pooling, because the heads were trained to be different. Figure 3 measures this change on a toy layer.

Figure 3. Mean pooling (26.6) without uptraining, computed live. A toy causal attention layer with \(n_h = 8\) heads of size 8, width 32 and 16 tokens (fixed random seed). Each head’s key and value weights are \(\sqrt{\rho}\,\) (shared part) \(+ \sqrt{1-\rho}\,\) (own part). The bars show the relative error \(\lVert \mathbf{y}_{\text{GQA}} - \mathbf{y}_{\text{MHA}} \rVert / \lVert \mathbf{y}_{\text{MHA}} \rVert\) of the layer output for each number of groups.
How to read Figure 3

With 8 groups nothing is pooled, so the error is exactly 0. Fewer groups average more heads together and the error grows; MQA (1 group) is worst. Now move \(\rho\). At \(\rho = 1\) all heads are already equal, so averaging loses nothing and every bar drops to 0. At \(\rho = 0\) the heads are unrelated and pooling destroys most of their information. Real trained heads lie in between, which is why the pooled model must be uptrained for a while, and why keeping more groups (GQA-8 rather than MQA) keeps more quality. The selected bar is outlined; the readout also shows how many cached numbers that choice keeps.

What GQA costs in quality

The GQA paper uptrained T5-XXL (an encoder–decoder model) and measured the average score over summarization, translation and question-answering tasks, with the inference time per sample: [Certain, numbers from the paper’s main table]

ModelAverage scoreTime per sampleSpeed-up vs MHA-XXL
MHA, T5-XXL47.21.511.0×
MQA, T5-XXL (uptrained)46.60.246.3×
GQA-8, T5-XXL (uptrained)47.10.285.4×
MHA, T5-Large (smaller model)46.00.374.1×

So GQA-8 keeps almost all of the MHA quality (47.1 vs 47.2) and almost all of the MQA speed. MQA is a bit faster but loses more (46.6). Both are much better than simply using a smaller MHA model of similar speed. This trade-off is why GQA became the default.

The picture is less rosy when models are trained from scratch with a fixed budget. DeepSeek trained three dense 7B models on 1.33T tokens that differed only in attention (MHA, GQA, MQA), and reported that “MHA demonstrates significant advantages over GQA and MQA” on their benchmarks. [Certain that the paper reports this] This was one motivation for MLA: keep a small cache without giving up per-head keys and values.

Multi-head latent attention (MLA)

GQA and MQA save memory by giving many heads the same key and value. MLA saves memory in a different way: it keeps every head’s key and value different, but stores them in compressed form. The idea is low-rank compression: the keys and values of all heads together (32,768 numbers per token in DeepSeek-V2) are generated from a much smaller vector (512 numbers). If the network is trained this way from the start, it learns to put what matters into that small vector.

Step 1: compress keys and values into one latent vector

For each token, MLA first projects the hidden vector down to a latent vector \(\mathbf{c}_t\) of size \(d_c\), shared by all heads. Then it projects back up to a key and a value for each head:

\[ \mathbf{c}^{KV}_t = W^{DKV} \mathbf{h}_t \in \R^{d_c}, \qquad \mathbf{k}^C_{t,i} = W^{UK}_i \mathbf{c}^{KV}_t, \qquad \mathbf{v}^C_{t,i} = W^{UV}_i \mathbf{c}^{KV}_t \tag{26.7} \]
Read the symbols
\(\mathbf{c}^{KV}_t\)
the compressed latent vector of token \(t\). Only this is stored in the cache (plus one small key, Step 3).
\(d_c\)
the latent size. In DeepSeek-V2 and V3, \(d_c = 512 = 4 d_h\), much smaller than \(2 n_h d_h = 32{,}768\).
\(W^{DKV}\)
the down-projection (D for down), shape \(d_c \times d\), shared by all heads
\(W^{UK}_i, W^{UV}_i\)
the up-projections (U for up) of head \(i\), each of shape \(d_h \times d_c\)
\(\mathbf{k}^C_{t,i}, \mathbf{v}^C_{t,i}\)
the “content” key and value of head \(i\). They are different for every head, as in MHA.

DeepSeek also compresses the queries in a similar way (into a latent of size \(d_c' = 1536\)), but that only saves activation memory during training; queries are never cached. [Certain]

Step 2: absorb the up-projections

If we rebuilt all the keys and values from the cache at every decode step, we would save memory but add a lot of compute. MLA avoids this with a neat use of matrix associativity. The score between the query of head \(i\) and the key of an earlier token \(j\) is:

\[ \mathbf{q}_{t,i}^{\top} \mathbf{k}^C_{j,i} = \mathbf{q}_{t,i}^{\top} W^{UK}_i \mathbf{c}^{KV}_j = \big( (W^{UK}_i)^{\top} \mathbf{q}_{t,i} \big)^{\top} \mathbf{c}^{KV}_j = \tilde{\mathbf{q}}_{t,i}^{\top}\, \mathbf{c}^{KV}_j \tag{26.8} \]

So we move \(W^{UK}_i\) to the query side: compute \(\tilde{\mathbf{q}}_{t,i} = (W^{UK}_i)^{\top} \mathbf{q}_{t,i} \in \R^{d_c}\) once for the new token, then take dot products directly with the cached latents. Since \(\mathbf{q}_{t,i} = W^Q_i \mathbf{h}_t\), the product \((W^{UK}_i)^\top W^Q_i\) can even be merged into one matrix ahead of time. The paper says \(W^{UK}\) can be absorbed into \(W^Q\). [Certain] The same works on the value side:

\[ \mathbf{o}_{t,i} = \sum_{j \le t} a_{t,j,i}\, W^{UV}_i \mathbf{c}^{KV}_j = W^{UV}_i \Big( \sum_{j \le t} a_{t,j,i}\, \mathbf{c}^{KV}_j \Big) \tag{26.9} \]

First average the latents with the attention weights \(a_{t,j,i}\), then up-project once. And since the next step multiplies by \(W^O\), the matrix \(W^{UV}_i\) can be merged into \(W^O\). The per-head keys and values never have to be built during decoding. Look at what is left: all 128 heads read the same cached vectors \(\mathbf{c}^{KV}_j\). During decoding, MLA behaves like MQA with one very wide key–value head, while training sees full per-head keys and values. [Likely, this is how fast MLA decode kernels are described]

Step 3: a separate small key for RoPE

There is one problem. Modern models encode position with RoPE (session 14), which rotates the query and the key by angles that depend on their positions. The score then has a rotation matrix between them: \(\mathbf{q}^{\top} R_{\,j - t}\, W^{UK}_i \mathbf{c}_j\). This matrix depends on the distance between the two tokens, so it is different for every pair, and \(W^{UK}_i\) can no longer be moved to the query side once and for all. The paper says RoPE is “incompatible” with the low-rank compression for this reason. [Certain]

The fix is decoupled RoPE: keep the compressed part free of RoPE, and add a small, separate key that carries the position. This RoPE key has \(d_h^R\) numbers, is shared by all heads (like MQA), and is cached as it is. Each head also gets a small RoPE query \(\mathbf{q}^R_{t,i}\). The score adds the two parts:

\[ \mathbf{k}^R_t = \operatorname{RoPE}\big(W^{KR} \mathbf{h}_t\big) \in \R^{d_h^R}, \qquad s_{t,j,i} = \frac{\mathbf{q}^{C\top}_{t,i}\, \mathbf{k}^C_{j,i} + \mathbf{q}^{R\top}_{t,i}\, \mathbf{k}^R_j}{\sqrt{d_h + d_h^R}} \tag{26.10} \]
Read the symbols
\(\mathbf{k}^R_t\)
the RoPE key of token \(t\): one small vector for all heads, rotated by its position, cached
\(d_h^R\)
its size: 64 in DeepSeek-V2, which is \(d_h / 2\)
\(\mathbf{q}^C, \mathbf{q}^R\)
the content part and the RoPE part of the query of head \(i\); the head’s full query and key are the two parts joined
\(\sqrt{d_h + d_h^R}\)
the usual scaling, now with the joined size \(128 + 64 = 192\)

The MLA cache

Per token and per layer, MLA stores the latent and the RoPE key:

\[ \text{MLA cache per token} = (d_c + d_h^R)\, L \ \text{numbers}, \qquad \text{GQA: } 2\, n_{kv}\, d_h\, L \tag{26.11} \]

With DeepSeek’s choices \(d_c = 4 d_h\) and \(d_h^R = d_h / 2\), MLA stores \(4.5\, d_h\) numbers per layer, the same as GQA with \(4.5 / 2 = 2.25\) groups. [Certain, as stated in the paper]

Worked example: the cache of DeepSeek-V2

From its config file: \(L = 60\), \(n_h = 128\), \(d_h = 128\), \(d_c = 512\) (kv_lora_rank), \(d_h^R = 64\) (qk_rope_head_dim).

MLA per token: \((512 + 64) \times 60 = 34{,}560\) numbers \(= 69{,}120\) bytes \(= 67.5\) KiB in bfloat16. At 128k tokens: 8.4 GiB.

MHA with the same heads would store \(2 \times 128 \times 128 = 32{,}768\) numbers per layer instead of 576: \(32{,}768 / 576 \approx 56.9\) times more, 3,840 KiB per token. At 128k tokens that is 480 GiB, which does not fit on a node of eight 80 GB GPUs.

The paper says DeepSeek-V2 “reduces the KV cache by 93.3%” compared with DeepSeek 67B. DeepSeek 67B uses GQA with \(L = 95\), \(n_{kv} = 8\), \(d_h = 128\): \(194{,}560\) numbers per token. In bfloat16 the ratio would be \(34{,}560 / 194{,}560 = 17.8\%\), an 82.2% reduction. The paper also quantizes the deployed cache to about 6 bits per number on average; with that, \(34{,}560 \times 0.75\) bytes against \(194{,}560 \times 2\) bytes gives exactly a 93.3% reduction. [Likely that this is how the 93.3% was computed; the paper does not show the calculation]

Figure 4. MLA step by step, with the real sizes of one DeepSeek-V2 layer (\(d = 5120\), 128 heads, \(d_h = 128\), \(d_c = 512\), \(d_h^R = 64\)). Boxes with a thick border are stored in the KV cache; dashed boxes are only computed when needed (training and prefill) and are never stored.
How to read Figure 4

Step through the five stages. Step 1 shows what MHA would store. Steps 2 and 3 show the compression (26.7): one 512-number latent is stored, and the 128 per-head keys and values are rebuilt from it. Step 4 adds the 64-number RoPE key (26.10), so the cache holds 576 numbers per layer per token. Step 5 shows decoding with absorbed matrices (26.8, 26.9): each head turns its query into a 512-number latent query and works on the cache directly; the dashed per-head keys and values disappear. The readout keeps a running count of the cache.

Figure 4 uses the real sizes but no real numbers. Figure 5 runs Algorithm 3 on a toy MLA layer in your browser: hidden size 6, 4 heads with \(d_h = 2\), a RoPE key of \(d_h^R = 2\) numbers rotated by 0.5 rad per position, and a latent of \(d_c\) numbers that you choose. At every head step it computes the score twice: the fast way, with the absorbed query \(\tilde{\mathbf{q}}_i\) and the cached latents (26.8), and the slow way, by rebuilding each key \(\mathbf{k}^C_{j,i} = W^{UK}_i \mathbf{c}_j\). They agree. Then turn on “RoPE on the content keys” to see why (26.10) is needed.

Latent size \(d_c\)
  1. \(\mathbf{c}_t \leftarrow W^{DKV} \mathbf{h}_t\); append to \(C\) (26.7)
  2. \(\mathbf{k}^R_t \leftarrow \operatorname{RoPE}_t(W^{KR} \mathbf{h}_t)\); append to \(K^R\) (26.10)
  3. for \(i = 0\) to \(n_h - 1\):
  4. \(\tilde{\mathbf{q}}_i \leftarrow (W^{UK}_i)^\top W^Q_i \mathbf{h}_t\) // (26.8)
  5. \(\mathbf{q}^R_i \leftarrow \operatorname{RoPE}_t(W^{QR}_i \mathbf{h}_t)\)
  6. \(s_j \leftarrow (\tilde{\mathbf{q}}_i \cdot \mathbf{c}_j + \mathbf{q}^R_i \cdot \mathbf{k}^R_j) / \sigma\) // \(\sigma = \sqrt{d_h + d_h^R}\)
  7. \(\mathbf{a} \leftarrow \softmax(\mathbf{s})\); \(\mathbf{o}_i \leftarrow W^{UV}_i \sum_j a_j \mathbf{c}_j\) // (26.9)
cached latent \(\mathbf{c}\)cached RoPE key \(\mathbf{k}^R\)head (query side)head output
Figure 5. MLA computed live on a toy layer: 4 tokens × 6 steps (compress, RoPE key, one step per head). The table is the whole cache: one row per token, \(d_c\) latent numbers plus 2 RoPE numbers. The bars compare it with what MHA, GQA-2 and MQA would store for the same 4 heads.
How to read Figure 5

The first two steps of each token add one row to the cache (first the purple latent, then the orange RoPE key). In each head step, all rows are shaded by the weight \(a_j\) this head gives them, and the weights are listed in the last column. Every head reads the same rows: at decode time MLA works like MQA with one wide head. In the panel, each score shows the absorbed value \(\tilde{\mathbf{q}}_i \cdot \mathbf{c}_j\) next to the check \(\mathbf{q}^C \cdot \mathbf{k}^C_j\) from rebuilt keys; they are equal, and so are the two outputs. Change \(d_c\): the MLA bar grows by \(d_c + 2\) per token (4 to 8, against 16 for MHA), and the outputs change because the low-rank bottleneck changes. Now tick “RoPE on the content keys”: the true score becomes \((R_t \mathbf{q}^C) \cdot (R_j \mathbf{k}^C_j)\), which depends on \(t - j\), and the absorbed score is wrong for every earlier row (red); only the row of the current token still matches. For head 3 at token 4 with \(d_c = 3\) the largest error in \(\mathbf{q} \cdot \mathbf{k}\) is 0.814.

Comparing the four designs

All four designs compute attention as in (26.1). They differ only in what is stored per token and how many query heads read each stored vector. The table uses one layer and \(n_h\) heads of size \(d_h\); the last column is the qualitative rating from the DeepSeek-V2 paper (its Table 1). [Certain that the paper gives these ratings; they are the authors’ summary, not a universal law]

DesignCached numbers per token per layerQuery heads per stored vectorExampleQuality (DeepSeek-V2 rating)
MHA\(2 n_h d_h\)1Llama 2 7B: 8,192Strong
GQA\(2 n_{kv} d_h\)\(n_h / n_{kv}\)Llama 3 8B: 2,048Moderate
MQA\(2 d_h\)\(n_h\)Falcon 7B: 128Weak
MLA\(d_c + d_h^R \approx 4.5\, d_h\)\(n_h\) (after absorption)DeepSeek-V2: 576Stronger

And for whole real models, from their config files (bfloat16 cache):

ModelAttention\(L\)\(n_h\) / \(n_{kv}\)\(d_h\)KV cache per tokenAt 128k tokens
Llama 2 7BMHA3232 / 32128512 KiB64 GiB
Llama 3 8BGQA3232 / 8128128 KiB16 GiB
Mistral 7BGQA3232 / 8128128 KiB16 GiB
Qwen2.5 7BGQA2828 / 412856 KiB7 GiB
Llama 3 70BGQA8064 / 8128320 KiB40 GiB
Falcon 7BMQA3271 / 1648 KiB1 GiB
DeepSeek-V2MLA60128 / latent12867.5 KiB8.4 GiB

The “at 128k tokens” column is a what-if for models whose real context is shorter (Llama 2 has 4,096 tokens; Falcon 7B 2,048); it shows how the design scales. Llama 3 70B is a good lesson: it is almost 9 times larger than Llama 3 8B, yet its cache per token is only 2.5 times larger, because both use 8 key–value heads. With MHA (64 key–value heads) its cache would be 2,560 KiB per token, 320 GiB at 128k tokens.

Figure 6. A KV-cache calculator, formula (26.2) and (26.11) in bfloat16. A preset loads a real model’s \(L\), \(n_h\), \(d_h\) and its real attention design (marked “real”). The other bars show what the same shape would need with the other designs: GQA uses the model’s own \(n_{kv}\), or 8 if the model does not use GQA; MLA uses DeepSeek’s sizes \(d_c = 4 d_h\), \(d_h^R = d_h / 2\). The dashed line is the memory of one 80 GB GPU.
How to read Figure 6

Start with Llama 3 8B at 131,072 tokens and batch 1: the real GQA bar is 16 GiB; MHA would be 64 GiB. Raise the batch to 4: MHA no longer fits on one GPU (the dashed line), GQA needs 64 GiB. Load Llama 3 70B: with MHA even one 128k-token sequence would need 320 GiB. Load DeepSeek-V2 and compare its real MLA bar with the 128-head MHA bar. The readout also gives the time to read the real cache once from memory at 3.35 TB/s (an H100 SXM), which is roughly the extra time each decode step spends on attention. Falcon has 71 heads, which cannot be split into 8 equal groups, so its GQA bar is shown as not available.

Effects on quality, training and serving

  • Quality. Fewer key–value heads means fewer distinct “views” of past tokens. Uptrained GQA-8 is within 0.1 points of MHA on T5-XXL, MQA loses 0.6 points (GQA paper). [Certain] Trained from scratch at 7B, DeepSeek saw MHA clearly ahead of GQA and MQA, and MLA ahead of MHA at 16B and 250B total parameters (mixture-of-experts models). [Certain that the paper reports this] Other groups may see different numbers; the general order MHA ≥ GQA ≥ MQA is the common finding. [Likely]
  • Parameters. GQA and MQA remove parameters from \(W^K\) and \(W^V\). Models usually give them back elsewhere (for example a wider feed-forward network) to keep the total size. [Likely] MLA adds the down- and up-projections but removes the large per-head \(W^K\) and \(W^V\).
  • Training cost. In training and prefill, all tokens are processed at once, the computation is limited by FLOPs, and the cache does not matter much. So the variants change training speed only a little. Their big effect is on decode.
  • Batch size and throughput. On a GPU with fixed memory, the cache limits how many sequences can be decoded at once. Shrinking it by 4 to 8 times allows 4 to 8 times larger batches, and larger batches make each read of the weights serve more users. DeepSeek reports a maximum generation throughput 5.76 times that of DeepSeek 67B (with FP8 weights and the quantized cache, on 8 H800 GPUs). [Certain that the paper reports this]
  • Combining with other tricks. These designs stack with cache quantization (session 110), paged memory (session 134) and sliding-window attention (session 27). Mistral 7B, for example, combines GQA with a sliding window. [Certain]

Where this is used

ApplicationWhy the KV-cache design mattersLink
Chat serving at scaleCache size sets the batch size per GPU and so the cost per token. GQA is the default in open models; MLA in DeepSeek-V2 and V3.session 80
Long-document QA and RAGPrompts of 32k to 128k tokens make the cache larger than the weights; the read time per token grows with the context (26.3).session 28
Long-video understanding and Video-RAGOne hour of video at 1 frame per second and 64 visual tokens per frame is 230,400 tokens. With a Llama-3-8B-like GQA cache that is 28.1 GiB; with MHA it would be 112.5 GiB; with an MLA-style cache like DeepSeek-V2’s, 14.8 GiB.session 206
Video question answering and key-moment detectionMany questions about the same video reuse one prefilled cache of the video tokens (prefix caching); a smaller cache lets more videos stay in memory.session 134
On-device and edge modelsPhones have a few GB of memory; small models like Llama 3.2 1B use GQA (32 query heads, 8 key–value heads, 32 KiB per token). [Certain]session 141
Agents with long tool historiesAgent traces grow over many steps, so the cache grows; a 4 to 8 times smaller cache means 4 to 8 times longer histories on the same GPU.session 141
Converting existing modelsMean-pool MHA heads into GQA (26.6) and uptrain with a small fraction of the original compute.session 97

Pseudo-code

Algorithm 1: one decode step of GQA with a KV cache (one layer)
  1. input: hidden vector \(\mathbf{h}_t\), cache \(K, V\) of shape \((n_{kv}, t-1, d_h)\), group size \(r = n_h / n_{kv}\)
  2. \(\mathbf{q}_i \leftarrow W^Q_i \mathbf{h}_t\) for \(i = 0, \dots, n_h - 1\)
  3. \(\mathbf{k}_g \leftarrow W^K_g \mathbf{h}_t\), \(\mathbf{v}_g \leftarrow W^V_g \mathbf{h}_t\) for \(g = 0, \dots, n_{kv} - 1\)
  4. append \(\mathbf{k}_g, \mathbf{v}_g\) to \(K_g, V_g\) // the cache grows by \(2 n_{kv} d_h\) numbers, formula (26.2)
  5. for \(i = 0\) to \(n_h - 1\) do
  6. \(g \leftarrow \lfloor i / r \rfloor\) // formula (26.5); \(r = 1\) is MHA, \(r = n_h\) is MQA
  7. \(\mathbf{a} \leftarrow \softmax(K_g \mathbf{q}_i / \sqrt{d_h})\) // reads \(K_g\); the \(r\) heads of a group share this read
  8. \(\mathbf{o}_i \leftarrow V_g^\top \mathbf{a}\)
  9. return \(W^O [\mathbf{o}_0; \dots; \mathbf{o}_{n_h - 1}]\)
Algorithm 2: convert MHA to GQA (Ainslie et al.)
  1. input: MHA checkpoint with \(W^K_i, W^V_i\) for \(i = 0, \dots, n_h - 1\); target \(n_{kv}\)
  2. \(r \leftarrow n_h / n_{kv}\)
  3. for \(g = 0\) to \(n_{kv} - 1\) do
  4. \(W^K_g \leftarrow \frac{1}{r}\sum_{i = g r}^{g r + r - 1} W^K_i\); \(\ W^V_g \leftarrow \frac{1}{r}\sum_{i = g r}^{g r + r - 1} W^V_i\) // formula (26.6)
  5. keep \(W^Q_i\) and \(W^O\) unchanged
  6. continue pre-training for about 5% of the original steps // “uptraining”
Algorithm 3: one decode step of MLA with absorbed matrices (one layer)
  1. input: \(\mathbf{h}_t\), position \(t\), cache of latents \(C\) (shape \((t-1, d_c)\)) and RoPE keys \(K^R\) (shape \((t-1, d_h^R)\))
  2. append \(\mathbf{c}_t = W^{DKV} \mathbf{h}_t\) to \(C\) and \(\mathbf{k}^R_t = \operatorname{RoPE}_t(W^{KR}\mathbf{h}_t)\) to \(K^R\) // \(d_c + d_h^R\) numbers, (26.11)
  3. for \(i = 0\) to \(n_h - 1\) do
  4. \(\tilde{\mathbf{q}}_i \leftarrow (W^{UK}_i)^\top W^Q_i \mathbf{h}_t\) // latent query, size \(d_c\), formula (26.8)
  5. \(\mathbf{q}^R_i \leftarrow \operatorname{RoPE}_t(W^{QR}_i \mathbf{h}_t)\)
  6. \(\mathbf{a} \leftarrow \softmax\big((C \tilde{\mathbf{q}}_i + K^R \mathbf{q}^R_i) / \sqrt{d_h + d_h^R}\big)\) // formula (26.10); all heads read the same C
  7. \(\mathbf{o}_i \leftarrow W^{UV}_i (C^\top \mathbf{a})\) // formula (26.9); \(W^{UV}\) can be merged into \(W^O\)
  8. return \(W^O [\mathbf{o}_0; \dots; \mathbf{o}_{n_h - 1}]\)

The same ideas in PyTorch. The script is deterministic and runs on a CPU in about a second. Each block checks one claim from this session:

import torch
import torch.nn as nn
torch.manual_seed(0)

# 1) one module for MHA, GQA and MQA: only n_kv changes, formula (26.5)
class GQA(nn.Module):
    def __init__(self, d, n_h, n_kv):
        super().__init__()
        assert n_h % n_kv == 0
        self.n_h, self.n_kv, self.dh = n_h, n_kv, d // n_h
        self.wq = nn.Linear(d, n_h * self.dh, bias=False)
        self.wk = nn.Linear(d, n_kv * self.dh, bias=False)
        self.wv = nn.Linear(d, n_kv * self.dh, bias=False)
        self.wo = nn.Linear(n_h * self.dh, d, bias=False)

    def forward(self, x, cache=None):               # x: (B, T, d)
        B, T, _ = x.shape
        q = self.wq(x).view(B, T, self.n_h, self.dh).transpose(1, 2)
        k = self.wk(x).view(B, T, self.n_kv, self.dh).transpose(1, 2)
        v = self.wv(x).view(B, T, self.n_kv, self.dh).transpose(1, 2)
        if cache is not None:                       # append the new K, V to the cache
            if "k" in cache:
                k = torch.cat([cache["k"], k], dim=2)
                v = torch.cat([cache["v"], v], dim=2)
            cache["k"], cache["v"] = k, v           # the cache holds n_kv heads only
        g = self.n_h // self.n_kv                   # query heads per KV head
        k = k.repeat_interleave(g, dim=1)           # (B, n_h, S, dh)
        v = v.repeat_interleave(g, dim=1)
        S = k.shape[2]
        mask = torch.ones(T, S, dtype=torch.bool).tril(S - T)
        a = (q @ k.transpose(-1, -2)) / self.dh ** 0.5
        a = a.masked_fill(~mask, float("-inf")).softmax(-1)
        y = (a @ v).transpose(1, 2).reshape(B, T, -1)
        return self.wo(y)

d, n_h = 64, 8
x = torch.randn(1, 10, d)
for n_kv in (8, 2, 1):                              # MHA, GQA-2, MQA
    m = GQA(d, n_h, n_kv)
    full = m(x)
    cache = {}
    steps = [m(x[:, t:t + 1], cache) for t in range(10)]   # decode one token at a time
    same = torch.allclose(full, torch.cat(steps, dim=1), atol=1e-5)
    print(n_kv, "kv heads | cached numbers per token per layer:",
          2 * cache["k"].shape[1] * cache["k"].shape[3], "| cached == full:", same)
# 8 kv heads: 128, 2: 32, 1: 16; cached == full: True each time

# 2) MHA -> GQA by mean-pooling the K and V projections in each group, formula (26.6)
mha = GQA(d, n_h, 8)
def pool(w, n_kv, dh):                              # w: (n_h*dh, d)
    return w.view(n_kv, -1, dh, w.shape[1]).mean(1).reshape(n_kv * dh, -1)
gqa = GQA(d, n_h, 2)
with torch.no_grad():
    gqa.wq.weight.copy_(mha.wq.weight); gqa.wo.weight.copy_(mha.wo.weight)
    gqa.wk.weight.copy_(pool(mha.wk.weight, 2, 8)); gqa.wv.weight.copy_(pool(mha.wv.weight, 2, 8))
err = (gqa(x) - mha(x)).norm() / mha(x).norm()
print("relative error before uptraining:", round(err.item(), 3))
# 0.911: these random heads are unrelated (rho = 0 in Figure 3), so pooling loses a lot

# 3) MLA: cache one latent c (d_c numbers) + one shared RoPE key (d_r numbers)
def rope(t, pos):                                   # t: (..., S, d_r), simple RoPE
    half = t.shape[-1] // 2
    f = 1.0 / (10000 ** (torch.arange(half) / half))
    ang = pos[:, None] * f[None]
    c, s = ang.cos(), ang.sin()
    t1, t2 = t[..., :half], t[..., half:]
    return torch.cat([t1 * c - t2 * s, t1 * s + t2 * c], -1)

d, n_h, dh, d_c, d_r, T = 64, 4, 16, 32, 8, 10
W_dkv = torch.randn(d_c, d) / d ** 0.5              # down-projection, shared by all heads
W_uk = torch.randn(n_h, dh, d_c) / d_c ** 0.5       # up-projection to keys, per head
W_uv = torch.randn(n_h, dh, d_c) / d_c ** 0.5       # up-projection to values, per head
W_q = torch.randn(n_h, dh, d) / d ** 0.5            # content part of the queries
W_qr = torch.randn(n_h, d_r, d) / d ** 0.5          # RoPE part of the queries, per head
W_kr = torch.randn(d_r, d) / d ** 0.5               # RoPE key: ONE for all heads
h = torch.randn(T, d)
pos = torch.arange(T).float()
c = h @ W_dkv.T                                     # (T, d_c)   cached, formula (26.7)
k_r = rope(h @ W_kr.T, pos)                         # (T, d_r)   cached, formula (26.10)
mask = torch.ones(T, T, dtype=torch.bool).tril()

# naive: rebuild the full K and V of every head from the latent
q_c = torch.einsum("hed,td->hte", W_q, h)
q_r = rope(torch.einsum("hed,td->hte", W_qr, h), pos)
K = torch.einsum("hec,tc->hte", W_uk, c)
V = torch.einsum("hec,tc->hte", W_uv, c)
s = (q_c @ K.transpose(-1, -2) + q_r @ k_r.T) / (dh + d_r) ** 0.5
o_naive = s.masked_fill(~mask, float("-inf")).softmax(-1) @ V      # (n_h, T, dh)

# absorbed: move W_uk into the query and W_uv after the sum; K and V never exist
q_lat = torch.einsum("hte,hec->htc", q_c, W_uk)     # (n_h, T, d_c), formula (26.8)
s2 = (q_lat @ c.T + q_r @ k_r.T) / (dh + d_r) ** 0.5
o_lat = s2.masked_fill(~mask, float("-inf")).softmax(-1) @ c       # (n_h, T, d_c)
o_abs = torch.einsum("htc,hec->hte", o_lat, W_uv)                 # formula (26.9)
print("absorbed == naive:", torch.allclose(o_naive, o_abs, atol=1e-5))  # True
print("MLA cache per token:", d_c + d_r, "numbers vs MHA", 2 * n_h * dh)  # 40 vs 128

# 4) KV-cache bytes per token of real configurations (bfloat16), formula (26.2)
def kv_bytes(layers, n_kv, dh, b=2):
    return 2 * layers * n_kv * dh * b
print(kv_bytes(32, 32, 128), kv_bytes(32, 8, 128), kv_bytes(80, 8, 128))
# 524288 (Llama 2 7B), 131072 (Llama 3 8B), 327680 (Llama 3 70B)
print((512 + 64) * 60 * 2)                          # 69120 bytes (DeepSeek-V2, MLA)

Common mistakes

  • Using \(n_h\) instead of \(n_{kv}\) in the cache formula. Llama 3 70B has 64 query heads but only 8 key–value heads. Using 64 gives 2.5 MiB per token instead of 320 KiB, eight times too much. Always read num_key_value_heads from the config.
  • Forgetting the factor 2 or the layers. The cache stores keys and values, in every layer. A frequent interview slip is \(n_{kv} d_h T b\) without \(2L\).
  • Assuming \(d_h = d / n_h\). Usually true, but not always: some configs set head_dim separately (Llama 3.2 1B has \(d = 2048\), 32 heads, \(d_h = 64\), which happens to match; other models do not). Read head_dim when it exists.
  • Thinking GQA saves FLOPs. The number of attention scores is the same. GQA saves memory and memory traffic, which is what decoding is limited by (26.3).
  • Grouping heads in the wrong order. Query head \(i\) must use key–value head \(\lfloor i / r \rfloor\) (repeat_interleave), not \(i \bmod n_{kv}\) (repeat), when the checkpoint was trained with the first layout. The two give silently wrong outputs, not errors.
  • Applying RoPE to the MLA latent. Rotating \(\mathbf{c}_t\) or the content keys breaks the absorption trick (26.8). Position must go through the separate RoPE key and query (26.10).
  • Caching the up-projected keys in MLA. Some early implementations rebuilt and cached the full per-head keys and values, which gives MHA-size memory and throws away the benefit. Cache only \(\mathbf{c}_t\) and \(\mathbf{k}^R_t\). [Likely]
  • Converting to GQA without uptraining. Mean pooling alone changes the outputs (Figure 3); the GQA paper relies on a short continued pre-training to recover quality.

Interview questions

Why do we cache keys and values during generation, but not queries?basic
With a causal mask, the key and value of token \(j\) depend only on tokens up to \(j\), so they never change once computed. Every later token needs them, so we store them and add one new row per step. The query of a token is used only in the step that token is processed; later tokens never attend from an old query. So old queries are thrown away.
Write the KV-cache size formula and compute it for Llama 3 8B at 8k tokens, batch 1.basic
\(M = 2 \cdot L \cdot n_{kv} \cdot d_h \cdot T \cdot B \cdot b\) (26.2). Llama 3 8B: \(L = 32\), \(n_{kv} = 8\), \(d_h = 128\), bfloat16 so \(b = 2\): per token \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes = 128 KiB. At \(T = 8{,}192\): 1 GiB. With MHA (32 KV heads) it would be 4 GiB.
What is the difference between MHA, MQA and GQA?basic
All have \(n_h\) query heads. MHA has \(n_h\) key–value heads (one per query head). MQA has one key–value head shared by all query heads. GQA has \(n_{kv}\) key–value heads, each shared by a group of \(n_h / n_{kv}\) query heads (26.5); MHA and MQA are the two ends. The cache scales with \(n_{kv}\), so MQA is smallest and MHA largest; quality usually goes the other way.
GQA barely changes the FLOPs. Why does it make decoding faster?core
Decoding is memory-bound. For each new token the GPU must read the whole cache, and attention does only about \(r = n_h / n_{kv}\) FLOPs per byte read (26.3), far below the ~295 FLOPs/byte an H100 needs to be compute-bound. So the time is about bytes ÷ bandwidth. GQA reads \(r\) times fewer bytes. The smaller cache also allows bigger batches, so each read of the weights serves more sequences, raising throughput. Prefill and training are compute-bound and gain little.
How do you convert a trained MHA model into a GQA model?core
Following Ainslie et al.: group the query heads, and set each group’s new key and value projection to the mean of the old projections of the heads in that group (26.6); keep \(W^Q\) and \(W^O\). Mean pooling beat picking one head and random init. Then uptrain, continuing pre-training for about 5% of the original steps, so the model adapts. Without uptraining the outputs change noticeably, more so when the original heads differ a lot (Figure 3).
Why is GQA with 8 key–value heads so common? What goes wrong with MQA under tensor parallelism?advanced
8 key–value heads is a good quality–memory trade-off (GQA-8 matched MHA within 0.1 points on T5-XXL while being nearly as fast as MQA). It also matches common tensor-parallel degrees: with 8 GPUs, each GPU holds its query heads and exactly one key–value head. With MQA, the single key–value head is needed on every GPU, so it is replicated 8 times and the memory saving per GPU shrinks; the KV projection compute is also duplicated. The GQA paper notes that standard sharding replicates the single key–value head across partitions and that GQA removes this waste.
Explain MLA. What is stored in the cache, and how can it be smaller than GQA while every head has its own key?advanced
MLA projects each hidden vector down to a latent \(\mathbf{c}_t\) of size \(d_c\) (512 in DeepSeek-V2), and each head’s key and value are up-projections of it (26.7). Only \(\mathbf{c}_t\) and a small shared RoPE key (64 numbers) are cached: 576 numbers per layer versus 32,768 for MHA with 128 heads. Keys stay per-head because \(W^{UK}_i\) differs per head; the information is shared through a low-rank bottleneck instead of by copying the same key. At decode time \(W^{UK}_i\) is absorbed into the query (26.8) and \(W^{UV}_i\) into the output (26.9), so the per-head keys and values are never built.
Why does MLA need a decoupled RoPE key?advanced
RoPE rotates queries and keys by position. With RoPE on the compressed keys, the score becomes \(\mathbf{q}^\top R_{j-t} W^{UK}_i \mathbf{c}_j\): a position-dependent rotation sits between the query and \(W^{UK}_i\), so \(W^{UK}_i\) cannot be absorbed into the query once for all positions, and you would need to rebuild keys for every cached token. DeepSeek instead keeps the latent part free of RoPE and adds a small extra key \(\mathbf{k}^R_t = \operatorname{RoPE}(W^{KR}\mathbf{h}_t)\) (64 numbers, shared by all heads) plus per-head RoPE queries; the score is the sum of both parts, scaled by \(1/\sqrt{d_h + d_h^R}\) (26.10).
DeepSeek-V2 says its KV cache equals GQA with 2.25 groups. Derive this.core
MLA caches \(d_c + d_h^R\) numbers per layer; with \(d_c = 4 d_h\) and \(d_h^R = d_h/2\) that is \(4.5\, d_h\). GQA caches \(2\, n_{kv}\, d_h\). Setting \(2 n_{kv} d_h = 4.5 d_h\) gives \(n_{kv} = 2.25\). For \(d_h = 128\): 576 numbers, between GQA-2 (512) and GQA-3 (768).
Llama 3 70B is 9 times bigger than Llama 3 8B. Is its KV cache 9 times bigger?core
No. Both use 8 key–value heads of size 128. The 70B model has 80 layers instead of 32, so its cache is \(80/32 = 2.5\) times larger: 320 KiB versus 128 KiB per token. The extra size of the 70B model went into width (8,192 vs 4,096) and query heads (64 vs 32), which do not enter the cache formula. With MHA it would have been 2,560 KiB per token.
You serve a VLM on long videos and run out of GPU memory during decoding. What do you check and change?advanced
First compute the cache with (26.2): visual tokens per frame × frames × \(2 L n_{kv} d_h b\); for example 230,400 visual tokens with a Llama-3-8B-like cache is 28.1 GiB per video. Options: pick a backbone with fewer KV heads (GQA) or MLA; quantize the cache to 8 or 4 bits; reduce visual tokens (fewer frames, pooling or token merging); use paged memory and share the prefilled video prefix across questions; lower the batch size. Changing \(n_{kv}\) of a trained model requires conversion plus uptraining, so it is a training-time decision.

Exercises

  1. Mistral 7B has \(L = 32\), \(n_h = 32\), \(n_{kv} = 8\), \(d_h = 128\). What is its KV cache per token in bfloat16, and for 16 sequences of 8,192 tokens?
  2. A model has 48 query heads. List all valid GQA group counts \(n_{kv}\) and the group size \(r\) for each.
  3. Using (26.3), what is the arithmetic intensity of decode attention for Llama 3 70B, Llama 2 7B and Falcon 7B? Is any of them compute-bound on an H100?
  4. In MLA, why can \(W^{UV}_i\) be merged into \(W^O\)? Write the product that shows it.
  5. DeepSeek-V3 has \(L = 61\) and the same \(d_c = 512\), \(d_h^R = 64\) as V2. What is its cache per token in bfloat16?
  6. A GPU has 80 GB. Llama 3 8B weights take 16 GB. Ignoring all other memory, how many sequences of 32,768 tokens fit in the rest with GQA (as built) and with a hypothetical MHA version?
  7. In MLA during decode, each of 128 heads does dot products with 576 cached numbers (latent plus RoPE key) and a weighted sum of 512 numbers, per cached token. What is the arithmetic intensity, counting the 576 cached numbers in bfloat16 read once per token?
Answerscheck
1. \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes = 128 KiB per token. \(16 \times 8{,}192 = 131{,}072\) tokens, so 16 GiB. 2. Divisors of 48: \(n_{kv} \in \{1, 2, 3, 4, 6, 8, 12, 16, 24, 48\}\) with \(r = 48, 24, 16, 12, 8, 6, 4, 3, 2, 1\). 3. \(r = n_h / n_{kv}\): Llama 3 70B \(64/8 = 8\); Llama 2 7B \(32/32 = 1\); Falcon 7B \(71/1 = 71\) FLOPs per byte. All are below the ~295 FLOPs/byte ridge of an H100, so all are memory-bound; MQA is closest. 4. \(W^O [\mathbf{o}_0; \dots] = \sum_i W^O_i \mathbf{o}_i = \sum_i W^O_i W^{UV}_i \big(\sum_j a_{t,j,i} \mathbf{c}_j\big)\), where \(W^O_i\) is the block of \(W^O\) that multiplies head \(i\). The product \(W^O_i W^{UV}_i\) (shape \(d \times d_c\)) can be computed once ahead of time. 5. \(576 \times 61 \times 2 = 70{,}272\) bytes \(\approx 68.6\) KiB. 6. 64 GB are free (about 59.6 GiB). GQA: 32,768 tokens × 128 KiB = 4 GiB per sequence, so 14 sequences. MHA: 16 GiB per sequence, so 3 sequences. 7. FLOPs per cached token: \(128 \times (2 \times 576 + 2 \times 512) = 278{,}528\); bytes: \(576 \times 2 = 1{,}152\); intensity \(\approx 242\) FLOPs per byte, close to the H100 ridge of ~295. This is why MLA decoding is much less memory-bound than GQA.

Key takeaways

  • Generation caches every earlier token’s keys and values. Its size is \(2 L n_{kv} d_h T B b\) bytes (26.2): linear in context and batch, and often larger than the weights.
  • Decode attention does only about \(n_h / n_{kv}\) FLOPs per byte read (26.3), so it is memory-bound: fewer cached bytes means faster tokens and bigger batches.
  • MQA shares one key–value head among all query heads (26.4): smallest GQA-style cache, but some quality loss and replication under tensor parallelism.
  • GQA shares one key–value head per group (26.5). GQA-8 nearly matches MHA quality at nearly MQA speed, and MHA checkpoints convert by mean pooling plus about 5% uptraining (26.6). It is the default in Llama 3, Mistral and Qwen2.5.
  • MLA caches a small latent vector plus a shared RoPE key, \((d_c + d_h^R) L\) numbers per token (26.11): 576 per layer in DeepSeek-V2, like GQA with 2.25 groups, while each head keeps its own key and value through up-projections (26.7) that are absorbed at decode time (26.8, 26.9).
  • RoPE does not commute with the absorption, so MLA puts position in a separate small key (26.10).

Key formulas

\[ M_{\text{KV}} = 2\, L\, n_{kv}\, d_h\, T\, B\, b, \qquad \text{intensity} \approx \frac{n_h}{n_{kv}}\ \text{FLOPs/byte}, \qquad g(i) = \left\lfloor \frac{i\, n_{kv}}{n_h} \right\rfloor \]
\[ \mathbf{c}_t = W^{DKV}\mathbf{h}_t, \quad \mathbf{q}_{t,i}^{\top} W^{UK}_i \mathbf{c}_j = \big((W^{UK}_i)^{\top}\mathbf{q}_{t,i}\big)^{\top} \mathbf{c}_j, \quad \text{MLA cache} = (d_c + d_h^R)\,L \]

References for this session

  1. Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need
  2. Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
  3. DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
  4. Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need
  5. Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
  6. DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
  7. DeepSeek-AI (2024). DeepSeek-V3 Technical Report
  8. Vaswani et al. (2017). Attention Is All You Need
  9. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  10. Llama Team, Meta (2024). The Llama 3 Herd of Models
  11. Jiang et al. (2023). Mistral 7B
  12. Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
  13. Almazrouei et al. (2023). The Falcon Series of Open Language Models
  14. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  15. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
  16. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale