Efficient attention 1: MQA, GQA and MLA, from the KV-cache view
When a model is served to many users, one number decides how many fit on a GPU and how many tokens per second it can sell: the KV bytes per token. This session looks at MQA, GQA and MLA through that number: what each stores, how to convert a trained MHA model by mean pooling and uptraining (or by an SVD into MLA), how MLA runs in prefill and decode, and what happens under tensor parallelism. Quality numbers from the papers sit next to the bytes.
What you will learn
- Compute KV bytes per token for MHA, GQA, MQA and MLA models from their config files, and avoid the MLA config trap
- Turn bytes per token into the largest batch and into decode throughput on a real GPU, and see why throughput at full memory is inversely proportional to it
- Convert an MHA checkpoint into GQA: why mean pooling is the least-squares start and what the GQA paper found about uptraining
- Show that GQA is a special case of MLA, and convert MHA weights to MLA with an SVD at the same cache size
- Explain the two ways to run MLA (naive for prefill, absorbed for decode) with their FLOPs and arithmetic intensity
- Work out where the cache lives under tensor parallelism, why GQA-8 fits 8-GPU nodes and why MLA is served with data-parallel attention
- Read the quality results of the MQA, GQA and DeepSeek-V2 papers next to the cache sizes
Why this matters
When a model serves many users, one number decides how many of them fit on a GPU and how fast each of them gets tokens: the KV bytes per token, the memory that one token of context takes in the key–value cache. This session looks at MQA, GQA and MLA through that one number. We ask what each design stores, how many users it lets you serve, how to get it from a model that was trained with ordinary attention, and what it costs in quality.
Session 26 introduced the four designs (MHA, MQA, GQA, MLA) and their math. Session 80 showed that peak memory is roughly weights plus cache, and session 82 showed that decoding is limited by memory traffic. Here we put the three together, as an engineer who must serve a model. New in this session:
- A table of real KV bytes per token, read from config files, from 8 KiB (Falcon 7B) to 512 KiB (Llama 2 7B).
- A serving model: how bytes per token turn into the largest batch and into tokens per second (Figure 2).
- Uptraining in detail: why mean pooling is the right start, what the GQA paper measured, and a small uptraining run you can watch (Figure 3).
- MLA as a superset of GQA: GQA is a special case of MLA with the same cache, and an SVD can turn MHA weights into MLA weights (Figure 4).
- Two ways to run MLA (prefill and decode), and what happens to all four designs under tensor parallelism (Figure 5).
- Bytes per token and quality side by side, with the numbers the papers report.
This is the multi-query attention part of the model-level optimizations in the efficient-inference survey (its section 5.1.2). [Certain] It comes up in interviews as: “How many concurrent 32k-token users fit on one H100 with Llama 3.1 8B?”, “How would you turn an MHA checkpoint into GQA?”, “Why is GQA-8 so popular with 8-GPU nodes?”, “Why does DeepSeek serve MLA with data-parallel attention instead of tensor parallelism?”. All of them come down to bytes per token.
In this part of the course, session 94 to session 96 made the feed-forward network cheaper. This session and the next two do the same for attention. Here we attack the memory cost of attention (the cache); session 98 attacks its compute cost, the \(T^2\) attention matrix.
One number: KV bytes per token
What each design stores
For every token of context and in every layer, the cache stores some numbers. MHA, GQA and MQA store \(n_{kv}\) key heads and \(n_{kv}\) value heads of size \(d_h\). MLA stores one latent vector of size \(d_c\) and one shared RoPE key of size \(d_R\). Multiply by the number of layers and by the bytes per number:
- Read the symbols
- \(\beta\)
- KV bytes per token: the memory one token of context takes in the cache, over all layers
- \(L\)
- the number of layers (
num_hidden_layersin the config) - \(n_{kv}\)
- the number of key–value heads (
num_key_value_heads): \(n_{kv} = n_h\) is MHA, \(n_{kv} = 1\) is MQA, anything between is GQA - \(d_h\)
- the size of one head (
head_dim, or hidden size divided by the number of heads) - \(d_c, d_R\)
- for MLA: the latent size (
kv_lora_rank) and the size of the shared RoPE key (qk_rope_head_dim) - \(b\)
- bytes per number: 2 for bfloat16, 1 for an 8-bit cache
Notice what is not in (97.1): the hidden size \(d\), the number of query heads \(n_h\), and the feed-forward width. A model can grow wider and add query heads without growing its cache. That is why model designers like these variants: they decouple model size from cache size.
The cache for a whole batch is \(\beta \cdot T \cdot B\) for \(B\) sequences of \(T\) tokens. Everything in this session follows from \(\beta\).
How to read Figure 1
Each bar is one model; longer means more cache per token. One grid step is a factor of 4. The colours show the design. Click Llama 3.1 8B: 128 KiB per token, so 1 GiB holds 8,192 tokens. Click Llama 2 7B, a model of almost the same size with MHA: 512 KiB, four times more. The three MLA models (DeepSeek-V2, DeepSeek-V3, Kimi K2) are huge mixture-of-experts models with 128 or 64 heads, yet their cache per token is about half of Llama 3.1 8B’s. Turn on the MHA versions: for DeepSeek-V3 the grey bar is 57 times longer than the real one. Falcon 7B (MQA) is the smallest: 8 KiB.
Llama 3.1 8B: num_hidden_layers = 32, num_key_value_heads = 8, head_dim = 128. \(\beta = 2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes \(= 128\) KiB.
Qwen3-235B-A22B (a mixture-of-experts model): 94 layers, 64 query heads, 4 key–value heads, \(d_h = 128\). \(\beta = 2 \times 94 \times 4 \times 128 \times 2 = 192{,}512\) bytes \(= 188\) KiB. A model with 235B parameters has a cache per token only 1.5 times larger than an 8B model.
DeepSeek-V3: 61 layers, kv_lora_rank = 512, qk_rope_head_dim = 64. Note that its config also says num_key_value_heads = 128; that number is not what is cached. \(\beta = 61 \times (512 + 64) \times 2 = 70{,}272\) bytes \(\approx 68.6\) KiB. With MHA and its 128 heads of 128 it would be \(2 \times 61 \times 128 \times 128 \times 2\) bytes \(= 3{,}904\) KiB, about 57 times more.
From bytes per token to users and tokens per second
How many sequences fit
A GPU has memory \(M\). We can use a fraction \(u\) of it (the rest goes to activations, the CUDA context and fragmentation; serving systems such as vLLM use \(u = 0.9\) by default). The weights take \(W\) bytes. What is left holds the cache:
- Read the symbols
- \(N_{\text{tok}}\)
- the total number of tokens of context that the GPU can hold, over all sequences
- \(B_{\max}\)
- the largest batch: how many sequences of \(T\) tokens can be decoded at the same time
- \(u M - W\)
- the memory left for the cache after the weights
For Llama 3.1 8B on an 80 GB H100: \(W = 16.06\) GB in bfloat16 and \(u M - W = 72 - 16.06 = 55.94\) GB. At \(T = 32{,}768\) tokens, (97.2) gives 13 sequences with its real GQA-8 cache, but only 3 with MHA, 104 with MQA, and 46 with a DeepSeek-style MLA cache (\(d_c = 4 d_h = 512\), \(d_R = 64\): 36 KiB per token over 32 layers).
How fast each step is
A decode step reads all weights once and the whole cache once. It also does about \(2 N_p\) FLOPs per sequence for the weights (\(N_p\) is the number of parameters) and some FLOPs for attention. The step can be no faster than the slower of the two limits:
- Read the symbols
- \(\text{BW}\)
- memory bandwidth: 3.35 TB/s on an H100 SXM
- \(F\)
- peak compute: about 989 TFLOP/s dense bfloat16 on an H100 SXM [Certain for both spec numbers]
- \(\phi\)
- attention FLOPs per cached token per new token: \(4 L n_h d_h\) for MHA, GQA and MQA; more for MLA (formula 97.8)
- throughput
- generated tokens per second, summed over the batch
Now fill the GPU: \(B = B_{\max}\). Then \(W + B T \beta \approx u M\): every step reads almost the whole usable memory, whatever the design. The step time is about \(uM / \text{BW} \approx 21.5\) ms, and only the number of sequences that share this step changes. If the memory term wins:
Throughput at full memory is inversely proportional to \(\beta\). Halve the bytes per token and you double the tokens per second the GPU can sell, as long as the step stays memory-bound. This is the serving case for MQA, GQA and MLA in one line.
How to read Figure 2
At batch 1 all four designs give almost the same speed: the step is dominated by reading 16 GB of weights. As the batch grows, the cache grows, and the curves split. The dot at the end of each curve is \(B_{\max}\); higher dots mean more tokens per second per GPU. Move the context slider: at 4,096 tokens MHA stops at batch 26, at 131,072 tokens it cannot even hold one sequence (no curve). The readout lists, for each design, \(\beta\), \(B_{\max}\), the step time and the throughput at \(B_{\max}\). For MQA at short context the compute term starts to matter; real servers also hit other limits first (activation memory, latency targets).
Two lessons from Figure 2. First, at small batch, the design barely matters: the weights dominate. A chatbot that serves one user does not get faster with GQA unless the context is long. Second, at full memory the design decides everything. The real Llama 3.1 8B (GQA-8) gives about 4 times the throughput of an MHA version at every context length. This is why every large open model since 2023 uses GQA or MLA. [Likely, as a statement about the major open model families]
Sharing K and V: MQA and GQA, briefly
As a recap from session 26: in MQA all \(n_h\) query heads read one key head and one value head. In GQA the query heads are split into \(n_{kv}\) groups of \(r = n_h / n_{kv}\), and query head \(i\) reads key–value head \(g(i) = \lfloor i / r \rfloor\). The FLOPs barely change; the cache shrinks by \(r\), and the arithmetic intensity of decode attention rises from 1 to \(r\) FLOPs per byte.
What did the first paper actually measure? Shazeer (2019) trained translation models on WMT14 English–German with 8 heads of size 128, and widened the feed-forward layers of the MQA model so that both had the same number of parameters. [Certain, numbers from the paper’s Tables 1 and 2]
| Attention | KV numbers per token per layer | Dev ln(PPL) | Dev BLEU | Test BLEU (greedy / beam 4) | Decoder time per token |
|---|---|---|---|---|---|
| MHA, \(h = 8\), \(d_k = 128\) | 2,048 | 1.424 | 26.7 | 27.7 / 28.4 | 46 µs |
| MQA, \(h = 8\), \(d_k = 128\) | 256 | 1.439 | 26.5 | 27.5 / 28.5 | 3.8 µs |
| MHA, \(h = 1\), \(d_k = 128\) | 256 | 1.518 | 25.8 | – | – |
| MHA, \(h = 2\), \(d_k = 64\) | 256 | 1.480 | 26.2 | 26.8 / 27.9 | – |
| MHA, \(h = 8\), \(d_k = 16\) | 256 | 1.513 | 25.8 | – | – |
The decoder time is in TPUv2 microseconds per token, amortized over a batch of 1,024 sequences. MQA made the decoder about 12 times faster (46 vs 3.8) and lost 0.2 BLEU on the dev set. The rows below it are the most useful part of the table and are often forgotten: they are MHA models with the same cache as MQA (fewer or smaller heads, so \(2 h d_k = 256\)). All of them are clearly worse than MQA. So for a fixed cache budget, keeping many query heads and sharing K and V beats simply having fewer heads.
The GQA paper (Ainslie et al., 2023) then showed that the middle ground is better still: on T5-XXL, uptrained GQA-8 scored 47.1 on average against 47.2 for MHA and 46.6 for MQA, at 0.28 s per sample against 1.51 s for MHA and 0.24 s for MQA. [Certain, Table 1 of the paper] It also reported that MQA can make fine-tuning unstable, while uptrained GQA appeared stable. [Certain that the paper reports this]
Uptraining: from an MHA checkpoint to GQA
The problem
Suppose you own a strong MHA model and serving it is too expensive. Training a GQA model from scratch costs as much as the original. The GQA paper’s answer is uptraining: change the architecture of the trained model, then train it a little more. It has two steps: build the new key and value heads from the old ones, and continue pre-training on the original data for a fraction \(\alpha\) of the original steps.
Step 1: why the mean is the natural start
For each group \(g\) we need one key matrix \(W^K_g\) that replaces the \(r\) old matrices \(W^K_i\) of that group. A natural goal is to stay as close as possible to all of them:
- Read the symbols
- \(\lVert \cdot \rVert_F\)
- the Frobenius norm: the square root of the sum of all squared entries of a matrix
- \(\argmin_W\)
- the matrix \(W\) that makes the sum smallest
- \(i : g(i) = g\)
- the \(r\) query heads of group \(g\)
The proof is one line: the gradient of the sum is \(2 \sum_i (W - W^K_i)\), which is zero exactly at the mean. The same holds for \(W^V_g\). So mean pooling is the least-squares choice in weight space. It is not the best choice for the output of the layer, because the softmax is not linear; that gap is what uptraining has to close. The query and output matrices \(W^Q_i\) and \(W^O\) are kept as they are.
Step 2: continue training
What the GQA paper reports about uptraining [Certain, from the paper’s text and figures]:
- The main setting is \(\alpha = 0.05\): 5% of the original pre-training steps, with the same data, optimizer (Adafactor) and learning-rate schedule as T5. For T5-XXL this took about 600 TPUv3 chip-days.
- Mean pooling worked best, then keeping the first head of each group, then random initialization (tested on T5-Large converted to MQA).
- GQA already gives reasonable quality right after conversion, while MQA needs uptraining to be useful. Both gain from 5% uptraining, with diminishing returns at 10%.
- Going from 1 group (MQA) to 8 groups adds only a little inference time; more groups cost more. The authors chose 8 groups as the middle ground.
Figure 3 lets you watch a tiny version of this. A real uptraining run continues pre-training with the language-model loss; our stand-in trains the shared key and value weights so that the converted layer reproduces the MHA layer’s output on fresh random inputs. It is a toy, but the order of the three starting points, and the floor that more groups lower, are the same as in the paper.
How to read Figure 3
Press Train (or Run to step 200). At step 0, mean pooling starts lowest, the first head next, random highest: this is the order of the GQA paper. With \(\rho = 0.5\) and 2 groups the three start near 0.65, 0.89 and 1.39. After about 100 to 150 steps the first-head start has caught up with mean pooling; random is still behind at step 200 (0.684 against 0.620). All curves flatten at a floor: shared heads cannot copy 8 different heads exactly. More groups lower the floor (try 4 groups), a larger \(\rho\) (more similar heads) lowers it too. The readout gives the three errors at the current step.
Two honest limits of the toy. Real uptraining also trains the queries, the output projection and every other layer, so the network can move work around and the floor is much lower. And real heads are neither random nor all alike; how similar they are inside a group decides how much is lost, which is why the paper found GQA-8 much easier to recover than MQA.
MLA from the cache view
What MLA caches, in one line
From session 26: MLA projects each hidden vector to a latent \(\mathbf{c}_t = W^{DKV} \mathbf{h}_t \in \R^{d_c}\) and caches it with one small shared RoPE key \(\mathbf{k}^R_t \in \R^{d_R}\). Each head \(i\) gets its own content key \(W^{UK}_i \mathbf{c}_t\) and value \(W^{UV}_i \mathbf{c}_t\). The DeepSeek-V2 paper sets \(d_c = 4 d_h\) and \(d_R = d_h / 2\), so the cache equals GQA with 2.25 groups. [Certain]
GQA is a special case of MLA
Here is a fact that explains why MLA can only be better than GQA at the same cache size (ignoring RoPE for now). Take any GQA layer. Stack all its cached keys and values into one vector, and call it the latent:
- Read the symbols
- \(\mathbf{c}_t\)
- the GQA cache of token \(t\) written as one vector: exactly the same numbers GQA stores
- \(E_m\)
- a selection matrix of shape \(d_h \times 2 n_{kv} d_h\): an identity block at block position \(m\), zeros elsewhere. It just picks block \(m\) of the latent.
- \(g(i)\)
- the group of query head \(i\)
With these up-projections, MLA gives every head exactly the key and value that GQA gives it. So GQA is MLA with \(d_c = 2 n_{kv} d_h\) and fixed 0/1 up-projections. MLA learns the up-projections instead, so with the same cache it can give each head a different key and value, mixed from the whole latent. The reverse is not true: an MLA layer generally cannot be written as GQA. The TransMLA paper (Meng et al., 2025) proves this, with a way to handle RoPE, and uses it to convert GQA models such as Llama and Qwen into MLA. [Certain that the paper proves and does this] The PyTorch code below checks (97.6) numerically.
Converting MHA to MLA with an SVD
(97.6) also suggests a conversion from MHA. Stack all key and value matrices of an MHA layer into one tall matrix \(W^{KV} \in \R^{2 n_h d_h \times d}\). The keys and values of a token are \(W^{KV} \mathbf{h}_t\). If we must squeeze them through a latent of size \(d_c\), the best rank-\(d_c\) approximation of \(W^{KV}\) (in Frobenius norm) comes from the singular value decomposition (SVD), by the Eckart–Young theorem:
- Read the symbols
- \(U \Sigma V^{\top}\)
- the SVD: \(\Sigma\) holds the singular values \(\sigma_1 \ge \sigma_2 \ge \dots\), \(V\) the right singular vectors
- \(V_{d_c}\)
- the first \(d_c\) columns of \(V\) (shape \(d \times d_c\))
- \(W^{DKV} = V_{d_c}^{\top}\)
- the down-projection: the latent is \(\mathbf{c}_t = V_{d_c}^{\top} \mathbf{h}_t\)
- \(W^{KV} V_{d_c}\)
- the up-projections of all heads, stacked; rows of head \(i\) give \(W^{UK}_i\) or \(W^{UV}_i\)
- right-hand fraction
- the relative weight error: the share of the “energy” \(\sum \sigma_j^2\) that the discarded directions carried
Mean-pooled GQA with \(n_{kv}\) groups is also a matrix of rank at most \(2 n_{kv} d_h\) (its rows repeat). So at the same cache size, the SVD is never worse than mean pooling in weight space. This is guaranteed by Eckart–Young. For the layer’s output there is no such guarantee, but in practice it usually follows; Figure 4 measures it. This joint SVD of keys and values, plus keeping RoPE only on a few dimensions (“partial RoPE”), is the method of MHA2MLA (Ji et al., 2025), which reports a 92.19% smaller cache for Llama 2 7B with a 0.5% drop on LongBench, after fine-tuning on a small fraction of the data. [Certain that the paper reports this]
How to read Figure 4
Lower is better. In every pair the MLA bar is lower: with the same number of cached numbers, the SVD keeps more of the eight heads than averaging them. At \(\rho = 0.5\) and 64 cached numbers, mean pooling leaves an output error of about 0.56, the SVD about 0.22. Move \(\rho\) to 1: all heads are equal, so both errors drop to zero. The readout also shows the weight errors; the SVD weight error is never larger than the pooling one (Eckart–Young). In this toy the latent is still smaller than \(d = 96\); if \(d_c \ge d\), the latent could simply be \(\mathbf{h}_t\) itself and nothing would be lost.
Two ways to run MLA: prefill and decode
MLA can be computed in two mathematically equal ways, and serving systems use both.
- Naive (expanded) form: up-project the latents into per-head keys (\(d_h + d_R = 192\) numbers in DeepSeek-V2/V3) and values (128), then run ordinary attention with 128 heads. Per (query, key) pair and head this costs \(2 \cdot 192 + 2 \cdot 128 = 640\) FLOPs.
- Absorbed form: fold \(W^{UK}_i\) into the query and \(W^{UV}_i\) after the weighted sum (see session 26, formulas 26.8 and 26.9). Each head now works directly on the 576 cached numbers, like MQA with one very wide key–value head: \(2 \cdot 576 + 2 \cdot 512 = 2{,}176\) FLOPs per pair and head.
So the absorbed form does 3.4 times more FLOPs per pair, but never builds per-head keys. Which one wins depends on the phase:
- Read the symbols
- \(d_v\)
- the value size per head (
v_head_dim, 128) - \(I_{\text{MLA}}\)
- the arithmetic intensity of absorbed decode attention: FLOPs per byte of cache read
- \(b\)
- bytes per cached number (2 in bfloat16)
\(n_h = 128\), \(d_c = 512\), \(d_R = 64\), \(d_h = d_v = 128\).
Decode. Absorbed: \(128 \times (2 \times 576 + 2 \times 512) = 278{,}528\) FLOPs per cached token. Naive: the cache holds only latents, so each step must rebuild K and V of every cached token: \(2 \times 128 \times 512 \times 256 = 33{,}554{,}432\) FLOPs per cached token, about 120 times more. (The other naive option, caching the rebuilt keys and values, brings back an MHA-sized cache.) So decode uses the absorbed form. Its intensity is \(278{,}528 / 1{,}152 \approx 241.8\) FLOPs per byte, close to the H100 ridge point of about 295. MLA decode is almost compute-bound, unlike GQA-8 at 8 FLOPs per byte.
Prefill of \(T = 4{,}096\) tokens. Both forms first pay about 33.6 MFLOPs per token for projections (rebuilding K and V, or absorbing into queries). The pair term is \(T(T+1)/2 \times 128 \times 640\) for naive and \(\times 2{,}176\) for absorbed. In total: \(8.25 \times 10^{11}\) FLOPs naive against \(2.47 \times 10^{12}\) absorbed per layer, 3 times more. Prefill is compute-bound, so it uses the naive form. [Likely that this is how the main serving engines implement MLA; DeepSeek’s own inference code offers both modes]
Tensor parallelism: whose cache is it?
Large models are split across GPUs with tensor parallelism (TP, see session 48): each of \(P\) GPUs holds \(n_h / P\) query heads of every layer, and the cache of those heads. This interacts with key–value sharing in a way that often surprises people.
- Read the symbols
- \(\beta_{\text{GPU}}\)
- cache bytes per token stored on each GPU
- \(P\)
- the tensor-parallel degree (number of GPUs that share one layer)
- \(P \beta_{\text{GPU}} / \beta\)
- the replication factor: how many copies of each cached number exist in total
Three consequences:
- MQA loses its edge under TP. With \(P = 8\), MQA stores one key–value head per GPU, and so does GQA-8. The per-GPU cache is the same; MQA just keeps 8 identical copies. The GQA paper points out this waste. [Certain]
- GQA-8 fits 8-GPU nodes. With \(n_{kv} = 8 = P\), each GPU holds exactly one key–value head and nothing is copied. With \(P = 16\) every head is stored twice.
- MLA does not shard under TP. Every head reads the full latent, so every GPU stores all of it. For a 70B-shaped model with MLA sizes, that is 90 KiB per token on each GPU, more than GQA-8’s 40 KiB per GPU at \(P = 8\). This is why SGLang serves DeepSeek models with data-parallel attention: each GPU runs attention for its own set of requests and keeps only their latents, while the MoE layers are still shared across GPUs. SGLang’s documentation says that without it, the KV cache is duplicated on all TP ranks. [Certain that the docs say this]
How to read Figure 5
Pick a number of GPUs and a design. The readout gives the cache per token on each GPU, the total over all GPUs, the replication factor and the arithmetic intensity of decode attention on one GPU (FLOPs per byte read). Start with GQA-8 on 8 GPUs: one head per GPU, 40 KiB, no copies. Switch to MQA: still 40 KiB per GPU, now 8 copies. Switch to MLA, TP: 90 KiB on every GPU, 8 copies, and the intensity falls from 120.9 to 15.1, because each GPU has only 8 heads reading the full latent. MLA with DP attention: each GPU keeps the latents of its own requests only, so the total is 90 KiB per token and the intensity stays at 120.9.
Same 70B shape, 8 GPUs, and say each GPU has \(C\) bytes free for cache. GQA-8 with TP: every token takes 40 KiB on every GPU, so the node holds \(C / 40\,\text{KiB}\) tokens in total. MLA with DP attention: a token takes 90 KiB on one GPU only, so the node holds \(8 C / 90\,\text{KiB}\) tokens. The ratio is \((8/90) / (1/40) = 320 / 90 \approx 3.56\): exactly the ratio of total bytes per token (320 KiB vs 90 KiB). With MLA under plain TP the node would hold only \(C / 90\,\text{KiB}\) tokens, less than GQA-8. The design and the parallel layout must be chosen together.
Bytes per token and quality, side by side
All the quality numbers below come from the papers; they used different models, data and tasks, so compare rows only within one block. [Certain that these are the reported numbers]
| Study | Design | KV cache | Quality | Speed |
|---|---|---|---|---|
| Shazeer 2019, WMT14 En–De, trained from scratch | MHA (\(8 \times 128\)) | 2,048 per layer | BLEU 26.7 (dev) | 46 µs/token |
| MQA | 256 per layer (8× smaller) | BLEU 26.5 | 3.8 µs/token | |
| Ainslie et al. 2023, T5-XXL, uptrained 5% | MHA | 1× | 47.2 average | 1.51 s/sample |
| GQA-8 | \(n_{kv} = 8\) | 47.1 | 0.28 s | |
| MQA | \(n_{kv} = 1\) | 46.6 | 0.24 s | |
| DeepSeek-V2 paper, dense 7B, 1.33T tokens from scratch | MHA | 1× | MMLU 45.2, BBH 37.0 | – |
| GQA (8 groups) | smaller | MMLU 41.2, BBH 35.6 | – | |
| MQA | smallest | MMLU 37.9, BBH 33.2 | – | |
| DeepSeek-V2 paper, small MoE (15.7B total), 1.33T tokens | MHA | 110.6K numbers per token | MMLU 48.7, BBH 37.9 | – |
| MLA | 15.6K (7.1× smaller) | MMLU 50.0, BBH 39.0 | – | |
| DeepSeek-V2 paper, large MoE (about 250B total), 420B tokens | MHA | 860.2K numbers per token | MMLU 57.5, BBH 46.6 | – |
| MLA | 34.6K (24.9× smaller) | MMLU 59.0, BBH 50.7 | – |
The picture is consistent in direction: sharing keys and values costs some quality (a lot for MQA when training from scratch at 7B: 7.3 MMLU points), uptrained GQA-8 costs very little, and MLA matched or beat MHA in DeepSeek’s own ablations while caching 7 to 25 times less. Two cautions. The DeepSeek ablation uses benchmarks where the differences can depend on the training recipe, and nobody outside DeepSeek has published a same-budget MLA vs GQA comparison of that size that we know of. [Likely] And the parameter counts are not equal: in the 7B ablation the MQA model has 7.1B parameters against 6.9B for the others, because the saved key–value weights were spent elsewhere. [Certain, from the table]
Pseudo-code
- input: MHA model with \(W^K_i, W^V_i\) per head and layer; target \(n_{kv}\) (a divisor of \(n_h\)); fraction \(\alpha\) (0.05 in the paper)
- \(r \leftarrow n_h / n_{kv}\)
- for each layer, for \(g = 0\) to \(n_{kv} - 1\) do
- \(W^K_g \leftarrow \frac{1}{r}\sum_{i = gr}^{gr + r - 1} W^K_i\); \(\ W^V_g \leftarrow \frac{1}{r}\sum_{i = gr}^{gr + r - 1} W^V_i\) // least squares, formula (97.5)
- keep all \(W^Q_i\), \(W^O\) and the rest of the model; set
num_key_value_heads\(= n_{kv}\) - continue pre-training on the original data for \(\alpha \times\) the original steps, same optimizer and schedule
- evaluate against the MHA model; if the gap is too large, raise \(n_{kv}\) or \(\alpha\) // GQA recovers faster than MQA
- input: config, GPU memory \(M\), weights \(W\), number of GPUs \(P\), expected context \(T\)
- \(\beta \leftarrow\) formula (97.1) from the config // read kv_lora_rank first; num_key_value_heads can mislead for MLA
- if MLA then use data-parallel attention, so each GPU stores latents of its own requests only // formula (97.9)
- else pick TP degree \(P \le n_{kv}\) where possible, so no key–value head is copied
- \(B_{\max} \leftarrow \lfloor (uM - W_{\text{GPU}}) / (T \beta_{\text{GPU}}) \rfloor\) // formula (97.2) per GPU
- for each step: if prefill then MLA in naive (expanded) form else MLA in absorbed form // formula (97.8)
The same ideas in PyTorch. The script is deterministic (fixed seed) and runs on a CPU in a few seconds. Each part checks one claim of this session; the printed values are in the comments.
import torch torch.manual_seed(0) # 1) KV bytes per token from config values, formula (97.1) def kv_bytes(cfg, b=2): L = cfg["num_hidden_layers"] if "kv_lora_rank" in cfg: # MLA: latent + shared RoPE key return L * (cfg["kv_lora_rank"] + cfg["qk_rope_head_dim"]) * b dh = cfg.get("head_dim", cfg["hidden_size"] // cfg["num_attention_heads"]) return 2 * L * cfg["num_key_value_heads"] * dh * b llama31_8b = dict(num_hidden_layers=32, hidden_size=4096, num_attention_heads=32, num_key_value_heads=8, head_dim=128) deepseek_v3 = dict(num_hidden_layers=61, kv_lora_rank=512, qk_rope_head_dim=64) print(kv_bytes(llama31_8b), kv_bytes(deepseek_v3)) # 131072 70272 bytes per token free = 0.9 * 80e9 - 16.06e9 # H100 80 GB, 90% usable, 8B weights in BF16 for n_kv in (32, 8, 1): # MHA, GQA-8 (the real model), MQA beta = kv_bytes(dict(llama31_8b, num_key_value_heads=n_kv)) print(n_kv, "KV heads:", beta // 1024, "KiB/token,", int(free // (32768 * beta)), "sequences of 32k fit") # 512 KiB: 3 sequences, 128 KiB: 13, 16 KiB: 104 (formula 97.2) # 2) a toy MHA layer and its GQA conversion, formula (97.5) d, n_h, dh, T = 64, 8, 8, 16 def heads(rho): # heads = shared part + own part shared = torch.randn(dh, d) / d ** 0.5 own = torch.randn(n_h, dh, d) / d ** 0.5 return rho ** 0.5 * shared + (1 - rho) ** 0.5 * own Wq, Wk, Wv = torch.randn(n_h, dh, d) / d ** 0.5, heads(0.5), heads(0.5) Wo = torch.randn(n_h, d, dh) / (n_h * dh) ** 0.5 mask = torch.ones(T, T, dtype=torch.bool).tril() def attn(x, Wk_, Wv_, group): # group[i] = KV head read by query head i q = torch.einsum("hed,btd->bhte", Wq, x) k = torch.einsum("hed,btd->bhte", Wk_, x)[:, group] v = torch.einsum("hed,btd->bhte", Wv_, x)[:, group] a = (q @ k.transpose(-1, -2) / dh ** 0.5).masked_fill(~mask, float("-inf")).softmax(-1) return torch.einsum("hde,bhte->btd", Wo, a @ v) def rel_err(y, y_ref): return ((y - y_ref).norm() / y_ref.norm()).item() x_test = torch.randn(64, T, d) y_mha = attn(x_test, Wk, Wv, list(range(n_h))) n_kv = 2; r = n_h // n_kv; group = [i // r for i in range(n_h)] inits = {"mean": (Wk.view(n_kv, r, dh, d).mean(1), Wv.view(n_kv, r, dh, d).mean(1)), "first": (Wk[::r].clone(), Wv[::r].clone()), "random": (torch.randn(n_kv, dh, d) / d ** 0.5, torch.randn(n_kv, dh, d) / d ** 0.5)} # 3) "uptraining": train only the shared K, V weights to match the MHA layer on fresh data for name, (k0, v0) in inits.items(): k_g, v_g = k0.clone().requires_grad_(), v0.clone().requires_grad_() opt = torch.optim.Adam([k_g, v_g], lr=3e-3) before = rel_err(attn(x_test, k_g, v_g, group), y_mha) for step in range(300): x = torch.randn(4, T, d) # a new batch each step: no overfitting loss = ((attn(x, k_g, v_g, group) - attn(x, Wk, Wv, list(range(n_h)))) ** 2).sum() opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): print(f"{name:6s} error before {before:.3f} after 300 steps {rel_err(attn(x_test, k_g, v_g, group), y_mha):.3f}") # mean 0.656 -> 0.616, first 0.918 -> 0.616, random 1.341 -> 0.632 # 4) GQA is a special case of MLA with the same cache, formula (97.6) c_dim = 2 * n_kv * dh # latent = all GQA keys and values W_down = torch.cat([Wk.view(n_kv, r, dh, d).mean(1).reshape(-1, d), Wv.view(n_kv, r, dh, d).mean(1).reshape(-1, d)]) # (c_dim, d) E = torch.eye(c_dim) W_uk = torch.stack([E[group[i] * dh:(group[i] + 1) * dh] for i in range(n_h)]) # selections W_uv = torch.stack([E[(n_kv + group[i]) * dh:(n_kv + group[i] + 1) * dh] for i in range(n_h)]) gqa = attn(x_test, *inits["mean"], group) mla = attn(x_test, W_uk @ W_down, W_uv @ W_down, list(range(n_h))) # per-head K, V from c print("GQA == MLA with selection up-projections:", torch.allclose(gqa, mla, atol=1e-5)) # True # 5) MHA -> MLA at the same cache size: joint SVD of the stacked K and V weights, formula (97.7) W_kv = torch.cat([Wk.reshape(-1, d), Wv.reshape(-1, d)]) # (2*n_h*dh, d) U, S, Vh = torch.linalg.svd(W_kv, full_matrices=False) Vr = Vh[:c_dim] # top right singular vectors = W_down P = Vr.T @ Vr # project h onto the latent space y_svd = attn(x_test, Wk @ P, Wv @ P, list(range(n_h))) print(f"same cache ({c_dim} numbers/token): GQA mean-pool error {rel_err(gqa, y_mha):.3f}, " f"MLA via SVD error {rel_err(y_svd, y_mha):.3f}") # same cache (32 numbers/token): GQA mean-pool error 0.656, MLA via SVD error 0.393 # 6) MLA decode: absorbed vs rebuilding K and V, per cached token, one layer, DeepSeek-V3 sizes n_h3, d_c, d_r, d_nope, d_v = 128, 512, 64, 128, 128 absorbed = n_h3 * (2 * (d_c + d_r) + 2 * d_c) # scores on c and k_R, weighted sum of c rebuild = n_h3 * 2 * d_c * (d_nope + d_v) # up-project one cached latent to K and V print("absorbed:", absorbed, "FLOPs; rebuild K,V:", rebuild, "FLOPs; intensity", round(absorbed / ((d_c + d_r) * 2), 1), "FLOPs/byte") # absorbed: 278528 FLOPs; rebuild K,V: 33554432 FLOPs; intensity 241.8 FLOPs/byte (formula 97.8)
Where this is used
| Application | Why bytes per token matter | Link |
|---|---|---|
| High-throughput chat serving | At full memory, throughput is inversely proportional to \(\beta\) (97.4). GQA-8 gives about 4 times the tokens per second of MHA on the same GPU. | session 135 |
| Serving DeepSeek and Kimi models | MLA caches about 68.6 KiB per token for 61 layers; data-parallel attention keeps it from being copied on every GPU (97.9). | session 137 |
| Cheaper serving of an existing model | Convert MHA to GQA by mean pooling plus a short uptraining (Algorithm 1), or MHA/GQA to MLA (MHA2MLA, TransMLA). | session 26 |
| Long-video understanding and Video-RAG | A 30-minute video at 1 frame per second and 256 visual tokens per frame is 460,800 tokens: 56.25 GiB of cache at 128 KiB per token (Llama-3.1-8B-like GQA), 225 GiB with MHA, 30.2 GiB at an MLA cost of 68.6 KiB per token. | session 206 |
| Video question answering and key-moment detection | Many questions reuse one prefilled video prefix; a smaller \(\beta\) keeps more videos’ caches resident for prefix caching. | session 134 |
| Agents and long tool traces | Context grows with every tool call; \(N_{\text{tok}}\) in (97.2) sets how many agent sessions one GPU can keep alive. | session 141 |
| Edge and on-device models | The weights are small, so the cache is a large share of memory even at short context; GQA with few KV heads is common in small models. [Likely] | session 140 |
Common mistakes
- Reading
num_key_value_headsfor an MLA model. DeepSeek-V3’s config says 128, which would give 3.8 MiB per token. The real cache is set bykv_lora_rankandqk_rope_head_dim: 68.6 KiB. - Expecting GQA to speed up batch-1 short-context decoding. With 16 GB of weights and a short context, the cache is a small part of the bytes read (Figure 2 at batch 1). The gain shows up at large batch or long context.
- Assuming MQA always beats GQA-8 in memory. Under tensor parallelism with \(P = 8\), both store one key–value head per GPU (97.9). MQA just stores 8 copies and loses quality.
- Running MLA under plain tensor parallelism and counting the cache once. Every GPU holds the full latent; the cache per GPU does not shrink with \(P\). Use data-parallel attention or count it \(P\) times.
- Using one MLA form for both phases. The absorbed form in prefill does about 3 times more FLOPs; the naive form in decode either rebuilds K and V for every cached token (about 120 times more FLOPs) or needs an MHA-sized cache.
- Mean-pooling heads that are not neighbours in the checkpoint’s layout. Groups must follow the layout the inference code will use (query head \(i\) reads KV head \(\lfloor i / r \rfloor\)); pooling heads \(0, 4, 8, \dots\) and then serving with
repeat_interleavegives silently wrong outputs. - Skipping uptraining. Mean pooling is least squares in weight space, not in output space; MQA in particular is not useful before uptraining, according to the GQA paper.
- Comparing quality numbers across papers. The MQA, GQA and DeepSeek studies used different models, data and metrics. Compare designs only inside one study.
Interview questions
What is “KV bytes per token” and how do you compute it from a config file?basic
num_hidden_layers, num_key_value_heads, head_dim (or hidden size / heads) and \(b = 2\) for bfloat16. For MLA: \(L (d_c + d_R) b\) with kv_lora_rank and qk_rope_head_dim (97.1). Llama 3.1 8B: \(2 \cdot 32 \cdot 8 \cdot 128 \cdot 2 = 131{,}072\) bytes = 128 KiB. DeepSeek-V3: \(61 \cdot 576 \cdot 2 = 70{,}272\) bytes.How many 32k-token sequences fit on one 80 GB H100 with Llama 3.1 8B?basic
GQA barely changes FLOPs. When does it make serving faster, and when not?core
How do you convert an MHA checkpoint to GQA? Why the mean?core
In the MQA paper, why compare MQA with MHA models that have 1 head of 128 or 8 heads of 16?core
Show that GQA is a special case of MLA. What does that imply?advanced
How would you convert an MHA model into MLA without training from scratch?advanced
Why do serving engines run MLA differently in prefill and decode?advanced
Why is GQA with 8 key–value heads such a common choice?core
You deploy an MLA model with tensor parallelism on 8 GPUs and the batch is smaller than expected. Why?advanced
Is MLA always better than GQA? Give the trade-offs.advanced
Exercises
- Qwen2.5-72B has 80 layers, 64 query heads, 8 key–value heads and \(d_h = 128\). Compute \(\beta\) in bfloat16 and the cache of one 32,768-token sequence.
- For the same model, compute \(\beta_{\text{GPU}}\) and the replication factor with \(P = 4\) and \(P = 16\).
- Use (97.4) to compute the full-memory throughput of the Llama-3.1-8B shape at \(T = 4{,}096\) with GQA-8 and with MHA (\(u M = 72\) GB, \(W = 16.06\) GB, 3.35 TB/s). Why is the ratio exactly 4?
- Prove (97.5): show that the mean of \(r\) matrices minimizes the sum of squared Frobenius distances to them.
- Kimi K2 uses MLA with 64 heads and the same \(d_c = 512\), \(d_R = 64\). What is the arithmetic intensity of absorbed decode attention (97.8)? What is it per GPU under TP with \(P = 8\)?
- The GQA paper’s uptraining with \(\alpha = 0.05\) took about 600 TPUv3 chip-days for T5-XXL. If cost is linear in steps, what did the original pre-training cost, and what would \(\alpha = 0.1\) cost?
- Why can an MLA latent with \(d_c \ge d\) always represent the MHA layer exactly? Why do real models still use \(d_c \ll d\)?
Answerscheck
Key takeaways
- KV bytes per token \(\beta\) is the serving currency: \(2 L n_{kv} d_h b\) for MHA, GQA and MQA, \(L (d_c + d_R) b\) for MLA (97.1). It does not depend on width or on the number of query heads.
- \(\beta\) sets the largest batch \((uM - W)/(T\beta)\) (97.2), and at full memory throughput is inversely proportional to \(\beta\) (97.4). At batch 1 and short context the design barely matters.
- For a fixed cache, sharing K and V across many query heads beats having fewer heads (MQA paper); GQA-8 keeps almost all MHA quality (GQA paper).
- Uptraining converts MHA to GQA: mean-pool each group (the least-squares choice, 97.5), then continue pre-training for about 5% of the original steps.
- GQA is MLA with 0/1 up-projections (97.6); an SVD converts MHA to MLA at equal cache and is never worse than mean pooling in weight space (97.7).
- MLA runs in naive form for prefill and absorbed form for decode (97.8); absorbed decode reaches about 242 FLOPs per byte in DeepSeek-V3.
- Under tensor parallelism MQA and MLA replicate their cache (97.9): GQA-8 fits 8-GPU nodes, and MLA is served with data-parallel attention.
Key formulas
References for this session
- Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need
- Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need
- Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- DeepSeek-AI (2024). DeepSeek-V3 Technical Report
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Meng, Tang, Tang, Yao, Sun, Zhang (2025). TransMLA: Multi-Head Latent Attention Is All You Need
- Ji et al. (2025). Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
- Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs
- SGLang team (2025). SGLang documentation: DeepSeek model optimizations (data-parallel attention)
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Moonshot AI (2025). Kimi-K2-Instruct config.json (Hugging Face)
- Qwen team (2025). Qwen3-235B-A22B config.json (Hugging Face)
- Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures