Where the cost is: FFN and attention, by the numbers
Before you choose an optimization, you must know which part of the model costs the most. This session splits every transformer layer into its two halves, the feed-forward network (FFN) and attention, and counts each one three ways: parameters, FLOPs per token and bytes read per token. The answer depends on the model size and, even more, on the context length, so short chat, long documents and video need different fixes.
What you will learn
- How to split the parameters of one layer into attention (4d² for MHA, less for GQA) and FFN (3·d·d_ff for SwiGLU)
- Why the survey reports 63.01% (LLaMA-7B) and 71.69% (LLaMA-70B) FFN parameters, and why a count from the 70B config file gives 81.7%
- FLOPs per token: the weight part grows with model size, the attention-score part grows with context length; the crossover is near t = 1.5·d_ff
- Bytes per token in decode: weights are shared by the batch, the KV cache is not; arithmetic intensity B for the FFN and n_h/n_kv for attention
- How to compute the split for your own model from config.json, and check it with a PyTorch FLOP counter
- Which part to optimize first for short chat, long documents and video
Why this matters
Every optimization in the rest of this track works on one part of the model. Some make the feed-forward network (FFN) cheaper: mixture of experts, pruning, weight quantization. Others make attention cheaper: GQA, sparse attention, FlashAttention, KV-cache quantization. Before you pick one, you must know where your cost is. If you shrink the part that is only 5% of the work, you gain at most 5%.
The efficient-inference survey starts its model-level chapter with this question. It reports that “the FFN module accounts for 63.01% of the parameters in the LLaMA-7B model and 71.69% in the LLaMA-70B model”, and that attention “demonstrates quadratic complexity in the input length”. [Certain, survey Section 5.1] These two facts give the plan of this session:
- FFN cost grows with model size. The FFN holds most of the weights. Every token must use all of them, whatever the context length.
- Attention cost grows with context length. Attention has fewer weights, but each new token compares itself with every earlier token. At long contexts this part takes over.
We will check the survey’s numbers ourselves (one of them does not match the model’s config file, and we will see why), and then count three kinds of cost, because they give different answers:
- Parameters: how much memory the weights need.
- FLOPs per token: how much arithmetic. This sets the speed of prefill, the phase that reads the prompt (session 78).
- Bytes per token: how much data the GPU must read from its memory. This sets the speed of decode, the phase that writes one token at a time (session 82).
Session 25 gave the basic formulas for the whole model. Here we split them into the two halves and ask which half wins in which situation. The data-level methods of the previous sessions, ending with session 92, made the input shorter. From now on we change the model, and this session is the map for that work. The next session, session 94, starts with the FFN.
Interviewers like this topic because it tests whether you can reason with numbers: “What fraction of a 70B model is the FFN?”, “At what context length does attention cost more than the MLP?”, “Your users send 100k-token documents; what do you optimize first?”, “Why does GQA not reduce attention FLOPs?”. By the end you can answer each with a formula and a number for a real model.
The two halves of a layer
What each half contains
A decoder layer of a modern model like Llama has two sub-blocks, each with a normalization before it and a residual connection around it (session 23):
- Attention: four weight matrices. \(W^Q\) makes queries, \(W^K\) keys, \(W^V\) values, and \(W^O\) maps the concatenated head outputs back to width \(d\). Between them is the part with no weights: the scores \(QK^\top\), the softmax, and the weighted sum of values.
- FFN (also called MLP): with SwiGLU, three matrices. \(W_{\text{gate}}\) and \(W_{\text{up}}\) expand the vector from \(d\) to \(d_{ff}\) numbers, and \(W_{\text{down}}\) brings it back to \(d\) (session 21).
Count the numbers in these matrices. With \(n_h\) query heads, \(n_{kv}\) key–value heads and head size \(d_h\):
- Read the symbols
- \(d\)
- the model width (
hidden_sizein the config file) - \(n_h, n_{kv}, d_h\)
- query heads, key–value heads and head size. For multi-head attention (MHA) \(n_{kv} = n_h\); with grouped-query attention (GQA) \(n_{kv}\) is smaller.
- \(d_{ff}\)
- the inner width of the FFN (
intermediate_size) - 3 and 2
- SwiGLU has three matrices (gate, up, down); the original transformer MLP has two (up, down). Biases are left out: Llama models have none.
In most models \(n_h d_h = d\). Then MHA gives \(P_{\text{attn}} = 4d^2\). GQA keeps the full \(W^Q\) and \(W^O\) but makes \(W^K\) and \(W^V\) smaller by the group factor \(g = n_h / n_{kv}\), so \(P_{\text{attn}} = (2 + 2/g)\, d^2\).
The whole model adds the embedding table, the output head (the “LM head”, which turns the last hidden vector into one score per vocabulary token) and the small normalization weights:
- Read the symbols
- \(L\)
- number of layers (
num_hidden_layers) - \(V\)
- vocabulary size; \(Vd\) numbers for the embedding table and \(Vd\) more for the head, unless
tie_word_embeddingsis true - \(2d\) and \(d\)
- the two RMSNorm weight vectors per layer and the final norm: tiny, but needed for an exact match
The FFN share of one layer
Divide (93.2) by the sum of (93.1) and (93.2). With \(n_h d_h = d\), write \(r = d_{ff}/d\) for the FFN expansion ratio:
- Read the symbols
- \(\rho_{\text{layer}}\)
- the fraction of one layer’s weights that sit in the FFN
- \(r\)
- the expansion ratio \(d_{ff}/d\): 2.6875 in Llama 2 7B (\(11{,}008/4{,}096\)), 3.5 in Llama 2 70B and Llama 3 (\(28{,}672/8{,}192\), \(14{,}336/4{,}096\))
- \(g\)
- the GQA group size \(n_h/n_{kv}\): 1 for MHA, 4 for Llama 3 8B, 8 for the 70B models
Three cases show the trend. Classic SwiGLU with \(r = 8/3\) and MHA gives \(\rho = 8/12 = 66.7\%\). Llama 2 7B (\(r = 2.6875\), MHA) gives 66.84%. Llama 2 70B (\(r = 3.5\), \(g = 8\)) gives \(10.5 / 12.75 = 82.35\%\). Larger models moved to GQA, which shrinks attention, and to wider FFNs, which grow the FFN. Both push the FFN share up. Mixture of experts (session 29) pushes it much further: with \(E\) experts, \(P_{\text{ffn}}\) is multiplied by \(E\).
Figure 1 draws the seven matrices of one layer at true scale, for six real models.
config.json on Hugging Face; the totals match the published sizes (6.74B, 68.98B, 8.03B, 70.55B, 7.62B, 46.70B).How to read Figure 1
Start with Llama 2 7B. The four blue boxes (attention) are all the same width, because MHA has \(n_{kv} = n_h\): each is \(4096 \times 4096\). The three teal boxes (FFN) are each \(4096 \times 11{,}008\), so the FFN is about two thirds of the layer. Switch to Llama 2 70B: \(W^K\) and \(W^V\) become thin slivers (8 key–value heads for 64 query heads), and the FFN grows to 82% of the layer. Switch to Llama 3 8B: the FFN share of the layer is 81%, but the bottom bar shows that the large vocabulary (128,256 tokens) puts 13% of the whole model in the embedding and head. Mixtral shows eight stacked copies of the FFN: 96.6% of its weights are experts. Now go back to Llama 2 7B and Llama 2 70B and tick the survey box: you get the survey’s 63.01% and 71.69%.
Checking the survey’s numbers
Numbers in a survey are a good start, but an engineer should be able to reproduce them. Let us count Llama 2 7B (LLaMA-7B has the same shapes) and Llama 2 70B from their config files.
Llama 2 7B: \(d = 4096\), \(L = 32\), \(d_{ff} = 11{,}008\), 32 heads (MHA), \(V = 32{,}000\), separate head. [Certain, from config.json]
Attention per layer: \(4 \times 4096^2 = 67{,}108{,}864\). FFN per layer: \(3 \times 4096 \times 11{,}008 = 135{,}266{,}304\). Times 32 layers: \(2{,}147{,}483{,}648\) and \(4{,}328{,}521{,}728\). Embedding plus head: \(2 \times 32{,}000 \times 4096 = 262{,}144{,}000\). Norms: \(266{,}240\). Total: \(6{,}738{,}415{,}616\), the familiar “6.74B”.
FFN share: \(4{,}328{,}521{,}728 / 6{,}738{,}415{,}616 = \mathbf{64.24\%}\). Attention: 31.87%. Embedding and head: 3.89%.
Llama 2 70B: \(d = 8192\), \(L = 80\), \(d_{ff} = 28{,}672\), 64 query heads but only 8 key–value heads. [Certain, from config.json] Attention per layer: \(2 \times 8192^2 + 2 \times 8192 \times 1024 = 150{,}994{,}944\). FFN per layer: \(704{,}643{,}072\). Total: 68.98B. FFN share: \(\mathbf{81.73\%}\).
So our count gives 64.24% and 81.73%, while the survey gives 63.01% and 71.69%. The 7B numbers are close; the 70B numbers are 10 points apart. Where does the gap come from? We can reproduce both survey numbers to two decimals with one recipe: count attention as \(4d^2\) per layer (as if the 70B model used MHA, with 64 key–value heads), and count \(Vd\) three times instead of twice. For the 7B model the denominator becomes \(6{,}869{,}221{,}376\) and the share \(63.01\%\); for the 70B model it becomes \(71.69\%\). [Likely: this is our reconstruction; the survey does not show its count, but both numbers match exactly.]
The lesson is not that the survey is careless. The lesson is that attention design changes the split a lot. GQA cut the attention weights of the 70B model from \(4d^2\) to \(2.25d^2\) per layer. If you ignore it, you overestimate attention by 78% and underestimate the FFN share by 10 points. Always count from the config file of the exact model you serve.
| Model | Attention | \(d_{ff}/d\) | Total | FFN | Attention | Embedding + head |
|---|---|---|---|---|---|---|
| Llama 2 7B | MHA, 32 heads | 2.69 | 6.74B | 64.2% | 31.9% | 3.9% |
| Llama 2 70B | GQA, 64 / 8 | 3.5 | 68.98B | 81.7% | 17.5% | 0.8% |
| Llama 3 8B | GQA, 32 / 8 | 3.5 | 8.03B | 70.2% | 16.7% | 13.1% |
| Llama 3 70B | GQA, 64 / 8 | 3.5 | 70.55B | 79.9% | 17.1% | 3.0% |
| Qwen2.5 7B | GQA, 28 / 4 | 5.29 | 7.62B | 74.9% | 10.8% | 14.3% |
| Qwen2.5 72B | GQA, 64 / 8 | 3.61 | 72.71B | 80.0% | 16.6% | 3.4% |
| Mixtral 8x7B | GQA, 32 / 8, 8 experts | 3.5 | 46.70B | 96.6% | 2.9% | 0.6% |
[Certain for the shapes, from the config files; the percentages are computed with (93.1) to (93.3).] For Mixtral, each token uses only 2 of the 8 experts, so the active parameters per token are 12.88B, and the FFN is 87.5% of those. The table shows a clear pattern: in every model of 7B or more, the FFN holds between 64% and 97% of the weights. For small models with a large vocabulary (Llama 3 8B, Qwen2.5 7B), the embedding and head are a real third part, more than 13%.
FLOPs per token: the context term
Two kinds of attention work
Parameters tell us about memory. Speed depends on work. A matrix multiply of a vector with a \(m \times n\) matrix costs \(2mn\) FLOPs (one multiply and one add per weight), so every weight costs 2 FLOPs per token (session 25). This is true for the FFN and for the four attention matrices.
But attention also has work without weights. The token at position \(t\) computes one score with each of the \(t\) keys (\(QK^\top\)), and then a weighted sum of \(t\) values. For one head, each of these costs \(2 d_h t\) FLOPs. Over all heads and layers, the forward FLOPs of one token at position \(t\) are
- Read the symbols
- \(F(t)\)
- forward FLOPs for one token that sits at position \(t\) (it attends to \(t\) tokens, itself included)
- \(2L P_{\text{ffn}}\), \(2L P_{\text{attn}}\)
- 2 FLOPs per weight. These do not depend on \(t\).
- \(4 L n_h d_h t\)
- \(QK^\top\) (\(2 n_h d_h t\)) plus the weighted sum of values (\(2 n_h d_h t\)), in every layer. This is the only term that grows with context.
- \(n_h\), not \(n_{kv}\)
- every query head computes its own scores, even when several heads share one key head. GQA does not reduce this term.
- \(2Vd\)
- the LM head; the embedding lookup costs no FLOPs
The softmax and the normalizations add a few FLOPs per score or per number, which we ignore. [Certain that they are small next to the matrix multiplies.]
The crossover length
The first terms of (93.5) are fixed; the score term grows linearly with \(t\). Set the score term equal to the FFN term and solve for \(t\):
- Read the symbols
- \(t^{*}\)
- the position at which one token’s attention scores cost as many FLOPs as its FFN
- \(1.5\, d_{ff}\)
- the crossover depends only on the FFN width: wider models need longer contexts before attention dominates
This is the formula behind the two key sentences of this session. For Llama 2 7B, \(t^* = 1.5 \times 11{,}008 = 16{,}512\). For Llama 3 8B, \(21{,}504\). For the 70B models, \(43{,}008\). Below \(t^*\), the FFN is the biggest single cost; above it, the scores are. If you compare the scores with all weight FLOPs of the layer (FFN plus projections), the crossover moves to \(t = 2d + 1.5 d_{ff}\): 24,704 for Llama 2 7B and 26,624 for Llama 3 8B.
Weight FLOPs per token: FFN \(2 \times 32 \times 176{,}160{,}768 = 11.27\) GFLOPs; attention projections \(2 \times 32 \times 41{,}943{,}040 = 2.68\) GFLOPs; head \(2 \times 128{,}256 \times 4096 = 1.05\) GFLOPs. Together 15.0 GFLOPs.
Score term: \(4 \times 32 \times 4096 \times t = 524{,}288\, t\) FLOPs. At \(t = 1024\): 0.54 GFLOPs, 3.5% of the total. At \(t = 4096\): 2.15 GFLOPs, 12.5%. At \(t = 32{,}768\): 17.18 GFLOPs, 53.4%, more than the FFN (35.0%).
The same rule also explains “FFN cost grows with model size”. Divide the score term by the weight term: roughly \(4 L d t / (2 L \cdot 12 d^2) = t / (6d)\) for a classic layer. At a fixed context, a wider model spends a smaller fraction of its work on attention scores. Big models are FFN-heavy; long contexts are attention-heavy.
How to read Figure 2
Each vertical slice adds up to 100%. On the left (short context) the teal FFN band and the blue projection band fill almost everything. Moving right, the magenta score band grows, and it passes the FFN at the dashed line. Move the slider to read exact numbers for one position. Compare models: Llama 3 70B and Mixtral have their crossover at 43,008, twice as far right as the 8B models, because their FFNs are wider. Note that some positions are beyond the trained context of a model (for example 4,096 for Llama 2 7B); the figure still shows the arithmetic.
Prefill: the whole prompt at once
Prefill processes \(P\) prompt tokens. Token \(t\) attends to \(t\) tokens (the causal mask hides the future), so we sum (93.5) over \(t = 1, \dots, P\):
- Read the symbols
- first term
- linear in \(P\): every prompt token goes through all weight matrices
- second term
- quadratic in \(P\): this is the “quadratic complexity in the input length” of the survey. It is \(\sum_t 4 L n_h d_h t\).
- \(2Vd\) once
- only the last prompt position needs logits; good servers skip the head for the other positions [Likely for vLLM-style servers; a naive loop computes all of them]
Doubling a long prompt doubles the first term but multiplies the second by four. For Llama 3 8B, a 100,000-token prompt costs \(4.0 \times 10^{15}\) FLOPs, and 65% of it is the score term. On an H100 (989 dense BF16 TFLOP/s [Certain, NVIDIA spec]) that is at least 4.1 seconds of prefill even at 100% utilization.
Bytes per token: what decode reads
Shared weights, private caches
In decode, each step makes one new token per sequence. The arithmetic per token is the same as (93.5), but the GPU is not limited by arithmetic. It is limited by how fast it can read data from its memory (HBM) (session 82). So in decode we must count bytes, and here the two halves behave very differently:
- Weights (FFN and attention projections): read once per step, and that one read serves all \(B\) sequences in the batch.
- KV cache (the input of the score term): every sequence has its own cache (session 80), so nothing is shared. A sequence of length \(T\) reads \(s \cdot T\) bytes per step.
- Read the symbols
- \(\beta\)
- bytes read from memory per generated token
- \(M_W\)
- bytes of all weights: \(N \times b\) (2 bytes each in bfloat16)
- \(B\)
- number of sequences decoded together
- \(s\)
- KV-cache bytes per token over all layers; \(n_{kv}\), not \(n_h\), appears here
- \(T\)
- current length of the sequence (prompt plus tokens written so far)
- \(M_W / s\)
- the “budget” of batch × context at which the cache costs as many bytes as the weights
For Llama 3 8B in bfloat16, \(M_W = 16.06\) GB and \(s = 131{,}072\) bytes, so \(M_W / s = 122{,}532\) tokens. One user at 4,096 tokens reads 16.06 GB of weights and 0.54 GB of cache per step: the weights are 96.8% of the traffic, and an H100 at 3.35 TB/s [Certain, NVIDIA spec] needs at least 4.95 ms per step. With 32 users at 4,096 tokens, the caches add up to 17.18 GB: now the cache is 51.7% of the traffic and the step takes at least 9.92 ms. Batching made the weights cheap per token, but it cannot make the cache cheap. (Strictly, a decode step reads only \(B\) rows of the embedding table, so the weight traffic is a little smaller, 15.0 GB for Llama 3 8B; like session 80 we keep \(M_W\) = all weights as a simple upper bound.)
Here GQA matters a lot. It does not change the score FLOPs (\(n_h\) in 93.5), but it divides \(s\) by \(g\) (\(n_{kv}\) in 93.8). Llama 2 7B (MHA) has \(s = 524{,}288\) bytes and \(M_W / s = 25{,}705\): its cache equals its weights already at one user with 25,705 tokens, or 8 users with 3,213 tokens.
Arithmetic intensity of the two halves
Arithmetic intensity is FLOPs divided by bytes read. A GPU is busy only when the intensity of the work is above its ridge point, its peak FLOP/s divided by its bandwidth: \(989 / 3.35 \approx 295\) FLOP/byte for an H100 (session 82, session 123). In decode:
- Read the symbols
- \(I_{\text{weights}}\)
- for the FFN and projections: each weight byte is used once per sequence in the batch. In bfloat16 (\(b = 2\)) this is just \(B\).
- \(I_{\text{attn}}\)
- for the score term: each cached key and value is used by the \(g\) query heads that share it. In bfloat16 this is \(g\): 1 for MHA, 4 for Llama 3 8B, 8 for the 70B models.
- no \(B\) in \(I_{\text{attn}}\)
- batching does not help attention in decode, because no two sequences share a cache
So in decode the FFN becomes efficient when the batch is large (near \(B \approx 295\) on an H100), but attention stays at an intensity of 1 to 8, far below the ridge, whatever you do with batching. [Certain, from the formula] This is why decode attention has its own kernels (session 125) and why the fixes for attention in decode are about bytes: fewer key–value heads, a smaller cache format, or fewer cached tokens.
How to read Figure 3
Light cells (left and top) are weight-bound: shrinking the weights (quantization, pruning, fewer experts) is what helps. Dark cells (right and bottom) are cache-bound: GQA, a quantized cache or a shorter context help, and smaller weights barely change the step time. The border between them is the line \(B \cdot T = M_W / s\). Compare Llama 2 7B (MHA) with Llama 3 8B (GQA): the same cell is much darker for Llama 2. Qwen2.5 7B, with only 4 key–value heads, is the lightest of the small models. Llama 3 70B has big weights, so it stays weight-bound longer.
Your workload decides: chat, long documents, video
Now combine prefill (FLOPs, formula 93.7) and decode (bytes, formula 93.8) for one request. Four typical workloads on Llama 3 8B give very different pictures. [Certain for the arithmetic; the workloads are illustrative examples.]
| Workload | Prompt \(P\) | Output \(N\) | Prefill: score share of FLOPs | Decode, \(B = 16\): cache share of bytes | Main cost |
|---|---|---|---|---|---|
| Short chat | 500 | 300 | 0.9% | 7.8% | FFN and projection weights |
| RAG with 8k context | 8,000 | 500 | 13.1% | 51.9% | Weights in prefill, half cache in decode |
| Long-document QA | 100,000 | 500 | 65.3% | 92.9% | Attention in both phases |
| Video QA, 10 min at 1 frame/s | 86,500 | 200 | 61.9% | 91.9% | Attention in both phases |
The video row assumes 600 frames with 144 visual tokens each (86,400 tokens) plus 100 text tokens. Real video LLMs use between tens and hundreds of tokens per frame, so this is an example, not a specific model. [Likely range] For decode we use the average length \(P + N/2\).
The advice follows directly:
- Short chat: almost all cost is the weights, mostly the FFN. Optimize the FFN and the weights: weight quantization (session 106), mixture of experts (session 95), structured pruning (session 115), and bigger batches. Making the attention scores or the cache faster gains less than 10%.
- Long documents: attention dominates prefill FLOPs and decode bytes. Optimize attention and the cache: FlashAttention for prefill, GQA or MLA (session 97), KV-cache quantization (session 110), sparse attention (session 116), or a shorter input with prompt compression (session 88).
- Video: like long documents, but the long part is visual tokens. The cheapest fix is often to send fewer visual tokens (pooling per frame, picking key frames, retrieving a few clips) before touching the model.
How to read Figure 4
Start with short chat on Llama 3 8B: both bars are almost all teal and blue (weights). Raise the batch: the magenta cache part of the bottom bar grows, because the weights are shared and the cache is not, but it stays small. Switch to Long document: the magenta part takes 65% of prefill and over 90% of decode bytes at batch 16. Try Llama 2 7B on the long document: its MHA cache is 4× larger per token, so decode is almost all cache even at batch 1. (Its trained context is only 4,096 tokens; the numbers show what the arithmetic would be.)
Pseudo-code
- input:
config.json; context length \(t\); batch \(B\); bytes per number \(b\) - read \(d, L, d_{ff}, n_h, n_{kv}, V\); \(d_h \leftarrow\)
head_dimif present, else \(d / n_h\); \(E, k \leftarrow\) experts and active experts (1, 1 if dense) - \(P_{\text{attn}} \leftarrow 2 d n_h d_h + 2 d n_{kv} d_h\) // (93.1); add biases if the model has them
- \(P_{\text{ffn}} \leftarrow 3 d d_{ff} E\) (+ router \(dE\)); \(\;P_{\text{ffn}}^{\text{act}} \leftarrow 3 d d_{ff} k\) // (93.2)
- \(N \leftarrow L(P_{\text{attn}} + P_{\text{ffn}} + 2d) + cVd + d\) // (93.3), \(c = 1\) if tied
- params: report \(L P_{\text{ffn}} / N\), \(L P_{\text{attn}} / N\), \(cVd / N\)
- FLOPs: \(F_{\text{ffn}} \leftarrow 2 L P_{\text{ffn}}^{\text{act}}\); \(F_{\text{proj}} \leftarrow 2 L P_{\text{attn}}\); \(F_{\text{sc}} \leftarrow 4 L n_h d_h t\); \(F_{\text{head}} \leftarrow 2Vd\) // (93.5)
- \(t^* \leftarrow 6 d\, d_{ff} k / (4 n_h d_h)\) // (93.6)
- bytes: \(M_W \leftarrow N b\); \(s \leftarrow 2 L n_{kv} d_h b\); \(\beta \leftarrow M_W / B + s t\) // (93.8)
- return the three splits, \(t^*\) and the batch–context budget \(M_W / s\)
- input: typical prompt length \(P\), output length \(N\), batch \(B\), and the splits of Algorithm 1
- \(f_{\text{pre}} \leftarrow\) score share of \(F_{\text{prefill}}(P)\) // (93.7)
- \(f_{\text{dec}} \leftarrow s(P + N/2) / (M_W / B + s(P + N/2))\) // cache share of decode bytes
- \(w \leftarrow\) time share of prefill (measure it, or estimate from FLOPs and bytes)
- if \(w f_{\text{pre}} + (1 - w) f_{\text{dec}} \lt 0.2\): // weights dominate
- optimize FFN and weights: quantize weights, MoE, prune FFN, larger batch
- else if \(f_{\text{dec}}\) is large: optimize the cache: GQA/MLA, KV quantization, eviction, paging
- if \(f_{\text{pre}}\) is large: FlashAttention, sparse or linear attention, prompt compression, fewer visual tokens
- always profile afterwards: the shares change after each fix
The threshold 0.2 is a rule of thumb, not a law: it says that if attention is less than a fifth of the cost, attention work can win at most 20%.
In code: count and check
The code builds one tiny Llama-style layer with GQA (8 query heads, 2 key–value heads) and SwiGLU in PyTorch. It counts the parameters of each half from the real tensors and compares them with (93.1) and (93.2). Then it runs one decode step of attention against a cache of \(t - 1\) tokens and one step of the FFN under PyTorch’s FlopCounterMode, and compares the measured FLOPs with (93.5). It is deterministic (fixed seed; the FLOP counts do not depend on the values anyway).
import torch, torch.nn as nn from torch.utils.flop_counter import FlopCounterMode torch.manual_seed(0) class Block(nn.Module): # one Llama-style layer: GQA + SwiGLU def __init__(s, d, n_h, n_kv, d_h, d_ff): super().__init__() s.n_h, s.n_kv, s.d_h = n_h, n_kv, d_h s.wq = nn.Linear(d, n_h * d_h, bias=False); s.wk = nn.Linear(d, n_kv * d_h, bias=False) s.wv = nn.Linear(d, n_kv * d_h, bias=False); s.wo = nn.Linear(n_h * d_h, d, bias=False) s.gate = nn.Linear(d, d_ff, bias=False); s.up = nn.Linear(d, d_ff, bias=False) s.down = nn.Linear(d_ff, d, bias=False) def attn(s, x, K, V): # x: (1, d) new token; K, V: (n_kv, t-1, d_h) cache q = s.wq(x).view(s.n_h, 1, s.d_h) K = torch.cat([K, s.wk(x).view(s.n_kv, 1, s.d_h)], 1) # append the new key and value: V = torch.cat([V, s.wv(x).view(s.n_kv, 1, s.d_h)], 1) # now t rows K = K.repeat_interleave(s.n_h // s.n_kv, 0); V = V.repeat_interleave(s.n_h // s.n_kv, 0) a = torch.softmax(q @ K.transpose(1, 2) / s.d_h**0.5, -1) # (n_h, 1, t) scores return s.wo((a @ V).reshape(1, -1)) def ffn(s, x): return s.down(nn.functional.silu(s.gate(x)) * s.up(x)) def count(mod, names): return sum(getattr(mod, n).weight.numel() for n in names) def flops(fn): with FlopCounterMode(display=False) as fc: fn() return fc.get_total_flops() d, n_h, n_kv, d_h, d_ff = 512, 8, 2, 64, 1408 # tiny model, GQA group size 4 blk = Block(d, n_h, n_kv, d_h, d_ff) P_attn = count(blk, ["wq", "wk", "wv", "wo"]); P_ffn = count(blk, ["gate", "up", "down"]) print("params attention", P_attn, "formula", 2*d*n_h*d_h + 2*d*n_kv*d_h) print("params FFN ", P_ffn, "formula", 3*d*d_ff, f" FFN share {P_ffn/(P_attn+P_ffn):.1%}") x = torch.randn(1, d) for t in [256, 1024, 4096, 16384]: K = torch.randn(n_kv, t - 1, d_h); V = torch.randn(n_kv, t - 1, d_h) fa, ff = flops(lambda: blk.attn(x, K, V)), flops(lambda: blk.ffn(x)) proj, score = 2 * P_attn, 4 * n_h * d_h * t # formula: projections + QK and AV print(f"t={t:6d} attention {fa:>10,} (formula {proj+score:>10,}) FFN {ff:>9,} attention share {fa/(fa+ff):.1%}") print("crossover t* where scores = FFN:", 6 * d * d_ff // (4 * n_h * d_h))
Output (PyTorch 2.x on CPU):
params attention 655360 formula 655360 params FFN 2162688 formula 2162688 FFN share 76.7% t= 256 attention 1,835,008 (formula 1,835,008) FFN 4,325,376 attention share 29.8% t= 1024 attention 3,407,872 (formula 3,407,872) FFN 4,325,376 attention share 44.1% t= 4096 attention 9,699,328 (formula 9,699,328) FFN 4,325,376 attention share 69.2% t= 16384 attention 34,865,152 (formula 34,865,152) FFN 4,325,376 attention share 89.0% crossover t* where scores = FFN: 2112
The measured FLOPs match (93.5) exactly. The FFN is 76.7% of the weights but its FLOPs stay fixed at 4.33 million, while attention passes it between 1,024 and 4,096 tokens. The crossover is early here (\(t^* = 1.5 \times 1408 = 2112\)) because the model is tiny; for real models it is \(1.5 d_{ff}\), tens of thousands of tokens. Note the counter does not see the softmax; that is the “small terms” we ignored. To apply the same check to a real model, replace the shapes with the values from its config file, or run FlopCounterMode around one forward pass of the Hugging Face model.
Where this is used
| Setting | Typical lengths | Where the cost is | What teams optimize |
|---|---|---|---|
| Chat assistants, customer support | Hundreds to a few thousand tokens | FFN and projection weights; decode is weight-bound | Weight quantization, MoE, large continuous batches |
| Code completion | A few thousand tokens of context, short output, low latency | Weights per step, at small batch | Small or quantized models, speculative decoding |
| RAG and long-document QA | 10k to 128k tokens of prompt | Attention scores in prefill, KV cache in decode | FlashAttention, prefix caching, prompt compression, GQA/MLA |
| Agents with tool calls | Long and growing context, many short turns | Prefill of the growing context; cache memory | Prefix caching, cache eviction, summarizing old turns |
| VLM image QA | Hundreds of visual tokens per image (576 for LLaVA-1.5) | Mostly weights for one image; attention for many images | Visual token pruning or pooling |
| Video QA, Video-RAG, key-moment detection | Thousands to over 100k visual tokens | Attention in prefill and cache in decode | Fewer tokens per frame, key-frame selection, clip retrieval before the LLM |
| Video classification and action recognition with an LLM head | Many frames, very short output | Prefill only; scores grow with frames squared | Token merging, sparse or windowed attention over frames |
| Batch summarization, offline jobs | Long inputs, large batches | Compute-bound prefill; scores at long inputs | Maximize utilization; chunked prefill |
Common mistakes
- Using one share for all models. “The FFN is two thirds” is true for MHA with \(d_{ff} = 8d/3\). GQA models with wide FFNs are at 80% or more; MoE models above 95%.
- Counting GQA attention as \(4d^2\). This is the error that turns 81.7% into 71.7% for Llama 2 70B. Use \(n_{kv}\) for \(W^K\) and \(W^V\).
- Thinking GQA reduces attention FLOPs. It reduces the K and V projections and the cache bytes; the score FLOPs still use all \(n_h\) query heads (93.5).
- Mixing parameters, FLOPs and bytes. The FFN can be 80% of parameters while attention is 90% of decode bytes. Say which cost you mean.
- Forgetting the embedding and head. For small models with large vocabularies (128k to 152k tokens), they are more than 13% of the weights and the head costs FLOPs on every decode step.
- Counting MoE total parameters as FLOPs. Mixtral has 46.7B weights but uses about 12.9B per token. Memory follows the total; FLOPs follow the active part.
- Believing FlashAttention removes the quadratic cost. It removes the quadratic memory for the score matrix, not the quadratic FLOPs.
- Optimizing from the paper, not from the workload. The same model is FFN-bound for chat and attention-bound for long documents. Measure your own length distribution first.
Interview questions
What fraction of a transformer’s parameters is in the FFN, and why?basic
Write the forward FLOPs of one token at position t.basic
At what context length does attention cost more FLOPs than the FFN?core
Why do people say “FFN cost grows with model size, attention cost grows with context”?core
Does GQA reduce attention FLOPs?core
In decode, why does batching help the FFN but not attention?core
Llama 3 8B, bfloat16: at what batch × context does the cache cost as many bytes as the weights?core
Your users send 100k-token documents and get 500-token answers. What do you optimize first?advanced
How does MoE change the split?advanced
Why is FlashAttention not a fix for the “quadratic complexity” of attention?advanced
Exercises
- Qwen2.5 72B has \(d = 8192\), \(d_{ff} = 29{,}568\), 64 query heads and 8 key–value heads of size 128. Compute the FFN share of one layer with (93.4) and the crossover \(t^*\) with (93.6).
- Reproduce the survey’s 63.01% for LLaMA-7B: compute the denominator with attention \(4d^2\) per layer and \(Vd\) counted three times. Then explain in one sentence why the correct count is 64.24%.
- Llama 2 7B in bfloat16 (6.74B parameters, MHA, 32 layers): compute \(s\) and the budget \(M_W / s\). At batch 8, at what context does the cache cost as many bytes as the weights?
- A video model built on Llama 3 8B receives 64 frames with 144 tokens each, plus 100 text tokens. Use (93.7) to compute the share of prefill FLOPs in the score term. Compare with the 600-frame case in the workload table.
- Using (93.9), what is the arithmetic intensity of decode attention for Llama 2 7B and for Llama 3 70B in bfloat16? How far are they from the H100 ridge point of 295 FLOP/byte? Does a larger batch help?
- For Mixtral 8x7B, explain why \(t^* = 43{,}008\) although \(d_{ff} = 14{,}336\), and why its decode at a large batch reads the weights of all 46.7B parameters.
Answerscheck
Key takeaways
- One layer = attention (\(2 d n_h d_h + 2 d n_{kv} d_h\) weights) + FFN (\(3 d d_{ff}\) with SwiGLU). The FFN holds 64% (Llama 2 7B) to 82% (70B GQA models) of the weights, and over 95% in MoE models.
- The survey’s 63.01% and 71.69% match a count with MHA attention and \(Vd\) three times; from the 70B config file the share is 81.7%. Count from the config of the model you serve.
- FLOPs per token: weights cost 2 FLOPs each, fixed; attention scores cost \(4 L n_h d_h t\), growing with context. The crossover is \(t^* = 1.5 d_{ff}\): about 21k tokens for Llama 3 8B.
- Prefill FLOPs have a term quadratic in the prompt length; FlashAttention saves memory traffic, not these FLOPs.
- Decode bytes: weights are shared by the batch, caches are not. The cache equals the weights when \(B \cdot T = M_W / s\) (122,532 tokens for Llama 3 8B). Attention intensity is \(g\) in bfloat16, whatever the batch.
- Short chat: optimize the FFN and the weights. Long documents and video: optimize attention, the cache and the input length. Measure your workload first.
Key formulas
References for this session
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Shazeer (2020). GLU Variants Improve Transformer
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Shazeer (2020). GLU Variants Improve Transformer
- Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Jiang et al. (2024). Mixtral of Experts
- Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Kaplan et al. (2020). Scaling Laws for Neural Language Models
- Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
- Yuan et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights