LLM & VLM courseBlog
93Track 5: LLM inference and serving · session 16 of 65

Where the cost is: FFN and attention, by the numbers

Before you choose an optimization, you must know which part of the model costs the most. This session splits every transformer layer into its two halves, the feed-forward network (FFN) and attention, and counts each one three ways: parameters, FLOPs per token and bytes read per token. The answer depends on the model size and, even more, on the context length, so short chat, long documents and video need different fixes.

Time: 2.5 to 3.5 hours (reading 2 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 25 and 80 to 82 (counting parameters and FLOPs, the KV cache, inference metrics, why inference is slow)

What you will learn

  • How to split the parameters of one layer into attention (4d² for MHA, less for GQA) and FFN (3·d·d_ff for SwiGLU)
  • Why the survey reports 63.01% (LLaMA-7B) and 71.69% (LLaMA-70B) FFN parameters, and why a count from the 70B config file gives 81.7%
  • FLOPs per token: the weight part grows with model size, the attention-score part grows with context length; the crossover is near t = 1.5·d_ff
  • Bytes per token in decode: weights are shared by the batch, the KV cache is not; arithmetic intensity B for the FFN and n_h/n_kv for attention
  • How to compute the split for your own model from config.json, and check it with a PyTorch FLOP counter
  • Which part to optimize first for short chat, long documents and video

Why this matters

Every optimization in the rest of this track works on one part of the model. Some make the feed-forward network (FFN) cheaper: mixture of experts, pruning, weight quantization. Others make attention cheaper: GQA, sparse attention, FlashAttention, KV-cache quantization. Before you pick one, you must know where your cost is. If you shrink the part that is only 5% of the work, you gain at most 5%.

The efficient-inference survey starts its model-level chapter with this question. It reports that “the FFN module accounts for 63.01% of the parameters in the LLaMA-7B model and 71.69% in the LLaMA-70B model”, and that attention “demonstrates quadratic complexity in the input length”. [Certain, survey Section 5.1] These two facts give the plan of this session:

  • FFN cost grows with model size. The FFN holds most of the weights. Every token must use all of them, whatever the context length.
  • Attention cost grows with context length. Attention has fewer weights, but each new token compares itself with every earlier token. At long contexts this part takes over.

We will check the survey’s numbers ourselves (one of them does not match the model’s config file, and we will see why), and then count three kinds of cost, because they give different answers:

  • Parameters: how much memory the weights need.
  • FLOPs per token: how much arithmetic. This sets the speed of prefill, the phase that reads the prompt (session 78).
  • Bytes per token: how much data the GPU must read from its memory. This sets the speed of decode, the phase that writes one token at a time (session 82).

Session 25 gave the basic formulas for the whole model. Here we split them into the two halves and ask which half wins in which situation. The data-level methods of the previous sessions, ending with session 92, made the input shorter. From now on we change the model, and this session is the map for that work. The next session, session 94, starts with the FFN.

Interviewers like this topic because it tests whether you can reason with numbers: “What fraction of a 70B model is the FFN?”, “At what context length does attention cost more than the MLP?”, “Your users send 100k-token documents; what do you optimize first?”, “Why does GQA not reduce attention FLOPs?”. By the end you can answer each with a formula and a number for a real model.

The two halves of a layer

What each half contains

A decoder layer of a modern model like Llama has two sub-blocks, each with a normalization before it and a residual connection around it (session 23):

  • Attention: four weight matrices. \(W^Q\) makes queries, \(W^K\) keys, \(W^V\) values, and \(W^O\) maps the concatenated head outputs back to width \(d\). Between them is the part with no weights: the scores \(QK^\top\), the softmax, and the weighted sum of values.
  • FFN (also called MLP): with SwiGLU, three matrices. \(W_{\text{gate}}\) and \(W_{\text{up}}\) expand the vector from \(d\) to \(d_{ff}\) numbers, and \(W_{\text{down}}\) brings it back to \(d\) (session 21).

Count the numbers in these matrices. With \(n_h\) query heads, \(n_{kv}\) key–value heads and head size \(d_h\):

\[ P_{\text{attn}} = \underbrace{d \cdot n_h d_h}_{W^Q} + \underbrace{2\, d \cdot n_{kv} d_h}_{W^K,\ W^V} + \underbrace{n_h d_h \cdot d}_{W^O} = 2\, d\, n_h d_h + 2\, d\, n_{kv} d_h \tag{93.1} \]
\[ P_{\text{ffn}} = 3\, d\, d_{ff} \ \ \text{(SwiGLU)}, \qquad P_{\text{ffn}} = 2\, d\, d_{ff} \ \ \text{(classic two-matrix MLP)} \tag{93.2} \]
Read the symbols
\(d\)
the model width (hidden_size in the config file)
\(n_h, n_{kv}, d_h\)
query heads, key–value heads and head size. For multi-head attention (MHA) \(n_{kv} = n_h\); with grouped-query attention (GQA) \(n_{kv}\) is smaller.
\(d_{ff}\)
the inner width of the FFN (intermediate_size)
3 and 2
SwiGLU has three matrices (gate, up, down); the original transformer MLP has two (up, down). Biases are left out: Llama models have none.

In most models \(n_h d_h = d\). Then MHA gives \(P_{\text{attn}} = 4d^2\). GQA keeps the full \(W^Q\) and \(W^O\) but makes \(W^K\) and \(W^V\) smaller by the group factor \(g = n_h / n_{kv}\), so \(P_{\text{attn}} = (2 + 2/g)\, d^2\).

The whole model adds the embedding table, the output head (the “LM head”, which turns the last hidden vector into one score per vocabulary token) and the small normalization weights:

\[ N = L\,(P_{\text{attn}} + P_{\text{ffn}} + 2d) + c\, V d + d, \qquad c = 2 \text{ (separate head) or } 1 \text{ (tied)} \tag{93.3} \]
Read the symbols
\(L\)
number of layers (num_hidden_layers)
\(V\)
vocabulary size; \(Vd\) numbers for the embedding table and \(Vd\) more for the head, unless tie_word_embeddings is true
\(2d\) and \(d\)
the two RMSNorm weight vectors per layer and the final norm: tiny, but needed for an exact match

The FFN share of one layer

Divide (93.2) by the sum of (93.1) and (93.2). With \(n_h d_h = d\), write \(r = d_{ff}/d\) for the FFN expansion ratio:

\[ \rho_{\text{layer}} = \frac{P_{\text{ffn}}}{P_{\text{attn}} + P_{\text{ffn}}} = \frac{3r}{2 + 2/g + 3r} \tag{93.4} \]
Read the symbols
\(\rho_{\text{layer}}\)
the fraction of one layer’s weights that sit in the FFN
\(r\)
the expansion ratio \(d_{ff}/d\): 2.6875 in Llama 2 7B (\(11{,}008/4{,}096\)), 3.5 in Llama 2 70B and Llama 3 (\(28{,}672/8{,}192\), \(14{,}336/4{,}096\))
\(g\)
the GQA group size \(n_h/n_{kv}\): 1 for MHA, 4 for Llama 3 8B, 8 for the 70B models

Three cases show the trend. Classic SwiGLU with \(r = 8/3\) and MHA gives \(\rho = 8/12 = 66.7\%\). Llama 2 7B (\(r = 2.6875\), MHA) gives 66.84%. Llama 2 70B (\(r = 3.5\), \(g = 8\)) gives \(10.5 / 12.75 = 82.35\%\). Larger models moved to GQA, which shrinks attention, and to wider FFNs, which grow the FFN. Both push the FFN share up. Mixture of experts (session 29) pushes it much further: with \(E\) experts, \(P_{\text{ffn}}\) is multiplied by \(E\).

Figure 1 draws the seven matrices of one layer at true scale, for six real models.

attention matricesFFN matricesembedding and LM head
Figure 1. Top: the weight matrices of one layer. Every matrix has \(d\) rows, so the width of each box is proportional to its number of parameters. Bottom: the split of the whole model, formula (93.3). Shapes come from each model’s config.json on Hugging Face; the totals match the published sizes (6.74B, 68.98B, 8.03B, 70.55B, 7.62B, 46.70B).
How to read Figure 1

Start with Llama 2 7B. The four blue boxes (attention) are all the same width, because MHA has \(n_{kv} = n_h\): each is \(4096 \times 4096\). The three teal boxes (FFN) are each \(4096 \times 11{,}008\), so the FFN is about two thirds of the layer. Switch to Llama 2 70B: \(W^K\) and \(W^V\) become thin slivers (8 key–value heads for 64 query heads), and the FFN grows to 82% of the layer. Switch to Llama 3 8B: the FFN share of the layer is 81%, but the bottom bar shows that the large vocabulary (128,256 tokens) puts 13% of the whole model in the embedding and head. Mixtral shows eight stacked copies of the FFN: 96.6% of its weights are experts. Now go back to Llama 2 7B and Llama 2 70B and tick the survey box: you get the survey’s 63.01% and 71.69%.

Checking the survey’s numbers

Numbers in a survey are a good start, but an engineer should be able to reproduce them. Let us count Llama 2 7B (LLaMA-7B has the same shapes) and Llama 2 70B from their config files.

Worked example: the FFN share of Llama 2 7B and 70B

Llama 2 7B: \(d = 4096\), \(L = 32\), \(d_{ff} = 11{,}008\), 32 heads (MHA), \(V = 32{,}000\), separate head. [Certain, from config.json]

Attention per layer: \(4 \times 4096^2 = 67{,}108{,}864\). FFN per layer: \(3 \times 4096 \times 11{,}008 = 135{,}266{,}304\). Times 32 layers: \(2{,}147{,}483{,}648\) and \(4{,}328{,}521{,}728\). Embedding plus head: \(2 \times 32{,}000 \times 4096 = 262{,}144{,}000\). Norms: \(266{,}240\). Total: \(6{,}738{,}415{,}616\), the familiar “6.74B”.

FFN share: \(4{,}328{,}521{,}728 / 6{,}738{,}415{,}616 = \mathbf{64.24\%}\). Attention: 31.87%. Embedding and head: 3.89%.

Llama 2 70B: \(d = 8192\), \(L = 80\), \(d_{ff} = 28{,}672\), 64 query heads but only 8 key–value heads. [Certain, from config.json] Attention per layer: \(2 \times 8192^2 + 2 \times 8192 \times 1024 = 150{,}994{,}944\). FFN per layer: \(704{,}643{,}072\). Total: 68.98B. FFN share: \(\mathbf{81.73\%}\).

So our count gives 64.24% and 81.73%, while the survey gives 63.01% and 71.69%. The 7B numbers are close; the 70B numbers are 10 points apart. Where does the gap come from? We can reproduce both survey numbers to two decimals with one recipe: count attention as \(4d^2\) per layer (as if the 70B model used MHA, with 64 key–value heads), and count \(Vd\) three times instead of twice. For the 7B model the denominator becomes \(6{,}869{,}221{,}376\) and the share \(63.01\%\); for the 70B model it becomes \(71.69\%\). [Likely: this is our reconstruction; the survey does not show its count, but both numbers match exactly.]

The lesson is not that the survey is careless. The lesson is that attention design changes the split a lot. GQA cut the attention weights of the 70B model from \(4d^2\) to \(2.25d^2\) per layer. If you ignore it, you overestimate attention by 78% and underestimate the FFN share by 10 points. Always count from the config file of the exact model you serve.

ModelAttention\(d_{ff}/d\)TotalFFNAttentionEmbedding + head
Llama 2 7BMHA, 32 heads2.696.74B64.2%31.9%3.9%
Llama 2 70BGQA, 64 / 83.568.98B81.7%17.5%0.8%
Llama 3 8BGQA, 32 / 83.58.03B70.2%16.7%13.1%
Llama 3 70BGQA, 64 / 83.570.55B79.9%17.1%3.0%
Qwen2.5 7BGQA, 28 / 45.297.62B74.9%10.8%14.3%
Qwen2.5 72BGQA, 64 / 83.6172.71B80.0%16.6%3.4%
Mixtral 8x7BGQA, 32 / 8, 8 experts3.546.70B96.6%2.9%0.6%

[Certain for the shapes, from the config files; the percentages are computed with (93.1) to (93.3).] For Mixtral, each token uses only 2 of the 8 experts, so the active parameters per token are 12.88B, and the FFN is 87.5% of those. The table shows a clear pattern: in every model of 7B or more, the FFN holds between 64% and 97% of the weights. For small models with a large vocabulary (Llama 3 8B, Qwen2.5 7B), the embedding and head are a real third part, more than 13%.

FLOPs per token: the context term

Two kinds of attention work

Parameters tell us about memory. Speed depends on work. A matrix multiply of a vector with a \(m \times n\) matrix costs \(2mn\) FLOPs (one multiply and one add per weight), so every weight costs 2 FLOPs per token (session 25). This is true for the FFN and for the four attention matrices.

But attention also has work without weights. The token at position \(t\) computes one score with each of the \(t\) keys (\(QK^\top\)), and then a weighted sum of \(t\) values. For one head, each of these costs \(2 d_h t\) FLOPs. Over all heads and layers, the forward FLOPs of one token at position \(t\) are

\[ F(t) = \underbrace{2L\, P_{\text{ffn}}}_{\text{FFN}} + \underbrace{2L\, P_{\text{attn}}}_{\text{projections}} + \underbrace{4\, L\, n_h d_h\, t}_{\text{scores and weighted sum}} + \underbrace{2Vd}_{\text{LM head}} \tag{93.5} \]
Read the symbols
\(F(t)\)
forward FLOPs for one token that sits at position \(t\) (it attends to \(t\) tokens, itself included)
\(2L P_{\text{ffn}}\), \(2L P_{\text{attn}}\)
2 FLOPs per weight. These do not depend on \(t\).
\(4 L n_h d_h t\)
\(QK^\top\) (\(2 n_h d_h t\)) plus the weighted sum of values (\(2 n_h d_h t\)), in every layer. This is the only term that grows with context.
\(n_h\), not \(n_{kv}\)
every query head computes its own scores, even when several heads share one key head. GQA does not reduce this term.
\(2Vd\)
the LM head; the embedding lookup costs no FLOPs

The softmax and the normalizations add a few FLOPs per score or per number, which we ignore. [Certain that they are small next to the matrix multiplies.]

The crossover length

The first terms of (93.5) are fixed; the score term grows linearly with \(t\). Set the score term equal to the FFN term and solve for \(t\):

\[ 4 L\, n_h d_h\, t^{*} = 6 L\, d\, d_{ff} \quad\Longrightarrow\quad t^{*} = \frac{3\, d\, d_{ff}}{2\, n_h d_h} = 1.5\, d_{ff} \ \ \text{ when } n_h d_h = d \tag{93.6} \]
Read the symbols
\(t^{*}\)
the position at which one token’s attention scores cost as many FLOPs as its FFN
\(1.5\, d_{ff}\)
the crossover depends only on the FFN width: wider models need longer contexts before attention dominates

This is the formula behind the two key sentences of this session. For Llama 2 7B, \(t^* = 1.5 \times 11{,}008 = 16{,}512\). For Llama 3 8B, \(21{,}504\). For the 70B models, \(43{,}008\). Below \(t^*\), the FFN is the biggest single cost; above it, the scores are. If you compare the scores with all weight FLOPs of the layer (FFN plus projections), the crossover moves to \(t = 2d + 1.5 d_{ff}\): 24,704 for Llama 2 7B and 26,624 for Llama 3 8B.

Worked example: Llama 3 8B at 1k, 4k and 32k

Weight FLOPs per token: FFN \(2 \times 32 \times 176{,}160{,}768 = 11.27\) GFLOPs; attention projections \(2 \times 32 \times 41{,}943{,}040 = 2.68\) GFLOPs; head \(2 \times 128{,}256 \times 4096 = 1.05\) GFLOPs. Together 15.0 GFLOPs.

Score term: \(4 \times 32 \times 4096 \times t = 524{,}288\, t\) FLOPs. At \(t = 1024\): 0.54 GFLOPs, 3.5% of the total. At \(t = 4096\): 2.15 GFLOPs, 12.5%. At \(t = 32{,}768\): 17.18 GFLOPs, 53.4%, more than the FFN (35.0%).

The same rule also explains “FFN cost grows with model size”. Divide the score term by the weight term: roughly \(4 L d t / (2 L \cdot 12 d^2) = t / (6d)\) for a classic layer. At a fixed context, a wider model spends a smaller fraction of its work on attention scores. Big models are FFN-heavy; long contexts are attention-heavy.

FFNattention projectionsattention scores and weighted sumLM head
Figure 2. The split of forward FLOPs for one token at position \(t\), formula (93.5), from 128 to about 1 million tokens (log scale). The dashed line marks \(t^*\) of formula (93.6), where scores and FFN cost the same. For Mixtral, only the 2 active experts count.
How to read Figure 2

Each vertical slice adds up to 100%. On the left (short context) the teal FFN band and the blue projection band fill almost everything. Moving right, the magenta score band grows, and it passes the FFN at the dashed line. Move the slider to read exact numbers for one position. Compare models: Llama 3 70B and Mixtral have their crossover at 43,008, twice as far right as the 8B models, because their FFNs are wider. Note that some positions are beyond the trained context of a model (for example 4,096 for Llama 2 7B); the figure still shows the arithmetic.

Prefill: the whole prompt at once

Prefill processes \(P\) prompt tokens. Token \(t\) attends to \(t\) tokens (the causal mask hides the future), so we sum (93.5) over \(t = 1, \dots, P\):

\[ F_{\text{prefill}}(P) = P \cdot 2L\,(P_{\text{ffn}} + P_{\text{attn}}) \;+\; 2 L\, n_h d_h\, P(P+1) \;+\; 2Vd \tag{93.7} \]
Read the symbols
first term
linear in \(P\): every prompt token goes through all weight matrices
second term
quadratic in \(P\): this is the “quadratic complexity in the input length” of the survey. It is \(\sum_t 4 L n_h d_h t\).
\(2Vd\) once
only the last prompt position needs logits; good servers skip the head for the other positions [Likely for vLLM-style servers; a naive loop computes all of them]

Doubling a long prompt doubles the first term but multiplies the second by four. For Llama 3 8B, a 100,000-token prompt costs \(4.0 \times 10^{15}\) FLOPs, and 65% of it is the score term. On an H100 (989 dense BF16 TFLOP/s [Certain, NVIDIA spec]) that is at least 4.1 seconds of prefill even at 100% utilization.

Bytes per token: what decode reads

Shared weights, private caches

In decode, each step makes one new token per sequence. The arithmetic per token is the same as (93.5), but the GPU is not limited by arithmetic. It is limited by how fast it can read data from its memory (HBM) (session 82). So in decode we must count bytes, and here the two halves behave very differently:

  • Weights (FFN and attention projections): read once per step, and that one read serves all \(B\) sequences in the batch.
  • KV cache (the input of the score term): every sequence has its own cache (session 80), so nothing is shared. A sequence of length \(T\) reads \(s \cdot T\) bytes per step.
\[ \beta = \frac{M_W}{B} + s \cdot T, \qquad s = 2 L\, n_{kv} d_h\, b, \qquad \frac{M_W}{B} = s\, T \iff B \cdot T = \frac{M_W}{s} \tag{93.8} \]
Read the symbols
\(\beta\)
bytes read from memory per generated token
\(M_W\)
bytes of all weights: \(N \times b\) (2 bytes each in bfloat16)
\(B\)
number of sequences decoded together
\(s\)
KV-cache bytes per token over all layers; \(n_{kv}\), not \(n_h\), appears here
\(T\)
current length of the sequence (prompt plus tokens written so far)
\(M_W / s\)
the “budget” of batch × context at which the cache costs as many bytes as the weights

For Llama 3 8B in bfloat16, \(M_W = 16.06\) GB and \(s = 131{,}072\) bytes, so \(M_W / s = 122{,}532\) tokens. One user at 4,096 tokens reads 16.06 GB of weights and 0.54 GB of cache per step: the weights are 96.8% of the traffic, and an H100 at 3.35 TB/s [Certain, NVIDIA spec] needs at least 4.95 ms per step. With 32 users at 4,096 tokens, the caches add up to 17.18 GB: now the cache is 51.7% of the traffic and the step takes at least 9.92 ms. Batching made the weights cheap per token, but it cannot make the cache cheap. (Strictly, a decode step reads only \(B\) rows of the embedding table, so the weight traffic is a little smaller, 15.0 GB for Llama 3 8B; like session 80 we keep \(M_W\) = all weights as a simple upper bound.)

Here GQA matters a lot. It does not change the score FLOPs (\(n_h\) in 93.5), but it divides \(s\) by \(g\) (\(n_{kv}\) in 93.8). Llama 2 7B (MHA) has \(s = 524{,}288\) bytes and \(M_W / s = 25{,}705\): its cache equals its weights already at one user with 25,705 tokens, or 8 users with 3,213 tokens.

Arithmetic intensity of the two halves

Arithmetic intensity is FLOPs divided by bytes read. A GPU is busy only when the intensity of the work is above its ridge point, its peak FLOP/s divided by its bandwidth: \(989 / 3.35 \approx 295\) FLOP/byte for an H100 (session 82, session 123). In decode:

\[ I_{\text{weights}} = \frac{2B \text{ FLOPs per weight}}{b \text{ bytes per weight}} = \frac{2B}{b}, \qquad I_{\text{attn}} = \frac{4\, n_h d_h T}{2\, n_{kv} d_h T\, b} = \frac{2g}{b} \tag{93.9} \]
Read the symbols
\(I_{\text{weights}}\)
for the FFN and projections: each weight byte is used once per sequence in the batch. In bfloat16 (\(b = 2\)) this is just \(B\).
\(I_{\text{attn}}\)
for the score term: each cached key and value is used by the \(g\) query heads that share it. In bfloat16 this is \(g\): 1 for MHA, 4 for Llama 3 8B, 8 for the 70B models.
no \(B\) in \(I_{\text{attn}}\)
batching does not help attention in decode, because no two sequences share a cache

So in decode the FFN becomes efficient when the batch is large (near \(B \approx 295\) on an H100), but attention stays at an intensity of 1 to 8, far below the ridge, whatever you do with batching. [Certain, from the formula] This is why decode attention has its own kernels (session 125) and why the fixes for attention in decode are about bytes: fewer key–value heads, a smaller cache format, or fewer cached tokens.

Figure 3. Decode traffic per step, formula (93.8), all in bfloat16. Each cell is one (batch, context) pair and shows the percentage of bytes that is KV cache; the rest is weights. Click a cell for the details and a lower bound on the step time with 3.35 TB/s of bandwidth. Mixtral counts all expert weights: in a big batch, the tokens together use all experts. [Likely for batches above a few tokens]
How to read Figure 3

Light cells (left and top) are weight-bound: shrinking the weights (quantization, pruning, fewer experts) is what helps. Dark cells (right and bottom) are cache-bound: GQA, a quantized cache or a shorter context help, and smaller weights barely change the step time. The border between them is the line \(B \cdot T = M_W / s\). Compare Llama 2 7B (MHA) with Llama 3 8B (GQA): the same cell is much darker for Llama 2. Qwen2.5 7B, with only 4 key–value heads, is the lightest of the small models. Llama 3 70B has big weights, so it stays weight-bound longer.

Your workload decides: chat, long documents, video

Now combine prefill (FLOPs, formula 93.7) and decode (bytes, formula 93.8) for one request. Four typical workloads on Llama 3 8B give very different pictures. [Certain for the arithmetic; the workloads are illustrative examples.]

WorkloadPrompt \(P\)Output \(N\)Prefill: score share of FLOPsDecode, \(B = 16\): cache share of bytesMain cost
Short chat5003000.9%7.8%FFN and projection weights
RAG with 8k context8,00050013.1%51.9%Weights in prefill, half cache in decode
Long-document QA100,00050065.3%92.9%Attention in both phases
Video QA, 10 min at 1 frame/s86,50020061.9%91.9%Attention in both phases

The video row assumes 600 frames with 144 visual tokens each (86,400 tokens) plus 100 text tokens. Real video LLMs use between tens and hundreds of tokens per frame, so this is an example, not a specific model. [Likely range] For decode we use the average length \(P + N/2\).

The advice follows directly:

  • Short chat: almost all cost is the weights, mostly the FFN. Optimize the FFN and the weights: weight quantization (session 106), mixture of experts (session 95), structured pruning (session 115), and bigger batches. Making the attention scores or the cache faster gains less than 10%.
  • Long documents: attention dominates prefill FLOPs and decode bytes. Optimize attention and the cache: FlashAttention for prefill, GQA or MLA (session 97), KV-cache quantization (session 110), sparse attention (session 116), or a shorter input with prompt compression (session 88).
  • Video: like long documents, but the long part is visual tokens. The cheapest fix is often to send fewer visual tokens (pooling per frame, picking key frames, retrieving a few clips) before touching the model.
FFNattention projectionsattention scores / KV cacheLM head (and embedding)
Figure 4. One request of each workload. Top bar: prefill FLOPs, formula (93.7). Bottom bar: decode bytes per generated token at batch \(B\), formula (93.8), with the weight bytes split into FFN, projections and head in proportion to their parameters. The text names the part to optimize first.
How to read Figure 4

Start with short chat on Llama 3 8B: both bars are almost all teal and blue (weights). Raise the batch: the magenta cache part of the bottom bar grows, because the weights are shared and the cache is not, but it stays small. Switch to Long document: the magenta part takes 65% of prefill and over 90% of decode bytes at batch 16. Try Llama 2 7B on the long document: its MHA cache is 4× larger per token, so decode is almost all cache even at batch 1. (Its trained context is only 4,096 tokens; the numbers show what the arithmetic would be.)

Pseudo-code

Algorithm 1: split a model’s cost from its config file
  1. input: config.json; context length \(t\); batch \(B\); bytes per number \(b\)
  2. read \(d, L, d_{ff}, n_h, n_{kv}, V\); \(d_h \leftarrow\) head_dim if present, else \(d / n_h\); \(E, k \leftarrow\) experts and active experts (1, 1 if dense)
  3. \(P_{\text{attn}} \leftarrow 2 d n_h d_h + 2 d n_{kv} d_h\) // (93.1); add biases if the model has them
  4. \(P_{\text{ffn}} \leftarrow 3 d d_{ff} E\) (+ router \(dE\)); \(\;P_{\text{ffn}}^{\text{act}} \leftarrow 3 d d_{ff} k\) // (93.2)
  5. \(N \leftarrow L(P_{\text{attn}} + P_{\text{ffn}} + 2d) + cVd + d\) // (93.3), \(c = 1\) if tied
  6. params: report \(L P_{\text{ffn}} / N\), \(L P_{\text{attn}} / N\), \(cVd / N\)
  7. FLOPs: \(F_{\text{ffn}} \leftarrow 2 L P_{\text{ffn}}^{\text{act}}\); \(F_{\text{proj}} \leftarrow 2 L P_{\text{attn}}\); \(F_{\text{sc}} \leftarrow 4 L n_h d_h t\); \(F_{\text{head}} \leftarrow 2Vd\) // (93.5)
  8. \(t^* \leftarrow 6 d\, d_{ff} k / (4 n_h d_h)\) // (93.6)
  9. bytes: \(M_W \leftarrow N b\); \(s \leftarrow 2 L n_{kv} d_h b\); \(\beta \leftarrow M_W / B + s t\) // (93.8)
  10. return the three splits, \(t^*\) and the batch–context budget \(M_W / s\)
Algorithm 2: choose what to optimize for a workload
  1. input: typical prompt length \(P\), output length \(N\), batch \(B\), and the splits of Algorithm 1
  2. \(f_{\text{pre}} \leftarrow\) score share of \(F_{\text{prefill}}(P)\) // (93.7)
  3. \(f_{\text{dec}} \leftarrow s(P + N/2) / (M_W / B + s(P + N/2))\) // cache share of decode bytes
  4. \(w \leftarrow\) time share of prefill (measure it, or estimate from FLOPs and bytes)
  5. if \(w f_{\text{pre}} + (1 - w) f_{\text{dec}} \lt 0.2\): // weights dominate
  6. optimize FFN and weights: quantize weights, MoE, prune FFN, larger batch
  7. else if \(f_{\text{dec}}\) is large: optimize the cache: GQA/MLA, KV quantization, eviction, paging
  8. if \(f_{\text{pre}}\) is large: FlashAttention, sparse or linear attention, prompt compression, fewer visual tokens
  9. always profile afterwards: the shares change after each fix

The threshold 0.2 is a rule of thumb, not a law: it says that if attention is less than a fifth of the cost, attention work can win at most 20%.

In code: count and check

The code builds one tiny Llama-style layer with GQA (8 query heads, 2 key–value heads) and SwiGLU in PyTorch. It counts the parameters of each half from the real tensors and compares them with (93.1) and (93.2). Then it runs one decode step of attention against a cache of \(t - 1\) tokens and one step of the FFN under PyTorch’s FlopCounterMode, and compares the measured FLOPs with (93.5). It is deterministic (fixed seed; the FLOP counts do not depend on the values anyway).

import torch, torch.nn as nn
from torch.utils.flop_counter import FlopCounterMode
torch.manual_seed(0)

class Block(nn.Module):                                  # one Llama-style layer: GQA + SwiGLU
    def __init__(s, d, n_h, n_kv, d_h, d_ff):
        super().__init__()
        s.n_h, s.n_kv, s.d_h = n_h, n_kv, d_h
        s.wq = nn.Linear(d, n_h * d_h, bias=False); s.wk = nn.Linear(d, n_kv * d_h, bias=False)
        s.wv = nn.Linear(d, n_kv * d_h, bias=False); s.wo = nn.Linear(n_h * d_h, d, bias=False)
        s.gate = nn.Linear(d, d_ff, bias=False); s.up = nn.Linear(d, d_ff, bias=False)
        s.down = nn.Linear(d_ff, d, bias=False)
    def attn(s, x, K, V):                                # x: (1, d) new token; K, V: (n_kv, t-1, d_h) cache
        q = s.wq(x).view(s.n_h, 1, s.d_h)
        K = torch.cat([K, s.wk(x).view(s.n_kv, 1, s.d_h)], 1)       # append the new key and value:
        V = torch.cat([V, s.wv(x).view(s.n_kv, 1, s.d_h)], 1)       # now t rows
        K = K.repeat_interleave(s.n_h // s.n_kv, 0); V = V.repeat_interleave(s.n_h // s.n_kv, 0)
        a = torch.softmax(q @ K.transpose(1, 2) / s.d_h**0.5, -1)   # (n_h, 1, t) scores
        return s.wo((a @ V).reshape(1, -1))
    def ffn(s, x):
        return s.down(nn.functional.silu(s.gate(x)) * s.up(x))

def count(mod, names): return sum(getattr(mod, n).weight.numel() for n in names)
def flops(fn):
    with FlopCounterMode(display=False) as fc: fn()
    return fc.get_total_flops()

d, n_h, n_kv, d_h, d_ff = 512, 8, 2, 64, 1408            # tiny model, GQA group size 4
blk = Block(d, n_h, n_kv, d_h, d_ff)
P_attn = count(blk, ["wq", "wk", "wv", "wo"]); P_ffn = count(blk, ["gate", "up", "down"])
print("params  attention", P_attn, "formula", 2*d*n_h*d_h + 2*d*n_kv*d_h)
print("params  FFN      ", P_ffn,  "formula", 3*d*d_ff, f" FFN share {P_ffn/(P_attn+P_ffn):.1%}")

x = torch.randn(1, d)
for t in [256, 1024, 4096, 16384]:
    K = torch.randn(n_kv, t - 1, d_h); V = torch.randn(n_kv, t - 1, d_h)
    fa, ff = flops(lambda: blk.attn(x, K, V)), flops(lambda: blk.ffn(x))
    proj, score = 2 * P_attn, 4 * n_h * d_h * t                # formula: projections + QK and AV
    print(f"t={t:6d}  attention {fa:>10,} (formula {proj+score:>10,})  FFN {ff:>9,}  attention share {fa/(fa+ff):.1%}")
print("crossover t* where scores = FFN:", 6 * d * d_ff // (4 * n_h * d_h))

Output (PyTorch 2.x on CPU):

params  attention 655360 formula 655360
params  FFN       2162688 formula 2162688  FFN share 76.7%
t=   256  attention  1,835,008 (formula  1,835,008)  FFN 4,325,376  attention share 29.8%
t=  1024  attention  3,407,872 (formula  3,407,872)  FFN 4,325,376  attention share 44.1%
t=  4096  attention  9,699,328 (formula  9,699,328)  FFN 4,325,376  attention share 69.2%
t= 16384  attention 34,865,152 (formula 34,865,152)  FFN 4,325,376  attention share 89.0%
crossover t* where scores = FFN: 2112

The measured FLOPs match (93.5) exactly. The FFN is 76.7% of the weights but its FLOPs stay fixed at 4.33 million, while attention passes it between 1,024 and 4,096 tokens. The crossover is early here (\(t^* = 1.5 \times 1408 = 2112\)) because the model is tiny; for real models it is \(1.5 d_{ff}\), tens of thousands of tokens. Note the counter does not see the softmax; that is the “small terms” we ignored. To apply the same check to a real model, replace the shapes with the values from its config file, or run FlopCounterMode around one forward pass of the Hugging Face model.

Where this is used

SettingTypical lengthsWhere the cost isWhat teams optimize
Chat assistants, customer supportHundreds to a few thousand tokensFFN and projection weights; decode is weight-boundWeight quantization, MoE, large continuous batches
Code completionA few thousand tokens of context, short output, low latencyWeights per step, at small batchSmall or quantized models, speculative decoding
RAG and long-document QA10k to 128k tokens of promptAttention scores in prefill, KV cache in decodeFlashAttention, prefix caching, prompt compression, GQA/MLA
Agents with tool callsLong and growing context, many short turnsPrefill of the growing context; cache memoryPrefix caching, cache eviction, summarizing old turns
VLM image QAHundreds of visual tokens per image (576 for LLaVA-1.5)Mostly weights for one image; attention for many imagesVisual token pruning or pooling
Video QA, Video-RAG, key-moment detectionThousands to over 100k visual tokensAttention in prefill and cache in decodeFewer tokens per frame, key-frame selection, clip retrieval before the LLM
Video classification and action recognition with an LLM headMany frames, very short outputPrefill only; scores grow with frames squaredToken merging, sparse or windowed attention over frames
Batch summarization, offline jobsLong inputs, large batchesCompute-bound prefill; scores at long inputsMaximize utilization; chunked prefill

Common mistakes

  • Using one share for all models. “The FFN is two thirds” is true for MHA with \(d_{ff} = 8d/3\). GQA models with wide FFNs are at 80% or more; MoE models above 95%.
  • Counting GQA attention as \(4d^2\). This is the error that turns 81.7% into 71.7% for Llama 2 70B. Use \(n_{kv}\) for \(W^K\) and \(W^V\).
  • Thinking GQA reduces attention FLOPs. It reduces the K and V projections and the cache bytes; the score FLOPs still use all \(n_h\) query heads (93.5).
  • Mixing parameters, FLOPs and bytes. The FFN can be 80% of parameters while attention is 90% of decode bytes. Say which cost you mean.
  • Forgetting the embedding and head. For small models with large vocabularies (128k to 152k tokens), they are more than 13% of the weights and the head costs FLOPs on every decode step.
  • Counting MoE total parameters as FLOPs. Mixtral has 46.7B weights but uses about 12.9B per token. Memory follows the total; FLOPs follow the active part.
  • Believing FlashAttention removes the quadratic cost. It removes the quadratic memory for the score matrix, not the quadratic FLOPs.
  • Optimizing from the paper, not from the workload. The same model is FFN-bound for chat and attention-bound for long documents. Measure your own length distribution first.

Interview questions

What fraction of a transformer’s parameters is in the FFN, and why?basic
Per layer, attention with MHA has \(4d^2\) weights (Q, K, V, O) and a SwiGLU FFN has \(3 d\, d_{ff}\). With \(d_{ff} = 8d/3\) that is \(8d^2\), so the FFN is two thirds of each layer. Real models differ: Llama 2 7B is 64.2% FFN over the whole model, Llama 2 70B is 81.7% because GQA shrinks attention to \(2.25 d^2\) and the FFN ratio is 3.5. The survey quotes 63.01% and 71.69%; the second number matches a count that treats the 70B model as MHA.
Write the forward FLOPs of one token at position t.basic
\(F(t) = 2L P_{\text{ffn}} + 2L P_{\text{attn}} + 4 L n_h d_h t + 2Vd\). Every weight costs 2 FLOPs per token. The score term counts \(QK^\top\) and the weighted sum of values, \(2 n_h d_h t\) each, in every layer. Only this term depends on \(t\). Softmax and norms are small extra terms.
At what context length does attention cost more FLOPs than the FFN?core
Set \(4 L n_h d_h t = 6 L d d_{ff}\). With \(n_h d_h = d\), \(t^* = 1.5\, d_{ff}\). For Llama 3 8B (\(d_{ff} = 14{,}336\)) that is 21,504 tokens; for Llama 2 7B 16,512; for 70B models 43,008. Against all weight FLOPs of the layer (FFN plus projections) the crossover is \(2d + 1.5 d_{ff}\), about 26,600 for Llama 3 8B. So most chat is FFN-dominated and long-document work is attention-dominated.
Why do people say “FFN cost grows with model size, attention cost grows with context”?core
The FFN cost per token is \(6 d d_{ff} L\): fixed for a model and proportional to its size, the same at position 10 or 100,000. The score cost is \(4 L n_h d_h t\): it grows linearly per token with the context and quadratically over a whole prompt. The ratio of scores to weights is roughly \(t / (6d)\), so a wider model at the same context spends a smaller share on attention. The crossover \(1.5 d_{ff}\) moves right as models grow.
Does GQA reduce attention FLOPs?core
Only slightly. It reduces the K and V projection weights (\(2 d n_{kv} d_h\) instead of \(2 d n_h d_h\)), but every query head still computes \(t\) scores and a weighted sum of \(t\) values, so the score FLOPs \(4 L n_h d_h t\) do not change. What GQA reduces a lot is the KV cache size and the bytes read per decode step, by the factor \(g = n_h / n_{kv}\). Since decode is memory-bound, that is where the speedup comes from.
In decode, why does batching help the FFN but not attention?core
The FFN weights are the same for all sequences: one read of each weight serves \(B\) tokens, so the arithmetic intensity is about \(2B/b\) FLOPs per byte (\(B\) in bfloat16) and the weight bytes per token are \(M_W / B\). Each sequence has its own KV cache, so its bytes per token stay \(sT\) whatever the batch, and the intensity is \(2g/b\), between 1 and 8 in bfloat16. With a large batch and long contexts, decode becomes cache-bound.
Llama 3 8B, bfloat16: at what batch × context does the cache cost as many bytes as the weights?core
\(M_W = 2 \times 8.03 \times 10^9 = 16.06\) GB. \(s = 2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes per token. \(B \cdot T = M_W / s = 122{,}532\). So one user at about 122k tokens, 16 users at about 7.7k, or 32 users at about 3.8k tokens. For Llama 2 7B (MHA, \(s = 524{,}288\)) the budget is only 25,705.
Your users send 100k-token documents and get 500-token answers. What do you optimize first?advanced
Count. For Llama 3 8B the prefill is about \(4 \times 10^{15}\) FLOPs, 65% of it attention scores, so at least 4 seconds on one H100. Decode reads about 13 GB of cache per token per user against 16 GB of weights shared by the batch; at batch 16 the cache is 93% of the bytes. So attention dominates both phases. First: reduce the input (better retrieval, prompt compression), cache shared prefixes, use FlashAttention-class kernels and chunked prefill. Then shrink the cache: FP8 KV cache, a model with fewer KV heads or MLA, eviction or sparse attention if quality allows. Weight quantization helps little here.
How does MoE change the split?advanced
MoE replaces one FFN by \(E\) expert FFNs and a router, and each token uses \(k\) of them. Parameters: the FFN share jumps (Mixtral 8x7B: 96.6% of 46.7B). FLOPs per token: only the \(k\) active experts count (Mixtral: 12.9B active, 87.5% FFN), so the crossover \(t^*\) is \(1.5 k d_{ff}\), 43,008 for Mixtral. Bytes: in a big batch, the tokens together touch all experts, so the weight bytes per step are those of the total model, and memory capacity is dominated by experts. MoE makes FFN compute cheap relative to memory, which is why expert offloading and expert parallelism exist.
Why is FlashAttention not a fix for the “quadratic complexity” of attention?advanced
FlashAttention computes exactly the same scores and sums, so the FLOPs stay \(2 L n_h d_h P(P+1)\) for a prompt of \(P\) tokens. It tiles the computation in on-chip memory and never writes the \(P \times P\) score matrix to HBM, so memory size becomes linear in \(P\) and memory traffic drops. Prefill becomes compute-bound and runs near peak, but doubling the prompt still costs about four times the score FLOPs. To reduce the FLOPs you need sparse, windowed or linear attention, or a shorter input.

Exercises

  1. Qwen2.5 72B has \(d = 8192\), \(d_{ff} = 29{,}568\), 64 query heads and 8 key–value heads of size 128. Compute the FFN share of one layer with (93.4) and the crossover \(t^*\) with (93.6).
  2. Reproduce the survey’s 63.01% for LLaMA-7B: compute the denominator with attention \(4d^2\) per layer and \(Vd\) counted three times. Then explain in one sentence why the correct count is 64.24%.
  3. Llama 2 7B in bfloat16 (6.74B parameters, MHA, 32 layers): compute \(s\) and the budget \(M_W / s\). At batch 8, at what context does the cache cost as many bytes as the weights?
  4. A video model built on Llama 3 8B receives 64 frames with 144 tokens each, plus 100 text tokens. Use (93.7) to compute the share of prefill FLOPs in the score term. Compare with the 600-frame case in the workload table.
  5. Using (93.9), what is the arithmetic intensity of decode attention for Llama 2 7B and for Llama 3 70B in bfloat16? How far are they from the H100 ridge point of 295 FLOP/byte? Does a larger batch help?
  6. For Mixtral 8x7B, explain why \(t^* = 43{,}008\) although \(d_{ff} = 14{,}336\), and why its decode at a large batch reads the weights of all 46.7B parameters.
Answerscheck
1. \(r = 29{,}568 / 8192 = 3.609\), \(g = 8\): \(\rho = 3r / (2 + 0.25 + 3r) = 10.83 / 13.08 = 82.8\%\). \(t^* = 1.5 \times 29{,}568 = 44{,}352\). 2. FFN: \(32 \times 3 \times 4096 \times 11{,}008 = 4{,}328{,}521{,}728\). Denominator: \(4{,}328{,}521{,}728 + 32 \times 4 \times 4096^2 + 3 \times 32{,}000 \times 4096 = 6{,}869{,}221{,}376\); share 63.01%. The correct count has the vocabulary twice (embedding and head), not three times, and includes the 266,240 norm weights, giving \(6{,}738{,}415{,}616\) and 64.24%. 3. \(s = 2 \times 32 \times 32 \times 128 \times 2 = 524{,}288\) bytes. \(M_W = 13.48\) GB, \(M_W / s = 25{,}705\) tokens; at batch 8, \(T = 3{,}213\), less than its 4,096-token context. 4. \(P = 9{,}316\): score share 14.9% (versus 61.9% at 86,500 tokens). With 9.3 times fewer tokens, the score term is about 86 times smaller (\(9.3^2\)) but the weight term only 9.3 times smaller. 5. \(I_{\text{attn}} = g\): 1 for Llama 2 7B (MHA), 8 for Llama 3 70B. That is 295× and 37× below the ridge. A larger batch does not help, because no two sequences share a cache; only GQA (larger \(g\)), a smaller cache format, or attention that shares a prefix cache across requests raises it. 6. Each token goes through \(k = 2\) experts, so the active FFN FLOPs are \(2 \times 6 d d_{ff}\) per layer and \(t^* = 1.5 \times 2 \times 14{,}336 = 43{,}008\). In a large batch, different tokens pick different experts, so together they touch all 8 experts in every layer, and each step reads all expert weights.

Key takeaways

  • One layer = attention (\(2 d n_h d_h + 2 d n_{kv} d_h\) weights) + FFN (\(3 d d_{ff}\) with SwiGLU). The FFN holds 64% (Llama 2 7B) to 82% (70B GQA models) of the weights, and over 95% in MoE models.
  • The survey’s 63.01% and 71.69% match a count with MHA attention and \(Vd\) three times; from the 70B config file the share is 81.7%. Count from the config of the model you serve.
  • FLOPs per token: weights cost 2 FLOPs each, fixed; attention scores cost \(4 L n_h d_h t\), growing with context. The crossover is \(t^* = 1.5 d_{ff}\): about 21k tokens for Llama 3 8B.
  • Prefill FLOPs have a term quadratic in the prompt length; FlashAttention saves memory traffic, not these FLOPs.
  • Decode bytes: weights are shared by the batch, caches are not. The cache equals the weights when \(B \cdot T = M_W / s\) (122,532 tokens for Llama 3 8B). Attention intensity is \(g\) in bfloat16, whatever the batch.
  • Short chat: optimize the FFN and the weights. Long documents and video: optimize attention, the cache and the input length. Measure your workload first.

Key formulas

\[ P_{\text{attn}} = 2 d\, n_h d_h + 2 d\, n_{kv} d_h, \qquad P_{\text{ffn}} = 3 d\, d_{ff}, \qquad \rho_{\text{layer}} = \frac{3r}{2 + 2/g + 3r} \]
\[ F(t) = 2L(P_{\text{ffn}} + P_{\text{attn}}) + 4 L\, n_h d_h\, t + 2Vd, \qquad t^* = 1.5\, d_{ff} \]
\[ \beta = \frac{M_W}{B} + s\,T, \qquad s = 2 L\, n_{kv} d_h\, b, \qquad I_{\text{weights}} = \frac{2B}{b}, \quad I_{\text{attn}} = \frac{2g}{b} \]

References for this session

  1. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  2. Shazeer (2020). GLU Variants Improve Transformer
  3. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  4. Shazeer (2020). GLU Variants Improve Transformer
  5. Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models
  6. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  7. Llama Team, Meta (2024). The Llama 3 Herd of Models
  8. Jiang et al. (2024). Mixtral of Experts
  9. Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
  10. Kaplan et al. (2020). Scaling Laws for Neural Language Models
  11. Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  12. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
  13. Yuan et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights