Counting: parameters, FLOPs and memory of a transformer
Before you train or serve a model, you can predict its size, its compute bill and its memory needs with a few lines of arithmetic. This session builds those formulas from one matrix multiplication: about 12·d² parameters per layer, 2N FLOPs per token forward, 6N per token in training, plus an attention term that grows with context. We then estimate an 8B model by hand and check the result against its real config file.
What you will learn
- Why a matrix multiplication of shapes (m × k) and (k × n) costs 2mkn FLOPs, and why every other count follows from this
- Parameters per layer: attention 4d², FFN 8d², so about 12·L·d² plus the embedding tables, and how GQA and SwiGLU change the constants
- Forward cost of about 2N FLOPs per token, training cost of about 6N per token, so C ≈ 6ND for a whole run
- The attention term 4·L·T·d per token, which grows with context length and passes the weight FLOPs near T = 6d
- Memory: bytes per parameter, 16 bytes per parameter for AdamW training, activation memory, and the KV cache
- How to turn a FLOP count into GPU-days with model FLOPs utilization (MFU)
- A full worked example: Llama 3 8B estimated by hand and checked against its config.json
Why this matters
Before anyone trains or serves a large model, they ask three questions. How many numbers (parameters) does it have? How much arithmetic (FLOPs) does it need? How much GPU memory does it need? You can answer all three on one sheet of paper, from the model’s config file and a few rules. This session gives you those rules.
The rules are short. A transformer layer of width \(d\) has about \(12d^2\) parameters. A forward pass costs about \(2N\) floating-point operations (FLOPs) per token, where \(N\) is the number of parameters. Training costs about \(6N\) per token, so a whole training run on \(D\) tokens costs about \(6ND\). Attention adds an extra term that grows with the context length. Memory is bytes per parameter, plus activations, plus (at inference) the KV cache.
- Planning. A team with 1,024 GPUs and one month must pick a model size and a token count that fit. The formula \(C \approx 6ND\) does that in one line. Scaling laws (session 35) are written in exactly these units.
- Debugging. If your training run reaches 15% of the GPU’s peak speed, is that bad? You can only tell if you know how many FLOPs one step should take. This ratio is called MFU (session 51).
- Serving. Will an 8B model with 32 users at 8k context fit on one 80 GB GPU? Weights plus KV cache tells you in a minute (session 80).
- Interviews. “How many parameters does Llama 3 8B have, from its config?”, “Why 6N?”, “How much memory to fine-tune a 7B model with Adam?”, “When does attention dominate the compute?”. These are among the most common LLM interview questions. By the end you can answer each one with a formula and a number.
In Session 23 we put the pieces together into one block and one stack. In Session 24 we saw a second family with cross-attention. Now we count everything inside a decoder-only stack. The next session, Session 26, uses these counts to explain why MQA, GQA and MLA exist: they shrink the KV cache that we measure here.
The one rule behind all counts: a matrix multiply
Almost all compute in a transformer is matrix multiplication. So we start there. Take \(A\) with shape \(m \times k\) and \(B\) with shape \(k \times n\). Each of the \(m \cdot n\) entries of \(AB\) is a dot product of length \(k\): \(k\) multiplications and \(k\) additions (strictly \(k-1\), but we round). One multiplication plus one addition is called a multiply-add, and we count it as 2 FLOPs:
- Read the symbols
- FLOP
- one floating-point operation: one addition or one multiplication. “FLOPs” (lower-case s) is a count; “FLOP/s” is a speed (operations per second).
- \(m\)
- the number of rows of the input. For a linear layer applied to a batch, \(m\) is the number of tokens.
- \(k\)
- the shared (inner) size that is summed over
- \(n\)
- the number of output columns
Now apply it to a linear layer \(\mathbf{y} = \mathbf{x} W\) with \(W \in \R^{d_{\text{in}} \times d_{\text{out}}}\), on one token. Here \(m = 1\), so the cost is \(2\, d_{\text{in}}\, d_{\text{out}}\). But \(d_{\text{in}} \cdot d_{\text{out}}\) is exactly the number of weights in \(W\). So:
Each weight of a linear layer costs 2 FLOPs per token in the forward pass: one multiply and one add.
This single sentence explains the “2N” rule later. Everything that is not a matrix multiply (softmax, normalization, activation functions, adding residuals) costs a few FLOPs per number in the activation, not per weight. With \(d\) in the thousands, these are small, and we ignore them. [Certain] They still take time on a GPU, because they read and write memory; that is a different story (session 82).
Take \(d = 4096\) and the query projection \(W_Q \in \R^{4096 \times 4096}\). It has \(4096^2 = 16{,}777{,}216\) weights. For one token the forward cost is \(2 \times 16{,}777{,}216 = 33{,}554{,}432\) FLOPs, about 33.6 MFLOPs. For a batch of 8 sequences of 4,096 tokens (\(m = 32{,}768\)) it is \(2 \times 32{,}768 \times 4096 \times 4096 \approx 1.10 \times 10^{12}\) FLOPs: about one second of work at 1 TFLOP/s, or about one millisecond on a modern GPU.
Counting parameters
One block, the classic way
Recall the decoder block from session 23: a normalization, multi-head attention, a residual add, another normalization, the feed-forward network (FFN), another residual add. We count the weights of each part, with model width \(d\).
Attention. There are four matrices: \(W_Q, W_K, W_V\) project the input to queries, keys and values, and \(W_O\) projects the concatenated heads back to width \(d\) (session 19). With \(h\) heads of size \(d_h = d/h\), each projection is \(d \times d\) in total, no matter how many heads. So:
The number of heads does not change the count. Splitting \(d\) into heads only changes how the numbers are grouped, not how many there are. [Certain]
FFN. The classic FFN (session 21) goes up from \(d\) to \(d_{\text{ff}} = 4d\), applies an activation, and comes back down: two matrices of size \(d \times 4d\):
Normalization. Each LayerNorm has a scale and a shift vector (\(2d\) numbers); an RMSNorm has only a scale (\(d\) numbers). These are tiny: \(2d\) to \(4d\) per block against \(12d^2\). Biases, where models have them, are also of size \(d\) or \(4d\). We ignore all of them in the estimate.
So one block has \(4d^2 + 8d^2 = 12d^2\) weights. A stack of \(L\) blocks has:
- Read the symbols
- \(L\)
- the number of transformer blocks (layers)
- \(d\)
- the model width: the length of the vector each token carries through the residual stream (
hidden_sizein a config file) - \(12 L d^2\)
- the non-embedding parameters: \(4d^2\) attention plus \(8d^2\) FFN, in each of \(L\) blocks. Kaplan et al. (2020) use exactly this count, \(N \approx 12\, n_{\text{layer}}\, d_{\text{model}}^2\). [Certain]
- \(|V|\)
- the vocabulary size. The input table and the output layer each have \(|V| \cdot d\) numbers. With weight tying they are the same matrix, counted once (session 11).
GPT-2 small has \(L = 12\), \(d = 768\), \(|V| = 50{,}257\), a learned position table of 1,024 rows, and tied embeddings. [Certain] It also has biases on every linear layer and LayerNorms with scale and shift.
Attention per block: \(4d^2 + 4d = 2{,}362{,}368\) (weights plus biases). FFN per block: \(8d^2 + 4d + d = 4{,}722{,}432\). Two LayerNorms: \(4d = 3{,}072\). One block: \(7{,}087{,}872\). Twelve blocks: \(85{,}054{,}464\). Token table: \(50{,}257 \times 768 = 38{,}597{,}376\). Position table: \(1{,}024 \times 768 = 786{,}432\). Final LayerNorm: \(1{,}536\). Total: 124,439,808, the famous “124M”.
The quick estimate: \(12 L d^2 = 84{,}934{,}656\) plus the two tables gives 124,318,464. It misses only the biases and norms: 0.1%.
Modern blocks: GQA and SwiGLU change the constants
Models from 2023 on, like Llama 3, Mistral and Qwen, change two parts of the block. The “12” moves a little, so it is worth knowing why.
- Grouped-query attention (GQA). The model keeps \(h\) query heads but only \(h_{kv}\) key/value heads, shared by groups of query heads (session 26). Then \(W_K\) and \(W_V\) are \(d \times h_{kv} d_h\) instead of \(d \times d\). Llama 3 8B has \(h = 32\) and \(h_{kv} = 8\), so \(W_K\) and \(W_V\) are each a quarter of \(d^2\). [Certain]
- SwiGLU FFN. A gated FFN has three matrices instead of two: a gate and an “up” matrix, both \(d \times d_{\text{ff}}\), and a “down” matrix \(d_{\text{ff}} \times d\) (Shazeer, 2020). That is \(3\, d\, d_{\text{ff}}\) weights. To keep the classic \(8d^2\), designers pick \(d_{\text{ff}} \approx \tfrac{8}{3} d\). Llama 2 7B uses \(d_{\text{ff}} = 11{,}008 \approx 2.69d\). Llama 3 8B uses a larger \(d_{\text{ff}} = 14{,}336 = 3.5d\). [Certain]
- No biases, RMSNorm. The Llama family has no biases in its linear layers and uses RMSNorm (\(d\) numbers each, two per block, plus one at the end). [Certain]
The general count per block becomes:
- Read the symbols
- \(h, d_h\)
- the number of query heads and the size of one head; usually \(h \cdot d_h = d\)
- \(h_{kv}\)
- the number of key/value heads. \(h_{kv} = h\) is classic multi-head attention, \(h_{kv} = 1\) is multi-query attention, anything in between is GQA.
- \(d_{\text{ff}}\)
- the inner width of the FFN (
intermediate_sizein a config file)
For Llama 3 8B: attention is \(2d^2 + 2 \cdot d \cdot d/4 = 2.5 d^2\) and the FFN is \(3 \times 3.5 d^2 = 10.5 d^2\). So the block has about \(13 d^2\), not \(12 d^2\). GQA saved \(1.5d^2\) in attention, and the bigger FFN spent \(2.5d^2\) more. This is why the quick rule \(12Ld^2\) is about 8% low for Llama 3 blocks. A full count is in the worked example below and in Figure 5.
How to read Figure 1
Start with Llama 3 8B: the exact total is 8,030,261,248, and the quick estimate is 7.49 B, 6.7% low, because the SwiGLU FFN is wider than the classic \(8d^2\). Now move the KV heads slider from 8 up to 32 (full multi-head attention): attention grows from \(2.5d^2\) to \(4d^2\) per layer, and the model gains 805 M parameters. Pick GPT-2 small: the embeddings are almost a third of the model, and the estimate is within 0.1%. Pick Llama 3 70B: the embeddings shrink to 3% of the total. In large models, the \(L d^2\) term is everything.
Three lessons from the calculator:
- Width matters more than depth. Doubling \(L\) doubles the block parameters; doubling \(d\) multiplies them by 4.
- Embeddings matter in small models only. They grow like \(|V| d\), blocks like \(L d^2\). For GPT-2 small they are 31% of the weights; for Llama 3 70B they are 3%.
- Config names differ. The same numbers appear as
hidden_size,n_embdord_model;intermediate_sizeorn_inner;num_key_value_heads. Read them carefully, and check whethertie_word_embeddingsis true.
Counting FLOPs: 2N forward, 6N for training
Forward: 2 FLOPs per weight per token
From formula (25.1), every weight used in a matrix multiply costs 2 FLOPs per token. Almost every weight in a transformer is used in exactly one matrix multiply per token. So:
Two details. First, the input embedding is a lookup, not a multiply, so it costs 0 FLOPs. Strictly, \(N\) here should be the weights used in matrix multiplies: everything minus the input table (for tied models, the table is used once, as the output layer). Kaplan et al. drop all embeddings and use the non-embedding count. [Certain] Second, attention itself has a part that is not a weight: the scores \(QK^\top\) and the weighted sum with \(V\). That part grows with context, and gets its own section below.
Backward: twice the forward
For a linear layer \(Y = XW\), the backward pass (session 4) needs two products, each as large as the forward one:
The first product sends the gradient to the earlier layers. The second gives the gradient of the weights. Each costs 2 FLOPs per weight per token, the same as the forward. So the backward pass costs \(4N\), and one training step costs \(2N + 4N = 6N\) per token. For a whole run on \(D\) tokens:
- Read the symbols
- \(C_{\text{train}}\)
- the total training compute in FLOPs
- \(N\)
- the number of parameters (strictly, those used in matrix multiplies)
- \(D\)
- the number of training tokens processed (tokens seen, counting repeats)
- \(6\)
- 2 for the forward pass plus 4 for the backward pass. Kaplan et al. define \(C \approx 6N\) per training token this way, and Chinchilla uses \(6ND\). [Certain]
\(N = 8.03 \times 10^9\) parameters, trained on more than 15 trillion tokens (Llama 3 paper). [Certain] So \(C \approx 6 \times 8.03 \times 10^9 \times 15 \times 10^{12} = 7.23 \times 10^{23}\) FLOPs.
An H100 SXM GPU has a peak of about 989 TFLOP/s for dense BF16 matrix multiplies (NVIDIA’s datasheet gives 1,979 “with sparsity”, which is twice the dense number). [Certain] Real training reaches perhaps 40% of peak; Meta reports 38% to 43% for Llama 3 405B. [Likely] At 40%: \(7.23 \times 10^{23} / (0.4 \times 989 \times 10^{12}) = 1.83 \times 10^{9}\) GPU-seconds \(\approx\) 507,000 GPU-hours, or about 21 days on 1,024 GPUs.
Compare GPT-3: \(6 \times 175 \times 10^9 \times 300 \times 10^9 = 3.15 \times 10^{23}\) FLOPs. The GPT-3 paper reports 3,640 petaflop/s-days, which is \(3.14 \times 10^{23}\). [Likely] So the 8B model of 2024 used more than twice the compute of the 175B model of 2020, because it saw 50 times more tokens.
To turn FLOPs into time, divide by the speed you actually reach:
- Read the symbols
- \(n_{\text{GPU}}\)
- the number of GPUs
- \(P_{\text{peak}}\)
- the peak FLOP/s of one GPU for your number format (dense, not “with sparsity”)
- MFU
- model FLOPs utilization: the useful FLOP/s you reach (from the formula, not from the hardware counter) divided by the peak. The PaLM paper defined it. [Certain] 35% to 55% is typical for large dense training. [Likely]
How to read Figure 2
The first card is the compute \(6ND\). The others turn it into GPU-hours and days. With the Llama 3 8B preset on 1,024 H100s at 40% MFU you get about 21 days. Switch to A100: the same run takes about 3.2 times longer, because 989/312 = 3.17. Change MFU from 40% to 20%: the days double. This is why engineers fight for every percent of MFU: at this scale, 10 points of MFU are weeks of a large cluster.
The attention term grows with context
The \(2N\) rule counts weights. But attention also multiplies activations with activations: the scores \(S = QK^\top\) and the output \(SV\) (session 17). These products have no weights, so \(2N\) misses them. Their size depends on how many tokens each query looks at.
Take one query token in a sequence of \(T\) tokens, in one layer. Its score against all \(T\) keys is \(T\) dot products of length \(d_h\), for each of the \(h\) heads: \(2 \cdot T \cdot h d_h = 2Td\) FLOPs. The weighted sum of the \(T\) values costs the same again. So per token, per layer, attention adds \(4Td\), and over \(L\) layers:
- Read the symbols
- \(T\)
- the context length: the number of keys each query attends to
- \(d\)
- here, \(h \cdot d_h\), the total query width (equal to the model width in most models)
- \(4\)
- 2 FLOPs per multiply-add, times 2 products (\(QK^\top\) and \(SV\))
- \(12\)
- 3 times the forward (forward plus a backward twice as large). PaLM writes the training cost per token as \(6N + 12LHQT\), with \(H\) heads of size \(Q\). [Certain]
Two remarks. With a causal mask, token \(t\) only looks at \(t\) keys, so on average half of the score matrix is masked. A kernel that skips masked blocks (FlashAttention does this, session 124) does about half of (25.10). Kaplan et al. use \(2LTd\) for this reason; PaLM counts the full \(4LTd\). [Certain] And GQA does not reduce these FLOPs: every query head still computes its own scores. GQA reduces the memory of K and V, not the arithmetic.
Now compare the two terms. With \(N \approx 12Ld^2\):
So attention-score FLOPs equal the weight FLOPs when \(T = 6d\): about 24,600 tokens for \(d = 4096\). Below that, the weights dominate; above it, attention does. With the causal saving the crossover moves to \(T = 12d\), which matches Kaplan’s remark that the context term is small when \(d_{\text{model}} \gg n_{\text{ctx}}/12\). [Certain] Note that the term is quadratic in \(T\) for a whole sequence (\(T\) tokens times \(4LTd\) each) but only linear in \(T\) per token.
How to read Figure 3
For Llama 3 8B at 8,192 tokens, attention is about 22% of the forward FLOPs. At 32,768 tokens it is more than half, and at about one million tokens it is 97%. Tick the causal box: every share drops, because the masked half is skipped, and the 50% point moves twice as far to the right. Switch to GPT-2 small (\(d = 768\)): the curve moves left; at its own context of 1,024 tokens, attention is already about 13% of the FLOPs in this count. Wide models push the crossover to longer contexts, because weight FLOPs grow like \(d^2\) and attention FLOPs only like \(d\).
Counting memory
FLOPs tell you how long. Memory tells you whether it runs at all. Memory comes in four parts: weights, gradients and optimizer states (training only), activations (training), and the KV cache (inference).
Weights and optimizer states
Each parameter takes 2 bytes in BF16 or FP16, 4 bytes in FP32, 1 byte in INT8 or FP8 (session 37). Llama 3 8B in BF16 is \(8.03 \times 10^9 \times 2 = 16.1\) GB (15.0 GiB). For training with AdamW in mixed precision, session 4 showed 16 bytes per parameter:
That is 128 GB for an 8B model and about 1.13 TB for a 70B model, before a single activation. [Certain for the arithmetic; the exact byte layout varies between frameworks.] This is the memory that ZeRO and FSDP split across GPUs (session 45).
Activations
Backpropagation needs the inputs of every layer, saved from the forward pass. This memory grows with batch size \(b\) and sequence length \(s\), not with \(N\). Korthikanti et al. (2022) count it for a GPT-style block, stored in 16-bit, with no parallelism:
- Read the symbols
- \(s, b\)
- sequence length and micro-batch size (sequences per step on one GPU)
- \(34\, s b d\)
- the inputs of the linear layers, norms, the GELU and the dropout masks: about 34 bytes per number of the residual stream, per layer
- \(5 h s \cdot s b\)
- the attention score matrices: \(h\) heads of \(s \times s\), stored before and after softmax and for dropout. This is the part that grows like \(s^2\).
The paper also shows that recomputing only the attention core in the backward pass (“selective recomputation”) removes the \(5hs/d\) term. FlashAttention has the same effect, because it never stores the score matrix. [Certain] The constant 34 is for the GPT-style block of the paper (\(4\times\) GELU FFN, dropout); a Llama-style block has a different constant. [Likely]
Take \(L = 32\), \(d = 4096\), \(h = 32\), \(s = 4096\), \(b = 1\). Per layer: \(4096 \times 1 \times 4096 \times (34 + 5 \times 32 \times 4096/4096) = 16{,}777{,}216 \times 194 = 3.25\) GB. For 32 layers: 104 GB, for one sequence! Without the score matrices (FlashAttention or selective recomputation): \(16{,}777{,}216 \times 34 = 0.57\) GB per layer, 18.3 GB in total. The score matrices were 82% of the activations.
The KV cache at inference
At inference there are no gradients and no optimizer. But to generate token \(t+1\), every layer needs the keys and values of all earlier tokens. We keep them in the KV cache (session 80). For each token, each layer stores one key and one value vector per KV head:
- Read the symbols
- \(2\)
- one key and one value
- \(h_{kv} d_h\)
- the width of K (and of V). With GQA this is smaller than \(d\): for Llama 3 8B, \(8 \times 128 = 1024\).
- \(T, b\)
- tokens per sequence (prompt plus generated) and the number of sequences in the batch
- bytes
- 2 for BF16, 1 for an 8-bit cache (session 110)
For Llama 3 8B in BF16: \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes = 128 KiB per token. An 8,192-token conversation needs 1 GiB; 32 such users need 32 GiB, twice the weights. With full multi-head attention (\(h_{kv} = 32\)) it would be four times more: 512 KiB per token. This is the whole motivation of session 26.
How to read Figure 4
In training mode, even with batch 1 the bar is far past 80 GB: the 128 GB of weights, gradients and Adam states alone do not fit. That is why full fine-tuning of an 8B model needs several GPUs with sharding, or LoRA. Untick FlashAttention: at 4,096 tokens the activations jump from 18 GB to 104 GB. Switch to inference: the 16 GB of weights fit easily; now raise the batch to 32 and the length to 8,192: the KV cache (32 GiB) is twice the weights. At inference, memory is about the cache, not the model.
Decode is limited by memory, not FLOPs
One more count connects FLOPs and memory. When generating one token for one user, the GPU must read all 16.1 GB of weights to do \(2N \approx 16\) GFLOPs. That is about 1 FLOP per byte read. An H100 SXM can do 989 TFLOP/s but read only about 3.35 TB/s from its memory, so it needs about 295 FLOPs per byte to be busy. [Certain for the datasheet numbers] At batch 1, decode is limited by memory speed: at best \(3.35 \times 10^{12} / 16.06 \times 10^9 \approx 209\) tokens per second. Batching many users reuses each weight read for many tokens, which is why serving systems batch (session 82).
Worked example: estimate an 8B model, then check its config
Here is the full exercise that interviewers like. You are given only this config (the real fields of Llama 3 8B’s config.json [Certain]):
{ "hidden_size": 4096, "intermediate_size": 14336, "num_hidden_layers": 32,
"num_attention_heads": 32, "num_key_value_heads": 8,
"vocab_size": 128256, "tie_word_embeddings": false, "max_position_embeddings": 8192 }
Step 1, the quick estimate. \(12 L d^2 = 12 \times 32 \times 4096^2 = 6.44\) B. Two untied tables: \(2 \times 128{,}256 \times 4096 = 1.05\) B. Estimate: 7.49 B.
Step 2, the exact count. Head size \(d_h = 4096/32 = 128\); K/V width \(8 \times 128 = 1024\). Per layer: \(W_Q\) and \(W_O\): \(2 \times 4096^2 = 33{,}554{,}432\). \(W_K\) and \(W_V\): \(2 \times 4096 \times 1024 = 8{,}388{,}608\). FFN: \(3 \times 4096 \times 14{,}336 = 176{,}160{,}768\). Two RMSNorms: 8,192. One layer: 218,112,000. Times 32: 6,979,584,000. Plus two tables of 525,336,576 and the final norm of 4,096: 8,030,261,248.
Step 3, check. The published name is “8B”, and the exact count matches the parameter count of the released checkpoint (the PyTorch code below builds the same shapes and gets the same number). [Certain] The quick estimate was 6.7% low; we now know why: the FFN is \(10.5d^2\) instead of \(8d^2\), partly paid back by GQA (\(2.5d^2\) instead of \(4d^2\)).
Consequences. Forward: \(2 \times 7.50\) B \(\approx 15.0\) GFLOPs per token for the weights (the 525 M input table is a lookup). Training: about \(6 \times 7.50\) B \(\approx 45\) GFLOPs per token. Memory: 16.1 GB in BF16, 128 GB for AdamW training states, 128 KiB of KV cache per token.
How to read Figure 5
Press Next. The highlighted row is the group being added; the readout shows the formula with the config values filled in. Watch the running total: after one layer it is only 0.74 B (most of it the embedding table), after all 32 layers it is 7.50 B, and the output layer brings it to 8,030,261,248. The last two steps reuse the count: \(2N\) and \(6N\) FLOPs per token, and the bytes for weights, Adam and the KV cache.
Pseudo-code
- input: \(d, L, h, h_{kv}, d_{\text{ff}}, |V|\), tied, gated, biases
- \(d_h \leftarrow d / h\)
- \(a \leftarrow 2 d (h d_h) + 2 d (h_{kv} d_h)\) // Q, O and K, V, formula (25.5)
- if gated then \(f \leftarrow 3 d d_{\text{ff}}\) else \(f \leftarrow 2 d d_{\text{ff}}\) // SwiGLU or classic FFN
- \(n \leftarrow\) norm and bias sizes (a few times \(d\) per block)
- \(N_{\text{blocks}} \leftarrow L \cdot (a + f + n)\)
- \(N_{\text{emb}} \leftarrow |V| d\) if tied else \(2 |V| d\) // formula (25.4)
- return \(N_{\text{blocks}} + N_{\text{emb}} +\) final norm
- input: \(N\), tokens \(D\), context \(T\), \(L, d\), GPUs \(n\), peak \(P\), MFU \(u\)
- \(c \leftarrow 6N + 12 L T d\) // FLOPs per token, (25.8) and (25.10)
- \(C \leftarrow c \cdot D\)
- days \(\leftarrow C / (n \cdot P \cdot u) / 86{,}400\) // formula (25.9)
- states \(\leftarrow 16 N\) bytes // formula (25.12); divide by \(n\) with ZeRO-3 or FSDP
- activations \(\leftarrow L \cdot s b d \cdot (34 + 5 h s / d)\) bytes // (25.13); drop the second term with FlashAttention
- if states / \(n\) + activations \(\gt\) GPU memory then shard more, lower \(b\), or recompute activations
- return \(C\), days, memory per GPU
The same counts in PyTorch. The model is a small Llama-style decoder (RMSNorm, GQA, SwiGLU, untied output). PyTorch’s FlopCounterMode counts the FLOPs of every matrix multiply, so we can check formulas (25.5), (25.6), (25.7) and (25.10) exactly:
import torch import torch.nn as nn import torch.nn.functional as F from torch.utils.flop_counter import FlopCounterMode torch.manual_seed(0) class Block(nn.Module): # Llama-style: RMSNorm, GQA attention, SwiGLU def __init__(s, d, h, kv, ff): super().__init__() s.h, s.kv, s.dh = h, kv, d // h s.n1, s.n2 = nn.RMSNorm(d), nn.RMSNorm(d) s.q = nn.Linear(d, h * s.dh, bias=False) s.k = nn.Linear(d, kv * s.dh, bias=False) s.v = nn.Linear(d, kv * s.dh, bias=False) s.o = nn.Linear(h * s.dh, d, bias=False) s.gate = nn.Linear(d, ff, bias=False) s.up = nn.Linear(d, ff, bias=False) s.down = nn.Linear(ff, d, bias=False) def forward(s, x): B, T, d = x.shape a = s.n1(x) q = s.q(a).view(B, T, s.h, s.dh).transpose(1, 2) k = s.k(a).view(B, T, s.kv, s.dh).transpose(1, 2).repeat_interleave(s.h // s.kv, 1) v = s.v(a).view(B, T, s.kv, s.dh).transpose(1, 2).repeat_interleave(s.h // s.kv, 1) att = (q @ k.transpose(-1, -2)) / s.dh ** 0.5 # (B, h, T, T): 2*T*T*d FLOPs mask = torch.ones(T, T, dtype=torch.bool).tril() att = att.masked_fill(~mask, float("-inf")).softmax(-1) y = (att @ v).transpose(1, 2).reshape(B, T, d) # another 2*T*T*d FLOPs x = x + s.o(y) b = s.n2(x) return x + s.down(F.silu(s.gate(b)) * s.up(b)) class LM(nn.Module): def __init__(s, V, d, L, h, kv, ff): super().__init__() s.emb = nn.Embedding(V, d) s.blocks = nn.ModuleList(Block(d, h, kv, ff) for _ in range(L)) s.norm = nn.RMSNorm(d) s.head = nn.Linear(d, V, bias=False) # untied output layer def forward(s, ids): x = s.emb(ids) for blk in s.blocks: x = blk(x) return s.head(s.norm(x)) def count_formula(V, d, L, h, kv, ff): # formula (25.5) plus tables dh = d // h per_layer = 2 * d * d + 2 * d * kv * dh + 3 * d * ff + 2 * d return L * per_layer + d + 2 * V * d # 1) parameter count: real modules vs formula cfg = dict(V=1000, d=256, L=4, h=8, kv=2, ff=704) m = LM(**cfg) n = sum(p.numel() for p in m.parameters()) print(n, count_formula(**cfg)) # 3332352 3332352 # 2) FLOPs of one forward and one forward+backward vs 2N and 6N B, T = 2, 128 ids = torch.randint(0, cfg["V"], (B, T)) N_mm = n - cfg["V"] * cfg["d"] - (2 * cfg["L"] + 1) * cfg["d"] # weights used in matmuls attn = 4 * cfg["L"] * T * cfg["d"] # QK^T and AV per token, full T x T with FlopCounterMode(display=False) as fc: m(ids) fwd = fc.get_total_flops() / (B * T) print(fwd, 2 * N_mm + attn) # 6672384.0 6672384 with FlopCounterMode(display=False) as fc: m(ids).float().logsumexp(-1).mean().backward() print(fc.get_total_flops() / (B * T), 3 * (2 * N_mm + attn)) # 20017152.0 20017152: exactly 3x # 3) Llama 3 8B on the meta device: shapes only, no memory is allocated with torch.device("meta"): big = LM(V=128256, d=4096, L=32, h=32, kv=8, ff=14336) print(sum(p.numel() for p in big.parameters())) # 8030261248
Notice that FlopCounterMode counts the full \(T \times T\) score matrix even with a mask: masking does not skip any multiply in this plain code. Only a causal-aware kernel saves the half. The printed values were produced with PyTorch 2.8 on CPU.
Where this is used
| Application | Which count | Link |
|---|---|---|
| Choosing model size and token count for a budget | \(C = 6ND\) with a scaling law; Chinchilla found about 20 tokens per parameter compute-optimal [Certain] | session 35 |
| Measuring training efficiency | \(\text{MFU} = (\text{tokens/s} \times 6N) / \text{peak}\); HFU adds recomputation | session 51 |
| Fitting training on GPUs | 16 bytes per parameter (25.12) plus activations (25.13); decides ZeRO stage, FSDP, checkpointing | session 38 |
| Serving capacity planning | Weights + KV cache (25.14) per user; decides batch size and how many users per GPU | session 81 |
| Long-context and video VLMs | The attention term \(4LTd\) per token: with thousands of visual tokens per clip, attention becomes a large share of prefill (video QA, Video-RAG over long recordings) | session 28 |
| Video retrieval and fingerprinting at scale | Encoder cost per clip \(= 2N \times\) tokens per clip; multiply by hours of video per day to size the indexing fleet | session 208 |
| Choosing attention variants and quantization | KV-cache bytes per token for MHA vs GQA vs MLA; bytes per weight for INT8/INT4 | session 26 |
Common mistakes
- Counting the number of heads in the parameter count. \(W_Q\) is \(d \times d\) whatever \(h\) is. Heads change the grouping, not the size. Only \(h_{kv}\) (GQA) changes the count of \(W_K\) and \(W_V\).
- Forgetting the untied output layer. Llama 3 8B has two 525 M tables. Count one and you get 7.5 B, not 8.03 B.
- Using \(8d^2\) for a SwiGLU FFN. A gated FFN has three matrices: \(3 d\, d_{\text{ff}}\). Read
intermediate_sizefrom the config. - Mixing FLOPs and FLOP/s, or using the “with sparsity” peak. Datasheets often print the 2:4-sparse number, which is twice the dense one. Your MFU will look twice as bad as it is, or your time estimate twice as optimistic.
- Saying 2N per token for training. Training is 6N: forward 2N plus backward 4N. And 8N of hardware work with full activation recomputation.
- Ignoring the attention term at long context. At \(T = 6d\) it equals the weight FLOPs (full count). For a 128k-context run with \(d = 4096\), it is the larger part.
- Thinking GQA reduces attention FLOPs. It reduces the K/V projection weights and the KV cache. Every query head still computes \(T\) scores.
- Confusing GB and GiB. 16.06 GB is 14.96 GiB. Tools print both; a 7% gap matters when you are close to the limit.
Interview questions
How many FLOPs is a matrix multiply of an \((m \times k)\) matrix and a \((k \times n)\) matrix?basic
Derive the parameter count of one transformer layer with width \(d\).basic
Count the parameters of Llama 3 8B from its config.core
Why does training cost about 6N FLOPs per token?core
When does attention dominate the FLOPs of a transformer?core
Estimate the training compute and time for an 8B model on 15T tokens.core
How much GPU memory do you need to fully fine-tune a 7B model with AdamW?core
How big is the KV cache of Llama 3 8B, and how does GQA change it?core
Why is single-user decoding slow even though it needs only about 16 GFLOPs per token?advanced
What are MFU and HFU, and why can activation checkpointing make MFU drop?advanced
Why are embedding parameters often left out of scaling-law counts?advanced
Exercises
- A model has \(d = 2048\), \(L = 24\), classic attention and a \(4\times\) GELU FFN, \(|V| = 50{,}000\), tied. Estimate its parameter count.
- Estimate the training compute of GPT-3 (175 B parameters, 300 B tokens). Compare it with Chinchilla (70 B parameters, 1.4 T tokens).
- Llama 2 7B has \(L = 32\), \(h = h_{kv} = 32\), \(d_h = 128\). What is its KV cache per token in BF16? For one 4,096-token sequence?
- For a model with \(d = 4096\), what is the ratio of attention-score FLOPs to weight FLOPs at \(T = 32{,}768\), using (25.11)? What share of the forward FLOPs is attention?
- How much memory do the AdamW training states of a 1 B model need? Of a 70 B model?
- In Figure 4 (training, FlashAttention on), what happens to the activation memory when you double the sequence length? And with FlashAttention off?
- Show that a SwiGLU FFN with \(d_{\text{ff}} = \tfrac{8}{3}d\) has the same parameter count as a classic FFN with \(d_{\text{ff}} = 4d\).
Answerscheck
Key takeaways
- Every count starts from one rule: a matrix multiply of \((m \times k)(k \times n)\) costs \(2mkn\) FLOPs (25.1), so each weight costs 2 FLOPs per token.
- Parameters: attention \(4d^2\) plus FFN \(8d^2\), so \(N \approx 12Ld^2\) plus \(|V|d\) per embedding table (25.4). GQA and SwiGLU move the 12 (Llama 3 8B has about \(13d^2\) per layer); the exact count is 8,030,261,248.
- Compute: about \(2N\) per token forward (25.6), \(6N\) per token in training (25.7, 25.8), so \(C \approx 6ND\). Time = \(6ND / (n \cdot P \cdot \mathrm{MFU})\) (25.9).
- Attention scores add \(4LTd\) per token forward (25.10). They match the weight FLOPs at \(T = 6d\) (25.11), or \(12d\) with causal skipping.
- Memory: 2 bytes per parameter for BF16 weights, 16 for AdamW training (25.12), activations that grow with \(s\) and \(s^2\) (25.13), and a KV cache of \(2Lh_{kv}d_h\) values per token (25.14).
- Decode at batch 1 is limited by memory bandwidth, not FLOPs: about 1 FLOP per byte of weights read.
Key formulas
References for this session
- Kaplan et al. (2020). Scaling Laws for Neural Language Models
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Stanford (2025). CS336: Language Modeling from Scratch (course)
- Hoffmann et al. (2022). Training Compute-Optimal Large Language Models
- Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
- Brown et al. (2020). Language Models are Few-Shot Learners
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Meta (mirrored by Unsloth) (2024). Llama 3 8B config.json on Hugging Face
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Shazeer (2020). GLU Variants Improve Transformer
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Micikevicius et al. (2017). Mixed Precision Training