LLM & VLM courseBlog
25Track 2: Attention and transformers · session 10 of 15

Counting: parameters, FLOPs and memory of a transformer

Before you train or serve a model, you can predict its size, its compute bill and its memory needs with a few lines of arithmetic. This session builds those formulas from one matrix multiplication: about 12·d² parameters per layer, 2N FLOPs per token forward, 6N per token in training, plus an attention term that grows with context. We then estimate an 8B model by hand and check the result against its real config file.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 4, 11, 19 and 21 to 23 (backprop and AdamW memory, embeddings, multi-head attention, the FFN, the full transformer)

What you will learn

  • Why a matrix multiplication of shapes (m × k) and (k × n) costs 2mkn FLOPs, and why every other count follows from this
  • Parameters per layer: attention 4d², FFN 8d², so about 12·L·d² plus the embedding tables, and how GQA and SwiGLU change the constants
  • Forward cost of about 2N FLOPs per token, training cost of about 6N per token, so C ≈ 6ND for a whole run
  • The attention term 4·L·T·d per token, which grows with context length and passes the weight FLOPs near T = 6d
  • Memory: bytes per parameter, 16 bytes per parameter for AdamW training, activation memory, and the KV cache
  • How to turn a FLOP count into GPU-days with model FLOPs utilization (MFU)
  • A full worked example: Llama 3 8B estimated by hand and checked against its config.json

Why this matters

Before anyone trains or serves a large model, they ask three questions. How many numbers (parameters) does it have? How much arithmetic (FLOPs) does it need? How much GPU memory does it need? You can answer all three on one sheet of paper, from the model’s config file and a few rules. This session gives you those rules.

The rules are short. A transformer layer of width \(d\) has about \(12d^2\) parameters. A forward pass costs about \(2N\) floating-point operations (FLOPs) per token, where \(N\) is the number of parameters. Training costs about \(6N\) per token, so a whole training run on \(D\) tokens costs about \(6ND\). Attention adds an extra term that grows with the context length. Memory is bytes per parameter, plus activations, plus (at inference) the KV cache.

  • Planning. A team with 1,024 GPUs and one month must pick a model size and a token count that fit. The formula \(C \approx 6ND\) does that in one line. Scaling laws (session 35) are written in exactly these units.
  • Debugging. If your training run reaches 15% of the GPU’s peak speed, is that bad? You can only tell if you know how many FLOPs one step should take. This ratio is called MFU (session 51).
  • Serving. Will an 8B model with 32 users at 8k context fit on one 80 GB GPU? Weights plus KV cache tells you in a minute (session 80).
  • Interviews. “How many parameters does Llama 3 8B have, from its config?”, “Why 6N?”, “How much memory to fine-tune a 7B model with Adam?”, “When does attention dominate the compute?”. These are among the most common LLM interview questions. By the end you can answer each one with a formula and a number.

In Session 23 we put the pieces together into one block and one stack. In Session 24 we saw a second family with cross-attention. Now we count everything inside a decoder-only stack. The next session, Session 26, uses these counts to explain why MQA, GQA and MLA exist: they shrink the KV cache that we measure here.

The one rule behind all counts: a matrix multiply

Almost all compute in a transformer is matrix multiplication. So we start there. Take \(A\) with shape \(m \times k\) and \(B\) with shape \(k \times n\). Each of the \(m \cdot n\) entries of \(AB\) is a dot product of length \(k\): \(k\) multiplications and \(k\) additions (strictly \(k-1\), but we round). One multiplication plus one addition is called a multiply-add, and we count it as 2 FLOPs:

\[ \mathrm{FLOPs}(A B) = 2\,m\,k\,n, \qquad A \in \R^{m \times k},\; B \in \R^{k \times n} \tag{25.1} \]
Read the symbols
FLOP
one floating-point operation: one addition or one multiplication. “FLOPs” (lower-case s) is a count; “FLOP/s” is a speed (operations per second).
\(m\)
the number of rows of the input. For a linear layer applied to a batch, \(m\) is the number of tokens.
\(k\)
the shared (inner) size that is summed over
\(n\)
the number of output columns

Now apply it to a linear layer \(\mathbf{y} = \mathbf{x} W\) with \(W \in \R^{d_{\text{in}} \times d_{\text{out}}}\), on one token. Here \(m = 1\), so the cost is \(2\, d_{\text{in}}\, d_{\text{out}}\). But \(d_{\text{in}} \cdot d_{\text{out}}\) is exactly the number of weights in \(W\). So:

Each weight of a linear layer costs 2 FLOPs per token in the forward pass: one multiply and one add.

This single sentence explains the “2N” rule later. Everything that is not a matrix multiply (softmax, normalization, activation functions, adding residuals) costs a few FLOPs per number in the activation, not per weight. With \(d\) in the thousands, these are small, and we ignore them. [Certain] They still take time on a GPU, because they read and write memory; that is a different story (session 82).

Worked example: one projection

Take \(d = 4096\) and the query projection \(W_Q \in \R^{4096 \times 4096}\). It has \(4096^2 = 16{,}777{,}216\) weights. For one token the forward cost is \(2 \times 16{,}777{,}216 = 33{,}554{,}432\) FLOPs, about 33.6 MFLOPs. For a batch of 8 sequences of 4,096 tokens (\(m = 32{,}768\)) it is \(2 \times 32{,}768 \times 4096 \times 4096 \approx 1.10 \times 10^{12}\) FLOPs: about one second of work at 1 TFLOP/s, or about one millisecond on a modern GPU.

Counting parameters

One block, the classic way

Recall the decoder block from session 23: a normalization, multi-head attention, a residual add, another normalization, the feed-forward network (FFN), another residual add. We count the weights of each part, with model width \(d\).

Attention. There are four matrices: \(W_Q, W_K, W_V\) project the input to queries, keys and values, and \(W_O\) projects the concatenated heads back to width \(d\) (session 19). With \(h\) heads of size \(d_h = d/h\), each projection is \(d \times d\) in total, no matter how many heads. So:

\[ N_{\text{attn}} = \underbrace{d^2}_{W_Q} + \underbrace{d^2}_{W_K} + \underbrace{d^2}_{W_V} + \underbrace{d^2}_{W_O} = 4d^2 \tag{25.2} \]

The number of heads does not change the count. Splitting \(d\) into heads only changes how the numbers are grouped, not how many there are. [Certain]

FFN. The classic FFN (session 21) goes up from \(d\) to \(d_{\text{ff}} = 4d\), applies an activation, and comes back down: two matrices of size \(d \times 4d\):

\[ N_{\text{ffn}} = \underbrace{d \cdot 4d}_{\text{up}} + \underbrace{4d \cdot d}_{\text{down}} = 8d^2 \tag{25.3} \]

Normalization. Each LayerNorm has a scale and a shift vector (\(2d\) numbers); an RMSNorm has only a scale (\(d\) numbers). These are tiny: \(2d\) to \(4d\) per block against \(12d^2\). Biases, where models have them, are also of size \(d\) or \(4d\). We ignore all of them in the estimate.

So one block has \(4d^2 + 8d^2 = 12d^2\) weights. A stack of \(L\) blocks has:

\[ N \;\approx\; 12\,L\,d^2 \;+\; \underbrace{|V|\,d}_{\text{input table}} \;+\; \underbrace{|V|\,d}_{\text{output layer, if untied}} \tag{25.4} \]
Read the symbols
\(L\)
the number of transformer blocks (layers)
\(d\)
the model width: the length of the vector each token carries through the residual stream (hidden_size in a config file)
\(12 L d^2\)
the non-embedding parameters: \(4d^2\) attention plus \(8d^2\) FFN, in each of \(L\) blocks. Kaplan et al. (2020) use exactly this count, \(N \approx 12\, n_{\text{layer}}\, d_{\text{model}}^2\). [Certain]
\(|V|\)
the vocabulary size. The input table and the output layer each have \(|V| \cdot d\) numbers. With weight tying they are the same matrix, counted once (session 11).
Worked example: GPT-2 small, exactly

GPT-2 small has \(L = 12\), \(d = 768\), \(|V| = 50{,}257\), a learned position table of 1,024 rows, and tied embeddings. [Certain] It also has biases on every linear layer and LayerNorms with scale and shift.

Attention per block: \(4d^2 + 4d = 2{,}362{,}368\) (weights plus biases). FFN per block: \(8d^2 + 4d + d = 4{,}722{,}432\). Two LayerNorms: \(4d = 3{,}072\). One block: \(7{,}087{,}872\). Twelve blocks: \(85{,}054{,}464\). Token table: \(50{,}257 \times 768 = 38{,}597{,}376\). Position table: \(1{,}024 \times 768 = 786{,}432\). Final LayerNorm: \(1{,}536\). Total: 124,439,808, the famous “124M”.

The quick estimate: \(12 L d^2 = 84{,}934{,}656\) plus the two tables gives 124,318,464. It misses only the biases and norms: 0.1%.

Modern blocks: GQA and SwiGLU change the constants

Models from 2023 on, like Llama 3, Mistral and Qwen, change two parts of the block. The “12” moves a little, so it is worth knowing why.

  • Grouped-query attention (GQA). The model keeps \(h\) query heads but only \(h_{kv}\) key/value heads, shared by groups of query heads (session 26). Then \(W_K\) and \(W_V\) are \(d \times h_{kv} d_h\) instead of \(d \times d\). Llama 3 8B has \(h = 32\) and \(h_{kv} = 8\), so \(W_K\) and \(W_V\) are each a quarter of \(d^2\). [Certain]
  • SwiGLU FFN. A gated FFN has three matrices instead of two: a gate and an “up” matrix, both \(d \times d_{\text{ff}}\), and a “down” matrix \(d_{\text{ff}} \times d\) (Shazeer, 2020). That is \(3\, d\, d_{\text{ff}}\) weights. To keep the classic \(8d^2\), designers pick \(d_{\text{ff}} \approx \tfrac{8}{3} d\). Llama 2 7B uses \(d_{\text{ff}} = 11{,}008 \approx 2.69d\). Llama 3 8B uses a larger \(d_{\text{ff}} = 14{,}336 = 3.5d\). [Certain]
  • No biases, RMSNorm. The Llama family has no biases in its linear layers and uses RMSNorm (\(d\) numbers each, two per block, plus one at the end). [Certain]

The general count per block becomes:

\[ N_{\text{block}} = \underbrace{2\,d\,(h\,d_h)}_{W_Q,\;W_O} + \underbrace{2\,d\,(h_{kv}\,d_h)}_{W_K,\;W_V} + \underbrace{3\,d\,d_{\text{ff}}}_{\text{SwiGLU}} + \underbrace{2d}_{\text{two RMSNorms}} \tag{25.5} \]
Read the symbols
\(h, d_h\)
the number of query heads and the size of one head; usually \(h \cdot d_h = d\)
\(h_{kv}\)
the number of key/value heads. \(h_{kv} = h\) is classic multi-head attention, \(h_{kv} = 1\) is multi-query attention, anything in between is GQA.
\(d_{\text{ff}}\)
the inner width of the FFN (intermediate_size in a config file)

For Llama 3 8B: attention is \(2d^2 + 2 \cdot d \cdot d/4 = 2.5 d^2\) and the FFN is \(3 \times 3.5 d^2 = 10.5 d^2\). So the block has about \(13 d^2\), not \(12 d^2\). GQA saved \(1.5d^2\) in attention, and the bigger FFN spent \(2.5d^2\) more. This is why the quick rule \(12Ld^2\) is about 8% low for Llama 3 blocks. A full count is in the worked example below and in Figure 5.

Figure 1. A parameter calculator. Each preset loads a real model’s configuration (from its public config file). The bar splits all parameters into attention, FFN, norms and biases, and the embedding tables. When you change \(d\), the head size and the ratio \(d_{\text{ff}}/d\) of the preset are kept. The readout compares the exact count with the quick estimate \(12Ld^2 + \) tables.
How to read Figure 1

Start with Llama 3 8B: the exact total is 8,030,261,248, and the quick estimate is 7.49 B, 6.7% low, because the SwiGLU FFN is wider than the classic \(8d^2\). Now move the KV heads slider from 8 up to 32 (full multi-head attention): attention grows from \(2.5d^2\) to \(4d^2\) per layer, and the model gains 805 M parameters. Pick GPT-2 small: the embeddings are almost a third of the model, and the estimate is within 0.1%. Pick Llama 3 70B: the embeddings shrink to 3% of the total. In large models, the \(L d^2\) term is everything.

Three lessons from the calculator:

  1. Width matters more than depth. Doubling \(L\) doubles the block parameters; doubling \(d\) multiplies them by 4.
  2. Embeddings matter in small models only. They grow like \(|V| d\), blocks like \(L d^2\). For GPT-2 small they are 31% of the weights; for Llama 3 70B they are 3%.
  3. Config names differ. The same numbers appear as hidden_size, n_embd or d_model; intermediate_size or n_inner; num_key_value_heads. Read them carefully, and check whether tie_word_embeddings is true.

Counting FLOPs: 2N forward, 6N for training

Forward: 2 FLOPs per weight per token

From formula (25.1), every weight used in a matrix multiply costs 2 FLOPs per token. Almost every weight in a transformer is used in exactly one matrix multiply per token. So:

\[ C_{\text{forward}} \approx 2N \ \text{FLOPs per token} \tag{25.6} \]

Two details. First, the input embedding is a lookup, not a multiply, so it costs 0 FLOPs. Strictly, \(N\) here should be the weights used in matrix multiplies: everything minus the input table (for tied models, the table is used once, as the output layer). Kaplan et al. drop all embeddings and use the non-embedding count. [Certain] Second, attention itself has a part that is not a weight: the scores \(QK^\top\) and the weighted sum with \(V\). That part grows with context, and gets its own section below.

Backward: twice the forward

For a linear layer \(Y = XW\), the backward pass (session 4) needs two products, each as large as the forward one:

\[ \underbrace{\frac{\partial \mathcal{L}}{\partial X} = \frac{\partial \mathcal{L}}{\partial Y}\, W^\top}_{\text{to pass the gradient back: } 2N} \qquad \underbrace{\frac{\partial \mathcal{L}}{\partial W} = X^\top\, \frac{\partial \mathcal{L}}{\partial Y}}_{\text{to update the weights: } 2N} \tag{25.7} \]

The first product sends the gradient to the earlier layers. The second gives the gradient of the weights. Each costs 2 FLOPs per weight per token, the same as the forward. So the backward pass costs \(4N\), and one training step costs \(2N + 4N = 6N\) per token. For a whole run on \(D\) tokens:

\[ C_{\text{train}} \approx 6\,N\,D \tag{25.8} \]
Read the symbols
\(C_{\text{train}}\)
the total training compute in FLOPs
\(N\)
the number of parameters (strictly, those used in matrix multiplies)
\(D\)
the number of training tokens processed (tokens seen, counting repeats)
\(6\)
2 for the forward pass plus 4 for the backward pass. Kaplan et al. define \(C \approx 6N\) per training token this way, and Chinchilla uses \(6ND\). [Certain]
Worked example: what Llama 3 8B cost to train

\(N = 8.03 \times 10^9\) parameters, trained on more than 15 trillion tokens (Llama 3 paper). [Certain] So \(C \approx 6 \times 8.03 \times 10^9 \times 15 \times 10^{12} = 7.23 \times 10^{23}\) FLOPs.

An H100 SXM GPU has a peak of about 989 TFLOP/s for dense BF16 matrix multiplies (NVIDIA’s datasheet gives 1,979 “with sparsity”, which is twice the dense number). [Certain] Real training reaches perhaps 40% of peak; Meta reports 38% to 43% for Llama 3 405B. [Likely] At 40%: \(7.23 \times 10^{23} / (0.4 \times 989 \times 10^{12}) = 1.83 \times 10^{9}\) GPU-seconds \(\approx\) 507,000 GPU-hours, or about 21 days on 1,024 GPUs.

Compare GPT-3: \(6 \times 175 \times 10^9 \times 300 \times 10^9 = 3.15 \times 10^{23}\) FLOPs. The GPT-3 paper reports 3,640 petaflop/s-days, which is \(3.14 \times 10^{23}\). [Likely] So the 8B model of 2024 used more than twice the compute of the 175B model of 2020, because it saw 50 times more tokens.

To turn FLOPs into time, divide by the speed you actually reach:

\[ \text{time} = \frac{6\,N\,D}{n_{\text{GPU}} \cdot P_{\text{peak}} \cdot \mathrm{MFU}} \tag{25.9} \]
Read the symbols
\(n_{\text{GPU}}\)
the number of GPUs
\(P_{\text{peak}}\)
the peak FLOP/s of one GPU for your number format (dense, not “with sparsity”)
MFU
model FLOPs utilization: the useful FLOP/s you reach (from the formula, not from the hardware counter) divided by the peak. The PaLM paper defined it. [Certain] 35% to 55% is typical for large dense training. [Likely]
Figure 2. From \(6ND\) to calendar days, formula (25.9). Peak numbers are dense BF16 matrix-multiply speeds from NVIDIA’s datasheets (A100: 312 TFLOP/s; H100 SXM: 989 TFLOP/s). The time ignores restarts, evaluation and data loading.
How to read Figure 2

The first card is the compute \(6ND\). The others turn it into GPU-hours and days. With the Llama 3 8B preset on 1,024 H100s at 40% MFU you get about 21 days. Switch to A100: the same run takes about 3.2 times longer, because 989/312 = 3.17. Change MFU from 40% to 20%: the days double. This is why engineers fight for every percent of MFU: at this scale, 10 points of MFU are weeks of a large cluster.

The attention term grows with context

The \(2N\) rule counts weights. But attention also multiplies activations with activations: the scores \(S = QK^\top\) and the output \(SV\) (session 17). These products have no weights, so \(2N\) misses them. Their size depends on how many tokens each query looks at.

Take one query token in a sequence of \(T\) tokens, in one layer. Its score against all \(T\) keys is \(T\) dot products of length \(d_h\), for each of the \(h\) heads: \(2 \cdot T \cdot h d_h = 2Td\) FLOPs. The weighted sum of the \(T\) values costs the same again. So per token, per layer, attention adds \(4Td\), and over \(L\) layers:

\[ C_{\text{attn}} = 4\,L\,T\,d \ \text{FLOPs per token (forward)}, \qquad 12\,L\,T\,d \ \text{(training)} \tag{25.10} \]
Read the symbols
\(T\)
the context length: the number of keys each query attends to
\(d\)
here, \(h \cdot d_h\), the total query width (equal to the model width in most models)
\(4\)
2 FLOPs per multiply-add, times 2 products (\(QK^\top\) and \(SV\))
\(12\)
3 times the forward (forward plus a backward twice as large). PaLM writes the training cost per token as \(6N + 12LHQT\), with \(H\) heads of size \(Q\). [Certain]

Two remarks. With a causal mask, token \(t\) only looks at \(t\) keys, so on average half of the score matrix is masked. A kernel that skips masked blocks (FlashAttention does this, session 124) does about half of (25.10). Kaplan et al. use \(2LTd\) for this reason; PaLM counts the full \(4LTd\). [Certain] And GQA does not reduce these FLOPs: every query head still computes its own scores. GQA reduces the memory of K and V, not the arithmetic.

Now compare the two terms. With \(N \approx 12Ld^2\):

\[ \frac{C_{\text{attn}}}{C_{\text{forward}}} = \frac{4LTd}{2 \cdot 12 L d^2} = \frac{T}{6d} \tag{25.11} \]

So attention-score FLOPs equal the weight FLOPs when \(T = 6d\): about 24,600 tokens for \(d = 4096\). Below that, the weights dominate; above it, attention does. With the causal saving the crossover moves to \(T = 12d\), which matches Kaplan’s remark that the context term is small when \(d_{\text{model}} \gg n_{\text{ctx}}/12\). [Certain] Note that the term is quadratic in \(T\) for a whole sequence (\(T\) tokens times \(4LTd\) each) but only linear in \(T\) per token.

Figure 3. Forward FLOPs per token, split into weight FLOPs (\(2N\), with \(N\) the weights used in matrix multiplies, so without the input table) and attention-score FLOPs (\(4LTd\), formula 25.10). The curve is the attention share for every context length from 512 to about one million tokens (log scale); the dot is the chosen \(T\).
How to read Figure 3

For Llama 3 8B at 8,192 tokens, attention is about 22% of the forward FLOPs. At 32,768 tokens it is more than half, and at about one million tokens it is 97%. Tick the causal box: every share drops, because the masked half is skipped, and the 50% point moves twice as far to the right. Switch to GPT-2 small (\(d = 768\)): the curve moves left; at its own context of 1,024 tokens, attention is already about 13% of the FLOPs in this count. Wide models push the crossover to longer contexts, because weight FLOPs grow like \(d^2\) and attention FLOPs only like \(d\).

Counting memory

FLOPs tell you how long. Memory tells you whether it runs at all. Memory comes in four parts: weights, gradients and optimizer states (training only), activations (training), and the KV cache (inference).

Weights and optimizer states

Each parameter takes 2 bytes in BF16 or FP16, 4 bytes in FP32, 1 byte in INT8 or FP8 (session 37). Llama 3 8B in BF16 is \(8.03 \times 10^9 \times 2 = 16.1\) GB (15.0 GiB). For training with AdamW in mixed precision, session 4 showed 16 bytes per parameter:

\[ M_{\text{train states}} = \big(\underbrace{2}_{\text{BF16 weights}} + \underbrace{2}_{\text{BF16 grads}} + \underbrace{4}_{\text{FP32 master}} + \underbrace{4 + 4}_{\text{Adam } m,\, v}\big)\, N = 16N \ \text{bytes} \tag{25.12} \]

That is 128 GB for an 8B model and about 1.13 TB for a 70B model, before a single activation. [Certain for the arithmetic; the exact byte layout varies between frameworks.] This is the memory that ZeRO and FSDP split across GPUs (session 45).

Activations

Backpropagation needs the inputs of every layer, saved from the forward pass. This memory grows with batch size \(b\) and sequence length \(s\), not with \(N\). Korthikanti et al. (2022) count it for a GPT-style block, stored in 16-bit, with no parallelism:

\[ M_{\text{act}} \approx L \cdot s\,b\,d\left(34 + \frac{5\,h\,s}{d}\right) \ \text{bytes} \tag{25.13} \]
Read the symbols
\(s, b\)
sequence length and micro-batch size (sequences per step on one GPU)
\(34\, s b d\)
the inputs of the linear layers, norms, the GELU and the dropout masks: about 34 bytes per number of the residual stream, per layer
\(5 h s \cdot s b\)
the attention score matrices: \(h\) heads of \(s \times s\), stored before and after softmax and for dropout. This is the part that grows like \(s^2\).

The paper also shows that recomputing only the attention core in the backward pass (“selective recomputation”) removes the \(5hs/d\) term. FlashAttention has the same effect, because it never stores the score matrix. [Certain] The constant 34 is for the GPT-style block of the paper (\(4\times\) GELU FFN, dropout); a Llama-style block has a different constant. [Likely]

Worked example: activations of a 7B-sized model

Take \(L = 32\), \(d = 4096\), \(h = 32\), \(s = 4096\), \(b = 1\). Per layer: \(4096 \times 1 \times 4096 \times (34 + 5 \times 32 \times 4096/4096) = 16{,}777{,}216 \times 194 = 3.25\) GB. For 32 layers: 104 GB, for one sequence! Without the score matrices (FlashAttention or selective recomputation): \(16{,}777{,}216 \times 34 = 0.57\) GB per layer, 18.3 GB in total. The score matrices were 82% of the activations.

The KV cache at inference

At inference there are no gradients and no optimizer. But to generate token \(t+1\), every layer needs the keys and values of all earlier tokens. We keep them in the KV cache (session 80). For each token, each layer stores one key and one value vector per KV head:

\[ M_{\text{KV}} = 2 \cdot L \cdot h_{kv} \cdot d_h \cdot T \cdot b \cdot \text{bytes} \tag{25.14} \]
Read the symbols
\(2\)
one key and one value
\(h_{kv} d_h\)
the width of K (and of V). With GQA this is smaller than \(d\): for Llama 3 8B, \(8 \times 128 = 1024\).
\(T, b\)
tokens per sequence (prompt plus generated) and the number of sequences in the batch
bytes
2 for BF16, 1 for an 8-bit cache (session 110)

For Llama 3 8B in BF16: \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes = 128 KiB per token. An 8,192-token conversation needs 1 GiB; 32 such users need 32 GiB, twice the weights. With full multi-head attention (\(h_{kv} = 32\)) it would be four times more: 512 KiB per token. This is the whole motivation of session 26.

Figure 4. Memory of Llama 3 8B on one GPU, without any sharding. Training: 16 bytes per parameter (25.12) plus activations from formula (25.13), with Llama 3’s \(L = 32\), \(d = 4096\), \(h = 32\); the GPT-style constant 34 makes the activation part an approximation. Inference: BF16 weights plus the KV cache (25.14). The dashed line is one 80 GB GPU.
How to read Figure 4

In training mode, even with batch 1 the bar is far past 80 GB: the 128 GB of weights, gradients and Adam states alone do not fit. That is why full fine-tuning of an 8B model needs several GPUs with sharding, or LoRA. Untick FlashAttention: at 4,096 tokens the activations jump from 18 GB to 104 GB. Switch to inference: the 16 GB of weights fit easily; now raise the batch to 32 and the length to 8,192: the KV cache (32 GiB) is twice the weights. At inference, memory is about the cache, not the model.

Decode is limited by memory, not FLOPs

One more count connects FLOPs and memory. When generating one token for one user, the GPU must read all 16.1 GB of weights to do \(2N \approx 16\) GFLOPs. That is about 1 FLOP per byte read. An H100 SXM can do 989 TFLOP/s but read only about 3.35 TB/s from its memory, so it needs about 295 FLOPs per byte to be busy. [Certain for the datasheet numbers] At batch 1, decode is limited by memory speed: at best \(3.35 \times 10^{12} / 16.06 \times 10^9 \approx 209\) tokens per second. Batching many users reuses each weight read for many tokens, which is why serving systems batch (session 82).

Worked example: estimate an 8B model, then check its config

Here is the full exercise that interviewers like. You are given only this config (the real fields of Llama 3 8B’s config.json [Certain]):

{ "hidden_size": 4096, "intermediate_size": 14336, "num_hidden_layers": 32,
  "num_attention_heads": 32, "num_key_value_heads": 8,
  "vocab_size": 128256, "tie_word_embeddings": false, "max_position_embeddings": 8192 }
Worked example: Llama 3 8B in three steps

Step 1, the quick estimate. \(12 L d^2 = 12 \times 32 \times 4096^2 = 6.44\) B. Two untied tables: \(2 \times 128{,}256 \times 4096 = 1.05\) B. Estimate: 7.49 B.

Step 2, the exact count. Head size \(d_h = 4096/32 = 128\); K/V width \(8 \times 128 = 1024\). Per layer: \(W_Q\) and \(W_O\): \(2 \times 4096^2 = 33{,}554{,}432\). \(W_K\) and \(W_V\): \(2 \times 4096 \times 1024 = 8{,}388{,}608\). FFN: \(3 \times 4096 \times 14{,}336 = 176{,}160{,}768\). Two RMSNorms: 8,192. One layer: 218,112,000. Times 32: 6,979,584,000. Plus two tables of 525,336,576 and the final norm of 4,096: 8,030,261,248.

Step 3, check. The published name is “8B”, and the exact count matches the parameter count of the released checkpoint (the PyTorch code below builds the same shapes and gets the same number). [Certain] The quick estimate was 6.7% low; we now know why: the FFN is \(10.5d^2\) instead of \(8d^2\), partly paid back by GQA (\(2.5d^2\) instead of \(4d^2\)).

Consequences. Forward: \(2 \times 7.50\) B \(\approx 15.0\) GFLOPs per token for the weights (the 525 M input table is a lookup). Training: about \(6 \times 7.50\) B \(\approx 45\) GFLOPs per token. Memory: 16.1 GB in BF16, 128 GB for AdamW training states, 128 KiB of KV cache per token.

Figure 5. Counting Llama 3 8B from its config, one line at a time. Each step adds one group of weights and shows the formula, the count for that group, and the running total. The last steps turn the count into FLOPs and memory.
How to read Figure 5

Press Next. The highlighted row is the group being added; the readout shows the formula with the config values filled in. Watch the running total: after one layer it is only 0.74 B (most of it the embedding table), after all 32 layers it is 7.50 B, and the output layer brings it to 8,030,261,248. The last two steps reuse the count: \(2N\) and \(6N\) FLOPs per token, and the bytes for weights, Adam and the KV cache.

Pseudo-code

Algorithm 1: count parameters from a config
  1. input: \(d, L, h, h_{kv}, d_{\text{ff}}, |V|\), tied, gated, biases
  2. \(d_h \leftarrow d / h\)
  3. \(a \leftarrow 2 d (h d_h) + 2 d (h_{kv} d_h)\) // Q, O and K, V, formula (25.5)
  4. if gated then \(f \leftarrow 3 d d_{\text{ff}}\) else \(f \leftarrow 2 d d_{\text{ff}}\) // SwiGLU or classic FFN
  5. \(n \leftarrow\) norm and bias sizes (a few times \(d\) per block)
  6. \(N_{\text{blocks}} \leftarrow L \cdot (a + f + n)\)
  7. \(N_{\text{emb}} \leftarrow |V| d\) if tied else \(2 |V| d\) // formula (25.4)
  8. return \(N_{\text{blocks}} + N_{\text{emb}} +\) final norm
Algorithm 2: compute and memory budget for a training run
  1. input: \(N\), tokens \(D\), context \(T\), \(L, d\), GPUs \(n\), peak \(P\), MFU \(u\)
  2. \(c \leftarrow 6N + 12 L T d\) // FLOPs per token, (25.8) and (25.10)
  3. \(C \leftarrow c \cdot D\)
  4. days \(\leftarrow C / (n \cdot P \cdot u) / 86{,}400\) // formula (25.9)
  5. states \(\leftarrow 16 N\) bytes // formula (25.12); divide by \(n\) with ZeRO-3 or FSDP
  6. activations \(\leftarrow L \cdot s b d \cdot (34 + 5 h s / d)\) bytes // (25.13); drop the second term with FlashAttention
  7. if states / \(n\) + activations \(\gt\) GPU memory then shard more, lower \(b\), or recompute activations
  8. return \(C\), days, memory per GPU

The same counts in PyTorch. The model is a small Llama-style decoder (RMSNorm, GQA, SwiGLU, untied output). PyTorch’s FlopCounterMode counts the FLOPs of every matrix multiply, so we can check formulas (25.5), (25.6), (25.7) and (25.10) exactly:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.flop_counter import FlopCounterMode
torch.manual_seed(0)

class Block(nn.Module):                      # Llama-style: RMSNorm, GQA attention, SwiGLU
    def __init__(s, d, h, kv, ff):
        super().__init__()
        s.h, s.kv, s.dh = h, kv, d // h
        s.n1, s.n2 = nn.RMSNorm(d), nn.RMSNorm(d)
        s.q = nn.Linear(d, h * s.dh, bias=False)
        s.k = nn.Linear(d, kv * s.dh, bias=False)
        s.v = nn.Linear(d, kv * s.dh, bias=False)
        s.o = nn.Linear(h * s.dh, d, bias=False)
        s.gate = nn.Linear(d, ff, bias=False)
        s.up = nn.Linear(d, ff, bias=False)
        s.down = nn.Linear(ff, d, bias=False)
    def forward(s, x):
        B, T, d = x.shape
        a = s.n1(x)
        q = s.q(a).view(B, T, s.h, s.dh).transpose(1, 2)
        k = s.k(a).view(B, T, s.kv, s.dh).transpose(1, 2).repeat_interleave(s.h // s.kv, 1)
        v = s.v(a).view(B, T, s.kv, s.dh).transpose(1, 2).repeat_interleave(s.h // s.kv, 1)
        att = (q @ k.transpose(-1, -2)) / s.dh ** 0.5          # (B, h, T, T): 2*T*T*d FLOPs
        mask = torch.ones(T, T, dtype=torch.bool).tril()
        att = att.masked_fill(~mask, float("-inf")).softmax(-1)
        y = (att @ v).transpose(1, 2).reshape(B, T, d)          # another 2*T*T*d FLOPs
        x = x + s.o(y)
        b = s.n2(x)
        return x + s.down(F.silu(s.gate(b)) * s.up(b))

class LM(nn.Module):
    def __init__(s, V, d, L, h, kv, ff):
        super().__init__()
        s.emb = nn.Embedding(V, d)
        s.blocks = nn.ModuleList(Block(d, h, kv, ff) for _ in range(L))
        s.norm = nn.RMSNorm(d)
        s.head = nn.Linear(d, V, bias=False)                   # untied output layer
    def forward(s, ids):
        x = s.emb(ids)
        for blk in s.blocks:
            x = blk(x)
        return s.head(s.norm(x))

def count_formula(V, d, L, h, kv, ff):                      # formula (25.5) plus tables
    dh = d // h
    per_layer = 2 * d * d + 2 * d * kv * dh + 3 * d * ff + 2 * d
    return L * per_layer + d + 2 * V * d

# 1) parameter count: real modules vs formula
cfg = dict(V=1000, d=256, L=4, h=8, kv=2, ff=704)
m = LM(**cfg)
n = sum(p.numel() for p in m.parameters())
print(n, count_formula(**cfg))                              # 3332352 3332352

# 2) FLOPs of one forward and one forward+backward vs 2N and 6N
B, T = 2, 128
ids = torch.randint(0, cfg["V"], (B, T))
N_mm = n - cfg["V"] * cfg["d"] - (2 * cfg["L"] + 1) * cfg["d"]   # weights used in matmuls
attn = 4 * cfg["L"] * T * cfg["d"]                          # QK^T and AV per token, full T x T
with FlopCounterMode(display=False) as fc:
    m(ids)
fwd = fc.get_total_flops() / (B * T)
print(fwd, 2 * N_mm + attn)                                 # 6672384.0 6672384
with FlopCounterMode(display=False) as fc:
    m(ids).float().logsumexp(-1).mean().backward()
print(fc.get_total_flops() / (B * T), 3 * (2 * N_mm + attn))  # 20017152.0 20017152: exactly 3x

# 3) Llama 3 8B on the meta device: shapes only, no memory is allocated
with torch.device("meta"):
    big = LM(V=128256, d=4096, L=32, h=32, kv=8, ff=14336)
print(sum(p.numel() for p in big.parameters()))             # 8030261248

Notice that FlopCounterMode counts the full \(T \times T\) score matrix even with a mask: masking does not skip any multiply in this plain code. Only a causal-aware kernel saves the half. The printed values were produced with PyTorch 2.8 on CPU.

Where this is used

ApplicationWhich countLink
Choosing model size and token count for a budget\(C = 6ND\) with a scaling law; Chinchilla found about 20 tokens per parameter compute-optimal [Certain]session 35
Measuring training efficiency\(\text{MFU} = (\text{tokens/s} \times 6N) / \text{peak}\); HFU adds recomputationsession 51
Fitting training on GPUs16 bytes per parameter (25.12) plus activations (25.13); decides ZeRO stage, FSDP, checkpointingsession 38
Serving capacity planningWeights + KV cache (25.14) per user; decides batch size and how many users per GPUsession 81
Long-context and video VLMsThe attention term \(4LTd\) per token: with thousands of visual tokens per clip, attention becomes a large share of prefill (video QA, Video-RAG over long recordings)session 28
Video retrieval and fingerprinting at scaleEncoder cost per clip \(= 2N \times\) tokens per clip; multiply by hours of video per day to size the indexing fleetsession 208
Choosing attention variants and quantizationKV-cache bytes per token for MHA vs GQA vs MLA; bytes per weight for INT8/INT4session 26

Common mistakes

  • Counting the number of heads in the parameter count. \(W_Q\) is \(d \times d\) whatever \(h\) is. Heads change the grouping, not the size. Only \(h_{kv}\) (GQA) changes the count of \(W_K\) and \(W_V\).
  • Forgetting the untied output layer. Llama 3 8B has two 525 M tables. Count one and you get 7.5 B, not 8.03 B.
  • Using \(8d^2\) for a SwiGLU FFN. A gated FFN has three matrices: \(3 d\, d_{\text{ff}}\). Read intermediate_size from the config.
  • Mixing FLOPs and FLOP/s, or using the “with sparsity” peak. Datasheets often print the 2:4-sparse number, which is twice the dense one. Your MFU will look twice as bad as it is, or your time estimate twice as optimistic.
  • Saying 2N per token for training. Training is 6N: forward 2N plus backward 4N. And 8N of hardware work with full activation recomputation.
  • Ignoring the attention term at long context. At \(T = 6d\) it equals the weight FLOPs (full count). For a 128k-context run with \(d = 4096\), it is the larger part.
  • Thinking GQA reduces attention FLOPs. It reduces the K/V projection weights and the KV cache. Every query head still computes \(T\) scores.
  • Confusing GB and GiB. 16.06 GB is 14.96 GiB. Tools print both; a 7% gap matters when you are close to the limit.

Interview questions

How many FLOPs is a matrix multiply of an \((m \times k)\) matrix and a \((k \times n)\) matrix?basic
\(2mkn\): there are \(mn\) outputs, each a dot product of length \(k\), which is \(k\) multiplications and about \(k\) additions. For a linear layer on one token (\(m = 1\)) this is 2 FLOPs per weight, which is where the \(2N\) rule comes from.
Derive the parameter count of one transformer layer with width \(d\).basic
Attention: \(W_Q, W_K, W_V, W_O\), each \(d \times d\): \(4d^2\). FFN with \(4\times\) expansion: \(d \times 4d\) up and \(4d \times d\) down: \(8d^2\). Total \(12d^2\), plus a few vectors of size \(d\) for norms and biases. The number of heads does not matter. A model has about \(12Ld^2\) plus \(|V|d\) per embedding table.
Count the parameters of Llama 3 8B from its config.core
\(d = 4096\), \(L = 32\), \(h = 32\), \(h_{kv} = 8\), \(d_{\text{ff}} = 14{,}336\), \(|V| = 128{,}256\), untied. \(d_h = 128\). Per layer: Q and O \(2 \times 4096^2 = 33.55\) M; K and V \(2 \times 4096 \times 1024 = 8.39\) M; SwiGLU \(3 \times 4096 \times 14{,}336 = 176.16\) M; norms 8,192. Layer: 218,112,000; \(\times 32 = 6.98\) B. Tables \(2 \times 525.3\) M = 1.05 B. Final norm 4,096. Total 8,030,261,248. The \(12Ld^2\) rule gives 6.44 B for the blocks, 8% low, because the FFN is \(10.5d^2\) and GQA attention is \(2.5d^2\).
Why does training cost about 6N FLOPs per token?core
Forward: each weight is used in one multiply-add per token, 2 FLOPs, so \(2N\). Backward: for \(Y = XW\) we need \(\partial L/\partial X = (\partial L/\partial Y) W^\top\) to continue backprop and \(\partial L/\partial W = X^\top (\partial L/\partial Y)\) for the update; each is the same size as the forward, so \(4N\). Total \(6N\), and \(6ND\) for \(D\) tokens. Not counted: attention scores (\(12LTd\) per token in training), elementwise operations, and recomputation from activation checkpointing.
When does attention dominate the FLOPs of a transformer?core
Per token, the score and value products cost \(4LTd\) forward (full count), against about \(2N \approx 24Ld^2\) for the weights. The ratio is \(T/(6d)\), so they are equal at \(T = 6d\) (about 24.6k tokens for \(d = 4096\)), or \(T = 12d\) if the kernel skips the causal-masked half. Above that, attention dominates. For Llama 3 8B at 8k tokens attention is about 22% of forward FLOPs; at 32k, more than half.
Estimate the training compute and time for an 8B model on 15T tokens.core
\(6 \times 8.03 \times 10^9 \times 15 \times 10^{12} = 7.2 \times 10^{23}\) FLOPs. One H100 at 989 TFLOP/s dense BF16 and 40% MFU does \(3.96 \times 10^{14}\) FLOP/s, so about \(1.83 \times 10^9\) GPU-seconds = 507k GPU-hours; on 1,024 GPUs about 21 days. State your MFU assumption; it changes the answer linearly.
How much GPU memory do you need to fully fine-tune a 7B model with AdamW?core
States: 16 bytes per parameter (BF16 weights 2, BF16 gradients 2, FP32 master 4, Adam m and v 4 + 4), so 112 GB for 7B, before activations. Activations add \(L\,s b d\,(34 + 5hs/d)\) bytes for a GPT-style block (about 18 GB at 4k tokens, batch 1, with FlashAttention; about 104 GB without). It does not fit on one 80 GB GPU: use ZeRO-3 or FSDP across GPUs, 8-bit optimizer states, activation checkpointing, or LoRA.
How big is the KV cache of Llama 3 8B, and how does GQA change it?core
\(2 \times L \times h_{kv} \times d_h \times\) bytes per token \(= 2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes = 128 KiB in BF16. At 8k tokens, 1 GiB per sequence; 32 sequences need 32 GiB, twice the weights. With full MHA (\(h_{kv} = 32\)) it would be 512 KiB per token, four times more. GQA cuts cache memory and bandwidth by \(h/h_{kv}\) but not the attention FLOPs.
Why is single-user decoding slow even though it needs only about 16 GFLOPs per token?advanced
Each decode step must read every weight from GPU memory (16 GB in BF16 for 8B) to do \(2N\) FLOPs: about 1 FLOP per byte. An H100 needs about 295 FLOPs per byte (989 TFLOP/s ÷ 3.35 TB/s) to be compute-bound. So decode at batch 1 is memory-bound: at most about 209 tokens/s from bandwidth alone. Batching \(b\) users reuses each weight read \(b\) times and raises the FLOPs per byte, until the KV-cache reads start to dominate.
What are MFU and HFU, and why can activation checkpointing make MFU drop?advanced
MFU = achieved useful FLOP/s ÷ peak FLOP/s, where useful FLOPs per token are fixed by the model: \(6N + 12LTd\) (the PaLM definition). HFU counts the FLOPs the hardware actually ran, including recomputed forward passes. Full activation checkpointing adds about one more forward (\(2N\)), so the GPU does about \(8N\) per token. The step takes longer for the same useful work, so MFU falls while HFU may stay high. MFU is the fair number to compare across systems.
Why are embedding parameters often left out of scaling-law counts?advanced
The input embedding is a lookup and costs no FLOPs; its size depends on the tokenizer, not on model capacity in the usual sense. Kaplan et al. found cleaner power laws using non-embedding parameters \(N \approx 12Ld^2\). In small models embeddings can be a quarter or more of all weights, so including them distorts the curves. The output layer does cost \(2|V|d\) FLOPs per token, so compute counts should include it when it matters.

Exercises

  1. A model has \(d = 2048\), \(L = 24\), classic attention and a \(4\times\) GELU FFN, \(|V| = 50{,}000\), tied. Estimate its parameter count.
  2. Estimate the training compute of GPT-3 (175 B parameters, 300 B tokens). Compare it with Chinchilla (70 B parameters, 1.4 T tokens).
  3. Llama 2 7B has \(L = 32\), \(h = h_{kv} = 32\), \(d_h = 128\). What is its KV cache per token in BF16? For one 4,096-token sequence?
  4. For a model with \(d = 4096\), what is the ratio of attention-score FLOPs to weight FLOPs at \(T = 32{,}768\), using (25.11)? What share of the forward FLOPs is attention?
  5. How much memory do the AdamW training states of a 1 B model need? Of a 70 B model?
  6. In Figure 4 (training, FlashAttention on), what happens to the activation memory when you double the sequence length? And with FlashAttention off?
  7. Show that a SwiGLU FFN with \(d_{\text{ff}} = \tfrac{8}{3}d\) has the same parameter count as a classic FFN with \(d_{\text{ff}} = 4d\).
Answerscheck
1. \(12 \times 24 \times 2048^2 = 1{,}207{,}959{,}552\) plus one table \(50{,}000 \times 2048 = 102{,}400{,}000\): about 1.31 B. 2. GPT-3: \(6 \times 175 \times 10^9 \times 300 \times 10^9 = 3.15 \times 10^{23}\) FLOPs. Chinchilla: \(6 \times 70 \times 10^9 \times 1.4 \times 10^{12} = 5.88 \times 10^{23}\), almost twice as much, for a model 2.5 times smaller. 3. \(2 \times 32 \times 32 \times 128 \times 2 = 524{,}288\) bytes = 512 KiB per token; 4,096 tokens = 2 GiB. 4. \(T/(6d) = 32{,}768 / 24{,}576 = 1.33\); share \(1.33/2.33 = 57\%\). (With the exact Llama 3 8B weight count, Figure 3 shows 53%.) 5. \(16 \times 10^9\) bytes = 16 GB; \(16 \times 70 \times 10^9 = 1.12\) TB. 6. With FlashAttention only the \(34sbd\) term is left, so it doubles (linear in \(s\)). Without it, the \(5hs^2b\) term grows four times, so the total grows by more than 2× and approaches 4× at long lengths. 7. Classic: \(2 \cdot d \cdot 4d = 8d^2\). SwiGLU: \(3 \cdot d \cdot \tfrac{8}{3}d = 8d^2\).

Key takeaways

  • Every count starts from one rule: a matrix multiply of \((m \times k)(k \times n)\) costs \(2mkn\) FLOPs (25.1), so each weight costs 2 FLOPs per token.
  • Parameters: attention \(4d^2\) plus FFN \(8d^2\), so \(N \approx 12Ld^2\) plus \(|V|d\) per embedding table (25.4). GQA and SwiGLU move the 12 (Llama 3 8B has about \(13d^2\) per layer); the exact count is 8,030,261,248.
  • Compute: about \(2N\) per token forward (25.6), \(6N\) per token in training (25.7, 25.8), so \(C \approx 6ND\). Time = \(6ND / (n \cdot P \cdot \mathrm{MFU})\) (25.9).
  • Attention scores add \(4LTd\) per token forward (25.10). They match the weight FLOPs at \(T = 6d\) (25.11), or \(12d\) with causal skipping.
  • Memory: 2 bytes per parameter for BF16 weights, 16 for AdamW training (25.12), activations that grow with \(s\) and \(s^2\) (25.13), and a KV cache of \(2Lh_{kv}d_h\) values per token (25.14).
  • Decode at batch 1 is limited by memory bandwidth, not FLOPs: about 1 FLOP per byte of weights read.

Key formulas

\[ \mathrm{FLOPs}(AB) = 2mkn, \qquad N \approx 12 L d^2 + |V| d\,(1 \text{ or } 2) \]
\[ C_{\text{fwd}} \approx 2N + 4LTd, \qquad C_{\text{train}} \approx (6N + 12LTd)\, D \approx 6ND \]
\[ M_{\text{train}} \approx 16N + L\,sbd\left(34 + \tfrac{5hs}{d}\right), \qquad M_{\text{KV}} = 2 L h_{kv} d_h T b \cdot \text{bytes} \]

References for this session

  1. Kaplan et al. (2020). Scaling Laws for Neural Language Models
  2. Austin et al., Google DeepMind (2025). How to Scale Your Model
  3. Stanford (2025). CS336: Language Modeling from Scratch (course)
  4. Hoffmann et al. (2022). Training Compute-Optimal Large Language Models
  5. Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
  6. Brown et al. (2020). Language Models are Few-Shot Learners
  7. Llama Team, Meta (2024). The Llama 3 Herd of Models
  8. Meta (mirrored by Unsloth) (2024). Llama 3 8B config.json on Hugging Face
  9. Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
  10. Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
  11. Shazeer (2020). GLU Variants Improve Transformer
  12. Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  13. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  14. Micikevicius et al. (2017). Mixed Precision Training