One training step, end to end: forward, loss, backward, update
This session follows one batch through one full training step of an LLM: token ids become embeddings, pass through the blocks, become logits and one loss. Then the gradient flows back layer by layer, the optimizer clips, reads the learning rate and updates every weight. At each moment we ask which tensors are in GPU memory, and why the peak comes where it does.
What you will learn
- The shapes and sizes of every tensor in the forward pass, from (B, T) ids to (B, T, V) logits and one loss
- Why the first gradient is prediction minus truth, (p − e_y)/N, and how to check it
- Where every weight gradient comes from: X^T G for linear layers, the residual copy, the softmax backward in attention, the scatter-add in the embedding
- The update in the right order: unscale, clip the global norm, AdamW with a warmup-cosine schedule, scheduler step, zero_grad
- What lives in GPU memory at each moment of the step, and why the peak is usually at the start of the backward pass
- How large logits, lazily created Adam states and checkpointing change the memory peak
- A deterministic PyTorch step that checks these claims
Why this matters
Training a language model is one small procedure repeated hundreds of thousands of times: take a batch of tokens, compute the loss, compute the gradient of every weight, change the weights a little. This session follows one such step from the first token id to the last freed byte of memory.
So far the course has shown the pieces one at a time: embeddings (session 11), attention and the MLP, the residual stream (session 22), the causal loss (session 31), backpropagation and AdamW (session 4), and the counts of parameters, FLOPs and memory (session 25). Here we put them in the order in which a GPU actually runs them. Three questions guide the session:
- Forward: what tensor exists after each stage, and what is its shape and size?
- Backward: where does the gradient of every weight come from? Which formula produces it, and which stored tensor does that formula need?
- Memory: what lives in GPU memory at each moment of the step, and when is the peak?
The answers explain most practical training problems. "CUDA out of memory" at the start of the backward pass, a loss that does not move because gradients were never cleared, a run that diverges because the clipping happened before un-scaling: these are all bugs in the order of the step. And the interview question "walk me through one training step of an LLM" is very common; a good answer gives shapes, gradients and memory, not only the four words "forward, backward, step, zero".
As a running example we use Llama 3 8B: \(L = 32\) layers, width \(d = 4096\), vocabulary \(V = 128{,}256\), MLP width 14,336, 32 query heads and 8 key-value heads, untied input and output embeddings, \(N = 8.03 \times 10^9\) parameters. [Certain: from the model's config.json.] We follow one sequence of \(T = 8192\) tokens, its pretraining context length.
The map of one step
One training step has five phases. Each phase reads some tensors and writes others:
- Load a batch. A tensor of token ids of shape \((B, T)\), and the targets, which are the same ids shifted by one position.
- Forward pass. Embedding lookup, \(L\) transformer blocks, a final norm, the output projection to logits, and the cross-entropy loss. Every operation saves the inputs it will need for its backward pass. These saved tensors are the activations.
- Backward pass. Start from the loss and go back through the same operations in reverse order. Each one turns the gradient of its output into the gradient of its input and of its weights, then frees its saved activations.
- Update. Clip the global gradient norm, read the learning rate from the schedule, and apply AdamW to every weight.
- Clean up. Free (or zero) the gradients so the next step starts fresh, and advance the schedule.
Step through Figure 1. The readout gives the shape and the size in memory of the main tensor at each moment, for Llama 3 8B with one sequence of 8,192 tokens.
How to read Figure 1
Press Next. Blue boxes are forward stages, orange boxes backward stages, green boxes the update. Watch the numbers in the readout: the hidden states are 64 MiB, but the logits are about 1.05 billion numbers (4.2 GB in FP32), larger than all the hidden states of one layer. The saved activations of all 32 blocks add up to tens of GB. These numbers come back in Figure 5. Figure 4 runs the same stages for real, with every number, on a model small enough to show all of its weights.
Forward: from token ids to one number
Batch and targets
The data loader gives a tensor of shape \((B, T + 1)\) of token ids. The input is the first \(T\) columns, and the target at each position is the next id:
With packing (session 18), every position is a real token, so a batch gives \(B \cdot T\) predictions. Llama 2 used batches of about 4 million tokens per step. [Certain] The ids are 64-bit integers: one sequence of 8,192 ids is only 64 KiB. Data is never the memory problem.
Embedding lookup
Each id selects one row of the embedding matrix \(E \in \R^{V \times d}\):
This is a lookup, not a matrix multiply: it costs almost no FLOPs. For one sequence, \(X_0\) has \(8192 \times 4096 = 33.6\) million numbers, 64 MiB in BF16. Llama has no position embedding to add here; position enters inside attention through RoPE (session 14).
The blocks
Each of the \(L\) blocks reads the residual stream and adds two updates to it (pre-norm, as in Llama):
The shape stays \((B, T, d)\) through all blocks. But inside a block many larger tensors appear: the queries, keys and values, the attention output, and the MLP's hidden layer (width 14,336, which is 3.5 times \(d\)). Each linear layer saves its input for the backward pass (it needs \(X\) to compute \(\partial L/\partial W = X^\top G\)). Each norm saves its input. The MLP saves the values before and after its activation function. Korthikanti et al. counted these saved tensors for a GPT-style block in 16-bit: about \(34\) bytes per number of the residual stream, per layer, when the attention scores are not stored (FlashAttention). [Certain for their GPT block; for a Llama block with SwiGLU the constant is similar but not identical. Likely.] For our sequence that is
1.14 GB per layer, 36.5 GB for the model, for a single sequence. The weights in BF16 are 16.1 GB. So for long sequences, the stored activations are larger than the model itself.
Logits and loss
After the last block, a final norm and the output matrix \(W_{\text{out}} \in \R^{d \times V}\) give one score per vocabulary entry at every position:
\(Z\) is the largest tensor of the forward pass: \(8192 \times 128{,}256 = 1.05\) billion numbers. Losses are computed in FP32 for accuracy, so this one tensor takes 4.2 GB. Then the cross-entropy loss averages the negative log-probability of each target over the \(N_{\text{tok}}\) positions that have a target:
- Read the symbols
- \(\mathbf{z}_{b,t} \in \R^V\)
- the logits at sequence \(b\), position \(t\)
- \(y_{b,t}\)
- the target id: the next token in the text
- \(m_{b,t}\)
- 1 if this position gets a loss, 0 if not (padding, or a prompt in SFT). PyTorch marks the 0 positions with the label \(-100\), the
ignore_index. - \(N_{\text{tok}}\)
- how many positions really get a loss; the average is over them only
At initialization the model knows nothing, all \(V\) tokens get about the same probability, and the loss is close to \(\ln V\). For Llama 3 that is \(\ln 128{,}256 = 11.76\) nats. If your first loss is much larger, the initialization or the output scale is wrong. This is a cheap and very useful check. [Certain]
The first gradient: softmax and cross-entropy
The backward pass starts at the scalar \(\mathcal{L}\), with \(\partial \mathcal{L}/\partial \mathcal{L} = 1\). The first real gradient is the one of the logits. For one position with logits \(\mathbf{z}\), probabilities \(\mathbf{p} = \softmax(\mathbf{z})\) and target \(y\), the loss is \(-\log p_y = -z_y + \log \sum_j e^{z_j}\). Its derivative with respect to \(z_i\) is \(-[i = y] + e^{z_i}/\sum_j e^{z_j}\). So:
- Read the symbols
- \(\mathbf{p}_{b,t}\)
- the predicted distribution at that position
- \(\mathbf{e}_{y}\)
- the one-hot vector of the target: 1 at position \(y\), 0 everywhere else
- \(m / N_{\text{tok}}\)
- the averaging: each position contributes with weight \(1/N_{\text{tok}}\), and ignored positions contribute 0
This is the cleanest formula in deep learning: prediction minus truth. The gradient pushes the target's logit up by \(1 - p_y\) and every other logit down by its own probability. It is never larger than 1 in size, and it is 0 only when the model puts probability 1 on the right token. Every gradient in the whole network comes from this one tensor, sent backward.
Logits \(\mathbf{z} = (2.0, 1.0, 0.1)\), target \(y = 0\), one position (\(N_{\text{tok}} = 1\)). Softmax: \(\mathbf{p} = (0.659, 0.242, 0.099)\). Loss: \(-\ln 0.659 = 0.417\).
Gradient: \(\mathbf{p} - \mathbf{e}_0 = (0.659 - 1,\ 0.242,\ 0.099) = (-0.341, 0.242, 0.099)\). The components sum to 0, as they always must: the probabilities and the one-hot vector both sum to 1. A step against this gradient raises \(z_0\) and lowers \(z_1\) and \(z_2\).
Figure 2 lets you do this by hand. Choose the target, then take gradient steps directly on the logits (as if the logits were the weights).
How to read Figure 2
The orange bar of the target is always negative (its logit must go up), the others are positive and equal to their probabilities. The bars always sum to zero. Press "Take one step" a few times: the loss falls fast at first, then slowly, because the gradient \(1 - p_y\) shrinks as \(p_y\) grows. Choose an unlikely target such as "sat": the first loss is large and the gradient is close to \(-1\), the largest it can be. The box under the readout shows the last update with its numbers, \(z_i \leftarrow z_i - \eta\,(p_i - [i = y])\). With the default target "mat" and \(\eta = 2\), press Play: \(p_y\) goes 0.406, 0.756, 0.851, 0.892 and the loss goes 0.902, 0.280, 0.162, 0.115. The first step gains the most; each later step gains less.
Backward: where every gradient comes from
Now we walk back through the network, in exactly the reverse order of the forward pass. Every operation uses the same two rules from session 4: the chain rule, and for a linear layer \(Y = XW\) with incoming gradient \(G = \partial \mathcal{L}/\partial Y\):
(For a batch of shape \((B, T, \cdot)\), the first formula also sums over all \(B \cdot T\) positions: every position adds its own small contribution to the weight gradient.) The first formula needs the stored input \(X\). The second needs the weight \(W\). This is why activations must be kept: the weight gradient of every linear layer is built from its forward input.
The output layer and the final norm
From \(G_Z = \partial \mathcal{L}/\partial Z\) (formula 32.7, shape \((B, T, V)\), another 4.2 GB in FP32), formula (32.8) gives \(\partial \mathcal{L}/\partial W_{\text{out}} = H^\top G_Z\), a \(4096 \times 128{,}256\) matrix, and \(\partial \mathcal{L}/\partial H = G_Z W_{\text{out}}^\top\), back to shape \((B, T, d)\). After this, the logits and their gradient can be freed. The final RMSNorm then turns \(\partial \mathcal{L}/\partial H\) into \(\partial \mathcal{L}/\partial X_L\), using its stored input, and gives the gradient of its scale vector.
The residual stream: a gradient highway
Write one sub-layer as \(X_{\text{out}} = X_{\text{in}} + F(X_{\text{in}})\). The chain rule gives
The first term is a plain copy. Whatever gradient arrives at the top of a block passes to the bottom unchanged, and the block only adds its own term. So the gradient of the loss reaches the embedding through 64 additions (two per block), not through 64 multiplications that could shrink it to zero. This is the main reason very deep transformers train at all (session 22). [Certain]
Inside a block: the MLP and attention
The MLP is three linear layers and an elementwise function. Its backward pass is formula (32.8) three times, plus the derivative of the activation (SiLU in Llama) at the stored pre-activation values. Attention is more interesting. With scores \(S = QK^\top/\sqrt{d_h}\) (causal mask applied), \(P = \softmax(S)\) row by row, output \(O = PV\), and incoming gradient \(G_O\):
- Read the symbols
- \(\odot\)
- elementwise product
- \(\operatorname{rowsum}(\cdot)\)
- the sum of each row, kept as a column and subtracted from every element of that row: the softmax backward step
- \(G_Q, G_K, G_V\)
- gradients that then flow into the query, key and value projections by formula (32.8), and through RoPE (a fixed rotation, whose backward is the inverse rotation)
The softmax backward needs \(P\), a \(T \times T\) matrix per head. Storing it for \(T = 8192\) and 32 heads would be 4.3 GB per layer in BF16. FlashAttention does not store it. It keeps only \(Q, K, V, O\) and one number per row (the log-sum-exp), and recomputes \(P\) block by block in the backward pass (session 124). [Certain] We checked formula (32.10) against PyTorch autograd in float64; the code is in the exercises.
The embedding: a scatter-add
The lookup (32.2) copied row \(x_{b,t}\) of \(E\) into position \((b, t)\). Its backward does the opposite: it adds the gradient of each position into the row it came from:
Rows of tokens that do not appear in the batch get exactly zero gradient. A token that appears 50 times gets the sum of 50 contributions. In PyTorch this is an index_add into a dense \(V \times d\) gradient (unless you ask for sparse gradients). If the model ties the input and output embeddings (GPT-2, many small models), the same matrix gets two gradients, one from (32.11) and one from the output layer, and autograd adds them. [Certain]
- input: loss \(\mathcal{L}\), saved activations of every layer, weights
- \(G_Z \leftarrow \frac{m}{N_{\text{tok}}}(\mathbf{p} - \mathbf{e}_y)\) // formula (32.7), shape \((B, T, V)\)
- \(\nabla W_{\text{out}} \leftarrow H^\top G_Z\); \(\;G \leftarrow G_Z W_{\text{out}}^\top\); free \(Z\), \(G_Z\) // formula (32.8)
- \(G \leftarrow\) backward of the final norm (uses its saved input); \(\nabla\gamma_{\text{final}}\)
- for \(\ell = L, L-1, \dots, 1\):
- \(G_U \leftarrow G + \) MLP-branch backward of \(G\) // formula (32.9); gives \(\nabla\) of 3 MLP matrices and the norm
- \(G \leftarrow G_U + \) attention-branch backward of \(G_U\) // formula (32.10); gives \(\nabla\) of \(W_q, W_k, W_v, W_o\) and the norm
- free the saved activations of layer \(\ell\); (DDP: start the all-reduce of this layer's gradients)
- \(\nabla E \leftarrow\) scatter-add of \(G\) into the rows of the input ids // formula (32.11)
- return a gradient for every weight
Count the work: each linear layer does two matrix multiplies in the backward pass (one for \(\nabla W\), one for \(\nabla X\)) and one in the forward pass. So backward costs about twice the forward, and one step costs about
For one sequence of 8,192 tokens on Llama 3 8B: \(6 \times 8.03 \times 10^9 \times 8192 = 3.95 \times 10^{14}\) FLOPs, without the attention-score term. A GPU that sustains 400 TFLOP/s needs about 1 second for it. For a 4-million-token batch the step is \(1.93 \times 10^{17}\) FLOPs, which is why such batches are spread over hundreds or thousands of GPUs (session 25).
The update: clipping, schedule, AdamW
When the backward pass ends, every weight \(\theta_i\) has a gradient \(g_i\) in its .grad field. Three things happen before the weights change, in this order.
1. Clip the global norm
Compute one number for the whole model, the length of the gradient vector over all 8 billion weights, and shrink the whole vector if it is too long:
The usual threshold is \(c = 1.0\), as in Llama 2. [Certain] The norm must be computed over all gradients, so with sharded training (FSDP) each GPU computes its part of the sum of squares and the parts are added with an all-reduce. Log the norm before clipping: it is one of the best health signals of a run (session 36).
2. Read the learning rate
The schedule turns the step number \(t\) into a learning rate. The common shape is linear warmup over \(W\) steps, then cosine decay to a fraction \(r\) of the peak at the last step \(T_{\max}\):
For a Llama 2 7B-like run, \(\eta_{\max} = 3 \times 10^{-4}\), \(W = 2000\), \(r = 0.1\) and about 500,000 steps (2 trillion tokens at 4 million per step). [Certain for the recipe; the step count is our arithmetic.]
3. AdamW
For each weight, AdamW updates its two running averages \(m\) and \(v\), corrects their start-up bias, and moves the weight (formula 4.10 in session 4):
The update of each weight is roughly \(\eta\) in size, whatever the size of its gradient. Clipping still matters with Adam: a huge gradient enters \(m\) and \(v\) and distorts them for many steps. AdamW is elementwise: weight \(i\) only needs \(\theta_i, g_i, m_i, v_i\). That makes it easy to shard: each GPU can update only its own slice of the weights (ZeRO, FSDP).
How to read Figure 3
Move the step: during the first 2,000 steps the learning rate grows linearly; halfway through the decay it is \(1.65 \times 10^{-4}\), the average of the peak and the minimum. Move the gradient norm above \(c\): the right bar is cut to \(c\), and the readout gives the scale factor applied to every gradient. The last line gives the largest move of one weight in this step (about \(\eta\), by formula 32.15) and the shrink from weight decay (\(\eta\lambda\) of each weight, with \(\lambda = 0.1\)).
4. Clean up
Call optimizer.zero_grad(). Since PyTorch 2.0 its default is set_to_none=True: the gradient tensors are freed, not filled with zeros, and the next backward pass allocates them again. [Certain] If you forget this call, the next backward pass adds its gradients to the old ones. That is exactly what gradient accumulation does on purpose (session 39), and a silent bug when it is not on purpose. Then call scheduler.step() to move to the next learning rate.
- input: model, optimizer, scheduler, batch of ids of shape \((B, T + 1)\)
- \(\mathbf{x}, \mathbf{y} \leftarrow\) ids[:, :-1], ids[:, 1:] // formula (32.1)
- with autocast(BF16): \(Z \leftarrow\) model(\(\mathbf{x}\)) // (32.2), (32.3), (32.5); activations saved
- \(\mathcal{L} \leftarrow\) cross_entropy(\(Z\) in FP32, \(\mathbf{y}\), ignore_index = −100) // formula (32.6)
- \(\mathcal{L}\).backward() // Algorithm 1; fills .grad of every weight, frees activations
- (FP16 only: unscale the gradients first, see below)
- \(\lVert\mathbf{g}\rVert \leftarrow\) clip_grad_norm_(params, \(c\)) // formula (32.13); log it
- optimizer.step() // AdamW (32.15) with \(\eta(t)\) from (32.14); creates \(m, v\) at the first step
- scheduler.step(); optimizer.zero_grad(set_to_none = True)
- every k steps: log \(\mathcal{L}\).item() and \(\lVert\mathbf{g}\rVert\) // .item() waits for the GPU, so not every step
Where does mixed precision fit? With BF16 autocast, the matrix multiplies run in BF16 but the weights, the gradients that the optimizer reads, and the optimizer states stay in FP32 or are kept as an FP32 master copy (session 37). With FP16, small gradients would underflow to zero, so the loss is multiplied by a large scale before backward(), and the gradients must be divided by it again (scaler.unscale_) before clipping. Otherwise the threshold \(c\) is compared with a norm that is thousands of times too large. BF16 has the same exponent range as FP32 and needs no loss scaling. [Certain]
Watch one step run, with every number
Formulas are easier to trust when you see them produce numbers. Figure 4 runs the whole step on a model small enough to show every tensor and every weight. The vocabulary has \(V = 6\) words (“the”, “cat”, “sat”, “on”, “mat”, “.”), the width is \(d = 4\), and there is one residual MLP block with hidden width 8 and a ReLU: \(X_1 = X_0 + \operatorname{relu}(X_0 W_1)\,W_2\), then \(Z = X_1 W_{\text{out}}\). That is 112 weights in four matrices. The batch has \(B = 2\) sequences of \(T = 3\) inputs, so \(N_{\text{tok}} = 6\). The weights start from a fixed seed, and the same batch is used at every step: this is the “overfit one batch” check that every training script should pass.
The browser computes everything: the forward pass, the loss, the backward pass by formulas (32.7)–(32.11), the global-norm clipping (32.13) and AdamW (32.15) with \(\beta_1 = 0.9\), \(\beta_2 = 0.95\), \(\varepsilon = 10^{-8}\), and a constant learning rate. The algorithm box highlights the line that is running, and the box next to it shows that line’s formula with the real numbers put in. We checked the gradients against finite differences in Python (largest error \(3 \times 10^{-10}\)) and all the numbers below against a Python copy of the same program.
- \(\mathbf{x}, \mathbf{y} \leftarrow\) ids[:, :−1], ids[:, 1:] // (32.1)
- \(X_0 \leftarrow E[\mathbf{x}]\) // (32.2), a row lookup
- \(H \leftarrow \operatorname{relu}(X_0 W_1)\)
- \(X_1 \leftarrow X_0 + H W_2\) // residual, as in (32.3)
- \(Z \leftarrow X_1 W_{\text{out}};\ \mathbf{p} \leftarrow \softmax(Z)\) // (32.5)
- \(\mathcal{L} \leftarrow -\frac{1}{N}\sum \log p_{y}\) // (32.6)
- \(G_Z \leftarrow (\mathbf{p} - \mathbf{e}_y)/N\) // (32.7), backward starts
- \(\nabla W_{\text{out}} \leftarrow X_1^\top G_Z;\ G_{X_1} \leftarrow G_Z W_{\text{out}}^\top\) // (32.8)
- \(\nabla W_2 \leftarrow H^\top G_{X_1};\ G_H \leftarrow (G_{X_1} W_2^\top) \odot [X_0 W_1 \gt 0]\)
- \(\nabla W_1 \leftarrow X_0^\top G_H;\ G_{X_0} \leftarrow G_{X_1} + G_H W_1^\top\) // (32.9)
- \(\nabla E[v] \leftarrow \sum_{x = v} G_{X_0}\) // (32.11), scatter-add
- \(\mathbf{g} \leftarrow \mathbf{g}\,\min(1, c/\lVert\mathbf{g}\rVert)\) // (32.13)
- AdamW: \(m, v \leftarrow\) averages of \(g, g^2\); \(\theta \leftarrow \theta - \eta\big(\frac{\hat m}{\sqrt{\hat v} + \varepsilon} + \lambda\theta\big)\) // (32.15)
- zero_grad; next step: forward again with the new weights
How to read Figure 4
Press Step and follow the highlighted line. The colors of each matrix are scaled to that matrix, so you can see the pattern of its values even when gradients are small. With the defaults, the first loss is 2.041, a little above \(\ln 6 = 1.792\) (uniform guessing). Watch the example position “cat” (target “sat”): \(p_{\text{sat}} = 0.255\), so its row of \(G_Z\) is \((0.028, 0.022, -0.124, 0.023, 0.026, 0.025)\). Line 9 shows that 26 of the 48 hidden values are not positive, so ReLU blocks the gradient there. Line 10 shows the residual copy: \(G_{X_0}\) is \(G_{X_1}\) plus the branch. Line 11 shows the scatter-add: the row of “the” gets two contributions, and the row of “.” stays exactly 0, because “.” is only a target, never an input. Line 12: the gradient norm is 0.919 (the output layer gives 0.403 of the squared norm 0.845, the embedding 0.298), below \(c = 1\), so nothing is clipped. Line 13 follows one weight, \(W_{\text{out}}[0, \text{sat}]\): its gradient is 0.0147, and the first AdamW step moves it from \(-0.300\) to \(-0.349\), a move of \(\eta(1 + \lambda\theta) = 0.05 \times 0.970\). Every weight moves by about \(\eta\) at the first step (the largest move is 0.055). Line 14: the loss falls to 1.743. After 12 steps it is 0.476. Now set \(c = 0.05\): every gradient is scaled down about 18 times, but the new loss after step 1 is again 1.743, because Adam divides \(\hat m\) by \(\sqrt{\hat v}\) and the scale cancels. Later steps differ only a little (1.524 instead of 1.526 at step 3). This is exercise 2 in action. Now set 30 steps: the loss stops near 0.231 and never reaches 0. This model sees only the current token (it has no attention), and “the” is followed once by “cat” and once by “mat”. The best it can do is \(p = 1/2\) for each, so the loss floor is \(2 \ln 2 / 6 = 0.231\). Raise \(\eta\) to 0.2: the loss reaches the floor after about 8 steps, then wobbles around it (0.232, 0.247, 0.236, …). The triangle in the figure marks the example position used in the live box: sequence 0, token 1, input “cat”, target “sat”.
What lives in GPU memory, moment by moment
Now we can answer the memory question exactly. Four kinds of tensors live on the GPU during a step:
| Tensor | Bytes | When it exists | Llama 3 8B, one sequence of 8,192 |
|---|---|---|---|
| BF16 weights | \(2N\) | always | 16.1 GB |
| FP32 master weights + Adam \(m\), \(v\) | \(12N\) | always (\(m, v\) created at the first step()) | 96.4 GB |
| Gradients (BF16) | \(2N\) | from the moment backward reaches a layer until zero_grad | 0 to 16.1 GB |
| Saved activations | \(\approx 34\, BTd\) per layer | grow during forward, shrink during backward | 0 to 36.5 GB |
| Logits and their gradient (FP32) | \(4BTV\) each | end of forward to the output-layer backward | 4.2 GB + 4.2 GB |
The four parts peak at different moments. Weights and optimizer states are a constant floor. Activations climb during the forward pass, layer by layer, and fall during the backward pass in reverse order. Gradients start at zero and climb during the backward pass. Because activations fall while gradients rise, the peak is usually at the start of the backward pass, where all activations, the logits and the logit gradient exist together:
With activation checkpointing (session 39), each layer keeps only its input (\(2BTd\) bytes) and recomputes the rest during backward. Then the activation term is small and the gradients, which keep growing until the end of backward, can move the peak to the end of the backward pass. Figure 5 draws the whole timeline and plays it.
- \(\mathbf{x}, \mathbf{y} \leftarrow\) ids[:, :−1], ids[:, 1:]
- \(Z \leftarrow\) model(\(\mathbf{x}\)) // each block saves its activations
- \(\mathcal{L} \leftarrow\) cross_entropy(\(Z\) in FP32, \(\mathbf{y}\)) // logits: \(4BTV\) bytes
- \(\mathcal{L}\).backward() // \(G_Z\), then blocks \(L \to 1\): grads appear, activations freed
- clip_grad_norm_(params, \(c\))
- optimizer.step() // creates \(m, v\) (\(8N\) bytes) at the first step
- scheduler.step(); zero_grad(set_to_none=True) // frees \(2N\) bytes of grads
zero_grad. The model of each part follows the table above: activations \(34BTd\) bytes per layer (Korthikanti et al., no stored attention scores), FP32 logits. The dashed line is one 80 GB GPU.How to read Figure 5
Press Play to move through the step moment by moment; the algorithm box highlights the line that is running, and the box next to it adds up the four parts of memory with the current numbers, formula (32.16) at that moment. The x-axis is time inside one step: forward through the layers, the loss, backward through the layers in reverse, the optimizer step, then zero_grad. Move "Read the moment" to see what is in memory at that moment. For Llama 3 8B the floor alone (112 GB) is above one 80 GB GPU: this is why such training is sharded with FSDP or ZeRO. Switch to GPT-2 small and raise the batch: the logits (vocabulary 50,257) become the largest single part. Turn on checkpointing and see the peak move to the end of the backward pass. Turn on "First step": the Adam states appear only at optimizer.step(), so the first step can fail with out-of-memory after a successful forward and backward.
Floor: \(14N = 14 \times 8.03 \times 10^9 = 112.4\) GB. Activations: 36.5 GB (formula 32.4). Logits and their gradient: \(8 \times 8192 \times 128{,}256 = 8.4\) GB. Peak at the start of backward: \(112.4 + 36.5 + 8.4 = 157.3\) GB. Gradients are still zero at that moment (they are freed by zero_grad and not yet recreated).
With checkpointing: stored layer inputs are \(32 \times 2 \times 8192 \times 4096 = 2.1\) GB. The start of backward is now \(112.4 + 2.1 + 8.4 = 123.0\) GB, but at the end of the backward pass the gradients have reached 15.0 GB and the last recomputed layer adds 1.2 GB: the new peak is 128.6 GB, at the end of backward. On 8 GPUs with FSDP the \(16N\) part is split 8 ways (16.1 GB each), while each GPU still holds the activations of its own micro-batch.
Two lessons from this table are worth remembering. First, the logits are a real memory problem for large vocabularies. Fused kernels such as Liger Kernel and Cut Cross-Entropy never build the full \((B, T, V)\) tensor; they compute the loss and its gradient chunk by chunk. Cut Cross-Entropy reports that the loss memory for Gemma 2 2B falls from 24 GB to 1 MB. [Certain: reported in that paper.] Second, the activations scale with \(B \cdot T\) and the states scale with \(N\). So you fix activation memory with smaller micro-batches, checkpointing or sequence parallelism, and state memory with sharding or smaller optimizers.
One step in PyTorch, checked
This program runs one full step of a tiny GPT on the CPU and checks three claims of this session: the logit gradient is \((\mathbf{p} - \mathbf{e}_y)/N_{\text{tok}}\) (32.7), only the rows of used ids get an embedding gradient (32.11), and the first AdamW step moves each weight by about \(\eta\). It also counts the bytes that autograd saves for the backward pass with a saved_tensors_hooks context. It is deterministic (seeded) and needs PyTorch 2.4 or newer for nn.RMSNorm.
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) V, d, h, L, T, B = 64, 32, 4, 2, 16, 4 # vocab, width, heads, layers, length, batch class Block(nn.Module): def __init__(s): super().__init__() s.n1, s.n2 = nn.RMSNorm(d), nn.RMSNorm(d) s.qkv, s.o = nn.Linear(d, 3 * d, bias=False), nn.Linear(d, d, bias=False) s.up, s.down = nn.Linear(d, 4 * d, bias=False), nn.Linear(4 * d, d, bias=False) def forward(s, x): q, k, v = s.qkv(s.n1(x)).view(B, T, 3, h, d // h).transpose(1, 3).unbind(2) a = F.scaled_dot_product_attention(q, k, v, is_causal=True) x = x + s.o(a.transpose(1, 2).reshape(B, T, d)) # residual 1 return x + s.down(F.gelu(s.up(s.n2(x)))) # residual 2 class TinyGPT(nn.Module): def __init__(s): super().__init__() s.tok, s.pos = nn.Embedding(V, d), nn.Embedding(T, d) s.blocks = nn.ModuleList(Block() for _ in range(L)) s.norm, s.head = nn.RMSNorm(d), nn.Linear(d, V, bias=False) def forward(s, x): hdn = s.tok(x) + s.pos(torch.arange(T)) # (B, T, d) for blk in s.blocks: hdn = blk(hdn) return s.head(s.norm(hdn)) # (B, T, V) model = TinyGPT() decay = [p for p in model.parameters() if p.ndim >= 2] nodecay = [p for p in model.parameters() if p.ndim < 2] opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.1}, {"params": nodecay, "weight_decay": 0.0}], lr=1e-3, betas=(0.9, 0.95), eps=1e-8) W, S = 10, 100 # warmup, total steps sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda t: (t + 1) / W if t < W else 0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * (t - W) / (S - W)))) data = torch.randint(0, V, (B, T + 1)) x, y = data[:, :-1], data[:, 1:] # shift by one, (32.1) # 1) forward, counting the bytes autograd saves for backward saved = {} def pack(t): saved[(t.data_ptr(), t.shape)] = t.numel() * t.element_size(); return t with torch.autograd.graph.saved_tensors_hooks(pack, lambda t: t): logits = model(x) loss = F.cross_entropy(logits.reshape(-1, V), y.reshape(-1)) logits.retain_grad() n_par = sum(p.numel() for p in model.parameters()) print(f"loss {loss.item():.4f} ln V = {math.log(V):.4f}") print(f"params {n_par:,} ({4 * n_par:,} bytes) saved activations {sum(saved.values()):,} bytes") # 2) backward loss.backward() p = logits.softmax(-1) expect = (p - F.one_hot(y, V).float()) / (B * T) # formula (32.7) print("dL/dlogits == (p - onehot)/N :", torch.allclose(logits.grad, expect, atol=1e-7)) rows = model.tok.weight.grad.abs().sum(1) > 0 used = torch.zeros(V, dtype=torch.bool); used[x.unique()] = True print("embedding rows with gradient:", rows.sum().item(), "of", V, " match used ids:", torch.equal(rows, used)) # 3) clip, update, schedule, zero norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) print(f"grad norm before clip {norm.item():.4f}") w0 = model.head.weight.detach().clone(); lr = opt.param_groups[0]["lr"] opt.step(); sched.step(); opt.zero_grad(set_to_none=True) dw = (model.head.weight.detach() - w0).abs().max().item() print(f"lr used {lr:.1e} largest |change| in head / lr = {dw / lr:.3f}") print("next lr", f"{opt.param_groups[0]['lr']:.1e}", " grads freed:", model.head.weight.grad is None) print("adam state tensors per weight:", sorted(opt.state[model.head.weight].keys()))
Output (PyTorch 2.8, CPU):
loss 4.3468 ln V = 4.1589 params 29,344 (117,376 bytes) saved activations 463,876 bytes dL/dlogits == (p - onehot)/N : True embedding rows with gradient: 37 of 64 match used ids: True grad norm before clip 0.7936 lr used 1.0e-04 largest |change| in head / lr = 1.018 next lr 2.0e-04 grads freed: True adam state tensors per weight: ['exp_avg', 'exp_avg_sq', 'step']
Read the output. The first loss, 4.35, is close to \(\ln 64 = 4.16\), as expected at initialization. The saved activations (464 KB) are already four times the FP32 weights (117 KB), even for this tiny model with 16 tokens per sequence. Only 37 of the 64 embedding rows got a gradient. The gradient norm, 0.79, was below 1, so clipping did nothing this time. The first step moved the largest weight by 1.018 times the learning rate: 1 from the bias-corrected Adam ratio \(\hat m/\sqrt{\hat v} = \pm 1\), plus a little from weight decay \(\lambda\theta\). After zero_grad the gradient is None, and Adam now holds two state tensors per weight (plus a step counter).
Where this is used
| Setting | What changes in the step | What stays the same |
|---|---|---|
| LLM pretraining (Llama, Qwen, DeepSeek) | Thousands of GPUs: gradients all-reduced (DDP) or reduce-scattered (FSDP) during backward; weights all-gathered during forward | Forward, loss (32.6), backward, clip, AdamW, schedule |
| Supervised fine-tuning (SFT) | Mask \(m = 0\) on prompt tokens, so (32.7) is zero there; smaller learning rate | Everything else |
| LoRA and other PEFT | Base weights frozen: no gradients and no Adam states for them; only small adapter matrices are updated | Activations are still stored, because the gradient must flow back through frozen layers |
| VLM training (LLaVA style) | Image tokens enter as embeddings from a vision encoder and projector; loss mask is 0 on them; the encoder is often frozen | The decoder step; activations now also grow with the number of image tokens |
| Video-LLMs for video QA, captioning and key-moment detection | Hundreds or thousands of frame tokens per sample make activations dominate memory; checkpointing and fewer tokens per frame are common | Causal loss only on the text answer |
| Contrastive video–text retrieval and video fingerprinting models | The loss is contrastive (a softmax over the batch) instead of next-token; larger batches give more negatives | Backward, clipping, AdamW and schedule |
| Debugging a training run | Check \(\mathcal{L} \approx \ln V\) at step 0; overfit one batch; log gradient norm; read peak memory | This session's order of operations |
Common mistakes
- Forgetting
zero_grad. Gradients add up across steps, the effective learning rate grows every step, and training diverges or behaves strangely. Clear gradients once per optimizer step (with accumulation, once per accumulation cycle). - Clipping scaled gradients. With FP16 and a
GradScaler, callscaler.unscale_(optimizer)beforeclip_grad_norm_. Otherwise the norm is multiplied by the scale (often 65,536) and every step is clipped to almost nothing. - Keeping the graph alive.
losses.append(loss)ortotal += losskeeps every step's activations. Memory grows each step until out-of-memory. Useloss.item()(sparingly) orloss.detach(). - Calling
.item()every step. It forces the CPU to wait for the GPU and removes all overlap. Log every few steps, or keep a GPU tensor and read it rarely. - Wrong averaging of the loss. Averaging over all positions including padding, or averaging per sequence and then over sequences, gives different weights to different tokens. Use
ignore_index=-100and divide by the real token count \(N_{\text{tok}}\) (32.6). With accumulation, divide by the token count of the whole accumulated batch. - Calling
scheduler.step()at the wrong place. It belongs afteroptimizer.step(), once per optimizer step. Calling it per epoch, or per micro-batch, gives a very different schedule from the one you designed. - Logits in BF16 for the loss. Computing softmax and log over 128,256 entries in BF16 loses precision; compute the loss in FP32 (or with a fused kernel that does so internally).
- Measuring memory after the first step only. Adam states are created at the first
optimizer.step(), and the allocator caches memory. Readtorch.cuda.max_memory_allocated()after two or three steps.
Interview questions
Walk me through one training step of a decoder-only LLM.basic
Derive the gradient of softmax cross-entropy with respect to the logits.basic
Why must activations be stored, and which ones?core
How does the residual connection help the backward pass?core
Which rows of the embedding matrix get a gradient in one step? What changes with tied embeddings?core
What is the correct order of backward, clipping, optimizer step, scheduler and zero_grad? What if you use FP16?core
When during a step is GPU memory at its peak, and what is in memory then?advanced
Why is the output layer a memory problem for large vocabularies, and how is it solved?advanced
Your first-step loss is 25 on a model with a 50K vocabulary. What do you check?basic
In data-parallel training, where does communication happen inside the step?advanced
Why can the first training step run out of memory even though the forward and backward passes succeeded?advanced
step(). For AdamW that is two FP32 tensors per weight, \(8N\) bytes, allocated after the backward pass, while the gradients still exist. If memory was tight, this is where it fails. Also, the allocator caches freed blocks, so measure peak memory after a few steps, and keep in mind that the optimizer may create temporary buffers during the update.Exercises
- A model with \(V = 4\) gives logits \((0, 0, 0, 0)\) at one position, with target \(y = 2\). Compute the loss and the logit gradient (one position, \(N_{\text{tok}} = 1\)).
- The gradient norm before clipping is 4.0 and \(c = 1.0\). What factor multiplies every gradient? By how much does the update of each weight change with AdamW at a late step? (Think before answering: AdamW divides by \(\sqrt{\hat v}\).)
- With the schedule of Figure 3, compute \(\eta\) at step 1,000 and at step 251,000.
- Logits for a micro-batch of \(B = 4\) sequences of \(T = 4096\) tokens, \(V = 128{,}256\), in FP32: how many GB? How many with the gradient?
- The input is "the cat sat on the mat" (6 tokens, one id each). Which embedding rows get a gradient, and how many contributions does each get? What changes if the embeddings are tied?
- Run the attention-backward check below and explain why the result does not depend on the random values. Then add a causal mask check: what is \(G_S\) at the masked positions?
import torch torch.manual_seed(0) T, dh = 5, 4 Q, K, V = [torch.randn(T, dh, dtype=torch.float64, requires_grad=True) for _ in range(3)] S = (Q @ K.T / dh**0.5).masked_fill(torch.triu(torch.ones(T, T, dtype=torch.bool), 1), float("-inf")) P = S.softmax(-1); O = P @ V G = torch.randn(T, dh, dtype=torch.float64); (O * G).sum().backward() # so dL/dO = G dV = P.T @ G; dP = G @ V.T dS = P * (dP - (dP * P).sum(-1, keepdim=True)) # formula (32.10) print(torch.allclose(dV, V.grad), torch.allclose(dS @ K / dh**0.5, Q.grad), torch.allclose(dS.T @ Q / dh**0.5, K.grad)) # True True True
Answerscheck
Key takeaways
- One step: ids \((B, T)\) → embeddings \((B, T, d)\) → \(L\) blocks → logits \((B, T, V)\) → one loss averaged over real target tokens (32.1–32.6). Initial loss ≈ \(\ln V\).
- Every gradient starts as prediction minus truth, \((\mathbf{p} - \mathbf{e}_y)/N_{\text{tok}}\) (32.7), and flows back by \(X^\top G\) and \(G W^\top\) (32.8). The residual stream copies it through every block (32.9); attention uses the softmax backward (32.10); the embedding gets a scatter-add (32.11).
- Backward costs about twice the forward; a step is about \(6N\) FLOPs per token (32.12).
- Update order: (unscale) → clip the global norm (32.13) → AdamW with \(\eta(t)\) from the schedule (32.14, 32.15) → scheduler.step → zero_grad.
- Memory: a floor of \(14N\) for weights, master copy and Adam; activations rise in forward and fall in backward; gradients rise in backward; logits cost \(8BTV\). Without checkpointing the peak is at the start of backward (32.16): about 157 GB for Llama 3 8B on one 8K sequence.
Key formulas
References for this session
- Karpathy (2023). Neural Networks: Zero to Hero (video course)
- Stanford (2025). CS336: Language Modeling from Scratch (course)
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Chen et al. (2016). Training Deep Nets with Sublinear Memory Cost
- Micikevicius et al. (2017). Mixed Precision Training
- Loshchilov & Hutter (2017). Decoupled Weight Decay Regularization
- Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Hsu, Dai, Kothapalli, Song, Tang, Zhu, Shimizu, Sahni, Ning, Chen (2024). Liger Kernel: Efficient Triton Kernels for LLM Training
- Wijmans, Huval, Hertzberg, Koltun, Krähenbühl (2024). Cut Your Losses in Large-Vocabulary Language Models