LLM & VLM courseBlog
32Track 3: Pretraining and scale · session 2 of 22

One training step, end to end: forward, loss, backward, update

This session follows one batch through one full training step of an LLM: token ids become embeddings, pass through the blocks, become logits and one loss. Then the gradient flows back layer by layer, the optimizer clips, reads the learning rate and updates every weight. At each moment we ask which tensors are in GPU memory, and why the peak comes where it does.

Time: 3 hours (reading 2 hours, exercises 1 hour)Level: BeginnerBefore this: Session 4 (backpropagation and AdamW), session 25 (counting parameters, FLOPs and memory) and session 31 (pretraining objectives)

What you will learn

  • The shapes and sizes of every tensor in the forward pass, from (B, T) ids to (B, T, V) logits and one loss
  • Why the first gradient is prediction minus truth, (p − e_y)/N, and how to check it
  • Where every weight gradient comes from: X^T G for linear layers, the residual copy, the softmax backward in attention, the scatter-add in the embedding
  • The update in the right order: unscale, clip the global norm, AdamW with a warmup-cosine schedule, scheduler step, zero_grad
  • What lives in GPU memory at each moment of the step, and why the peak is usually at the start of the backward pass
  • How large logits, lazily created Adam states and checkpointing change the memory peak
  • A deterministic PyTorch step that checks these claims

Why this matters

Training a language model is one small procedure repeated hundreds of thousands of times: take a batch of tokens, compute the loss, compute the gradient of every weight, change the weights a little. This session follows one such step from the first token id to the last freed byte of memory.

So far the course has shown the pieces one at a time: embeddings (session 11), attention and the MLP, the residual stream (session 22), the causal loss (session 31), backpropagation and AdamW (session 4), and the counts of parameters, FLOPs and memory (session 25). Here we put them in the order in which a GPU actually runs them. Three questions guide the session:

  • Forward: what tensor exists after each stage, and what is its shape and size?
  • Backward: where does the gradient of every weight come from? Which formula produces it, and which stored tensor does that formula need?
  • Memory: what lives in GPU memory at each moment of the step, and when is the peak?

The answers explain most practical training problems. "CUDA out of memory" at the start of the backward pass, a loss that does not move because gradients were never cleared, a run that diverges because the clipping happened before un-scaling: these are all bugs in the order of the step. And the interview question "walk me through one training step of an LLM" is very common; a good answer gives shapes, gradients and memory, not only the four words "forward, backward, step, zero".

As a running example we use Llama 3 8B: \(L = 32\) layers, width \(d = 4096\), vocabulary \(V = 128{,}256\), MLP width 14,336, 32 query heads and 8 key-value heads, untied input and output embeddings, \(N = 8.03 \times 10^9\) parameters. [Certain: from the model's config.json.] We follow one sequence of \(T = 8192\) tokens, its pretraining context length.

The map of one step

One training step has five phases. Each phase reads some tensors and writes others:

  1. Load a batch. A tensor of token ids of shape \((B, T)\), and the targets, which are the same ids shifted by one position.
  2. Forward pass. Embedding lookup, \(L\) transformer blocks, a final norm, the output projection to logits, and the cross-entropy loss. Every operation saves the inputs it will need for its backward pass. These saved tensors are the activations.
  3. Backward pass. Start from the loss and go back through the same operations in reverse order. Each one turns the gradient of its output into the gradient of its input and of its weights, then frees its saved activations.
  4. Update. Clip the global gradient norm, read the learning rate from the schedule, and apply AdamW to every weight.
  5. Clean up. Free (or zero) the gradients so the next step starts fresh, and advance the schedule.

Step through Figure 1. The readout gives the shape and the size in memory of the main tensor at each moment, for Llama 3 8B with one sequence of 8,192 tokens.

forward (active)backward (active)update and clean-up
Figure 1. One training step of Llama 3 8B on one sequence of 8,192 tokens, stage by stage. Forward goes left to right along the top row, backward right to left, then the optimizer updates every weight.
How to read Figure 1

Press Next. Blue boxes are forward stages, orange boxes backward stages, green boxes the update. Watch the numbers in the readout: the hidden states are 64 MiB, but the logits are about 1.05 billion numbers (4.2 GB in FP32), larger than all the hidden states of one layer. The saved activations of all 32 blocks add up to tens of GB. These numbers come back in Figure 5. Figure 4 runs the same stages for real, with every number, on a model small enough to show all of its weights.

Forward: from token ids to one number

Batch and targets

The data loader gives a tensor of shape \((B, T + 1)\) of token ids. The input is the first \(T\) columns, and the target at each position is the next id:

\[ \mathbf{x} = \text{ids}[:, 0{:}T], \qquad \mathbf{y} = \text{ids}[:, 1{:}T{+}1], \qquad \mathbf{x}, \mathbf{y} \in \{0, \dots, V{-}1\}^{B \times T} \tag{32.1} \]

With packing (session 18), every position is a real token, so a batch gives \(B \cdot T\) predictions. Llama 2 used batches of about 4 million tokens per step. [Certain] The ids are 64-bit integers: one sequence of 8,192 ids is only 64 KiB. Data is never the memory problem.

Embedding lookup

Each id selects one row of the embedding matrix \(E \in \R^{V \times d}\):

\[ X_0[b, t, :] = E[\,x_{b,t}, :\,], \qquad X_0 \in \R^{B \times T \times d} \tag{32.2} \]

This is a lookup, not a matrix multiply: it costs almost no FLOPs. For one sequence, \(X_0\) has \(8192 \times 4096 = 33.6\) million numbers, 64 MiB in BF16. Llama has no position embedding to add here; position enters inside attention through RoPE (session 14).

The blocks

Each of the \(L\) blocks reads the residual stream and adds two updates to it (pre-norm, as in Llama):

\[ U_\ell = X_{\ell-1} + \operatorname{Attn}_\ell\big(\operatorname{Norm}(X_{\ell-1})\big), \qquad X_\ell = U_\ell + \operatorname{MLP}_\ell\big(\operatorname{Norm}(U_\ell)\big) \tag{32.3} \]

The shape stays \((B, T, d)\) through all blocks. But inside a block many larger tensors appear: the queries, keys and values, the attention output, and the MLP's hidden layer (width 14,336, which is 3.5 times \(d\)). Each linear layer saves its input for the backward pass (it needs \(X\) to compute \(\partial L/\partial W = X^\top G\)). Each norm saves its input. The MLP saves the values before and after its activation function. Korthikanti et al. counted these saved tensors for a GPT-style block in 16-bit: about \(34\) bytes per number of the residual stream, per layer, when the attention scores are not stored (FlashAttention). [Certain for their GPT block; for a Llama block with SwiGLU the constant is similar but not identical. Likely.] For our sequence that is

\[ M_{\text{act}} \approx L \cdot 34 \cdot B\,T\,d \ \text{bytes} = 32 \times 34 \times 8192 \times 4096 \approx 36.5 \ \text{GB} \tag{32.4} \]

1.14 GB per layer, 36.5 GB for the model, for a single sequence. The weights in BF16 are 16.1 GB. So for long sequences, the stored activations are larger than the model itself.

Logits and loss

After the last block, a final norm and the output matrix \(W_{\text{out}} \in \R^{d \times V}\) give one score per vocabulary entry at every position:

\[ H = \operatorname{Norm}(X_L), \qquad Z = H\,W_{\text{out}} \in \R^{B \times T \times V} \tag{32.5} \]

\(Z\) is the largest tensor of the forward pass: \(8192 \times 128{,}256 = 1.05\) billion numbers. Losses are computed in FP32 for accuracy, so this one tensor takes 4.2 GB. Then the cross-entropy loss averages the negative log-probability of each target over the \(N_{\text{tok}}\) positions that have a target:

\[ \mathcal{L} = -\frac{1}{N_{\text{tok}}} \sum_{b, t} m_{b,t}\, \log \softmax(\mathbf{z}_{b,t})_{y_{b,t}}, \qquad N_{\text{tok}} = \sum_{b,t} m_{b,t} \tag{32.6} \]
Read the symbols
\(\mathbf{z}_{b,t} \in \R^V\)
the logits at sequence \(b\), position \(t\)
\(y_{b,t}\)
the target id: the next token in the text
\(m_{b,t}\)
1 if this position gets a loss, 0 if not (padding, or a prompt in SFT). PyTorch marks the 0 positions with the label \(-100\), the ignore_index.
\(N_{\text{tok}}\)
how many positions really get a loss; the average is over them only

At initialization the model knows nothing, all \(V\) tokens get about the same probability, and the loss is close to \(\ln V\). For Llama 3 that is \(\ln 128{,}256 = 11.76\) nats. If your first loss is much larger, the initialization or the output scale is wrong. This is a cheap and very useful check. [Certain]

The first gradient: softmax and cross-entropy

The backward pass starts at the scalar \(\mathcal{L}\), with \(\partial \mathcal{L}/\partial \mathcal{L} = 1\). The first real gradient is the one of the logits. For one position with logits \(\mathbf{z}\), probabilities \(\mathbf{p} = \softmax(\mathbf{z})\) and target \(y\), the loss is \(-\log p_y = -z_y + \log \sum_j e^{z_j}\). Its derivative with respect to \(z_i\) is \(-[i = y] + e^{z_i}/\sum_j e^{z_j}\). So:

\[ \frac{\partial \mathcal{L}}{\partial \mathbf{z}_{b,t}} = \frac{m_{b,t}}{N_{\text{tok}}} \big(\mathbf{p}_{b,t} - \mathbf{e}_{y_{b,t}}\big) \tag{32.7} \]
Read the symbols
\(\mathbf{p}_{b,t}\)
the predicted distribution at that position
\(\mathbf{e}_{y}\)
the one-hot vector of the target: 1 at position \(y\), 0 everywhere else
\(m / N_{\text{tok}}\)
the averaging: each position contributes with weight \(1/N_{\text{tok}}\), and ignored positions contribute 0

This is the cleanest formula in deep learning: prediction minus truth. The gradient pushes the target's logit up by \(1 - p_y\) and every other logit down by its own probability. It is never larger than 1 in size, and it is 0 only when the model puts probability 1 on the right token. Every gradient in the whole network comes from this one tensor, sent backward.

Worked example: the logit gradient for three tokens

Logits \(\mathbf{z} = (2.0, 1.0, 0.1)\), target \(y = 0\), one position (\(N_{\text{tok}} = 1\)). Softmax: \(\mathbf{p} = (0.659, 0.242, 0.099)\). Loss: \(-\ln 0.659 = 0.417\).

Gradient: \(\mathbf{p} - \mathbf{e}_0 = (0.659 - 1,\ 0.242,\ 0.099) = (-0.341, 0.242, 0.099)\). The components sum to 0, as they always must: the probabilities and the one-hot vector both sum to 1. A step against this gradient raises \(z_0\) and lowers \(z_1\) and \(z_2\).

Figure 2 lets you do this by hand. Choose the target, then take gradient steps directly on the logits (as if the logits were the weights).

Context: "the cat sat on the ..." Choose the target token:
probability \(p_i\)gradient \(p_i - [i = y]\)target
Figure 2. The first gradient of the backward pass, for a vocabulary of six tokens. Top: the probabilities. Bottom: the gradient of the loss with respect to each logit, formula (32.7).
How to read Figure 2

The orange bar of the target is always negative (its logit must go up), the others are positive and equal to their probabilities. The bars always sum to zero. Press "Take one step" a few times: the loss falls fast at first, then slowly, because the gradient \(1 - p_y\) shrinks as \(p_y\) grows. Choose an unlikely target such as "sat": the first loss is large and the gradient is close to \(-1\), the largest it can be. The box under the readout shows the last update with its numbers, \(z_i \leftarrow z_i - \eta\,(p_i - [i = y])\). With the default target "mat" and \(\eta = 2\), press Play: \(p_y\) goes 0.406, 0.756, 0.851, 0.892 and the loss goes 0.902, 0.280, 0.162, 0.115. The first step gains the most; each later step gains less.

Backward: where every gradient comes from

Now we walk back through the network, in exactly the reverse order of the forward pass. Every operation uses the same two rules from session 4: the chain rule, and for a linear layer \(Y = XW\) with incoming gradient \(G = \partial \mathcal{L}/\partial Y\):

\[ \frac{\partial \mathcal{L}}{\partial W} = X^\top G, \qquad \frac{\partial \mathcal{L}}{\partial X} = G\, W^\top \tag{32.8} \]

(For a batch of shape \((B, T, \cdot)\), the first formula also sums over all \(B \cdot T\) positions: every position adds its own small contribution to the weight gradient.) The first formula needs the stored input \(X\). The second needs the weight \(W\). This is why activations must be kept: the weight gradient of every linear layer is built from its forward input.

The output layer and the final norm

From \(G_Z = \partial \mathcal{L}/\partial Z\) (formula 32.7, shape \((B, T, V)\), another 4.2 GB in FP32), formula (32.8) gives \(\partial \mathcal{L}/\partial W_{\text{out}} = H^\top G_Z\), a \(4096 \times 128{,}256\) matrix, and \(\partial \mathcal{L}/\partial H = G_Z W_{\text{out}}^\top\), back to shape \((B, T, d)\). After this, the logits and their gradient can be freed. The final RMSNorm then turns \(\partial \mathcal{L}/\partial H\) into \(\partial \mathcal{L}/\partial X_L\), using its stored input, and gives the gradient of its scale vector.

The residual stream: a gradient highway

Write one sub-layer as \(X_{\text{out}} = X_{\text{in}} + F(X_{\text{in}})\). The chain rule gives

\[ \frac{\partial \mathcal{L}}{\partial X_{\text{in}}} = \frac{\partial \mathcal{L}}{\partial X_{\text{out}}} + \Big(\frac{\partial F}{\partial X_{\text{in}}}\Big)^{\!\top} \frac{\partial \mathcal{L}}{\partial X_{\text{out}}} \tag{32.9} \]

The first term is a plain copy. Whatever gradient arrives at the top of a block passes to the bottom unchanged, and the block only adds its own term. So the gradient of the loss reaches the embedding through 64 additions (two per block), not through 64 multiplications that could shrink it to zero. This is the main reason very deep transformers train at all (session 22). [Certain]

Inside a block: the MLP and attention

The MLP is three linear layers and an elementwise function. Its backward pass is formula (32.8) three times, plus the derivative of the activation (SiLU in Llama) at the stored pre-activation values. Attention is more interesting. With scores \(S = QK^\top/\sqrt{d_h}\) (causal mask applied), \(P = \softmax(S)\) row by row, output \(O = PV\), and incoming gradient \(G_O\):

\[ \begin{aligned} G_V &= P^\top G_O, \qquad G_P = G_O V^\top \\ G_S &= P \odot \big(G_P - \operatorname{rowsum}(G_P \odot P)\big) \\ G_Q &= G_S K / \sqrt{d_h}, \qquad G_K = G_S^\top Q / \sqrt{d_h} \end{aligned} \tag{32.10} \]
Read the symbols
\(\odot\)
elementwise product
\(\operatorname{rowsum}(\cdot)\)
the sum of each row, kept as a column and subtracted from every element of that row: the softmax backward step
\(G_Q, G_K, G_V\)
gradients that then flow into the query, key and value projections by formula (32.8), and through RoPE (a fixed rotation, whose backward is the inverse rotation)

The softmax backward needs \(P\), a \(T \times T\) matrix per head. Storing it for \(T = 8192\) and 32 heads would be 4.3 GB per layer in BF16. FlashAttention does not store it. It keeps only \(Q, K, V, O\) and one number per row (the log-sum-exp), and recomputes \(P\) block by block in the backward pass (session 124). [Certain] We checked formula (32.10) against PyTorch autograd in float64; the code is in the exercises.

The embedding: a scatter-add

The lookup (32.2) copied row \(x_{b,t}\) of \(E\) into position \((b, t)\). Its backward does the opposite: it adds the gradient of each position into the row it came from:

\[ \frac{\partial \mathcal{L}}{\partial E[v, :]} = \sum_{(b,t)\,:\,x_{b,t} = v} \frac{\partial \mathcal{L}}{\partial X_0[b, t, :]} \tag{32.11} \]

Rows of tokens that do not appear in the batch get exactly zero gradient. A token that appears 50 times gets the sum of 50 contributions. In PyTorch this is an index_add into a dense \(V \times d\) gradient (unless you ask for sparse gradients). If the model ties the input and output embeddings (GPT-2, many small models), the same matrix gets two gradients, one from (32.11) and one from the output layer, and autograd adds them. [Certain]

Algorithm 1: the backward pass of a pre-norm decoder, layer by layer
  1. input: loss \(\mathcal{L}\), saved activations of every layer, weights
  2. \(G_Z \leftarrow \frac{m}{N_{\text{tok}}}(\mathbf{p} - \mathbf{e}_y)\) // formula (32.7), shape \((B, T, V)\)
  3. \(\nabla W_{\text{out}} \leftarrow H^\top G_Z\); \(\;G \leftarrow G_Z W_{\text{out}}^\top\); free \(Z\), \(G_Z\) // formula (32.8)
  4. \(G \leftarrow\) backward of the final norm (uses its saved input); \(\nabla\gamma_{\text{final}}\)
  5. for \(\ell = L, L-1, \dots, 1\):
  6.   \(G_U \leftarrow G + \) MLP-branch backward of \(G\) // formula (32.9); gives \(\nabla\) of 3 MLP matrices and the norm
  7.   \(G \leftarrow G_U + \) attention-branch backward of \(G_U\) // formula (32.10); gives \(\nabla\) of \(W_q, W_k, W_v, W_o\) and the norm
  8.   free the saved activations of layer \(\ell\); (DDP: start the all-reduce of this layer's gradients)
  9. \(\nabla E \leftarrow\) scatter-add of \(G\) into the rows of the input ids // formula (32.11)
  10. return a gradient for every weight

Count the work: each linear layer does two matrix multiplies in the backward pass (one for \(\nabla W\), one for \(\nabla X\)) and one in the forward pass. So backward costs about twice the forward, and one step costs about

\[ C_{\text{step}} \approx 6\, N \cdot B\,T \ \text{FLOPs} \tag{32.12} \]

For one sequence of 8,192 tokens on Llama 3 8B: \(6 \times 8.03 \times 10^9 \times 8192 = 3.95 \times 10^{14}\) FLOPs, without the attention-score term. A GPU that sustains 400 TFLOP/s needs about 1 second for it. For a 4-million-token batch the step is \(1.93 \times 10^{17}\) FLOPs, which is why such batches are spread over hundreds or thousands of GPUs (session 25).

The update: clipping, schedule, AdamW

When the backward pass ends, every weight \(\theta_i\) has a gradient \(g_i\) in its .grad field. Three things happen before the weights change, in this order.

1. Clip the global norm

Compute one number for the whole model, the length of the gradient vector over all 8 billion weights, and shrink the whole vector if it is too long:

\[ \lVert \mathbf{g} \rVert = \sqrt{\textstyle\sum_{i=1}^{N} g_i^2}, \qquad \mathbf{g} \leftarrow \mathbf{g} \cdot \min\!\Big(1, \frac{c}{\lVert \mathbf{g} \rVert}\Big) \tag{32.13} \]

The usual threshold is \(c = 1.0\), as in Llama 2. [Certain] The norm must be computed over all gradients, so with sharded training (FSDP) each GPU computes its part of the sum of squares and the parts are added with an all-reduce. Log the norm before clipping: it is one of the best health signals of a run (session 36).

2. Read the learning rate

The schedule turns the step number \(t\) into a learning rate. The common shape is linear warmup over \(W\) steps, then cosine decay to a fraction \(r\) of the peak at the last step \(T_{\max}\):

\[ \eta(t) = \begin{cases} \eta_{\max}\, t / W, & t \lt W \\ \eta_{\max} \Big( r + (1 - r)\, \tfrac12 \big(1 + \cos \pi \tfrac{t - W}{T_{\max} - W}\big) \Big), & t \ge W \end{cases} \tag{32.14} \]

For a Llama 2 7B-like run, \(\eta_{\max} = 3 \times 10^{-4}\), \(W = 2000\), \(r = 0.1\) and about 500,000 steps (2 trillion tokens at 4 million per step). [Certain for the recipe; the step count is our arithmetic.]

3. AdamW

For each weight, AdamW updates its two running averages \(m\) and \(v\), corrects their start-up bias, and moves the weight (formula 4.10 in session 4):

\[ \theta \leftarrow \theta - \eta(t) \Big( \frac{\hat m}{\sqrt{\hat v} + \varepsilon} + \lambda\, \theta \Big) \tag{32.15} \]

The update of each weight is roughly \(\eta\) in size, whatever the size of its gradient. Clipping still matters with Adam: a huge gradient enters \(m\) and \(v\) and distorts them for many steps. AdamW is elementwise: weight \(i\) only needs \(\theta_i, g_i, m_i, v_i\). That makes it easy to shard: each GPU can update only its own slice of the weights (ZeRO, FSDP).

Figure 3. The update phase of one step, with the Llama 2 7B recipe: peak learning rate \(3 \times 10^{-4}\), 2,000 warmup steps, cosine decay to 10% over 500,000 steps, clipping at \(c\). Left: the schedule (32.14). Right: the gradient norm before and after clipping (32.13).
How to read Figure 3

Move the step: during the first 2,000 steps the learning rate grows linearly; halfway through the decay it is \(1.65 \times 10^{-4}\), the average of the peak and the minimum. Move the gradient norm above \(c\): the right bar is cut to \(c\), and the readout gives the scale factor applied to every gradient. The last line gives the largest move of one weight in this step (about \(\eta\), by formula 32.15) and the shrink from weight decay (\(\eta\lambda\) of each weight, with \(\lambda = 0.1\)).

4. Clean up

Call optimizer.zero_grad(). Since PyTorch 2.0 its default is set_to_none=True: the gradient tensors are freed, not filled with zeros, and the next backward pass allocates them again. [Certain] If you forget this call, the next backward pass adds its gradients to the old ones. That is exactly what gradient accumulation does on purpose (session 39), and a silent bug when it is not on purpose. Then call scheduler.step() to move to the next learning rate.

Algorithm 2: one complete training step (BF16 autocast, AdamW, clipping, schedule)
  1. input: model, optimizer, scheduler, batch of ids of shape \((B, T + 1)\)
  2. \(\mathbf{x}, \mathbf{y} \leftarrow\) ids[:, :-1], ids[:, 1:] // formula (32.1)
  3. with autocast(BF16): \(Z \leftarrow\) model(\(\mathbf{x}\)) // (32.2), (32.3), (32.5); activations saved
  4. \(\mathcal{L} \leftarrow\) cross_entropy(\(Z\) in FP32, \(\mathbf{y}\), ignore_index = −100) // formula (32.6)
  5. \(\mathcal{L}\).backward() // Algorithm 1; fills .grad of every weight, frees activations
  6. (FP16 only: unscale the gradients first, see below)
  7. \(\lVert\mathbf{g}\rVert \leftarrow\) clip_grad_norm_(params, \(c\)) // formula (32.13); log it
  8. optimizer.step() // AdamW (32.15) with \(\eta(t)\) from (32.14); creates \(m, v\) at the first step
  9. scheduler.step(); optimizer.zero_grad(set_to_none = True)
  10. every k steps: log \(\mathcal{L}\).item() and \(\lVert\mathbf{g}\rVert\) // .item() waits for the GPU, so not every step

Where does mixed precision fit? With BF16 autocast, the matrix multiplies run in BF16 but the weights, the gradients that the optimizer reads, and the optimizer states stay in FP32 or are kept as an FP32 master copy (session 37). With FP16, small gradients would underflow to zero, so the loss is multiplied by a large scale before backward(), and the gradients must be divided by it again (scaler.unscale_) before clipping. Otherwise the threshold \(c\) is compared with a norm that is thousands of times too large. BF16 has the same exponent range as FP32 and needs no loss scaling. [Certain]

Watch one step run, with every number

Formulas are easier to trust when you see them produce numbers. Figure 4 runs the whole step on a model small enough to show every tensor and every weight. The vocabulary has \(V = 6\) words (“the”, “cat”, “sat”, “on”, “mat”, “.”), the width is \(d = 4\), and there is one residual MLP block with hidden width 8 and a ReLU: \(X_1 = X_0 + \operatorname{relu}(X_0 W_1)\,W_2\), then \(Z = X_1 W_{\text{out}}\). That is 112 weights in four matrices. The batch has \(B = 2\) sequences of \(T = 3\) inputs, so \(N_{\text{tok}} = 6\). The weights start from a fixed seed, and the same batch is used at every step: this is the “overfit one batch” check that every training script should pass.

The browser computes everything: the forward pass, the loss, the backward pass by formulas (32.7)–(32.11), the global-norm clipping (32.13) and AdamW (32.15) with \(\beta_1 = 0.9\), \(\beta_2 = 0.95\), \(\varepsilon = 10^{-8}\), and a constant learning rate. The algorithm box highlights the line that is running, and the box next to it shows that line’s formula with the real numbers put in. We checked the gradients against finite differences in Python (largest error \(3 \times 10^{-10}\)) and all the numbers below against a Python copy of the same program.

value > 0value < 0gradient > 0gradient < 0written nowread nowupdated weight
  1. \(\mathbf{x}, \mathbf{y} \leftarrow\) ids[:, :−1], ids[:, 1:] // (32.1)
  2. \(X_0 \leftarrow E[\mathbf{x}]\) // (32.2), a row lookup
  3. \(H \leftarrow \operatorname{relu}(X_0 W_1)\)
  4. \(X_1 \leftarrow X_0 + H W_2\) // residual, as in (32.3)
  5. \(Z \leftarrow X_1 W_{\text{out}};\ \mathbf{p} \leftarrow \softmax(Z)\) // (32.5)
  6. \(\mathcal{L} \leftarrow -\frac{1}{N}\sum \log p_{y}\) // (32.6)
  7. \(G_Z \leftarrow (\mathbf{p} - \mathbf{e}_y)/N\) // (32.7), backward starts
  8. \(\nabla W_{\text{out}} \leftarrow X_1^\top G_Z;\ G_{X_1} \leftarrow G_Z W_{\text{out}}^\top\) // (32.8)
  9. \(\nabla W_2 \leftarrow H^\top G_{X_1};\ G_H \leftarrow (G_{X_1} W_2^\top) \odot [X_0 W_1 \gt 0]\)
  10. \(\nabla W_1 \leftarrow X_0^\top G_H;\ G_{X_0} \leftarrow G_{X_1} + G_H W_1^\top\) // (32.9)
  11. \(\nabla E[v] \leftarrow \sum_{x = v} G_{X_0}\) // (32.11), scatter-add
  12. \(\mathbf{g} \leftarrow \mathbf{g}\,\min(1, c/\lVert\mathbf{g}\rVert)\) // (32.13)
  13. AdamW: \(m, v \leftarrow\) averages of \(g, g^2\); \(\theta \leftarrow \theta - \eta\big(\frac{\hat m}{\sqrt{\hat v} + \varepsilon} + \lambda\theta\big)\) // (32.15)
  14. zero_grad; next step: forward again with the new weights
Figure 4. One training step of a tiny model, computed live and repeated. Top row: the forward tensors, with shapes \((B, T, \cdot)\). Middle row: their gradients, written right to left by the backward pass. Bottom row: the weights; each one is replaced by its gradient when the backward pass reaches it, and by its new value after AdamW. Activations fade when they are freed. The panel at the bottom right plots the loss after every step.
How to read Figure 4

Press Step and follow the highlighted line. The colors of each matrix are scaled to that matrix, so you can see the pattern of its values even when gradients are small. With the defaults, the first loss is 2.041, a little above \(\ln 6 = 1.792\) (uniform guessing). Watch the example position “cat” (target “sat”): \(p_{\text{sat}} = 0.255\), so its row of \(G_Z\) is \((0.028, 0.022, -0.124, 0.023, 0.026, 0.025)\). Line 9 shows that 26 of the 48 hidden values are not positive, so ReLU blocks the gradient there. Line 10 shows the residual copy: \(G_{X_0}\) is \(G_{X_1}\) plus the branch. Line 11 shows the scatter-add: the row of “the” gets two contributions, and the row of “.” stays exactly 0, because “.” is only a target, never an input. Line 12: the gradient norm is 0.919 (the output layer gives 0.403 of the squared norm 0.845, the embedding 0.298), below \(c = 1\), so nothing is clipped. Line 13 follows one weight, \(W_{\text{out}}[0, \text{sat}]\): its gradient is 0.0147, and the first AdamW step moves it from \(-0.300\) to \(-0.349\), a move of \(\eta(1 + \lambda\theta) = 0.05 \times 0.970\). Every weight moves by about \(\eta\) at the first step (the largest move is 0.055). Line 14: the loss falls to 1.743. After 12 steps it is 0.476. Now set \(c = 0.05\): every gradient is scaled down about 18 times, but the new loss after step 1 is again 1.743, because Adam divides \(\hat m\) by \(\sqrt{\hat v}\) and the scale cancels. Later steps differ only a little (1.524 instead of 1.526 at step 3). This is exercise 2 in action. Now set 30 steps: the loss stops near 0.231 and never reaches 0. This model sees only the current token (it has no attention), and “the” is followed once by “cat” and once by “mat”. The best it can do is \(p = 1/2\) for each, so the loss floor is \(2 \ln 2 / 6 = 0.231\). Raise \(\eta\) to 0.2: the loss reaches the floor after about 8 steps, then wobbles around it (0.232, 0.247, 0.236, …). The triangle in the figure marks the example position used in the live box: sequence 0, token 1, input “cat”, target “sat”.

What lives in GPU memory, moment by moment

Now we can answer the memory question exactly. Four kinds of tensors live on the GPU during a step:

TensorBytesWhen it existsLlama 3 8B, one sequence of 8,192
BF16 weights\(2N\)always16.1 GB
FP32 master weights + Adam \(m\), \(v\)\(12N\)always (\(m, v\) created at the first step())96.4 GB
Gradients (BF16)\(2N\)from the moment backward reaches a layer until zero_grad0 to 16.1 GB
Saved activations\(\approx 34\, BTd\) per layergrow during forward, shrink during backward0 to 36.5 GB
Logits and their gradient (FP32)\(4BTV\) eachend of forward to the output-layer backward4.2 GB + 4.2 GB

The four parts peak at different moments. Weights and optimizer states are a constant floor. Activations climb during the forward pass, layer by layer, and fall during the backward pass in reverse order. Gradients start at zero and climb during the backward pass. Because activations fall while gradients rise, the peak is usually at the start of the backward pass, where all activations, the logits and the logit gradient exist together:

\[ M_{\text{peak}} \approx \underbrace{14N}_{\text{weights + master + Adam}} + \underbrace{34\,L\,BTd}_{\text{activations}} + \underbrace{8\,BTV}_{\text{logits + their gradient}} \ \text{bytes} \tag{32.16} \]

With activation checkpointing (session 39), each layer keeps only its input (\(2BTd\) bytes) and recomputes the rest during backward. Then the activation term is small and the gradients, which keep growing until the end of backward, can move the peak to the end of the backward pass. Figure 5 draws the whole timeline and plays it.

weights + master + Adamgradientssaved activationslogits and their gradient
  1. \(\mathbf{x}, \mathbf{y} \leftarrow\) ids[:, :−1], ids[:, 1:]
  2. \(Z \leftarrow\) model(\(\mathbf{x}\)) // each block saves its activations
  3. \(\mathcal{L} \leftarrow\) cross_entropy(\(Z\) in FP32, \(\mathbf{y}\)) // logits: \(4BTV\) bytes
  4. \(\mathcal{L}\).backward() // \(G_Z\), then blocks \(L \to 1\): grads appear, activations freed
  5. clip_grad_norm_(params, \(c\))
  6. optimizer.step() // creates \(m, v\) (\(8N\) bytes) at the first step
  7. scheduler.step(); zero_grad(set_to_none=True) // frees \(2N\) bytes of grads
Figure 5. GPU memory during one training step, without any sharding, from the start of the forward pass to zero_grad. The model of each part follows the table above: activations \(34BTd\) bytes per layer (Korthikanti et al., no stored attention scores), FP32 logits. The dashed line is one 80 GB GPU.
How to read Figure 5

Press Play to move through the step moment by moment; the algorithm box highlights the line that is running, and the box next to it adds up the four parts of memory with the current numbers, formula (32.16) at that moment. The x-axis is time inside one step: forward through the layers, the loss, backward through the layers in reverse, the optimizer step, then zero_grad. Move "Read the moment" to see what is in memory at that moment. For Llama 3 8B the floor alone (112 GB) is above one 80 GB GPU: this is why such training is sharded with FSDP or ZeRO. Switch to GPT-2 small and raise the batch: the logits (vocabulary 50,257) become the largest single part. Turn on checkpointing and see the peak move to the end of the backward pass. Turn on "First step": the Adam states appear only at optimizer.step(), so the first step can fail with out-of-memory after a successful forward and backward.

Worked example: the peak for Llama 3 8B, one sequence

Floor: \(14N = 14 \times 8.03 \times 10^9 = 112.4\) GB. Activations: 36.5 GB (formula 32.4). Logits and their gradient: \(8 \times 8192 \times 128{,}256 = 8.4\) GB. Peak at the start of backward: \(112.4 + 36.5 + 8.4 = 157.3\) GB. Gradients are still zero at that moment (they are freed by zero_grad and not yet recreated).

With checkpointing: stored layer inputs are \(32 \times 2 \times 8192 \times 4096 = 2.1\) GB. The start of backward is now \(112.4 + 2.1 + 8.4 = 123.0\) GB, but at the end of the backward pass the gradients have reached 15.0 GB and the last recomputed layer adds 1.2 GB: the new peak is 128.6 GB, at the end of backward. On 8 GPUs with FSDP the \(16N\) part is split 8 ways (16.1 GB each), while each GPU still holds the activations of its own micro-batch.

Two lessons from this table are worth remembering. First, the logits are a real memory problem for large vocabularies. Fused kernels such as Liger Kernel and Cut Cross-Entropy never build the full \((B, T, V)\) tensor; they compute the loss and its gradient chunk by chunk. Cut Cross-Entropy reports that the loss memory for Gemma 2 2B falls from 24 GB to 1 MB. [Certain: reported in that paper.] Second, the activations scale with \(B \cdot T\) and the states scale with \(N\). So you fix activation memory with smaller micro-batches, checkpointing or sequence parallelism, and state memory with sharding or smaller optimizers.

One step in PyTorch, checked

This program runs one full step of a tiny GPT on the CPU and checks three claims of this session: the logit gradient is \((\mathbf{p} - \mathbf{e}_y)/N_{\text{tok}}\) (32.7), only the rows of used ids get an embedding gradient (32.11), and the first AdamW step moves each weight by about \(\eta\). It also counts the bytes that autograd saves for the backward pass with a saved_tensors_hooks context. It is deterministic (seeded) and needs PyTorch 2.4 or newer for nn.RMSNorm.

import math, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
V, d, h, L, T, B = 64, 32, 4, 2, 16, 4          # vocab, width, heads, layers, length, batch

class Block(nn.Module):
    def __init__(s):
        super().__init__()
        s.n1, s.n2 = nn.RMSNorm(d), nn.RMSNorm(d)
        s.qkv, s.o = nn.Linear(d, 3 * d, bias=False), nn.Linear(d, d, bias=False)
        s.up, s.down = nn.Linear(d, 4 * d, bias=False), nn.Linear(4 * d, d, bias=False)
    def forward(s, x):
        q, k, v = s.qkv(s.n1(x)).view(B, T, 3, h, d // h).transpose(1, 3).unbind(2)
        a = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        x = x + s.o(a.transpose(1, 2).reshape(B, T, d))       # residual 1
        return x + s.down(F.gelu(s.up(s.n2(x))))              # residual 2

class TinyGPT(nn.Module):
    def __init__(s):
        super().__init__()
        s.tok, s.pos = nn.Embedding(V, d), nn.Embedding(T, d)
        s.blocks = nn.ModuleList(Block() for _ in range(L))
        s.norm, s.head = nn.RMSNorm(d), nn.Linear(d, V, bias=False)
    def forward(s, x):
        hdn = s.tok(x) + s.pos(torch.arange(T))               # (B, T, d)
        for blk in s.blocks:
            hdn = blk(hdn)
        return s.head(s.norm(hdn))                            # (B, T, V)

model = TinyGPT()
decay   = [p for p in model.parameters() if p.ndim >= 2]
nodecay = [p for p in model.parameters() if p.ndim < 2]
opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.1},
                         {"params": nodecay, "weight_decay": 0.0}],
                        lr=1e-3, betas=(0.9, 0.95), eps=1e-8)
W, S = 10, 100                                                # warmup, total steps
sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda t: (t + 1) / W if t < W
            else 0.1 + 0.9 * 0.5 * (1 + math.cos(math.pi * (t - W) / (S - W))))

data = torch.randint(0, V, (B, T + 1))
x, y = data[:, :-1], data[:, 1:]                              # shift by one, (32.1)

# 1) forward, counting the bytes autograd saves for backward
saved = {}
def pack(t):
    saved[(t.data_ptr(), t.shape)] = t.numel() * t.element_size(); return t
with torch.autograd.graph.saved_tensors_hooks(pack, lambda t: t):
    logits = model(x)
    loss = F.cross_entropy(logits.reshape(-1, V), y.reshape(-1))
logits.retain_grad()
n_par = sum(p.numel() for p in model.parameters())
print(f"loss {loss.item():.4f}   ln V = {math.log(V):.4f}")
print(f"params {n_par:,}  ({4 * n_par:,} bytes)   saved activations {sum(saved.values()):,} bytes")

# 2) backward
loss.backward()
p = logits.softmax(-1)
expect = (p - F.one_hot(y, V).float()) / (B * T)              # formula (32.7)
print("dL/dlogits == (p - onehot)/N :", torch.allclose(logits.grad, expect, atol=1e-7))
rows = model.tok.weight.grad.abs().sum(1) > 0
used = torch.zeros(V, dtype=torch.bool); used[x.unique()] = True
print("embedding rows with gradient:", rows.sum().item(), "of", V, " match used ids:", torch.equal(rows, used))

# 3) clip, update, schedule, zero
norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
print(f"grad norm before clip {norm.item():.4f}")
w0 = model.head.weight.detach().clone(); lr = opt.param_groups[0]["lr"]
opt.step(); sched.step(); opt.zero_grad(set_to_none=True)
dw = (model.head.weight.detach() - w0).abs().max().item()
print(f"lr used {lr:.1e}   largest |change| in head / lr = {dw / lr:.3f}")
print("next lr", f"{opt.param_groups[0]['lr']:.1e}", "  grads freed:", model.head.weight.grad is None)
print("adam state tensors per weight:", sorted(opt.state[model.head.weight].keys()))

Output (PyTorch 2.8, CPU):

loss 4.3468   ln V = 4.1589
params 29,344  (117,376 bytes)   saved activations 463,876 bytes
dL/dlogits == (p - onehot)/N : True
embedding rows with gradient: 37 of 64  match used ids: True
grad norm before clip 0.7936
lr used 1.0e-04   largest |change| in head / lr = 1.018
next lr 2.0e-04   grads freed: True
adam state tensors per weight: ['exp_avg', 'exp_avg_sq', 'step']

Read the output. The first loss, 4.35, is close to \(\ln 64 = 4.16\), as expected at initialization. The saved activations (464 KB) are already four times the FP32 weights (117 KB), even for this tiny model with 16 tokens per sequence. Only 37 of the 64 embedding rows got a gradient. The gradient norm, 0.79, was below 1, so clipping did nothing this time. The first step moved the largest weight by 1.018 times the learning rate: 1 from the bias-corrected Adam ratio \(\hat m/\sqrt{\hat v} = \pm 1\), plus a little from weight decay \(\lambda\theta\). After zero_grad the gradient is None, and Adam now holds two state tensors per weight (plus a step counter).

Where this is used

SettingWhat changes in the stepWhat stays the same
LLM pretraining (Llama, Qwen, DeepSeek)Thousands of GPUs: gradients all-reduced (DDP) or reduce-scattered (FSDP) during backward; weights all-gathered during forwardForward, loss (32.6), backward, clip, AdamW, schedule
Supervised fine-tuning (SFT)Mask \(m = 0\) on prompt tokens, so (32.7) is zero there; smaller learning rateEverything else
LoRA and other PEFTBase weights frozen: no gradients and no Adam states for them; only small adapter matrices are updatedActivations are still stored, because the gradient must flow back through frozen layers
VLM training (LLaVA style)Image tokens enter as embeddings from a vision encoder and projector; loss mask is 0 on them; the encoder is often frozenThe decoder step; activations now also grow with the number of image tokens
Video-LLMs for video QA, captioning and key-moment detectionHundreds or thousands of frame tokens per sample make activations dominate memory; checkpointing and fewer tokens per frame are commonCausal loss only on the text answer
Contrastive video–text retrieval and video fingerprinting modelsThe loss is contrastive (a softmax over the batch) instead of next-token; larger batches give more negativesBackward, clipping, AdamW and schedule
Debugging a training runCheck \(\mathcal{L} \approx \ln V\) at step 0; overfit one batch; log gradient norm; read peak memoryThis session's order of operations

Common mistakes

  • Forgetting zero_grad. Gradients add up across steps, the effective learning rate grows every step, and training diverges or behaves strangely. Clear gradients once per optimizer step (with accumulation, once per accumulation cycle).
  • Clipping scaled gradients. With FP16 and a GradScaler, call scaler.unscale_(optimizer) before clip_grad_norm_. Otherwise the norm is multiplied by the scale (often 65,536) and every step is clipped to almost nothing.
  • Keeping the graph alive. losses.append(loss) or total += loss keeps every step's activations. Memory grows each step until out-of-memory. Use loss.item() (sparingly) or loss.detach().
  • Calling .item() every step. It forces the CPU to wait for the GPU and removes all overlap. Log every few steps, or keep a GPU tensor and read it rarely.
  • Wrong averaging of the loss. Averaging over all positions including padding, or averaging per sequence and then over sequences, gives different weights to different tokens. Use ignore_index=-100 and divide by the real token count \(N_{\text{tok}}\) (32.6). With accumulation, divide by the token count of the whole accumulated batch.
  • Calling scheduler.step() at the wrong place. It belongs after optimizer.step(), once per optimizer step. Calling it per epoch, or per micro-batch, gives a very different schedule from the one you designed.
  • Logits in BF16 for the loss. Computing softmax and log over 128,256 entries in BF16 loses precision; compute the loss in FP32 (or with a fused kernel that does so internally).
  • Measuring memory after the first step only. Adam states are created at the first optimizer.step(), and the allocator caches memory. Read torch.cuda.max_memory_allocated() after two or three steps.

Interview questions

Walk me through one training step of a decoder-only LLM.basic
Load ids \((B, T{+}1)\); inputs are the first \(T\), targets are shifted by one. Forward: embedding lookup to \((B, T, d)\), \(L\) pre-norm blocks (attention and MLP, each added to the residual stream), final norm, output projection to logits \((B, T, V)\), cross-entropy averaged over the real target tokens. Each operation saves what its backward needs. Backward: start with \((\mathbf{p} - \mathbf{e}_y)/N\) at the logits, go back layer by layer, each layer producing weight gradients (\(X^\top G\)) and the input gradient (\(G W^\top\)) and freeing its activations; the embedding gets a scatter-add. Update: clip the global norm to 1.0, read \(\eta(t)\) from warmup plus cosine, apply AdamW. Clean up: zero_grad (set to None), scheduler.step. Cost about \(6N\) FLOPs per token.
Derive the gradient of softmax cross-entropy with respect to the logits.basic
\(\ell = -\log p_y = -z_y + \log\sum_j e^{z_j}\). Then \(\partial\ell/\partial z_i = -[i=y] + e^{z_i}/\sum_j e^{z_j} = p_i - [i = y]\). As a vector: \(\mathbf{p} - \mathbf{e}_y\). With averaging over \(N\) tokens, divide by \(N\). It sums to zero, is bounded in \([-1, 1]\) per component, and is why frameworks fuse softmax and cross-entropy: the fused backward is one subtraction and is numerically stable.
Why must activations be stored, and which ones?core
The weight gradient of a linear layer is \(X^\top G\): it needs the layer's forward input \(X\). Norms need their input (and the RMS), activation functions need their pre-activation values, attention needs \(Q, K, V\), the output and the log-sum-exp (with FlashAttention; otherwise the full probability matrix). For a GPT-style block this is about 34 bytes per residual-stream element per layer in 16-bit, which for long sequences exceeds the weights. Checkpointing keeps only block inputs and recomputes the rest, at about one extra forward pass.
How does the residual connection help the backward pass?core
For \(X_{\text{out}} = X_{\text{in}} + F(X_{\text{in}})\), the gradient is \(G_{\text{in}} = G_{\text{out}} + J_F^\top G_{\text{out}}\). The identity term carries the gradient from the loss to every layer unchanged, so it does not vanish through depth; each block only adds a correction. Without residuals, the gradient would be a product of 2L Jacobians, which tends to shrink or explode.
Which rows of the embedding matrix get a gradient in one step? What changes with tied embeddings?core
Only the rows of token ids that appear in the batch: the lookup's backward is a scatter-add of the position gradients into their rows (32.11); a row used \(k\) times receives the sum of \(k\) vectors. With tied input and output embeddings, the matrix also receives \(H^\top G_Z\) from the output layer, which is dense: every row gets a gradient, because every vocabulary entry has a logit at every position. Autograd sums the two contributions. Note that AdamW with weight decay still changes unused rows (decay, and momentum from earlier steps).
What is the correct order of backward, clipping, optimizer step, scheduler and zero_grad? What if you use FP16?core
backward → (FP16: scaler.unscale_) → clip_grad_norm_ → optimizer.step (FP16: scaler.step, then scaler.update) → scheduler.step → zero_grad. zero_grad may also be at the start of the next step, as long as it comes before the next backward. Clipping must see the true gradients; with FP16 they are multiplied by the loss scale until unscaled, so clipping first would cut almost everything.
When during a step is GPU memory at its peak, and what is in memory then?advanced
Without checkpointing, usually at the start of the backward pass: the weights, master copy and Adam states (\(14N\) bytes with mixed precision), all saved activations of all layers, the FP32 logits and the logit gradient (\(8BTV\) bytes). Gradients are still near zero then. For Llama 3 8B on one 8,192-token sequence that is about 112 + 36.5 + 8.4 = 157 GB. With checkpointing the activation term is small, and since gradients grow until the end of the backward pass, the peak can move there. There are also temporary buffers (all-reduce buckets, optimizer temporaries) and allocator fragmentation on top.
Why is the output layer a memory problem for large vocabularies, and how is it solved?advanced
The logits have shape \((B, T, V)\); with \(V = 128{,}256\) and one 8K sequence that is 1.05 billion numbers, 4.2 GB in FP32, and the gradient has the same size. This can exceed the activations of several layers. Fused linear cross-entropy kernels (Liger Kernel, Cut Cross-Entropy) compute the logits chunk by chunk, compute the loss and the logit gradient for each chunk immediately, and never store the full tensor. Cut Cross-Entropy reports 24 GB to 1 MB of loss memory for Gemma 2 2B. Another simple fix is computing the loss over chunks of positions in a loop.
Your first-step loss is 25 on a model with a 50K vocabulary. What do you check?basic
At initialization the loss should be near \(\ln V = \ln 50{,}257 = 10.8\). A loss of 25 means the logits are far too large: wrong initialization scale (for example of the output layer or the embedding), a missing final norm, or a missing scaling factor. Also check that the targets are shifted correctly and that the ignore_index masks padding. Then overfit one batch: the loss should drop toward zero quickly.
In data-parallel training, where does communication happen inside the step?advanced
In DDP, every GPU has the full model and its own micro-batch. During backward, as soon as a bucket of gradients is ready (the last layers first), an all-reduce averages it across GPUs, overlapped with the backward of earlier layers. After backward, every GPU has the same averaged gradient and runs the same clip and AdamW. In FSDP/ZeRO-3, forward and backward also all-gather each layer's weights just before use, and gradients are reduce-scattered, so each GPU keeps and updates only its shard of weights, gradients and Adam states. The clipping norm needs one extra all-reduce of the sum of squares.
Why can the first training step run out of memory even though the forward and backward passes succeeded?advanced
PyTorch optimizers create their state lazily, at the first step(). For AdamW that is two FP32 tensors per weight, \(8N\) bytes, allocated after the backward pass, while the gradients still exist. If memory was tight, this is where it fails. Also, the allocator caches freed blocks, so measure peak memory after a few steps, and keep in mind that the optimizer may create temporary buffers during the update.

Exercises

  1. A model with \(V = 4\) gives logits \((0, 0, 0, 0)\) at one position, with target \(y = 2\). Compute the loss and the logit gradient (one position, \(N_{\text{tok}} = 1\)).
  2. The gradient norm before clipping is 4.0 and \(c = 1.0\). What factor multiplies every gradient? By how much does the update of each weight change with AdamW at a late step? (Think before answering: AdamW divides by \(\sqrt{\hat v}\).)
  3. With the schedule of Figure 3, compute \(\eta\) at step 1,000 and at step 251,000.
  4. Logits for a micro-batch of \(B = 4\) sequences of \(T = 4096\) tokens, \(V = 128{,}256\), in FP32: how many GB? How many with the gradient?
  5. The input is "the cat sat on the mat" (6 tokens, one id each). Which embedding rows get a gradient, and how many contributions does each get? What changes if the embeddings are tied?
  6. Run the attention-backward check below and explain why the result does not depend on the random values. Then add a causal mask check: what is \(G_S\) at the masked positions?
import torch
torch.manual_seed(0)
T, dh = 5, 4
Q, K, V = [torch.randn(T, dh, dtype=torch.float64, requires_grad=True) for _ in range(3)]
S = (Q @ K.T / dh**0.5).masked_fill(torch.triu(torch.ones(T, T, dtype=torch.bool), 1), float("-inf"))
P = S.softmax(-1); O = P @ V
G = torch.randn(T, dh, dtype=torch.float64); (O * G).sum().backward()   # so dL/dO = G
dV = P.T @ G; dP = G @ V.T
dS = P * (dP - (dP * P).sum(-1, keepdim=True))                       # formula (32.10)
print(torch.allclose(dV, V.grad), torch.allclose(dS @ K / dh**0.5, Q.grad),
      torch.allclose(dS.T @ Q / dh**0.5, K.grad))                       # True True True
Answerscheck
1. All probabilities are 0.25, so the loss is \(\ln 4 = 1.386\) and the gradient is \((0.25, 0.25, -0.75, 0.25)\). 2. The factor is \(1/4 = 0.25\). With AdamW at a late step, the update is \(\eta\,\hat m/(\sqrt{\hat v} + \varepsilon)\); one clipped step changes \(\hat m\) and \(\hat v\) only by their small \((1-\beta)\) shares, so the update barely changes. Clipping with Adam mainly protects the running averages from one huge gradient, rather than scaling the step directly. (At the very first step, \(\hat m/\sqrt{\hat v} = \pm 1\) whatever the scale.) 3. Step 1,000: \(3\times10^{-4} \times 1000/2000 = 1.5\times10^{-4}\). Step 251,000 is exactly halfway through the decay (\((251{,}000 - 2000)/498{,}000 = 0.5\)), so \(\eta = 3\times10^{-4}(0.1 + 0.9 \times 0.5) = 1.65\times10^{-4}\). 4. \(4 \times 4096 \times 128{,}256 \times 4 = 8.4\) GB; with the gradient, 16.8 GB. 5. Five rows: "the" gets two contributions (positions 1 and 5), "cat", "sat", "on", "mat" one each; all other rows get zero. With tied embeddings every row gets a gradient from the output layer, since every token has a logit at every position. 6. Formula (32.10) is an exact identity of calculus, so it matches autograd for any values (up to float64 rounding). At masked positions \(P = 0\), so \(G_S = P \odot (\dots) = 0\): no gradient flows to future keys.

Key takeaways

  • One step: ids \((B, T)\) → embeddings \((B, T, d)\) → \(L\) blocks → logits \((B, T, V)\) → one loss averaged over real target tokens (32.1–32.6). Initial loss ≈ \(\ln V\).
  • Every gradient starts as prediction minus truth, \((\mathbf{p} - \mathbf{e}_y)/N_{\text{tok}}\) (32.7), and flows back by \(X^\top G\) and \(G W^\top\) (32.8). The residual stream copies it through every block (32.9); attention uses the softmax backward (32.10); the embedding gets a scatter-add (32.11).
  • Backward costs about twice the forward; a step is about \(6N\) FLOPs per token (32.12).
  • Update order: (unscale) → clip the global norm (32.13) → AdamW with \(\eta(t)\) from the schedule (32.14, 32.15) → scheduler.step → zero_grad.
  • Memory: a floor of \(14N\) for weights, master copy and Adam; activations rise in forward and fall in backward; gradients rise in backward; logits cost \(8BTV\). Without checkpointing the peak is at the start of backward (32.16): about 157 GB for Llama 3 8B on one 8K sequence.

Key formulas

\[ \frac{\partial \mathcal{L}}{\partial \mathbf{z}} = \frac{\mathbf{p} - \mathbf{e}_y}{N_{\text{tok}}}, \qquad \frac{\partial \mathcal{L}}{\partial W} = X^\top G, \qquad \frac{\partial \mathcal{L}}{\partial X} = G W^\top, \qquad G_{\text{in}} = G_{\text{out}} + J_F^\top G_{\text{out}} \]
\[ \mathbf{g} \leftarrow \mathbf{g}\min\!\Big(1, \frac{c}{\lVert\mathbf{g}\rVert}\Big), \qquad C_{\text{step}} \approx 6NBT, \qquad M_{\text{peak}} \approx 14N + 34LBTd + 8BTV \]

References for this session

  1. Karpathy (2023). Neural Networks: Zero to Hero (video course)
  2. Stanford (2025). CS336: Language Modeling from Scratch (course)
  3. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  4. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  5. Llama Team, Meta (2024). The Llama 3 Herd of Models
  6. Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
  7. Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  8. Chen et al. (2016). Training Deep Nets with Sublinear Memory Cost
  9. Micikevicius et al. (2017). Mixed Precision Training
  10. Loshchilov & Hutter (2017). Decoupled Weight Decay Regularization
  11. Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
  12. Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
  13. Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  14. Hsu, Dai, Kothapalli, Song, Tang, Zhu, Shimizu, Sahni, Ning, Chen (2024). Liger Kernel: Efficient Triton Kernels for LLM Training
  15. Wijmans, Huval, Hertzberg, Koltun, Krähenbühl (2024). Cut Your Losses in Large-Vocabulary Language Models