Gradient accumulation and gradient checkpointing
Two tricks let you train with a batch, or a model, that does not fit in GPU memory. Gradient accumulation runs many small batches and takes one optimizer step, so you get a large batch with the memory of a small one. Gradient checkpointing throws most activations away in the forward pass and recomputes them in the backward pass, so you trade about one extra forward pass for a large cut in activation memory.
What you will learn
- Why gradient accumulation gives the same gradient as one big batch, and how to compute the effective batch size in sequences and in tokens
- How to scale the loss across micro-batches, and the token-count trap that silently changed the gradients of many fine-tuning runs in 2024
- Where accumulation is not exactly the same as a big batch: normalization over the batch, data-parallel synchronization, numerics
- Gradient (activation) checkpointing: keep only the block inputs, recompute the rest in the backward pass
- The memory and compute trade-off: the square-root rule of Chen et al., and why one checkpoint per transformer block is usually best
- Selective recomputation (Korthikanti et al.): recompute only the attention core, for a few percent of extra compute
- How to use both in PyTorch and Hugging Face, and how to check that the gradients did not change
Why this matters
In session 38 we added up the bytes of one training step and saw that they often do not fit. Two parts of the bill grow with the work you do: the activations grow with the batch size and with the sequence length. This session gives you the two standard tools that attack exactly that part, without buying more GPUs.
Gradient accumulation solves the batch problem. The recipe says “4 million tokens per step”, but your GPU can hold the activations of only 8,192 tokens. So you run many small batches, add up their gradients, and take one optimizer step at the end. Gradient checkpointing (also called activation checkpointing or activation recomputation) solves the depth and length problem. It keeps only a few activations during the forward pass and recomputes the others when the backward pass needs them. You pay with extra compute, about one more forward pass.
Both tools are everywhere. Llama 2 was trained with a global batch of about 4 million tokens per step [Certain], far more than one GPU can hold, so the batch is split across GPUs and, when needed, across accumulation steps. Hugging Face Trainer has the flags gradient_accumulation_steps and gradient_checkpointing; Megatron-LM, DeepSpeed and PyTorch FSDP all support both. [Certain] And a small mistake in the first one, the token-count trap, changed the gradients of many fine-tuning runs until it was found and fixed in October 2024. [Certain]
Gradient accumulation: one big batch, in pieces
Recall from session 4 that the mini-batch gradient is an average of per-example gradients. A micro-batch is the batch that actually goes through the GPU in one forward and backward pass. The global batch (or effective batch) is the set of examples that together produce one optimizer step. Accumulation makes the global batch \(K\) times larger than the micro-batch:
- Run forward and backward on micro-batch 1. PyTorch adds the new gradient into each parameter’s
.gradbuffer. - Do the same for micro-batches 2, 3, …, \(K\). Do not call
optimizer.step()and do not zero the gradients in between. - After micro-batch \(K\): clip, call
optimizer.step(), thenzero_grad().
With data parallelism on \(N_{\text{dp}}\) GPUs (session 44), each GPU processes its own micro-batches and the gradients are averaged across GPUs before the step. So the effective batch is:
- Read the symbols
- \(b\)
- micro-batch size: sequences per forward pass on one GPU
- \(K\)
- number of accumulation steps (micro-batches per optimizer step)
- \(N_{\text{dp}}\)
- number of data-parallel copies of the model (often the number of GPUs)
- \(s\)
- sequence length in tokens (for packed sequences, every token is real)
Only \(b\) and \(s\) decide the activation memory on one GPU. \(K\) and \(N_{\text{dp}}\) change the batch the optimizer sees, not the memory. That is the whole point.
Take the Llama 2 setting: sequences of \(s = 4{,}096\) tokens and about 4 million tokens per step. \(4{,}194{,}304 / 4{,}096 = 1{,}024\) sequences per step. Say you have 64 GPUs and each one fits a micro-batch of \(b = 2\) sequences. Then one round of all GPUs covers \(2 \times 64 = 128\) sequences, so you need \(K = 1{,}024 / 128 = 8\) accumulation steps. With 128 GPUs you would need \(K = 4\); with 512 GPUs and \(b = 2\), no accumulation at all.
Why the gradient is exactly the same
The gradient is a linear operation: the gradient of a sum is the sum of the gradients. Split a batch of \(B\) examples into \(K\) micro-batches \(\mathcal{B}_1, \dots, \mathcal{B}_K\) of equal size \(B/K\). Then
In words: if each micro-batch computes its mean loss, divide it by \(K\) before backward(). The .grad buffers then hold exactly the full-batch gradient after \(K\) micro-batches. Forget the \(1/K\) and the gradient is \(K\) times too large. With Adam this is partly hidden (Adam divides by \(\sqrt{\hat v}\), which also grows), but gradient clipping, the \(\varepsilon\) term and weight decay all see a different scale, so the run is not the run you designed.
Figure 1 runs this loop in the browser on a tiny linear model: 6 weights, a batch of 32 seeded examples, and the squared-error loss \(\ell_i = \tfrac12 (\mathbf{x}_i^\top \mathbf{w} - y_i)^2\). Its gradient is \(e_i \mathbf{x}_i\), where \(e_i = \mathbf{x}_i^\top \mathbf{w} - y_i\) is the error. The figure takes 3 optimizer steps of plain gradient descent with learning rate \(\eta = 0.2\). Next to it, the browser also trains a second copy of the model with the whole batch of 32 at once, so you can compare the two runs at every step. The algorithm box highlights the line that is running; the box next to it shows that line’s formula with the real numbers put in.
- \(\mathbf{b} \leftarrow \mathbf{0}\) // zero_grad()
- for \(k = 1, \dots, K\) do
- \(L_k \leftarrow \frac{1}{m} \sum_{i \in \mathcal{B}_k} \tfrac12 e_i^2\) // forward on micro-batch k only
- \(\mathbf{b} \leftarrow \mathbf{b} + \nabla L_k / K\) // backward; .grad adds up, (39.2)
- end for
- \(\mathbf{w} \leftarrow \mathbf{w} - \eta\, \mathbf{b}\) // the one optimizer step
.grad of the six weights). Each backward pass adds one orange piece. After \(K\) micro-batches the blue bars fill the dashed boxes, and only then the weights move. Here \(m = 32/K\) is the micro-batch size.How to read Figure 1
Press Step and follow the highlighted line. With \(K = 4\), each micro-batch has 8 examples. The first full-batch loss is 1.550. After the 4th micro-batch the buffer equals the full-batch gradient up to about \(10^{-16}\), which is floating-point rounding; for example its second number is 0.792. Then line 6 moves the weights once: \(w_2 = 0 - 0.2 \times 0.792 = -0.158\). The loss after each step is 0.908, then 0.557, and 0.360 after the third step, exactly the same as the big-batch copy (the meter shows the largest difference between the two weight vectors). Now untick “divide each loss by K”: the buffer ends at exactly \(K\) times the target, which acts like a learning rate \(K\) times larger. With \(K = 4\) the loss happens to fall faster (0.056 after 3 steps) but it is no longer the run you designed. With \(K = 8\) the effective learning rate \(8 \times 0.2 = 1.6\) is too large: the loss goes 1.550, 1.930, 4.062, 10.466 and the run diverges.
What accumulation does not give you for free
Formula (39.2) holds when the loss is a plain sum or average over examples. Three things can break it:
- Normalization over the batch. BatchNorm computes a mean and variance over the micro-batch, so 8 micro-batches of 4 are not one batch of 32. Transformers use LayerNorm or RMSNorm, which work per token, so LLMs are safe. [Certain] Vision backbones with BatchNorm are not.
- Losses that compare examples inside the batch. A contrastive loss like CLIP’s (session 166) scores every image against every text in the batch. Accumulation does not give you more negatives; you need tricks such as gradient caching instead. [Certain]
- Different numbers of tokens per micro-batch. This is the most common one in LLM work, and it gets its own section next.
Small differences also come from rounding (sums in a different order), from dropout masks (different random numbers), and from the data order. These are noise, not bias. [Certain]
Scaling the loss: the token-count trap
For language models the loss is an average over tokens, not over sequences. When the sequences are padded, only the real (non-padding) tokens count: their labels are real token ids, and the padding positions get the label −100, which cross_entropy ignores. The correct loss for one optimizer step is the sum over all real tokens in all \(K\) micro-batches, divided by their total number:
- Read the symbols
- \(\ell_t\)
- the cross-entropy of one real token \(t\)
- \(n_k\)
- the number of real tokens in micro-batch \(k\)
- \(N_{\text{tot}}\)
- the number of real tokens in the whole optimizer step (on all GPUs)
The natural code does something else. Each micro-batch calls cross_entropy(..., reduction="mean"), which divides by its own \(n_k\), and then the result is divided by \(K\):
In the correct loss every token has weight \(1/N_{\text{tot}}\). In the naive loss, a token in a short micro-batch counts much more than a token in a long one. The two agree only when all \(n_k\) are equal, which is true for packed pretraining data and false for padded fine-tuning data with sequences of very different lengths.
Micro-batch A has 1,000 real tokens with mean loss 2.0. Micro-batch B has 10 real tokens with mean loss 5.0. The naive loss is \((2.0 + 5.0)/2 = 3.5\). The correct loss is \((1{,}000 \times 2.0 + 10 \times 5.0)/1{,}010 = 2{,}050/1{,}010 = 2.030\). The 10 tokens of B have half of the total weight in the naive loss (0.5) instead of \(10/1{,}010 = 0.0099\): each of them counts \(0.5/0.0099 = 50.5\) times too much. The gradient follows the same wrong weights.
This is exactly the bug that Hugging Face described in its blog post “Fixing Gradient Accumulation” (October 16, 2024): the Trainer averaged the loss per micro-batch, so training with accumulation gave different (higher) losses than the same batch without it. The fix: compute the loss with reduction="sum" and divide by the number of real tokens in the whole accumulation step. [Certain] Explore it in Figure 2.
How to read Figure 2
With the default counts (500, 480, 60 and 8 tokens), the naive loss is 3.000 and the correct one is 2.305. The 8 tokens of micro-batch 4 have high loss and get 32.75 times their fair weight, so they pull the naive loss up. Now set all four counts equal: the two bars of each micro-batch become the same height and the two losses agree. That is why packed pretraining (every micro-batch full) does not suffer from the trap, while padded instruction tuning does.
The same trap across GPUs
Data parallelism averages the gradients over the \(N_{\text{dp}}\) GPUs. If every GPU divides by its own token count, you get the same mean-of-means error, now across GPUs. The fix: before the backward passes, all-reduce (sum) the real-token count over all micro-batches and all GPUs to get \(N_{\text{tot}}\). Each GPU then uses \(\text{sum of its token losses} \times N_{\text{dp}} / N_{\text{tot}}\); the factor \(N_{\text{dp}}\) cancels the averaging that DDP does. [Certain for the arithmetic]
The cost of accumulation
Accumulation adds no FLOPs: \(K\) micro-batches of size \(b\) cost the same as one batch of size \(Kb\) would. It costs time in another way: the \(K\) micro-batches run one after another, so the step takes \(K\) times longer, and small micro-batches can use the GPU less efficiently than large ones. With data parallelism, you should not all-reduce the gradients after every micro-batch, only after the last one. DDP’s no_sync() context manager skips the communication for the first \(K - 1\) micro-batches. [Certain] With ZeRO-2/3 or FSDP the gradients are sharded, so frameworks usually reduce after every micro-batch anyway, or keep an extra unsharded buffer. [Likely] Gradients are often accumulated in FP32 even when the backward pass runs in BF16, because adding many small BF16 numbers loses precision (session 37). [Likely]
- input: micro-batches \(\mathcal{B}_1, \dots, \mathcal{B}_K\) on each of \(N_{\text{dp}}\) GPUs, optimizer, clip value \(c\)
- \(N_{\text{tot}} \leftarrow \text{all\_reduce\_sum}\big(\sum_k n_k\big)\) // real tokens in the whole step, all GPUs
- for \(k = 1, \dots, K\) do
- \(S_k \leftarrow \sum_{t \in \mathcal{B}_k} \ell_t\) // forward, cross-entropy with reduction = sum
- backward of \(S_k \cdot N_{\text{dp}} / N_{\text{tot}}\), with gradient sync only if \(k = K\) // .grad adds up; formula (39.3)
- end for
- clip the global gradient norm to \(c\);
optimizer.step();scheduler.step(); zero the gradients - log \(\sum_k S_k / N_{\text{tot}}\) (summed over GPUs) as the step loss
Gradient checkpointing: forget now, recompute later
Backpropagation needs values from the forward pass. For \(Y = XW\), the weight gradient is \(X^\top G\), so \(X\) must still be in memory when the backward pass reaches this layer (session 4, formula 4.4). Normally PyTorch saves every such tensor during the forward pass and frees it during the backward pass. The peak, just before the backward pass starts, holds the saved tensors of every layer. For a GPT-style block, Korthikanti et al. count them:
- Read the symbols
- \(s, b, d, h\)
- sequence length, micro-batch size, model width, number of attention heads
- \(34\,sbd\)
- inputs of the linear layers, norms, GELU and dropout masks
- \(5\,h s^2 b\)
- the attention scores, softmax output and its dropout mask: the part that grows with \(s^2\)
(This is formula 25.13 of session 25, written for one layer.) Gradient checkpointing keeps only a few tensors, the checkpoints, and drops the rest. In the backward pass, when a dropped tensor is needed, it runs the forward pass of that part again, starting from the nearest checkpoint. The usual choice for transformers is one checkpoint per block: keep only the block input, a tensor of \(s \times b \times d\) numbers (\(2sbd\) bytes in BF16).
With one checkpoint per block, the peak memory is the \(L\) stored block inputs plus the full activations of the one block that is being recomputed:
Korthikanti et al. write the first term, \(2sbdL\), as the memory of “full activation recomputation”. [Certain] The second term is the working space of one block, which we add because it is also in memory at the peak.
Take \(L = 32\), \(d = 4{,}096\), \(h = 32\), \(s = 4{,}096\), \(b = 1\) (the shape of Llama 2 7B and Llama 3 8B, with the GPT-style constant 34 as an approximation). Here \(sbd = 16{,}777{,}216\).
No checkpointing: \(16{,}777{,}216 \times (34 + 5 \times 32) = 3.25\) GB per layer, 104.2 GB for 32 layers. Not storing the score matrices (FlashAttention, or selective recomputation): \(16{,}777{,}216 \times 34 = 0.57\) GB per layer, 18.3 GB in total. One checkpoint per block: \(2 \times 16{,}777{,}216 \times 32 = 1.07\) GB of block inputs, plus 0.57 GB for the block being recomputed: 1.64 GB. That is 63 times less than no checkpointing, and 11 times less than FlashAttention alone.
Figure 3 runs the whole schedule, with real numbers, on a small chain of 8 layers. Layer \(i\) computes \(t_i = \tanh(w_i x_{i-1})\) and \(x_i = x_{i-1} + t_i\) (a residual layer), and the loss is \(\mathcal{L} = \tfrac12 x_8^2\). To backpropagate through layer \(i\), you need its input \(x_{i-1}\) and its inside value \(t_i\): the weight gradient is \(g_i (1 - t_i^2)\, x_{i-1}\), and the gradient passed down is \(g_{i-1} = g_i \big(1 + (1 - t_i^2) w_i\big)\), where \(g_i = \partial \mathcal{L} / \partial x_i\). Each number stands for a tensor. For the memory count, a layer input costs 1 unit, and the inside of a layer costs 16 units (like a transformer block) or 1 unit (a simple layer). Time is counted as 1 unit per layer forward and 2 per layer backward, the \(2N : 4N\) ratio of session 25.
- for \(i = 1, \dots, n\): // forward
\(t_i \leftarrow \tanh(w_i x_{i-1})\); \(x_i \leftarrow x_{i-1} + t_i\) - save \(x_{i-1}\) only if layer \(i\) starts a segment of \(c\) layers // no checkpointing: save \(x_{i-1}\) and \(t_i\)
- \(\mathcal{L} \leftarrow \tfrac12 x_n^2;\ g \leftarrow x_n\)
- for each segment, last to first do
- recompute its layers from the checkpoint, saving \(x_{j-1}\) and \(t_j\) // skipped without checkpointing
- for \(j\) = last … first: // backward, then free \(x_{j-1}, t_j\)
\(\nabla w_j \leftarrow g (1 - t_j^2)\, x_{j-1}\)
\(g \leftarrow g\big(1 + (1 - t_j^2)\, w_j\big)\)
How to read Figure 3
Press Play, or move the slider. Start with “No checkpointing”: memory climbs to 136 units at the end of the forward pass and falls during the backward pass; time is \(8 + 16 = 24\) units. Click “Every layer”: the forward pass stores only the 8 inputs (the orange row stays empty). In the backward pass each layer is recomputed (amber) just before it is needed, so the peak is 24 units, the count of layer forwards goes from 8 to 16, and time goes from 24 to 32 units: \(32/24 = 1.33\), formula (39.8). The recomputed \(t_i\) are exactly the values of the forward pass, so the gradients do not change: \(\nabla w_1 = 0.2924, \dots, \nabla w_8 = 0.4875\) in every setting, and the loss is 0.1762. “Every 2 layers” and “Every 4 layers” store fewer checkpoints but rebuild bigger segments: the peaks are 37 and 69. Now switch to “Simple layer”: no checkpointing gives 16, every layer 9, every 2 layers 7, every 4 layers 9. When the inside of a layer is small, the best segment length is about \(\sqrt{8} \approx 2.8\); when the inside is large, checkpoint every layer.
How many checkpoints? The square-root rule
Chen et al. (2016) asked how to place checkpoints in a chain of \(n\) layers. Put one checkpoint every \(c\) layers. Let \(I\) be the size of one checkpoint (a layer input) and \(A\) the full saved size of one layer. At the peak you hold \(n/c\) checkpoints and one rebuilt segment of \(c\) layers:
The first term falls with \(c\), the second grows; setting the derivative \(-nI/c^2 + A\) to zero gives \(c^*\). When \(I\) and \(A\) are similar (a chain of simple layers), \(c^* \approx \sqrt{n}\) and memory grows only as \(\sqrt{n}\) instead of \(n\). That is the result of Chen et al.: \(O(\sqrt{n})\) memory for the cost of one extra forward pass; they trained a 1,000-layer residual network with memory going from 48 GB to 7 GB, for about 30% more running time. [Certain]
For a transformer block, the inside is much larger than the input. In units of \(sbd\) bytes with FlashAttention: \(I = 2\), \(A = 34\), \(n = 32\). Then \(c^* = \sqrt{32 \times 2 / 34} = 1.37\). Checking whole numbers: \(c = 1\) gives \(64 + 34 = 98\), \(c = 2\) gives \(32 + 68 = 100\), \(c = 4\) gives \(16 + 136 = 152\). So one checkpoint per block is the best choice, which is why every framework offers exactly that. [Certain for the arithmetic]
The cost: about one extra forward pass
Recomputation runs the forward pass of every checkpointed segment a second time. In FLOPs per token, training normally costs \(2N\) (forward) \(+ 4N\) (backward) \(= 6N\) (session 25). With full recomputation:
So the GPU does about 33% more matrix work. The attention-score FLOPs scale the same way (forward, recompute, backward in the ratio 1 : 1 : 2), so the ratio stays at 4/3 for any sequence length. In measured time the overhead is usually 30–40%, the range Korthikanti et al. quote; they measured 36% per layer for their 530B and 1T models. [Certain] This is also why MFU falls with recomputation: the recomputed FLOPs are real work for the hardware but not “useful” model FLOPs (session 51).
Selective recomputation
Look again at formula (39.5). The \(5hs/d\) part (the attention scores and softmax) is most of the memory at long sequence lengths: 160 out of 194 units in the worked example, 82%. But it is cheap to recompute: the scores \(QK^\top\) and the product with \(V\) are a small share of the FLOPs when \(s\) is not much larger than \(d\). The linear layers are the reverse: little memory each, most of the FLOPs. Korthikanti et al. therefore recompute only the attention core (\(QK^\top\), softmax, softmax dropout, and the product with \(V\)) and store everything else. [Certain] This removes the \(5hs/d\) term and costs only the attention-score forward again, about \(4\,s\,d\) FLOPs per token per layer:
At \(s = d = 4{,}096\) this is \(88/84 = 1.048\), about 5% more FLOPs in this simple count. The paper reports 2.7% extra FLOPs for GPT-3 175B and 1.6% for the 530B model (their \(d\) is much larger than \(s\)), and, together with sequence parallelism, 29–32% higher throughput than full recomputation. [Certain]
Figure 4 puts the formulas together for the 7B-sized shape. You pick the length, the micro-batch and the strategy; it gives the activation memory and the extra compute.
How to read Figure 4
At \(s = 4{,}096\), \(b = 1\): 104 GB, 18.3 GB and 1.6 GB, with 1.00×, 1.05× and 1.33× the FLOPs. Push \(s\) to 32,768: without recomputation the score matrices need thousands of GB; selective recomputation brings it to 146 GB at 1.19× the FLOPs (the attention core is no longer cheap); full recomputation needs 13.2 GB. Raise \(b\) instead: every bar scales linearly with \(b\), and the FLOP ratios do not change. This is where accumulation and checkpointing meet: checkpointing lets you raise \(b\); accumulation lets you keep \(b\) small.
Using both in practice
- input: blocks \(f_1, \dots, f_L\), input \(x_0\)
- for \(i = 1, \dots, L\) do // forward, no internal activations saved
- save \(x_{i-1}\) (and the random-number state, for dropout); \(x_i \leftarrow f_i(x_{i-1})\) without saving anything inside \(f_i\)
- end for; compute the loss from \(x_L\) and its gradient \(g_L = \partial \mathcal{L} / \partial x_L\)
- for \(i = L, \dots, 1\) do // backward
- restore the random-number state; recompute \(x_i = f_i(x_{i-1})\), this time saving the internal activations
- backprop through \(f_i\): \(g_{i-1} \leftarrow\) input gradient, add the weight gradients into
.grad - free the internal activations of \(f_i\) and the checkpoint \(x_{i-1}\)
- end for
In PyTorch this is torch.utils.checkpoint.checkpoint(block, x, use_reentrant=False). The non-reentrant version is the one PyTorch recommends, and by default it saves and restores the random-number state so that dropout makes the same mask both times (preserve_rng_state=True). [Certain] In Hugging Face Transformers, model.gradient_checkpointing_enable() wraps every decoder layer this way; the KV cache must be off during training (use_cache=False), and the library warns and turns it off for you. [Likely]
The code below checks three claims on a tiny 6-layer model, on the CPU, in about a second: correct accumulation equals the big batch, the naive version does not, and checkpointing gives the same gradients with fewer saved activations. It measures saved activations with saved_tensors_hooks, which sees every tensor that autograd stores for the backward pass.
import torch, torch.nn as nn, torch.nn.functional as F from torch.utils.checkpoint import checkpoint torch.manual_seed(0) V, D, NL, T = 50, 64, 6, 16 # vocab, width, layers, max length class Block(nn.Module): # pre-norm MLP block with a residual def __init__(s): super().__init__(); s.n = nn.LayerNorm(D) s.f = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D)) def forward(s, x): return x + s.f(s.n(x)) class TinyLM(nn.Module): def __init__(s): super().__init__(); s.emb = nn.Embedding(V, D) s.blocks = nn.ModuleList(Block() for _ in range(NL)); s.out = nn.Linear(D, V) def forward(s, x, ckpt=False): h = s.emb(x) for b in s.blocks: # checkpoint = keep only the block input h = checkpoint(b, h, use_reentrant=False) if ckpt else b(h) return s.out(h) # 4 micro-batches of 2 sequences each; real lengths differ a lot, the rest is padding lens = [[16, 14], [3, 2], [15, 16], [4, 1]] batches = [] for pair in lens: x = torch.randint(0, V, (2, T)); y = torch.randint(0, V, (2, T)) for i, n in enumerate(pair): y[i, n:] = -100 # -100 = ignored by cross_entropy batches.append((x, y)) model = TinyLM() params = list(model.parameters()) def grads(): return torch.cat([p.grad.flatten() for p in params]) # (1) reference: all 8 sequences in ONE batch, mean over all real tokens model.zero_grad() X = torch.cat([b[0] for b in batches]); Y = torch.cat([b[1] for b in batches]) F.cross_entropy(model(X).flatten(0, 1), Y.flatten(), ignore_index=-100).backward() g_ref = grads().clone() # (2) correct accumulation: sum of token losses / total real tokens of the WHOLE step n_total = sum((y != -100).sum() for _, y in batches) model.zero_grad() for x, y in batches: loss = F.cross_entropy(model(x).flatten(0, 1), y.flatten(), ignore_index=-100, reduction="sum") (loss / n_total).backward() # .grad adds up across micro-batches print("correct :", (grads() - g_ref).abs().max().item()) # about 3e-8: rounding only # (3) the token-count trap: mean per micro-batch, then divide by K model.zero_grad() for x, y in batches: loss = F.cross_entropy(model(x).flatten(0, 1), y.flatten(), ignore_index=-100) (loss / len(batches)).backward() print("naive :", (grads() - g_ref).abs().max().item(), "norm ratio", (grads().norm() / g_ref.norm()).item()) # about 0.15 and 1.42: a different gradient # (4) checkpointing: same gradients, fewer saved activations def saved_bytes(ckpt): total = [0] def pack(t): total[0] += t.numel() * t.element_size(); return t model.zero_grad() with torch.autograd.graph.saved_tensors_hooks(pack, lambda t: t): loss = F.cross_entropy(model(X, ckpt).flatten(0, 1), Y.flatten(), ignore_index=-100) loss.backward() return total[0], grads().clone() b0, g0 = saved_bytes(False); b1, g1 = saved_bytes(True) print("saved KB :", b0 // 1024, b1 // 1024, (g0 - g1).abs().max().item()) # 2793 288 0.0
With PyTorch 2.8 on a CPU this prints a difference of about \(3 \times 10^{-8}\) for the correct accumulation, about 0.15 (and a gradient norm 1.42 times too large) for the naive one, and 2,793 KB versus 288 KB of saved tensors with checkpointing, with identical gradients. [Certain: we ran it] The naive gradient is too large here because the two short micro-batches (5 and 5 real tokens) get the same weight as the long ones (30 and 31 tokens).
Other ways to save activation memory, all covered later: offloading activations to CPU memory, sequence parallelism, which splits the \(34\,sbd\) part across tensor-parallel GPUs (session 48), and lower-precision activations. In session 41 you measure all of this on a real GPU.
Where this is used
| Setting | Typical use | Why |
|---|---|---|
| LLM pretraining (millions of tokens per step) | Data parallelism first; accumulation for the rest of the global batch; selective or full recomputation depending on length | The batch is set by the training recipe, not by the GPU; packed sequences make the token-count trap disappear |
| Full fine-tuning on one node | Micro-batch 1–4, accumulation 8–64, checkpointing on [Likely] | Weights and Adam states already fill most of the memory (session 38) |
| SFT with padded chat data | Accumulation with token-sum normalization (39.3) | Lengths vary a lot; the naive loss over-weights short answers (session 54) |
| LoRA and QLoRA on a single consumer GPU | Checkpointing almost always on, plus accumulation [Likely] | Frozen weights are small in 4-bit, so activations become the biggest part (session 58) |
| Long-context training (32k tokens and more) | FlashAttention plus full block checkpointing, often with context parallelism | Even \(34\,sbd\) per layer is too much at long \(s\) |
| Video models: action recognition, video classification, video-text retrieval | Checkpointing in the video backbone; accumulation for classification losses | Many frames make very long token sequences; but a contrastive retrieval loss needs real in-batch negatives, so accumulation alone does not help it |
| VLM fine-tuning (image or video tokens + text) | Checkpointing in both the vision encoder and the LLM; token-correct loss over text tokens only | Image and video tokens make sequences long, and the number of text tokens per sample varies (session 188) |
Common mistakes
- Forgetting the \(1/K\). The gradient becomes \(K\) times too large; clipping and weight decay now act differently, and the learning rate you tuned is wrong.
- The token-count trap. Averaging per micro-batch and then over micro-batches (39.4) when the micro-batches have different numbers of real tokens. Use
reduction="sum"and divide by the total real-token count of the whole step, on all GPUs. - Calling
optimizer.step()or the scheduler every micro-batch. The step and the learning-rate schedule count optimizer steps, not micro-batches. Also: clip the gradient once, after the last micro-batch. - Syncing gradients every micro-batch in DDP. Correct but slow; use
no_sync()for the first \(K - 1\) micro-batches. - Assuming accumulation works with BatchNorm or contrastive losses. Batch statistics and in-batch negatives depend on the micro-batch, not on the effective batch.
- Checkpointing with dropout but without restoring the random state. The recomputed forward then uses a different dropout mask and the gradients are wrong. PyTorch’s default (
preserve_rng_state=True) handles it; custom recomputation code must too. - Checkpointing a block whose forward has side effects (it updates a counter, a cache, or a running statistic). The side effect happens twice. Keep the KV cache off in training.
- Expecting checkpointing to help the weights. It only reduces activation memory. If weights, gradients and optimizer states do not fit, you need sharding, 8-bit states or LoRA instead.
Interview questions
What is gradient accumulation, and what is the effective batch size?basic
.grad buffers, and call optimizer.step() once at the end. The optimizer sees the gradient of all \(K\) micro-batches together. The effective batch is \(b \cdot K \cdot N_{\text{dp}}\) sequences, or that times \(s\) in tokens (39.1). Memory depends only on the micro-batch \(b\).Is accumulation exactly equivalent to a larger batch?core
Explain the token-count trap with numbers.core
reduction="sum" and divide by the total number of real tokens across all micro-batches (and all GPUs). Hugging Face fixed this in its Trainer in October 2024.How does the token-count trap appear with data parallelism, and how do you fix it?advanced
What does gradient checkpointing store, and what does it recompute?basic
What does full checkpointing cost in compute and time?core
Derive the square-root rule. Why do transformers usually checkpoint every block instead?advanced
What is selective activation recomputation?advanced
Your 7B fine-tune runs out of memory. In what order do you try accumulation, checkpointing and other options?core
Why must recomputation restore the random-number state?core
preserve_rng_state=True by default).Exercises
- You want 2 million tokens per step with sequences of 8,192 tokens, 32 GPUs and a micro-batch of 1. How many accumulation steps do you need?
- Three micro-batches have 200, 50 and 750 real tokens and mean losses 3.0, 4.0 and 2.0. Compute the naive loss (39.4) and the correct loss (39.3).
- In the same example, by what factor is one token of the 50-token micro-batch over-weighted by the naive loss?
- With formula (39.7), \(n = 64\) simple layers and \(I = A\): what segment length is best, and how much memory (in units of \(I\)) do you need, compared with no checkpointing?
- For \(L = 32\), \(d = 4{,}096\), \(s = 8{,}192\), \(b = 1\): how much memory do the stored block inputs (\(2sbdL\)) take in BF16?
- Use formula (39.9) to compute the FLOP overhead of selective recomputation at \(d = 4{,}096\) and \(s = 16{,}384\).
Answerscheck
Key takeaways
- Gradient accumulation runs \(K\) micro-batches and steps once; the effective batch is \(b K N_{\text{dp}}\) sequences (39.1), and memory depends only on \(b\).
- Because the gradient is linear, accumulation equals the big batch (39.2), if the loss is normalized by the total count of the whole step.
- For tokens with padding, use the sum of token losses divided by the step’s real-token count (39.3); the mean of micro-batch means (39.4) over-weights short micro-batches. Do the same across GPUs.
- Checkpointing keeps only block inputs and recomputes the rest in the backward pass: memory goes from \(L \cdot sbd(34 + 5hs/d)\) to about \(2sbdL\) plus one block (39.5, 39.6), 104 GB to 1.6 GB in the 7B example.
- The price is one extra forward pass: \(8N\) instead of \(6N\) FLOPs, 33% (39.8). Selective recomputation of the attention core costs only a few percent (39.9).
- The square-root rule (39.7) says checkpoint every \(\sqrt{n}\) layers for simple chains; for transformer blocks, one checkpoint per block is best.
Key formulas
References for this session
- Chen et al. (2016). Training Deep Nets with Sublinear Memory Cost
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Chen et al. (2016). Training Deep Nets with Sublinear Memory Cost
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Hugging Face (Transformers team) (2024). Fixing Gradient Accumulation