measure memory, accumulation and checkpointing
A hands-on lab. You take a model with the exact shape of Llama 3.2 1B, measure the peak memory of one AdamW training step at different batch sizes, and switch on BF16, gradient checkpointing and gradient accumulation one by one, recording memory and speed. You check that accumulation gives the same gradient as one big batch, and you write a memory calculator that matches all 60 measurements within 0.3%.
What you will learn
- How to measure the peak memory of a training step correctly: after a warm-up step, with max_memory_allocated or MemTracker, and with FakeTensorMode for steps that do not fit
- A memory calculator with three peak moments (end of forward, start of backward, end of backward) that matches the tracker for the Llama 3.2 1B shape
- Why BF16 autocast made training 2.46 times faster but saved almost no memory for this model
- What checkpointing really costs (about 30% more time here) and why gradient accumulation can raise the peak
- Why the logits of a 128k vocabulary take 1.54 MB per token, and how the attention kernel can add a T-squared term
- How to test that accumulation gives the big-batch gradient, with the right tolerance for FP32 and BF16
- How to choose micro-batch, accumulation and checkpointing for a given GPU
Why this matters
In session 38 you added up the bytes of a training step on paper, and in session 39 you learned two tools that cut the activation part of that bill. In this lab you check the paper against the machine. You take a model with the exact shape of Llama 3.2 1B, measure the peak memory of one full training step at different batch sizes, switch on BF16, checkpointing and accumulation one by one, time each setting, check that accumulation really gives the same gradient as one big batch, and write a memory calculator that matches the measurements.
Why spend a day on this? Because “CUDA out of memory” is the most common error in LLM work, and the usual reaction (halve the batch, try again) wastes GPU hours. An engineer who can say “the states take 19.8 GB, the logits take 1.5 MB per token, so this run peaks at 32.6 GB” can choose the batch size, the sequence length and the tricks before the job starts. The lab also shows some facts that surprise most people the first time:
- For this 1B model, turning on BF16 autocast saves almost no memory at small batch sizes. It saves time.
- Gradient accumulation can make the peak higher, not lower, for a small model with a large vocabulary.
- The output layer and the loss, not the transformer blocks, hold 42% of the activation memory, because the vocabulary has 128,256 tokens.
Real systems do the same accounting. The Ultra-Scale Playbook from Hugging Face builds its whole guide to parallelism on this memory bill. [Certain] PyTorch ships a memory tracker and a memory snapshot tool for exactly this job, and Megatron-LM, DeepSpeed and torchtitan expose switches for recomputation, mixed precision and accumulation. [Certain]
Setup and plan
The model is a Llama-style decoder written in about 50 lines of PyTorch (part 1 of the program below). Its shape is copied from the config.json of Llama 3.2 1B on Hugging Face: [Certain]
| Setting | Value | Meaning |
|---|---|---|
| \(V\) | 128,256 | vocabulary size |
| \(d\) | 2,048 | width of the residual stream |
| \(L\) | 16 | number of transformer blocks |
| \(h\), \(h_{kv}\) | 32, 8 | query heads and key/value heads (grouped-query attention, head size 64) |
| \(d_{ff}\) | 8,192 | hidden size of the SwiGLU feed-forward network |
| tied embeddings | yes | the output layer reuses the input embedding matrix |
| \(N\) | 1,235,814,400 | parameters (counted by PyTorch and by our formula) |
We leave out the “llama3” RoPE scaling, which does not change memory. The optimizer is AdamW. Weights are kept in FP32; “BF16” in this lab means mixed precision with torch.autocast: matrix multiplications run in BF16, while the weights, the norms and the loss stay in FP32 (session 37).
How we measure. On an NVIDIA GPU you would call torch.cuda.reset_peak_memory_stats() before the step and torch.cuda.max_memory_allocated() after it. We wrote this page on a laptop (Apple M5 Pro, 48 GB of unified memory, PyTorch 2.8.0), so we used PyTorch’s MemTracker (in torch.distributed._tools). It follows every tensor that is created and freed and reports the peak of their total size. [Certain] This is very close to what max_memory_allocated reports; the CUDA allocator also rounds each allocation up to a multiple of 512 bytes. [Likely] And we used one more trick: MemTracker also works under FakeTensorMode, where tensors have shapes and dtypes but no data. The program runs as usual, every allocation is counted, and nothing is really allocated. So you can measure a step that does not fit in your memory at all.
Is a fake measurement a real measurement? We checked. On a smaller model of the same family (4 layers, \(d = 512\), \(V = 32{,}000\)) we ran each setting twice, once with real tensors and once with fake ones, on the CPU and on the Apple GPU. The peaks were identical to the byte in every case, for example 997,034,148 bytes both ways for FP32 at 4 sequences of 256 tokens on the CPU. [Certain: we ran it] All memory numbers on this page for the 1B model come from this tracker. All times come from real runs of the 1B model on the Apple GPU.
The plan has five steps: (1) write the calculator from first principles, (2) measure peak memory at several batch sizes and compare, (3) switch on BF16, checkpointing and accumulation and record memory and speed, (4) check that accumulation gives the same gradient as one big batch, (5) put it together and pick a configuration for a real GPU.
Step 1: what is in memory at the peak
Memory during training has three parts. We write \(n = B \cdot T\) for the number of tokens in one micro-batch (\(B\) sequences of \(T\) tokens).
The states. Every parameter has an FP32 weight (4 bytes), an FP32 gradient (4 bytes) and two FP32 Adam moments (8 bytes): 16 bytes per parameter (session 4). But the gradients do not always exist. With zero_grad(set_to_none=True), the gradient buffers are freed after each optimizer step and created again during the next backward pass. So during the forward pass only 12 bytes per parameter are held, unless you accumulate: then the gradients of the earlier micro-batches stay in memory during the forward pass of the later ones.
The saved activations. These are the tensors that autograd keeps for the backward pass. We counted them for our code with saved_tensors_hooks (a PyTorch hook that sees every tensor saved for backward), block by block. Per token and per block, in bytes:
- Read the symbols
- \(a\)
- bytes saved for the backward pass by one block for one token
- \(40d\)
- about ten \(d\)-sized FP32 tensors: the block input, the two RMSNorm inputs and outputs, \(Q\), \(K\), \(V\) after the GQA repeat, and the attention output
- \(16 d_{ff}\)
- four FP32 tensors in the SwiGLU network: gate output, SiLU output, up output, their product
- \(4h\)
- the softmax statistics (one FP32 number per head) that a fused attention kernel keeps instead of the \(T \times T\) score matrix
- \(L, n\)
- number of blocks, tokens in the micro-batch
For the 1B model this is \(a = 213{,}128\) bytes in FP32, \(135{,}304\) in BF16, and \(8{,}192\) with checkpointing. Two details explain why BF16 does not halve \(a\). The residual stream and the RMSNorm math stay in FP32 (the code casts to FP32 inside the norm, as Llama does), and autocast casts the norm output to BF16 once for each linear layer that reads it, so the same input is stored three times for \(Q\), \(K\) and \(V\). Under checkpointing, the stored block input is the FP32 residual stream, so it costs \(4d\), not \(2d\).
The big temporary buffers. Two more things are large. First, under autocast PyTorch keeps a BF16 copy of every weight matrix for the whole forward pass (the autocast cache): \(C = 2 N_{\text{mat}} = 2.47\) GB, where \(N_{\text{mat}}\) is the number of parameters in matrices (all but the norm weights). Second, the logits: for every token the output layer produces \(V = 128{,}256\) scores. The loss casts them to FP32 and computes a log-softmax, and its backward pass makes the gradient of both. Counted per token, the output layer and the loss hold about \(10V\) bytes at the end of the forward pass and \(12V\) bytes at the start of the backward pass. With \(V = 128{,}256\), \(12V = 1{,}539{,}072\) bytes: 1.54 MB per token, more than half of the 2.16 MB that all 16 BF16 blocks together save per token.
The peak is the largest of three moments in the step:
- Read the symbols
- \(C\)
- BF16 weight copies of autocast, \(2N_{\text{mat}}\) bytes; 0 in FP32
- \(C'\)
- the copies still alive at the start of backward: all of \(C\) without checkpointing, only the output-layer copy (\(2Vd\)) with checkpointing, because the blocks recast their weights when they recompute
- \(10V + 10d\), \(12V + 10d\)
- the output layer and the loss, per token (BF16 case; in FP32 the forward term is \(8V + 12d\) and the backward term \(12V + 12d\))
- \(8Vd\)
- two FP32 temporary buffers of the size of the embedding matrix at the very end of the backward pass, when every other gradient already exists (with \(K \gt 1\) we measured \(12Vd\))
We found the last term by measurement, not by theory: the tracker showed a floor of \(16N + 8Vd\) that no setting could go below. With an untied output layer the same floor appears, so it comes from the embedding backward and its gradient accumulation, not from the tied weights. [Likely: the exact source of the two buffers is our reading of the tracker’s per-module report]
- input: model shape \((V, d, L, h, d_{ff})\), micro-batch \(B \times T\), precision, checkpointing on or off, accumulation steps \(K\)
- \(N, N_{\text{mat}} \leftarrow\) count parameters; \(n \leftarrow B \cdot T\)
- \(S \leftarrow 12N\) if \(K = 1\), else \(16N\) // (41.1)
- \(a \leftarrow\) bytes per token per block for this precision and checkpointing; \(A \leftarrow L\, n\, a\) // (41.2)
- \(C \leftarrow 2N_{\text{mat}}\) if BF16, else 0
- compute \(P_{\text{fw}}, P_{\text{bw}}, P_{\text{end}}\) // (41.3)
- return the maximum and the name of the moment that wins // (41.4)
In code (part 3 of the program):
def n_params(c): m = c.d * c.d * 2 + 2 * c.d * c.kv * (c.d // c.h) + 3 * c.d * c.ff # matrices of one block return c.vocab * c.d * (1 if c.tied else 2) + c.L * (m + 2 * c.d) + c.d, c.L * m + c.vocab * c.d def calc(c, B, T, bf16, ckpt, K=1, math_attn=False): N, Nmat = n_params(c); n = B * T; V, d = c.vocab, c.d S = (16 if K > 1 else 12) * N # FP32 weights + Adam m, v (+ grads if K > 1) if ckpt: a = 4 * d # only the FP32 block input elif math_attn: a = 40 * d + (8 if bf16 else 16) * c.ff + 4 * c.h * T + 8 elif bf16: a = 34 * d + 8 * c.ff + 4 * c.h + 8 else: a = 40 * d + 16 * c.ff + 4 * c.h + 8 acts = c.L * n * a # saved by the blocks C = 2 * Nmat if bf16 else 0 # BF16 copies of the weights (autocast) p_fw = S + acts + C + n * ((10 * V + 10 * d) if bf16 else (8 * V + 12 * d)) p_bw = S + acts + (2 * V * d if bf16 and ckpt else C) + n * (12 * V + (10 if bf16 else 12) * d) p_end = 16 * N + (8 if K == 1 else 12) * V * d return max(p_fw, p_bw, p_end)
\(n = 4{,}096\) tokens. States: \(S = 12N = 14.83\) GB. Blocks: \(A = 16 \times 4{,}096 \times 135{,}304 = 8.87\) GB. Weight copies: \(C = 2.47\) GB. Logits at the start of backward: \(4{,}096 \times (12 \times 128{,}256 + 10 \times 2{,}048) = 6.39\) GB. So \(P_{\text{bw}} = 14.83 + 8.87 + 2.47 + 6.39 = 32.56\) GB. At the end of the forward pass the logits term is \(10V\) instead of \(12V\), so \(P_{\text{fw}} = 31.51\) GB, and \(P_{\text{end}} = 21.87\) GB. The peak is \(P_{\text{bw}} = 32.56\) GB. The tracker measured 32.561 GB.
Now turn on checkpointing. \(A\) falls to \(16 \times 4{,}096 \times 8{,}192 = 0.54\) GB, and \(C'\) to \(2Vd = 0.53\) GB, so \(P_{\text{bw}} = 22.28\) GB. But \(P_{\text{fw}}\) still holds all the weight copies: \(14.83 + 0.54 + 2.47 + 5.34 = 23.18\) GB. Now the peak is at the end of the forward pass. The tracker measured 23.176 GB.
Figure 1 is this calculator. Change the settings and watch which moment sets the peak.
How to read Figure 1
Start at BF16, \(B = 4\), \(T = 1{,}024\): the middle bar (start of backward) is the peak, 32.56 GB, and the tracker says 32.56 GB. The orange part (logits) is 6.39 GB, almost as big as the blue part (all 16 blocks, 8.87 GB). Click “BF16 + checkpointing”: the blue part almost disappears and the left bar wins, because the green BF16 copies are still there at the end of the forward pass. Set \(B = 1\): all three settings give 21.87 GB, the right bar, the floor set by the states. Move \(K\) to 4: the dark gradient block now also appears in the first two bars, so the peak grows even though nothing else changed. Finally click “Math attention” and push \(T\) to 4,096: the blue part explodes, because the \(T \times T\) attention probabilities are stored.
Step 2: measure peak memory at different batch sizes
The measurement code is part 2 of the program. It builds the model and AdamW, runs one warm-up step so that the Adam moments exist, runs \(K - 1\) micro-batches so that the gradients exist when \(K \gt 1\), and then tracks the last micro-batch together with the optimizer step.
from torch._subclasses.fake_tensor import FakeTensorMode from torch.distributed._tools.mem_tracker import MemTracker def measure(c, B, T, bf16, ckpt, K=1, dev="cpu", fake=True): """Peak bytes of one optimizer step with K micro-batches of B x T tokens.""" with FakeTensorMode() if fake else contextlib.nullcontext(): torch.manual_seed(0); model = LM(c).to(dev); model.ckpt = ckpt opt = torch.optim.AdamW(model.parameters(), lr=1e-4) amp = lambda: torch.autocast(dev, dtype=torch.bfloat16, enabled=bf16) x = torch.randint(0, c.vocab, (B, T), device=dev) def micro(): with amp(): loss = model(x, x) (loss / K).backward() # .grad adds up for _ in range(K): micro() # warm-up step: creates Adam states opt.step(); opt.zero_grad(set_to_none=True) for _ in range(K - 1): micro() # grads now exist (K > 1) mt = MemTracker(); mt.track_external(model, opt) with mt: # track the last micro-batch + step micro(); opt.step(); opt.zero_grad(set_to_none=True) return list(mt.get_tracker_snapshot("peak").values())[0]["Total"]
Why the warm-up step? AdamW creates its two moment buffers lazily, at the first opt.step(). A measurement of the very first step therefore misses \(8N\) bytes of states during the forward and backward pass and is too optimistic. Always measure a later step.
On an NVIDIA GPU the same measurement with real tensors is four lines:
torch.cuda.reset_peak_memory_stats()
step() # one full step, after a warm-up step
torch.cuda.synchronize()
print(torch.cuda.max_memory_allocated() / 1e9, torch.cuda.max_memory_reserved() / 1e9)
(We did not run this CUDA snippet; it is the standard API. [Certain for the API]) Here are the tracker results for \(T = 1{,}024\), in GB (\(10^9\) bytes), with the fused attention kernel. The calculator gives the same numbers to within 0.3% in all 60 settings we measured (\(T\) = 1,024 and 2,048; \(B\) = 1, 2, 4, 8; \(K\) = 1 and 4; three precisions; two attention kernels). [Certain: we ran it]
| Micro-batch \(B\) (\(T = 1{,}024\)) | 1 | 2 | 4 | 8 |
|---|---|---|---|---|
| FP32 | 21.87 | 25.02 | 35.20 | 55.57 |
| BF16 autocast | 21.87 | 24.93 | 32.56 | 47.82 |
| BF16 + checkpointing | 21.87 | 21.87 | 23.18 | 29.21 |
Read the table from left to right. At \(B = 1\) all three are the same: the peak is \(P_{\text{end}}\), the states. Activations start to matter at about 1,200 (BF16) to 1,400 (FP32) tokens per micro-batch, where \(P_{\text{bw}}\) crosses the floor. Then each extra sequence of 1,024 tokens costs about 5.1 GB in FP32 (from \(B = 4\) to \(B = 8\): \((55.57 - 35.20)/4\)), 3.8 GB in BF16, and 1.5 GB with checkpointing. With checkpointing almost all of that 1.5 GB is the logits.
How to read Figure 2
The circles sit on the lines: the calculator is right. All three lines start on the same floor (21.87 GB) and bend upward when activations become larger than the gap to the floor. On an 80 GB GPU at \(T = 2{,}048\), FP32 fits 6 sequences, BF16 fits 8 and BF16 with checkpointing fits 18. On a 24 GB GPU only checkpointing makes \(T = 2{,}048\) fit at all (\(B \le 2\)). Now click “Math attention”: this is what the Apple GPU backend did in PyTorch 2.8, and the FP32 and BF16 lines get much steeper at \(T = 2{,}048\), because \(4hT\) bytes per token per block are now attention probabilities. Checkpointing hides almost all of that difference.
A surprise from the measurement: the attention kernel
Our first comparison on the Apple GPU did not match the calculator: the error grew with \(T\), and at \(T = 2{,}048\), \(B = 8\) in FP32 the Apple GPU needed 165 GB against 96 GB with the fused kernel. An error that grows like \(T^2\) per sequence has one usual suspect: the \(T \times T\) attention matrix. Listing the saved tensors confirmed it. On the CPU, scaled_dot_product_attention used a fused kernel that saves \(Q\), \(K\), \(V\), the output and \(h\) softmax statistics per token. On the Apple GPU it used the plain “math” path, which saves the FP32 attention probabilities, shape \((B, h, T, T)\), even under autocast. [Certain: we listed them with saved_tensors_hooks] That is \(4hT\) bytes per token per block, 131 KB at \(T = 1{,}024\): about as much as the whole rest of the BF16 block (135,304 bytes). On NVIDIA GPUs PyTorch normally picks the FlashAttention or memory-efficient kernel for this call, which behaves like our fused case. [Likely] Lesson: when measurement and calculator disagree, the size of the error and how it scales (with \(n\), with \(T^2\), or constant) tell you which term is missing.
Step 3: BF16, checkpointing and accumulation, measured
Now the speed. We timed the forward and backward passes of the real 1B model on the Apple GPU for an optimizer step of 2,048 tokens (\(T = 512\)), in eight ways. Each time is the best of three runs after one warm-up run. Other jobs were running on the same machine, so expect about ±10% noise. [Certain: we ran it] The memory columns are tracker peaks for the full step with AdamW at the same setting.
| Setting (2,048 tokens per step) | Time (s) | Tokens/s | Peak, fused attention (GB) | Peak, Apple GPU (GB) |
|---|---|---|---|---|
| FP32, \(B = 4\) | 3.473 | 590 | 25.02 | 27.16 |
| FP32, \(B = 4\), checkpointing | 4.064 | 504 | 21.87 | 21.87 |
| BF16, \(B = 4\) | 1.410 | 1,452 | 24.93 | 27.48 |
| BF16, \(B = 4\), checkpointing | 1.831 | 1,119 | 21.87 | 21.87 |
| BF16, \(B = 2\), \(K = 2\) | 1.399 | 1,464 | 26.06 | 27.33 |
| BF16, \(B = 2\), \(K = 2\), checkpointing | 1.973 | 1,038 | 23.71 | 23.71 |
| BF16, \(B = 1\), \(K = 4\) | 1.773 | 1,155 | 24.42 | 25.05 |
| BF16, \(B = 1\), \(K = 4\), checkpointing | 2.200 | 931 | 22.98 | 22.98 |
Four lessons, each with its number:
- BF16 buys speed, not memory, at this size. BF16 is 2.46 times faster than FP32 (3.473 s against 1.410 s), but the peak drops only from 25.02 to 24.93 GB. The activations shrink by 0.64× (135,304 against 213,128 bytes per token per block), but autocast adds 2.47 GB of BF16 weight copies. The two cancel at about 2,000 tokens per micro-batch. On NVIDIA GPUs with tensor cores the speed gap between FP32 and BF16 is usually larger. [Likely]
- Checkpointing costs about one forward pass. The time ratio is 1.30 for BF16 at \(B = 4\) (1.831 / 1.410), close to the 4/3 that the FLOP count predicts (session 39, formula 39.8). In FP32 it was 1.17. In return the peak falls to the 21.87 GB floor.
- Accumulation keeps the gradients alive. \(B = 2, K = 2\) peaks at 26.06 GB, more than \(B = 4, K = 1\) (24.93 GB). Halving the micro-batch removes about 3.8 GB of activations, but the 4.94 GB of gradients (\(4N\)) now stay in memory during the second forward pass. For a 1B model, where activations are small next to the states, accumulation is not a memory tool until the micro-batch is large.
- Small micro-batches are slower. \(B = 1, K = 4\) took 1.773 s against 1.410 s for \(B = 4\): 26% slower for the same tokens and the same FLOPs, because small matrix multiplications use the GPU less well. \(B = 2, K = 2\) was as fast as \(B = 4\).
What does 1,452 tokens per second mean in FLOPs? With the rule of \(6N\) FLOPs per token (session 25):
About 10.8 TFLOP/s on a laptop GPU (the attention FLOPs are not included). With checkpointing the GPU does \(8N\) FLOPs per token, but only \(6N\) count as useful work: this is why MFU falls when you recompute, even if the GPU is just as busy (session 42).
How to read Figure 3
The two FP32 points are far up: slow. Among the BF16 points, the ones without checkpointing take 1.4–1.8 s but need 24.4–26.1 GB; the checkpointed ones sit on the left (21.9–23.7 GB) but higher, at 1.8–2.2 s. No point is both lowest and leftmost: you buy memory with time. Click “BF16, B = 2, K = 2” and compare it with “BF16, B = 4”: same speed, but more memory, because of the gradients that accumulation keeps.
Step 4: does accumulation give the same gradient?
session 39 proved that \(K\) micro-batches with the right weights give exactly the gradient of one big batch. Now we test it on our Llama code. We use a tiny model of the same code (\(V = 256\), \(d = 64\), 2 blocks, 8 sequences of 32 tokens), because we need to store several full gradients. We compare two gradient vectors with the relative difference
- Read the symbols
- \(\mathbf{g}_1\)
- all parameter gradients of one batch of 8 sequences, flattened into one vector
- \(\mathbf{g}_K\)
- the same after \(K\) micro-batches of \(8/K\) sequences, added up in
.grad - \(\delta\)
- 0 means identical; about \(10^{-7}\) is FP32 rounding (one FP32 unit is \(2^{-23} \approx 1.2 \times 10^{-7}\))
With padding, the right weight for micro-batch \(k\) is its share of the real tokens, \(n_k / N_{\text{tot}}\), applied to its mean loss. The naive weight \(1/K\) is only right when all micro-batches have the same number of real tokens:
- input: model, batch \((x, y)\) with padding labels −100, number of micro-batches \(K\)
- \(\mathbf{g}_1 \leftarrow\) gradient of the mean loss over all real tokens of the whole batch
- zero the gradients; \(N_{\text{tot}} \leftarrow\) number of real tokens in the batch
- for each micro-batch \((x_k, y_k)\) do
- \(\bar\ell_k \leftarrow\) mean loss over the real tokens of micro-batch \(k\); backward of \(\bar\ell_k \cdot n_k / N_{\text{tot}}\) // (41.7), .grad adds up
- end for
- \(\mathbf{g}_K \leftarrow\) the
.gradbuffers; report \(\delta\) (41.6) // expect about 1e-7 in FP32 - repeat with weights \(1/K\), without any weight, with BF16 autocast, and with checkpointing on
class Tiny(Cfg): vocab, d, L, h, kv, ff = 256, 64, 2, 4, 2, 128 def grad_vec(model): return torch.cat([p.grad.flatten() for p in model.parameters()]) def grads(model, x, y, K, weight="tokens", bf16=False): model.zero_grad(set_to_none=True); total = int((y != -100).sum()) for xb, yb in zip(x.chunk(K), y.chunk(K)): with torch.autocast("cpu", dtype=torch.bfloat16, enabled=bf16): loss = model(xb, yb) # mean over real tokens of xb w = int((yb != -100).sum()) / total if weight == "tokens" else 1 / K (loss * w).backward() # .grad adds up return grad_vec(model) def rel(a, b): return ((a - b).norm() / b.norm()).item()
The results (CPU, FP32 unless stated, seed 0): [Certain: we ran it]
| Test | \(\delta\) | Verdict |
|---|---|---|
| Packed (no padding), \(K = 2, 4, 8\), weight \(1/K\) | \(1.3 \times 10^{-7}\), \(1.4 \times 10^{-7}\), \(1.4 \times 10^{-7}\) | same gradient, up to FP32 rounding |
| Packed, \(K = 4\), no division by \(K\) | norm ratio 4.0 | gradient 4 times too large |
| Padded (62, 53, 14, 9 real tokens), token weights | \(1.2 \times 10^{-7}\) | same gradient |
| Padded, naive weight \(1/K\) | 0.60 (norm ratio 1.23) | a different gradient |
| Checkpointing on vs off | 0.0 | bit-for-bit the same |
| BF16 autocast, \(K = 2, 4, 8\) vs BF16 big batch | \(9.3 \times 10^{-4}\), \(9.2 \times 10^{-4}\), \(9.1 \times 10^{-4}\) | same within BF16 noise |
| BF16 big batch vs FP32 big batch | \(2.3 \times 10^{-3}\) | the size of BF16 noise itself |
Three things to notice. The packed differences do not grow with \(K\): adding 8 numbers in a different order costs the same rounding as adding 2. Checkpointing gives exactly the same bits, because the recomputed forward runs the same kernels on the same inputs (and with dropout, the same random state). And under BF16 the accumulated gradient differs from the big-batch one by \(9 \times 10^{-4}\), less than BF16 differs from FP32; so a BF16 test must use a BF16 tolerance, not an FP32 one.
How to read Figure 4
Click through the tests. The packed and token-weighted bars stop at about \(10^{-7}\), on the FP32 line: correct. The naive padded bar reaches 0.60, almost seven orders of magnitude higher: this is the token-count trap, a real bug. The BF16 bars sit near \(10^{-3}\), below the BF16 line: correct for BF16. Checkpointing has no bar at all, because \(\delta = 0\) exactly.
Step 5: pick a configuration for a real GPU
Now use the calculator for a decision. You want 32,768 tokens per optimizer step (16 sequences of 2,048 tokens) on one 80 GB GPU, with AdamW and BF16.
The tracker results (the calculator agrees to 0.01 GB):
- \(B = 16, K = 1\): 139.35 GB. Does not fit.
- \(B = 8, K = 2\): 83.27 GB. Does not fit, by a little.
- \(B = 4, K = 4\): 52.76 GB. Fits, no recomputation.
- \(B = 16, K = 1\), checkpointing: 70.75 GB. Fits, about 30% slower.
- \(B = 8, K = 2\), checkpointing: 48.00 GB. Fits.
The best choice is \(B = 4, K = 4\) without checkpointing: no recomputation, and \(B = 4\) was large enough to keep the Apple GPU busy in Step 3. Checkpointing would cost about 30% more time to avoid accumulation that costs little. On a 40 GB GPU, \(B = 4, K = 4\) does not fit, and checkpointing with \(B = 4, K = 4\) (34.15 GB) is the answer.
The general recipe, in order: (1) compute the states; if \(16N + 8Vd\) does not fit, no batch size will, and you need sharding, 8-bit optimizer states or LoRA (session 45, session 58); (2) choose the largest micro-batch that fits without checkpointing and is big enough to keep the GPU busy; (3) fill the global batch with accumulation; (4) only if even a small micro-batch does not fit (long sequences), turn on checkpointing. For very large vocabularies, a fused or chunked cross-entropy kernel that never stores all \(n \times V\) logits at once removes the orange part of Figure 1. [Likely]
To run everything, put the four parts in one file, memlab.py, and add this main block. With fake tensors it needs no GPU and about 30 seconds on a laptop CPU; it prints the 12 numbers of the table in Step 2 next to the calculator, then the gradient tests.
if __name__ == "__main__": c = Cfg(); N, _ = n_params(c); print("parameters:", N) for bf16, ckpt in [(False, False), (True, False), (True, True)]: for B in [1, 2, 4, 8]: m, p = measure(c, B, 1024, bf16, ckpt), calc(c, B, 1024, bf16, ckpt) print(f"bf16={bf16:d} ckpt={ckpt:d} B={B}: measured {m/1e9:6.2f} GB calc {p/1e9:6.2f} GB") torch.manual_seed(0); tiny = LM(Tiny()) x = torch.randint(0, 256, (8, 32)); y = torch.randint(0, 256, (8, 32)) g1 = grads(tiny, x, y, K=1) for K in [2, 4, 8]: print("packed, K =", K, "rel. diff", f"{rel(grads(tiny, x, y, K), g1):.1e}") y[torch.arange(32)[None] >= torch.tensor([32, 30, 28, 25, 8, 6, 5, 4])[:, None]] = -100 # padding g1 = grads(tiny, x, y, K=1) print("padded, per-token weights", f"{rel(grads(tiny, x, y, 4), g1):.1e}", "| naive 1/K", f"{rel(grads(tiny, x, y, 4, weight='naive'), g1):.2f}") tiny.ckpt = True; print("checkpointing changes the gradient by", rel(grads(tiny, x, y, 1), g1))
And here is part 1, the model. It is a plain Llama block: RMSNorm in FP32, RoPE, grouped-query attention through scaled_dot_product_attention, a SwiGLU network, a tied output layer, and an optional checkpoint around each block with use_reentrant=False (the version PyTorch recommends). [Certain]
import contextlib, time, torch, torch.nn as nn, torch.nn.functional as F from torch.utils.checkpoint import checkpoint class Cfg: # Llama 3.2 1B, from its config.json (RoPE scaling left out) vocab, d, L, h, kv, ff, tied = 128256, 2048, 16, 32, 8, 8192, True class RMSNorm(nn.Module): def __init__(s, d): super().__init__(); s.w = nn.Parameter(torch.ones(d)) def forward(s, x): x32 = x.float() # norm math in FP32, like Llama return (x32 * torch.rsqrt(x32.pow(2).mean(-1, keepdim=True) + 1e-5)).type_as(x) * s.w def rope(T, dh, dev, base=500000.0): inv = base ** (-torch.arange(0, dh, 2, device=dev).float() / dh) ang = torch.arange(T, device=dev).float()[:, None] * inv[None] ang = torch.cat([ang, ang], -1) return ang.cos(), ang.sin() def rot_half(x): a, b = x.chunk(2, -1); return torch.cat([-b, a], -1) class Block(nn.Module): def __init__(s, c): super().__init__(); s.c = c; dh = c.d // c.h s.n1, s.n2 = RMSNorm(c.d), RMSNorm(c.d) s.q = nn.Linear(c.d, c.h * dh, bias=False); s.k = nn.Linear(c.d, c.kv * dh, bias=False) s.v = nn.Linear(c.d, c.kv * dh, bias=False); s.o = nn.Linear(c.h * dh, c.d, bias=False) s.gate = nn.Linear(c.d, c.ff, bias=False); s.up = nn.Linear(c.d, c.ff, bias=False) s.down = nn.Linear(c.ff, c.d, bias=False) def forward(s, x, cos, sin): c = s.c; B, T, _ = x.shape; dh = c.d // c.h y = s.n1(x) q = s.q(y).view(B, T, c.h, dh).transpose(1, 2) k = s.k(y).view(B, T, c.kv, dh).transpose(1, 2) v = s.v(y).view(B, T, c.kv, dh).transpose(1, 2) cos, sin = cos.to(q.dtype), sin.to(q.dtype) q = q * cos + rot_half(q) * sin; k = k * cos + rot_half(k) * sin r = c.h // c.kv; k, v = k.repeat_interleave(r, 1), v.repeat_interleave(r, 1) # GQA a = F.scaled_dot_product_attention(q, k, v, is_causal=True) x = x + s.o(a.transpose(1, 2).reshape(B, T, c.d)) y = s.n2(x) return x + s.down(F.silu(s.gate(y)) * s.up(y)) # SwiGLU FFN class LM(nn.Module): def __init__(s, c): super().__init__(); s.c = c; s.ckpt = False s.emb = nn.Embedding(c.vocab, c.d) s.blocks = nn.ModuleList(Block(c) for _ in range(c.L)) s.norm = RMSNorm(c.d); s.head = nn.Linear(c.d, c.vocab, bias=False) if c.tied: s.head.weight = s.emb.weight def forward(s, idx, tgt): cos, sin = rope(idx.shape[1], s.c.d // s.c.h, idx.device) x = s.emb(idx) for b in s.blocks: # checkpointing: keep only each block's input x = checkpoint(b, x, cos, sin, use_reentrant=False) if s.ckpt else b(x, cos, sin) logits = s.head(s.norm(x)) return F.cross_entropy(logits.float().view(-1, s.c.vocab), tgt.view(-1))
Where this is used
| Setting | What you measure | Why |
|---|---|---|
| Planning a pretraining run | Peak per GPU for each micro-batch, length and recomputation choice, before renting the cluster | The global batch is fixed by the recipe; the micro-batch and accumulation are chosen to fit (Ultra-Scale Playbook) |
| Full fine-tuning of 1B–8B models on one node | States vs activations; whether accumulation helps or hurts | For small models with big vocabularies the states and logits dominate, as in this lab |
| Long-context training | The \(T^2\) term, and which attention kernel runs | A math attention path stores \(4hT\) bytes per token per block, which explodes at long \(T\) |
| Memory-saving kernels (fused cross-entropy, fused norms) | The logits term \(12V\) per token | With \(V = 128{,}256\) the logits are 42% of the activation bytes in BF16 |
| VLM fine-tuning (image or video tokens + text) | Activations of the vision encoder and of the many visual tokens in the LLM | A few images or frames add thousands of tokens, so \(n\) grows fast (session 188) |
| Video models: action recognition, video classification, key-moment detection | Peak memory per clip length; checkpointing in the video backbone | Frames × patches make very long sequences; the same per-token accounting applies |
| Contrastive video-text retrieval | Batch size that fits, since accumulation does not add in-batch negatives | Here memory sets the number of negatives, so checkpointing matters more than accumulation |
Common mistakes
- Measuring the first step. Adam creates its moments at the first
opt.step(), so the first step misses \(8N\) bytes (9.9 GB here). Measure after a warm-up step. - Reading
nvidia-smiinstead of the allocator. It includes the CUDA context and the cache of the caching allocator. Usemax_memory_allocatedfor the model’s needs andmax_memory_reservedfor what the process holds. - Expecting BF16 autocast to halve memory. The FP32 master weights, the FP32 norms and residual stream, and the BF16 weight copies all remain. For this model BF16 saved 0.09 GB at \(B = 4\), \(T = 512\).
- Expecting accumulation to always lower the peak. The gradients (\(4N\)) stay alive across micro-batches. Lower \(B\) only lowers the peak when the activations you remove are larger than \(4N\).
- Forgetting the logits. \(12V\) bytes per token is easy to miss and, with a 128k vocabulary, as large as many transformer blocks.
- Ignoring which attention kernel runs. The same code can save \(4h\) or \(4hT\) bytes per token per block depending on the backend, dtype and mask. Check with the profiler or with
saved_tensors_hooks. - Comparing gradients with the wrong tolerance. FP32 accumulation differs by about \(10^{-7}\), BF16 by about \(10^{-3}\). A test that demands \(10^{-7}\) under BF16 fails for no reason; a test that accepts \(10^{-1}\) hides the token-count trap.
- Timing without synchronizing. GPU calls are asynchronous; call
torch.cuda.synchronize()(ortorch.mps.synchronize()) before reading the clock, and skip the first run, which includes warm-up work.
Interview questions
How do you measure the peak GPU memory of a training step in PyTorch?basic
torch.cuda.reset_peak_memory_stats(), run one full step (all micro-batches and opt.step()), synchronize, and read torch.cuda.max_memory_allocated(). Also read max_memory_reserved() to see the allocator cache. For a breakdown, use the memory snapshot tool or MemTracker, which splits the peak into parameters, gradients, optimizer states and activations; under FakeTensorMode it measures steps that do not even fit.Estimate the memory of full fine-tuning a 1.24B model with AdamW. Where is the floor?basic
Why did BF16 autocast barely reduce memory for the 1B model?core
Can gradient accumulation increase peak memory? Give numbers.core
Why are the logits such a large part of activation memory, and what can you do?core
Your calculator and your measurement disagree, and the error grows with \(T^2\). What is missing?advanced
saved_tensors_hooks or the profiler, and force a fused kernel (or use FlashAttention) if one is available.How would you test that your accumulation code is correct?core
Does gradient checkpointing change the gradients?basic
preserve_rng_state=True default) and the forward has no side effects. It changes only memory and time.Checkpointing made the step 30% slower. Is that expected, and how does it affect MFU?core
You need 32k tokens per step on an 80 GB GPU for the 1B model at \(T = 2{,}048\). What do you choose?advanced
Exercises
- Llama 3.1 8B has \(N = 8{,}030{,}261{,}248\) parameters. How much memory do the AdamW states take with FP32 master weights?
- Compute \(a\) (41.2) for BF16 autocast with \(d = 2{,}048\), \(d_{ff} = 8{,}192\), \(h = 32\). How much do all 16 blocks save per token?
- Llama 2 has \(V = 32{,}000\). How many bytes per token do the logits take at the start of backward (\(12V\)), and how does that compare with Llama 3’s vocabulary?
- From the Step 3 table, compute the time overhead of checkpointing for BF16 at \(B = 1, K = 4\), and the slowdown of \(B = 1, K = 4\) against \(B = 4, K = 1\) without checkpointing.
- In the padded gradient test the micro-batches have 62, 53, 14 and 9 real tokens. With the naive weight \(1/K\), how many times too much does one token of the last micro-batch count?
- With checkpointing and \(B = 4\), \(T = 1{,}024\), the peak is \(P_{\text{fw}}\). Which term would you remove first to lower it, and by how much would the peak fall if you removed it completely?
Answerscheck
Key takeaways
- Measure a later step, not the first one, with
max_memory_allocatedorMemTracker; underFakeTensorModethe tracker measures steps that do not fit, and it matched real runs to the byte. - The peak is the largest of three moments (41.3): end of forward, start of backward, end of backward. A calculator with states, per-token block activations, weight copies and logits matched all 60 measured settings within 0.3%.
- For the Llama 3.2 1B shape the floor is 21.87 GB (\(16N + 8Vd\)); the logits take 1.54 MB per token, 42% of the BF16 activation bytes.
- BF16 autocast was 2.46 times faster but saved almost no memory at small batches; checkpointing reached the floor for about 30% more time; accumulation kept 4.94 GB of gradients alive and could raise the peak.
- Accumulation with token weights gives the big-batch gradient up to \(10^{-7}\) in FP32; the naive \(1/K\) with padding was off by 0.60; checkpointing changed nothing, bit for bit.
- When the calculator and the measurement disagree, the way the error scales points to the missing term; here a \(T^2\) error revealed a non-fused attention kernel.
Key formulas
References for this session
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Chen et al. (2016). Training Deep Nets with Sublinear Memory Cost
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- Micikevicius et al. (2017). Mixed Precision Training
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Meta (2024). Llama 3.2 1B model card and config.json
- PyTorch (2024). torch.distributed._tools.mem_tracker: MemTracker (source and docstring)
- Hugging Face (Transformers team) (2024). Fixing Gradient Accumulation