Memory math for training
Before a training run starts, you can predict how much GPU memory it needs and where every gigabyte goes. This session builds the full bill of one training step: 16 bytes per parameter for mixed-precision Adam, activations that grow with batch, sequence length and depth, the large logits tensor, and the overheads that formulas miss, such as temporary buffers and fragmentation. With it you can explain why full fine-tuning of a 7B model does not fit on one 24 GB GPU, and what does.
What you will learn
- The four parts of training memory: model states, activations, temporary buffers, overhead and fragmentation
- Why mixed-precision AdamW costs 16 bytes per parameter, and how other optimizers and precisions change it (8, 10, 12, 20 bytes)
- How to count the activations of one Llama-style block, about sb(12h + 4g + 8I) bytes, and how they grow with batch, sequence length and depth
- The two hidden giants: the s² attention scores without FlashAttention, and the logits with a large vocabulary
- Why full fine-tuning of a 7B model does not fit on one 24 GB GPU, and why LoRA or QLoRA with checkpointing does
- Where the memory peak is in one step, and why a run can survive step 1 and crash in step 2
- The PyTorch caching allocator (allocated vs reserved), fragmentation, and how to measure and fix an OOM
Why this matters
Most training runs do not fail because the math is wrong. They fail with one line: CUDA out of memory. Before you start a run, you should be able to predict, with a pen and a few numbers, how many gigabytes it needs and where they go.
In session 37 we saw that one number can take 1, 2 or 4 bytes, and that mixed precision keeps some copies in BF16 and some in FP32. In session 25 we counted parameters and FLOPs and gave a first, short memory estimate. This session is the full memory bill of one training step. We answer four questions: how many bytes each parameter costs (16 for mixed-precision Adam), how activation memory grows with batch, sequence length and depth, why full fine-tuning of a 7B model cannot fit on one 24 GB GPU, and where the “missing” gigabytes go when your estimate says the run fits but it still crashes.
This bill decides real engineering choices. It is the reason ZeRO and FSDP shard optimizer states across GPUs (session 45), the reason gradient checkpointing exists (session 39), and the reason most people fine-tune 7B models with LoRA or QLoRA on a single card (session 57, session 58). The ZeRO paper starts with exactly this analysis and calls it “Where did all the memory go?”. [Certain]
The four parts of the bill
During training, GPU memory holds four kinds of things:
- Model states: the weights, their gradients, and the optimizer states (for Adam: two running averages per weight). Their size depends only on the number of parameters \(N\), not on the data.
- Activations: the intermediate results of the forward pass that the backward pass needs. Their size depends on the batch size, the sequence length and the depth, not on \(N\) directly.
- Temporary buffers: tensors that live for a short time, for example a large FP32 copy of the logits, a buffer that groups gradients for communication, or scratch space of a kernel.
- Overhead and waste: the CUDA context and kernels, memory that the allocator has reserved but not used, and fragmentation (free memory cut into pieces that are too small to use).
The ZeRO paper uses the same split: model states on one side, and “residual states” (activations, temporary buffers, fragmented memory) on the other. [Certain]
Model states: 16 bytes per parameter
Recall the standard mixed-precision recipe (session 37, session 4). The forward and backward passes run in BF16, so we keep a BF16 copy of the weights (2 bytes each) and get BF16 gradients (2 bytes each). The optimizer works in FP32: it keeps an FP32 master copy of the weights, so small updates are not lost to rounding, and Adam’s two moments \(\mathbf{m}\) and \(\mathbf{v}\), also in FP32. Add it up:
The ZeRO paper writes this as \(2\Psi + 2\Psi + K\Psi\) with \(K = 12\) for mixed-precision Adam, where \(\Psi\) is the number of parameters. [Certain] Its example: a 1.5B-parameter GPT-2 needs only 3 GB for its 16-bit weights, but at least 24 GB for its model states. [Certain]
The general form is useful, because every optimizer and every precision choice just changes the three numbers:
- Read the symbols
- \(N\)
- number of trainable parameters (for LoRA, frozen weights count only in \(b_w\))
- \(b_w\)
- bytes per parameter for the weights used in the forward pass (2 for BF16, 4 for FP32, about 0.5 for 4-bit)
- \(b_g\)
- bytes per parameter for the gradients (2 for BF16, 4 for FP32, 6 if you keep both)
- \(b_o\)
- bytes per parameter for everything the optimizer keeps: master weights and moments
Two rows of Figure 1 surprise many people. First, plain PyTorch torch.autocast keeps the weights and gradients in FP32 and makes BF16 copies of the weights on the fly during the forward pass: 4 + 4 + 4 + 4 is also 16 bytes, with a different layout. [Likely] Second, many large-scale frameworks keep an extra FP32 gradient buffer to add up gradients precisely, which gives 20 bytes per parameter; the Ultra-Scale Playbook lists 112 GB for a 7B model without this buffer and 140 GB with it. [Certain]
| Setup | \(b_w + b_g + b_o\) | Llama 2 7B (6.74B) | Llama 3 8B (8.03B) |
|---|---|---|---|
| Mixed-precision AdamW (38.1) | \(2 + 2 + 12 = 16\) | 107.8 GB | 128.5 GB |
| Same, plus FP32 gradient accumulation | \(2 + 6 + 12 = 20\) | 134.8 GB | 160.6 GB |
| 8-bit AdamW (moments in 8 bits) | \(2 + 2 + 6 = 10\) | 67.4 GB | 80.3 GB |
| SGD with momentum, mixed precision | \(2 + 2 + 8 = 12\) | 80.9 GB | 96.4 GB |
| Pure BF16 AdamW (no master copy) | \(2 + 2 + 4 = 8\) | 53.9 GB | 64.2 GB |
| Plain SGD in BF16 (no state) | \(2 + 2 + 0 = 4\) | 27.0 GB | 32.1 GB |
| Inference only (BF16 weights) | 2 | 13.5 GB | 16.1 GB |
Here GB means \(10^9\) bytes. A “24 GB” or “80 GB” GPU actually has 24 or 80 GiB (\(2^{30}\) bytes each), so 24 GiB = 25.8 GB, but part of it goes to overhead (see below). In this session we compare GB against the GPU’s label, which is a safe, slightly pessimistic rule.
Activation memory
Backpropagation needs, for every operation, some of its inputs or outputs from the forward pass (session 4). For \(Y = XW\) the weight gradient is \(X^\top G\), so \(X\) must be kept. For \(\text{SiLU}(z)\) the local derivative needs \(z\). These saved tensors are the activations. Each one has the shape of the data flowing through the network: (batch \(b\)) × (sequence length \(s\)) × (some width). That is why activation memory is proportional to \(b \cdot s\), and to the number of layers \(L\).
Counting one Llama-style block
Take a modern decoder block: RMSNorm, attention with grouped-query attention (GQA) and FlashAttention, a residual add, RMSNorm, a SwiGLU feed-forward network, a residual add (session 23). Everything is stored in BF16 (2 bytes). Let \(h\) be the hidden size, \(g = h_{kv} d_h\) the width of K and V, and \(I\) the FFN’s inner size. Walk through the block and write down what each step must save:
| Operation | What it saves for backward | Bytes |
|---|---|---|
| RMSNorm 1 | its input | \(2sbh\) |
| Q, K, V projections | their shared input (the norm output) | \(2sbh\) |
| FlashAttention | Q, K, V and its output (plus one small FP32 number per row) | \(2sbh + 4sbg + 2sbh\) |
| Output projection | its input = the attention output (already counted) | 0 |
| RMSNorm 2 | its input | \(2sbh\) |
| Gate and up projections | their shared input | \(2sbh\) |
| SiLU and the product | the gate output, the up output, \(\operatorname{SiLU}(\text{gate})\) | \(6sbI\) |
| Down projection | its input, \(\operatorname{SiLU}(\text{gate}) \times \text{up}\) | \(2sbI\) |
The residual adds need nothing. The sum is the activation memory of one block:
- Read the symbols
- \(s, b\)
- sequence length and micro-batch size (sequences per forward pass on one GPU)
- \(12h\)
- the inputs of the two norms, of the two groups of projections, plus Q and the attention output: six tensors of width \(h\), 2 bytes each
- \(4g\)
- K and V, which are narrower than \(h\) with GQA (\(g = h_{kv} d_h\))
- \(8I\)
- four tensors of the FFN’s inner width \(I\), 2 bytes each
Figure 2 runs this walk as a small program. Each step adds one row of the table to a running total \(M\), with the real sizes of the chosen model put into the formula. Then it multiplies by the number of blocks, adds the logits (formula 38.5, below), and puts the result next to the 16 bytes per parameter of the model states.
- \(M \leftarrow 0\) // bytes one block saves
- \(M \mathrel{+}= 2sbh\) // RMSNorm 1 input
- \(M \mathrel{+}= 2sbh\) // Q, K, V share one input
- \(M \mathrel{+}= 4sbh + 4sbg\) // Flash: Q, O and K, V
- \(M \mathrel{+}= 0\) // output proj.: input is O
- \(M \mathrel{+}= 2sbh\) // RMSNorm 2 input
- \(M \mathrel{+}= 2sbh\) // gate and up share one input
- \(M \mathrel{+}= 6sbI\) // gate, up, \(\operatorname{SiLU}(\text{gate})\)
- \(M \mathrel{+}= 2sbI\) // input of down
- \(A \leftarrow L \cdot M\) // each block keeps its own
- \(A \mathrel{+}= 8sbV\) // logits (38.5)
- \(\text{total} \leftarrow 16N + A\) // (38.1) + (38.6)
How to read Figure 2
Press Play, or Step through the 12 lines. The right panel shows each line with the numbers put in. For Llama 3 8B with \(s = 4096\) and \(b = 1\), the block grows 33.6 MB at a time for the \(h\)-wide tensors, then jumps by 352 MB at line 8: the FFN is the biggest part. It ends at 0.688 GB, which is \(41h\) bytes per token. Line 10 copies it 32 times (22.0 GB), line 11 adds 4.2 GB of logits, and line 12 sets it next to 128.5 GB of model states. Now move \(s\) or \(b\): the top two bars scale with them, the bottom bar does not. At \(s = 32{,}768\) the activations of one sequence (210 GB) are larger than the model states.
For Llama 3 8B (\(h = 4096\), \(g = 8 \times 128 = 1024\), \(I = 14336\)), the bracket is \(49{,}152 + 4{,}096 + 114{,}688 = 167{,}936 = 41h\). So one block stores about 41 bytes per number of the residual stream, and two thirds of it is the FFN. We checked this by hand-counting with PyTorch: a hook on the tensors that autograd saves gives 41.2 bytes per \(sbh\) for this block with a fused RMSNorm (the extra 0.2 are the small FP32 numbers). [Certain for this implementation] A plain, unfused RMSNorm keeps one more copy per norm and gives 45; the code below shows it.
Korthikanti et al. did the same count for a GPT-style block (GELU FFN with \(4h\) inner size, dropout, 16-bit storage) and found \(sbh(34 + 5as/h)\) bytes per layer, where \(a\) is the number of heads. [Certain] The constant differs (34 vs 41) because SwiGLU is wider and has three projections, while dropout masks are gone. The method is the same: walk through the block and add up what is saved.
The attention scores: the term that grows like \(s^2\)
Without FlashAttention, “eager” attention builds the full score matrix: \(a\) heads, each \(s \times s\). The common Hugging Face pattern computes the softmax in FP32 and casts the result to BF16; both copies are saved:
We measured exactly \(6\) bytes per score with saved-tensor hooks for this pattern. [Certain for this implementation] For Llama 3 8B at \(s = 4096\): \(6 \times 32 \times 4096^2 = 3.22\) GB per layer, almost five times the 0.69 GB of formula (38.3). FlashAttention never stores the score matrix; it recomputes it block by block in the backward pass, so the \(s^2\) term disappears (session 39 explains the same idea as “selective recomputation”).
The logits: a big tensor at the very end
After the last block, the output layer produces one score per vocabulary entry for every token: \(s \cdot b \cdot V\) numbers. With a large vocabulary this is huge. A common pattern is logits = lm_head(x).float() followed by cross-entropy. The FP32 logits (4 bytes) stay alive because the model returns them, and the log-softmax output (4 bytes) is saved for the backward pass:
For Llama 3 8B (\(V = 128{,}256\)) and one sequence of 4,096 tokens, \(sbV = 525\) million, so 4.2 GB: as much as six transformer blocks. In the backward pass the FP32 gradient of the logits adds another \(4sbV\) for a moment. [Likely] For small models with big vocabularies this part can dominate: Cut Cross-Entropy (Wijmans et al., 2024) reports that the loss computation of Gemma 2 (2B) can take 24 GB, and that computing it piece by piece, without ever writing the full logits to GPU memory, brings it to 1 MB. [Certain]
The total, and how it grows
- Batch \(b\): linear. Twice the batch, twice the activations. Model states do not change.
- Sequence length \(s\): linear with FlashAttention, quadratic without it.
- Depth \(L\): linear. Each block keeps its own saved tensors until the backward pass reaches it.
- Width \(h\): linear (through \(h\), \(g\) and \(I\)), while the parameters grow like \(h^2\). So for big models the states dominate at short context, and activations dominate at long context.
One block: \(4096 \times 1 \times 167{,}936 = 0.688\) GB. Thirty-two blocks: 22.0 GB. Logits (38.5): 4.2 GB. Total activations about 26.3 GB, on top of 128.5 GB of model states.
Without FlashAttention, add \(32 \times 3.22 = 103.1\) GB of scores: 129 GB of activations for one sequence. With block checkpointing (keep only each block’s input, \(2sbh\) = 34 MB per block, and recompute one block at a time), the blocks cost \(32 \times 0.034 + 0.69 = 1.76\) GB instead of 22.0 GB.
Figure 3 puts all of this together. Choose a model, a training method and a GPU, and see which part of the bill breaks the budget.
- \(M \leftarrow b_w N\) // weights
- \(M \mathrel{+}= b_g N\) // gradients
- \(M \mathrel{+}= 4N\) if FP32 master // master copy
- \(M \mathrel{+}= b_{mv} N\) // Adam \(m\) and \(v\)
- \(M \mathrel{+}= 16\,P_{\text{LoRA}}\) // adapters (38.7)
- for \(l = 1..L\): \(M \mathrel{+}= M_{\text{layer}}\) or \(2sbh\) // (38.3) / (38.8)
- if eager: \(M \mathrel{+}= 6as^2b\) // (38.4)
- if checkpointing: \(M \mathrel{+}= M_{\text{layer}}\) // recompute
- \(M \mathrel{+}= 4sbh + 8sbV\) // final norm, logits (38.5)
- return \(M\); compare with the GPU
How to read Figure 3
The figure opens on the finished bill. Press Play to build it again: each line of the program adds one colored part, the right panel shows the line with your numbers, and the banner tells you at which line the total passes the GPU size. Start with the default: Llama 2 7B, AdamW, 24 GB. The model states alone (108 GB) are four and a half times the card. Switch through the methods: only LoRA and QLoRA bring the states below 24 GB. Now pick LoRA: states are about 14.1 GB, and with \(s = 2048\) the block activations (10.1 GB) and logits (0.5 GB) push the total just over 24 GB. Tick “Checkpoint every block”: it fits. Then raise the sequence length to 32,768 and untick FlashAttention to see the quadratic term take over. Finally pick Llama 3 70B with AdamW on 80 GB: 1.13 TB of states, so you need at least 15 such GPUs just to store them (with sharding, session 45).
Why full fine-tuning of a 7B model does not fit on one 24 GB GPU
Many people own or rent a 24 GB GPU (RTX 3090, RTX 4090, L4, A10). [Certain] A 7B model in BF16 is 13.5 GB, so it fits for inference. Can we fine-tune it? Go down the table above:
AdamW, mixed precision: \(16 \times 6.74 \times 10^9 = 107.8\) GB. Over four times the card, before one activation.
8-bit AdamW: 67.4 GB. Pure BF16 AdamW: 53.9 GB. Still far too much.
Even plain SGD with no optimizer state at all: weights 13.5 GB + gradients 13.5 GB = 27.0 GB. It does not fit even with zero activations. Full fine-tuning needs a gradient for every weight, and the gradients alone are as big as the model.
LoRA, rank 16 on all seven linear layers of every block: the adapters have \(39{,}976{,}960\) parameters (0.59% of the model, formula 38.7). Frozen BF16 weights 13.48 GB + adapters with full AdamW states \(16 \times 0.040 = 0.64\) GB = 14.1 GB. Activations at \(s = 2048\), \(b = 1\): 10.1 GB for the blocks plus 0.5 GB of logits, so 24.7 GB in total: just over. With block checkpointing, the blocks drop to 0.9 GB and the total is 15.5 GB: it fits, with room for \(b = 4\) (19.8 GB).
QLoRA: the frozen linear weights in 4-bit NF4 with double quantization take about 4.127 bits each, \(6.48 \times 10^9 \times 4.127 / 8 = 3.3\) GB, plus 0.5 GB for the BF16 embedding and output layers. [Likely for the exact layout]
The number of LoRA parameters comes from adding two thin matrices \(A \in \R^{d_{\text{in}} \times r}\) and \(B \in \R^{r \times d_{\text{out}}}\) to each adapted weight:
For Llama 2 7B with \(r = 16\): \(32 \times 16 \times \big(4 \times 8192 + 3 \times 15104\big) = 39{,}976{,}960\). The point is not the exact count. It is that LoRA removes \(b_g\) and \(b_o\) from almost all parameters, and leaves only \(b_w = 2\) (or about 0.5 with QLoRA). The activations stay, because the backward pass still has to flow through every frozen layer to reach the adapters. [Likely]
There is research that pushes full-parameter training into 24 GB: GaLore projects gradients to low rank and, with 8-bit states and per-layer updates, shows pre-training a 7B model on a 24 GB RTX 4090. [Certain] But the standard answer in an interview is: states are 16 bytes per parameter, so 7B needs about 112 GB before activations; use LoRA or QLoRA on one card, or shard the states across several GPUs.
Checkpointing changes formula (38.3) into “one input per block, plus one block being recomputed”:
The cost is about one extra forward pass, around 33% more compute (session 39).
One step over time: where is the peak?
Formulas (38.1) to (38.8) give sizes. But a GPU runs out of memory at one moment: the peak. Memory changes a lot during one step:
- Forward: each block adds its activations. Memory climbs steadily, then jumps at the logits.
- Start of backward: the gradient of the logits appears. Usually this is the highest point.
- Backward through the blocks: each block frees its activations and creates its weight gradients. If the activations of a block are larger than its gradients, memory falls; otherwise it rises.
- Optimizer step: reads all gradients, updates the states. Then
zero_grad(set_to_none=True)frees the gradients.
So, with \(A\) the block activations, \(G = b_g N\) the gradients and \(S = (b_w + b_o) N\) the persistent part, the peak is roughly
The first term inside the max is the start of the backward pass; the second is the end of it, when all gradients exist and the activations are gone. Long sequences make the first term win; short sequences with a big model make the second win.
There is a trap with Adam: PyTorch creates \(\mathbf{m}\) and \(\mathbf{v}\) lazily, inside the first call to optimizer.step(). So the first step runs with 8 fewer bytes per parameter. The Ultra-Scale Playbook notes that this is why training sometimes succeeds in the first step and then runs out of memory in the following ones. [Certain] Figure 4 shows a step of full fine-tuning of Llama 3.2 1B with \(s = 2048\) and \(b = 4\).
- \(M \leftarrow 6N\) (+ \(8N\) after step 1) // \(w\), master, \(m\), \(v\)
- for \(l = 1..L\): \(A \mathrel{+}= M_{\text{layer}}\) or \(2sbh\) // forward
- logits \(\mathrel{+}= 8sbV\) // .float(), log-softmax
- loss.backward(): \(\mathrel{+}= 4sbV\) // FP32 dlogits
- free logits; \(G \mathrel{+}= 2Vh\) // output layer
- for \(l = L..1\):
- if ckpt: \(A \mathrel{+}= M_{\text{layer}}\) // recompute
- \(A \mathrel{-}= M_{\text{layer}}\); \(G \mathrel{+}= 2P_{\text{layer}}\)
- if first step: \(M \mathrel{+}= 8N\) // create \(m\), \(v\)
- optimizer.step() // reads all of \(G\)
- zero_grad(set_to_none=True): \(G \leftarrow 0\)
How to read Figure 4
Press Play to run the step from the start, or drag the slider. The program on the left highlights the line that runs at this moment; the right panel shows the change it makes and the sum \(M = W + \text{Adam} + G + A + \text{logits}\) with live numbers. By default the peak is 42.1 GB, at the start of the backward pass. Notice how big the red part is: for this small model with a large vocabulary, the logits (8.4 GB stored, plus 4.2 GB for their gradient, 12.6 GB together) are larger than all 16 blocks (12.1 GB). Tick “Chunked loss” and the peak falls to 29.9 GB. Tick “First step”: the bars start 9.9 GB lower and the Adam states appear only at the end, so a first step that fits does not prove that step 2 fits. Tick “Keep gradients”: 2.5 GB more for the whole step. With checkpointing and chunked loss together, the peak moves to the end of the backward pass (20.4 GB), where all gradients exist.
Where memory really goes
You add up formulas (38.2) and (38.6), the total says 70 GB, the GPU has 80 GB, and the job still crashes. The difference is in the two parts of the bill we have not counted yet: temporary buffers and waste.
Overhead you cannot avoid
- CUDA context and kernels. Just initializing CUDA and loading the kernel libraries takes memory. The Ultra-Scale Playbook says CUDA kernels typically need 1–2 GB, and suggests checking with
nvidia-smiafter creating one small tensor. [Certain] - Library workspaces. cuBLAS and attention kernels use scratch space. Communication libraries (NCCL) allocate buffers when you train on several GPUs. [Likely]
Temporary tensors
A temporary tensor is created, used and freed within one operation or one phase. It never shows up in a formula for “what is stored”, but it does count at the peak. Common examples:
- Upcasts.
.float()on the logits creates a full FP32 copy (formula 38.5 already counts it). An eager RMSNorm that computes in FP32 creates FP32 copies of the hidden states; we measured about 51 bytes per \(sbh\) for a block with such norms, versus 41 with fused norms. [Certain for that implementation] - Flat buffers. Some systems copy all gradients into one contiguous buffer for communication or clipping. The ZeRO paper’s example: for a 1.5B-parameter model, a flattened FP32 buffer needs 6 GB. [Certain] PyTorch DDP groups gradients into smaller “buckets” for the same reason (session 44).
- Optimizer temporaries. The default “foreach” implementations of Adam and AdamW work on lists of tensors at once and create intermediate lists; the fused versions avoid most of them. [Likely]
- Recomputation. With checkpointing, one block’s activations are rebuilt during backward; this is the \(M_{\text{layer}}\) term in (38.8).
The caching allocator: allocated vs reserved
Asking the GPU driver for memory (cudaMalloc) is slow, so PyTorch uses a caching allocator: when a tensor is freed, PyTorch keeps its memory block and reuses it for later tensors. The PyTorch documentation says this unused cached memory still appears as used in nvidia-smi. [Certain] So there are two numbers:
torch.cuda.memory_allocated(): bytes occupied by live tensors. This is what our formulas predict.torch.cuda.memory_reserved(): bytes the allocator holds, used or not. This is what the GPU sees, and it is always at least as large.
The allocator also rounds sizes up (in multiples of 512 bytes, and large blocks are managed separately from small ones). [Certain] Use max_memory_allocated() after a step to read the real peak.
Fragmentation
Fragmentation means that free memory exists, but it is cut into pieces that are each too small for the tensor you need. A tensor needs one contiguous block of addresses. If short-lived tensors (temporaries, recomputed activations) are placed between long-lived ones (stored activations, gradients), freeing the short-lived ones leaves holes. The ZeRO paper reports running out of memory with over 30% of memory still free in some extreme cases, and explains it by exactly this mix of short and long lifetimes. [Certain] Step through Figure 5.
How to read Figure 5
Watch two numbers in the readout: total free memory and the largest contiguous free block. An allocation succeeds only if the largest block is big enough. In step 6 the total is 4 GB but the largest block is 2 GB, so the 3 GB request fails. In step 7 the same tensors are placed so that the short-lived ones sit together at the end; freeing them leaves one 4 GB block, and the request succeeds. This is the idea of ZeRO’s memory defragmentation: pre-allocate contiguous space for long-lived tensors so they are not mixed with short-lived ones.
Two practical fixes in PyTorch. The option PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True lets the allocator grow a memory segment instead of creating new fixed-size ones; the documentation says this leaves fewer “slivers” of unusable memory, especially when tensor sizes change from batch to batch. [Certain] The option max_split_size_mb stops the allocator from splitting large blocks, which the documentation says can reduce fragmentation. [Certain] Also: keep tensor shapes stable (pad or pack to fixed lengths), because changing sizes every step creates new block sizes again and again.
Estimating and debugging memory: the procedure
- input: parameters \(N\) (trainable and frozen), config \(L, h, g, I, V, a\), micro-batch \(b\), length \(s\), method, attention kernel
- \(S + G \leftarrow (b_w + b_g + b_o) N\) // formula (38.2); for LoRA: \(2N\) + 16 \(P_{\text{LoRA}}\), formula (38.7)
- \(A \leftarrow L \cdot s b (12h + 4g + 8I)\) // formula (38.3); with checkpointing use (38.8)
- if eager attention: \(A \leftarrow A + L \cdot 6 a s^2 b\) // formula (38.4)
- \(M_{\text{logits}} \leftarrow 8 s b V\) unless the loss is chunked // formula (38.5)
- \(M_{\text{peak}} \leftarrow S + \max(A + M_{\text{logits}} + 4sbV,\ G)\) // formula (38.9)
- if data parallel with ZeRO or FSDP: divide the sharded parts of \(S + G\) by the number of GPUs
- add 1–2 GB for CUDA and libraries, and keep about 10% free for fragmentation and temporaries // rule of thumb [Likely]
- return \(M_{\text{peak}}\); compare with the GPU size
- Measure first: run one or two steps with
reset_peak_memory_stats(), then readmax_memory_allocated()andmax_memory_reserved() - if it fails only from step 2 on: the Adam states (8 B per parameter) are the cause; plan for them
- if reserved is much larger than allocated: fragmentation; set
expandable_segments:True, keep shapes fixed - if the model states are too large: 8-bit optimizer, LoRA or QLoRA, or shard with ZeRO / FSDP // session 45
- if the activations are too large: FlashAttention, then a smaller micro-batch with gradient accumulation, then checkpointing // session 39
- if the peak is at the logits: chunked or fused cross-entropy; do not keep FP32 logits you do not need
- still too large: CPU offloading, or more GPUs with tensor / sequence parallelism
The code below does the paper estimate, then measures what autograd really saves for one Llama-style block on the CPU. It needs only PyTorch and runs in a second. The attention core is written as a small function that saves only what FlashAttention saves (Q, K, V, output, one log-sum-exp per row), so the count does not depend on which attention kernel your machine has.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) The bill on paper: formulas (38.1), (38.3), (38.5) and (38.8) def train_memory_gb(N, L, h, g, I, V, s, b, bytes_per_param=16, ckpt=False): states = bytes_per_param * N layer = s * b * (12 * h + 4 * g + 8 * I) # one block, FlashAttention, BF16 acts = L * 2 * s * b * h + layer if ckpt else L * layer logits = 8 * s * b * V # FP32 logits + saved log-softmax return states / 1e9, acts / 1e9, logits / 1e9 print(train_memory_gb(8.03e9, 32, 4096, 1024, 14336, 128256, 4096, 1)) # (128.48, 22.01..., 4.20...): Llama 3 8B, one sequence of 4,096 tokens # 2) Measure what autograd really saves for one Llama-style block h, nh, nkv, I, s, b = 256, 8, 2, 896, 64, 2 dh = h // nh class FlashLikeAttention(torch.autograd.Function): """Saves only q, k, v, the output and one log-sum-exp per row, like FlashAttention.""" @staticmethod def forward(ctx, q, k, v): rep = q.shape[1] // k.shape[1] # GQA: share each K/V head kk, vv = k.repeat_interleave(rep, 1), v.repeat_interleave(rep, 1) sc = (q.float() @ kk.float().transpose(-1, -2)) / dh ** 0.5 sc = sc.masked_fill(torch.ones(s, s, dtype=torch.bool).triu(1), float("-inf")) lse = torch.logsumexp(sc, -1) o = (torch.exp(sc - lse[..., None]) @ vv.float()).to(q.dtype) o = o.transpose(1, 2).contiguous() # (b, s, heads, dh) ctx.save_for_backward(q, k, v, o, lse) return o.view(b, s, h) class RMSNorm(nn.Module): # plain, unfused def __init__(self, d): super().__init__(); self.w = nn.Parameter(torch.ones(d)) def forward(self, x): return self.w * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + 1e-5) class Block(nn.Module): def __init__(self): super().__init__() lin = lambda i, o: nn.Linear(i, o, bias=False) self.n1, self.n2 = RMSNorm(h), RMSNorm(h) self.q, self.k, self.v, self.o = lin(h, h), lin(h, nkv * dh), lin(h, nkv * dh), lin(h, h) self.g, self.u, self.d = lin(h, I), lin(h, I), lin(I, h) def forward(self, x): y = self.n1(x) q = self.q(y).view(b, s, nh, dh).transpose(1, 2) k = self.k(y).view(b, s, nkv, dh).transpose(1, 2) v = self.v(y).view(b, s, nkv, dh).transpose(1, 2) x = x + self.o(FlashLikeAttention.apply(q, k, v)) y = self.n2(x) return x + self.d(F.silu(self.g(y)) * self.u(y)) block = Block().to(torch.bfloat16) params = {p.untyped_storage().data_ptr() for p in block.parameters()} saved = {} def pack(t): # called for every tensor autograd keeps st = t.untyped_storage() if st.data_ptr() not in params: # skip weights: they are not activations saved[st.data_ptr()] = st.nbytes() # count each storage once return t x = torch.randn(b, s, h, dtype=torch.bfloat16, requires_grad=True) with torch.autograd.graph.saved_tensors_hooks(pack, lambda t: t): out = block(x) print(f"saved: {sum(saved.values()) / (s * b * h):.1f} bytes per (s*b*h)") # 45.1 print(f"formula (38.3): {(12 * h + 4 * nkv * dh + 8 * I) / h:.1f}") # 41.0 # The gap of about 4 is the unfused RMSNorm: it keeps x * rsqrt(...) as an extra copy, # once per norm. A fused RMSNorm kernel saves only x and one FP32 number per row. # 3) On a GPU, measure the real peak of one step if torch.cuda.is_available(): torch.cuda.reset_peak_memory_stats() # ... one forward, backward and optimizer.step() here ... print(torch.cuda.max_memory_allocated() / 1e9, "GB allocated at peak") print(torch.cuda.max_memory_reserved() / 1e9, "GB reserved by the caching allocator")
Two more tools are worth knowing. torch.cuda.memory_summary() prints a table of allocated, reserved and freed memory. torch.cuda.memory._record_memory_history() with torch.cuda.memory._dump_snapshot() records every allocation with its stack trace; the PyTorch memory visualizer then draws a timeline like Figure 4 for your real run. [Likely] You will use them in session 41.
Where this is used
| Setting | Which part of the bill dominates | Typical answer |
|---|---|---|
| Pretraining a large model on a cluster | Model states (16–20 B per parameter): 1.13 TB for 70B | Shard states with ZeRO-3 / FSDP; tensor and pipeline parallelism (session 46, session 48) |
| Long-context training (32K+ tokens) | Activations, linear in \(s\) with FlashAttention | Checkpointing, sequence or context parallelism (session 49) |
| Fine-tuning a 7B–8B model on one 24 GB GPU | Model states (108–129 GB for full fine-tuning) | LoRA or QLoRA, checkpointing, micro-batch 1–4 with accumulation (session 62) |
| Small models with big vocabularies (1B–3B, 128K vocab) | Logits: 8.4 GB for Llama 3.2 1B at \(b = 4\), \(s = 2048\) | Chunked or fused cross-entropy |
| Training a Video-LLM (e.g. 64 frames × 196 tokens = 12,544 visual tokens) | Activations: about 67 GB for an 8B language model per sequence, without checkpointing | Token compression or pooling, checkpointing, context parallelism (session 212, session 206) |
| Fine-tuning video encoders for action recognition or video retrieval | Activations of many frames × patches; batch size for contrastive loss | Frame sampling, checkpointing, gradient caching for large contrastive batches [Likely] (session 208) |
| Capacity planning for a training budget | All parts | Algorithm 1 before you rent GPUs; measure one step before a long run |
Common mistakes
- Counting only the weights. “7B in BF16 is 14 GB, my GPU has 24 GB, so I can fine-tune it.” The gradients alone add another 14 GB, and Adam adds 8 bytes more per parameter.
- Forgetting the first-step trap. Adam creates its states in the first
step(). A run that survives step 1 can crash at step 2. Always run at least two steps before you trust a configuration. - Ignoring the logits. With a 128K vocabulary, the FP32 logits and their gradient can be larger than several transformer blocks. Do not call
.float()on the full logits if a fused loss is available. - Reading
nvidia-smias “memory used by tensors”. It shows what the caching allocator reserved. Usememory_allocatedandmax_memory_allocatedto compare with your estimate. - Using eager attention for long sequences. The \(6as^2b\) term (38.4) is five times the rest of a block at 4,096 tokens and grows four times with each doubling of \(s\). Check which attention kernel you actually run.
- Mixing GB and GiB. A 24 GB card has 25.8 × \(10^9\) bytes; tools print different units. Pick one and keep a margin.
- Variable shapes every step. Different sequence lengths create different block sizes and grow fragmentation. Pack or bucket sequences; use
expandable_segments. - Thinking gradient accumulation reduces model-state memory. It reduces only activations (smaller micro-batch). The 16 bytes per parameter stay.
Interview questions
Where does the “16 bytes per parameter” for Adam come from?basic
How much memory do you need to fully fine-tune a 7B model with AdamW? Does it fit on a 24 GB GPU?basic
How does activation memory scale with batch size, sequence length, layers and hidden size?core
Derive the activation memory of one transformer block.core
Training succeeds in step 1 and runs out of memory in step 2. Why?core
optimizer.step(). Step 1’s forward and backward ran without them; step 2 runs with 8 more bytes per parameter permanently allocated. Fragmentation from the first step can make it worse. Fix: plan for the full 16 bytes, or test two steps when tuning batch size.Where is the memory peak in a training step?core
Your estimate says 70 GB, the GPU has 80 GB, and you still get OOM. What do you check?advanced
max_memory_allocated vs max_memory_reserved. The gap is caching and fragmentation; try expandable_segments:True and fixed shapes. Then look for things the formula missed: 1–2 GB of CUDA context and kernels, FP32 upcasts (logits, norms), flat gradient buffers or DDP buckets, optimizer temporaries (foreach), recomputation buffers, eager attention instead of FlashAttention, Adam states from step 2. A memory snapshot shows which allocation is at the peak.What is memory fragmentation, and why does it happen in training?advanced
max_split_size_mb.Which memory does gradient accumulation reduce, and which does checkpointing reduce?core
Why can the output layer be a memory problem, and how do fused losses help?advanced
Why do LoRA and QLoRA still need activation memory?core
Exercises
- How much memory do the model states of Llama 3.2 3B (3.21B parameters) take with mixed-precision AdamW? Does it fit on an 80 GB GPU, before activations?
- Llama 3 70B has 70.55B parameters. How much do the model states take with AdamW, and how much per GPU if ZeRO-3 shards them evenly across 8 GPUs?
- For Llama 3 8B, how does activation memory change when the sequence length goes from 2,048 to 4,096 tokens (a) with FlashAttention, (b) for the score term (38.4) without it?
- Compute the bracket of formula (38.3) for Llama 3.2 3B (\(h = 3072\), \(g = 1024\), \(I = 8192\)) in units of \(h\).
- Llama 3.2 1B with \(s = 2048\), \(b = 4\): compute the logit memory (38.5). Compare it with the block activations (12.1 GB).
- In Figure 5, which single change in the placement of the tensors would make the 3 GB request succeed?
Answerscheck
Key takeaways
- Training memory = model states + activations + temporary buffers + overhead and fragmentation.
- Model states are \((b_w + b_g + b_o)N\) (38.2): 16 bytes per parameter for mixed-precision AdamW (38.1), 20 with FP32 gradient accumulation. A 7B model needs about 112 GB before any activation.
- Activations grow linearly with batch, sequence length and depth: about \(sb(12h + 4g + 8I)\) bytes per Llama-style block (38.3), plus an \(s^2\) term without FlashAttention (38.4) and a big logits term with large vocabularies (38.5).
- Full fine-tuning of a 7B model cannot fit on 24 GB: even weights plus gradients are 27 GB. LoRA or QLoRA plus checkpointing does fit.
- The OOM happens at the peak (38.9), usually at the start of backward; Adam states appear only after step 1.
- Measure with
max_memory_allocatedandmax_memory_reserved; the gap is caching and fragmentation.
Key formulas
References for this session
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- Dettmers et al. (2021). 8-bit Optimizers via Block-wise Quantization
- Chen et al. (2016). Training Deep Nets with Sublinear Memory Cost
- Zhao et al. (2024). GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
- Wijmans, Huval, Hertzberg, Koltun, Krähenbühl (2024). Cut Your Losses in Large-Vocabulary Language Models
- PyTorch (docs). CUDA semantics: memory management and the caching allocator