LLM & VLM courseBlog
38Track 3: Pretraining and scale · session 8 of 22

Memory math for training

Before a training run starts, you can predict how much GPU memory it needs and where every gigabyte goes. This session builds the full bill of one training step: 16 bytes per parameter for mixed-precision Adam, activations that grow with batch, sequence length and depth, the large logits tensor, and the overheads that formulas miss, such as temporary buffers and fragmentation. With it you can explain why full fine-tuning of a 7B model does not fit on one 24 GB GPU, and what does.

Time: 2.5 to 3.5 hours (reading 1.5 to 2 hours, exercises 1 to 1.5 hours)Level: BeginnerBefore this: Sessions 4, 25 and 37 (backprop and AdamW, counting parameters and FLOPs, numbers on GPUs and mixed precision)

What you will learn

  • The four parts of training memory: model states, activations, temporary buffers, overhead and fragmentation
  • Why mixed-precision AdamW costs 16 bytes per parameter, and how other optimizers and precisions change it (8, 10, 12, 20 bytes)
  • How to count the activations of one Llama-style block, about sb(12h + 4g + 8I) bytes, and how they grow with batch, sequence length and depth
  • The two hidden giants: the s² attention scores without FlashAttention, and the logits with a large vocabulary
  • Why full fine-tuning of a 7B model does not fit on one 24 GB GPU, and why LoRA or QLoRA with checkpointing does
  • Where the memory peak is in one step, and why a run can survive step 1 and crash in step 2
  • The PyTorch caching allocator (allocated vs reserved), fragmentation, and how to measure and fix an OOM

Why this matters

Most training runs do not fail because the math is wrong. They fail with one line: CUDA out of memory. Before you start a run, you should be able to predict, with a pen and a few numbers, how many gigabytes it needs and where they go.

In session 37 we saw that one number can take 1, 2 or 4 bytes, and that mixed precision keeps some copies in BF16 and some in FP32. In session 25 we counted parameters and FLOPs and gave a first, short memory estimate. This session is the full memory bill of one training step. We answer four questions: how many bytes each parameter costs (16 for mixed-precision Adam), how activation memory grows with batch, sequence length and depth, why full fine-tuning of a 7B model cannot fit on one 24 GB GPU, and where the “missing” gigabytes go when your estimate says the run fits but it still crashes.

This bill decides real engineering choices. It is the reason ZeRO and FSDP shard optimizer states across GPUs (session 45), the reason gradient checkpointing exists (session 39), and the reason most people fine-tune 7B models with LoRA or QLoRA on a single card (session 57, session 58). The ZeRO paper starts with exactly this analysis and calls it “Where did all the memory go?”. [Certain]

The four parts of the bill

During training, GPU memory holds four kinds of things:

  1. Model states: the weights, their gradients, and the optimizer states (for Adam: two running averages per weight). Their size depends only on the number of parameters \(N\), not on the data.
  2. Activations: the intermediate results of the forward pass that the backward pass needs. Their size depends on the batch size, the sequence length and the depth, not on \(N\) directly.
  3. Temporary buffers: tensors that live for a short time, for example a large FP32 copy of the logits, a buffer that groups gradients for communication, or scratch space of a kernel.
  4. Overhead and waste: the CUDA context and kernels, memory that the allocator has reserved but not used, and fragmentation (free memory cut into pieces that are too small to use).

The ZeRO paper uses the same split: model states on one side, and “residual states” (activations, temporary buffers, fragmented memory) on the other. [Certain]

Model states: 16 bytes per parameter

Recall the standard mixed-precision recipe (session 37, session 4). The forward and backward passes run in BF16, so we keep a BF16 copy of the weights (2 bytes each) and get BF16 gradients (2 bytes each). The optimizer works in FP32: it keeps an FP32 master copy of the weights, so small updates are not lost to rounding, and Adam’s two moments \(\mathbf{m}\) and \(\mathbf{v}\), also in FP32. Add it up:

\[ M_{\text{states}} = \big(\underbrace{2}_{\text{BF16 weights}} + \underbrace{2}_{\text{BF16 grads}} + \underbrace{4}_{\text{FP32 master}} + \underbrace{4}_{\text{Adam } m} + \underbrace{4}_{\text{Adam } v}\big)\, N = 16N \ \text{bytes} \tag{38.1} \]

The ZeRO paper writes this as \(2\Psi + 2\Psi + K\Psi\) with \(K = 12\) for mixed-precision Adam, where \(\Psi\) is the number of parameters. [Certain] Its example: a 1.5B-parameter GPT-2 needs only 3 GB for its 16-bit weights, but at least 24 GB for its model states. [Certain]

The general form is useful, because every optimizer and every precision choice just changes the three numbers:

\[ M_{\text{states}} = (b_w + b_g + b_o)\, N \tag{38.2} \]
Read the symbols
\(N\)
number of trainable parameters (for LoRA, frozen weights count only in \(b_w\))
\(b_w\)
bytes per parameter for the weights used in the forward pass (2 for BF16, 4 for FP32, about 0.5 for 4-bit)
\(b_g\)
bytes per parameter for the gradients (2 for BF16, 4 for FP32, 6 if you keep both)
\(b_o\)
bytes per parameter for everything the optimizer keeps: master weights and moments
weightsgradientsFP32 master weightsoptimizer moments \(m, v\)
Figure 1. Bytes per parameter for seven common setups. Each small block is one byte. The total on the right times \(N\) gives the model-state memory (38.2). The first row is the classic 16 bytes of formula (38.1).

Two rows of Figure 1 surprise many people. First, plain PyTorch torch.autocast keeps the weights and gradients in FP32 and makes BF16 copies of the weights on the fly during the forward pass: 4 + 4 + 4 + 4 is also 16 bytes, with a different layout. [Likely] Second, many large-scale frameworks keep an extra FP32 gradient buffer to add up gradients precisely, which gives 20 bytes per parameter; the Ultra-Scale Playbook lists 112 GB for a 7B model without this buffer and 140 GB with it. [Certain]

Setup\(b_w + b_g + b_o\)Llama 2 7B (6.74B)Llama 3 8B (8.03B)
Mixed-precision AdamW (38.1)\(2 + 2 + 12 = 16\)107.8 GB128.5 GB
Same, plus FP32 gradient accumulation\(2 + 6 + 12 = 20\)134.8 GB160.6 GB
8-bit AdamW (moments in 8 bits)\(2 + 2 + 6 = 10\)67.4 GB80.3 GB
SGD with momentum, mixed precision\(2 + 2 + 8 = 12\)80.9 GB96.4 GB
Pure BF16 AdamW (no master copy)\(2 + 2 + 4 = 8\)53.9 GB64.2 GB
Plain SGD in BF16 (no state)\(2 + 2 + 0 = 4\)27.0 GB32.1 GB
Inference only (BF16 weights)213.5 GB16.1 GB

Here GB means \(10^9\) bytes. A “24 GB” or “80 GB” GPU actually has 24 or 80 GiB (\(2^{30}\) bytes each), so 24 GiB = 25.8 GB, but part of it goes to overhead (see below). In this session we compare GB against the GPU’s label, which is a safe, slightly pessimistic rule.

Activation memory

Backpropagation needs, for every operation, some of its inputs or outputs from the forward pass (session 4). For \(Y = XW\) the weight gradient is \(X^\top G\), so \(X\) must be kept. For \(\text{SiLU}(z)\) the local derivative needs \(z\). These saved tensors are the activations. Each one has the shape of the data flowing through the network: (batch \(b\)) × (sequence length \(s\)) × (some width). That is why activation memory is proportional to \(b \cdot s\), and to the number of layers \(L\).

Counting one Llama-style block

Take a modern decoder block: RMSNorm, attention with grouped-query attention (GQA) and FlashAttention, a residual add, RMSNorm, a SwiGLU feed-forward network, a residual add (session 23). Everything is stored in BF16 (2 bytes). Let \(h\) be the hidden size, \(g = h_{kv} d_h\) the width of K and V, and \(I\) the FFN’s inner size. Walk through the block and write down what each step must save:

OperationWhat it saves for backwardBytes
RMSNorm 1its input\(2sbh\)
Q, K, V projectionstheir shared input (the norm output)\(2sbh\)
FlashAttentionQ, K, V and its output (plus one small FP32 number per row)\(2sbh + 4sbg + 2sbh\)
Output projectionits input = the attention output (already counted)0
RMSNorm 2its input\(2sbh\)
Gate and up projectionstheir shared input\(2sbh\)
SiLU and the productthe gate output, the up output, \(\operatorname{SiLU}(\text{gate})\)\(6sbI\)
Down projectionits input, \(\operatorname{SiLU}(\text{gate}) \times \text{up}\)\(2sbI\)

The residual adds need nothing. The sum is the activation memory of one block:

\[ M_{\text{layer}} = s\, b\, \big(12h + 4g + 8I\big) \ \text{bytes} \tag{38.3} \]
Read the symbols
\(s, b\)
sequence length and micro-batch size (sequences per forward pass on one GPU)
\(12h\)
the inputs of the two norms, of the two groups of projections, plus Q and the attention output: six tensors of width \(h\), 2 bytes each
\(4g\)
K and V, which are narrower than \(h\) with GQA (\(g = h_{kv} d_h\))
\(8I\)
four tensors of the FFN’s inner width \(I\), 2 bytes each

Figure 2 runs this walk as a small program. Each step adds one row of the table to a running total \(M\), with the real sizes of the chosen model put into the formula. Then it multiplies by the number of blocks, adds the logits (formula 38.5, below), and puts the result next to the 16 bytes per parameter of the model states.

tensors of width \(h\) (the \(12h\) term)K and V, width \(g\) (the \(4g\) term)FFN tensors, width \(I\) (the \(8I\) term)logitsmodel states
  1. \(M \leftarrow 0\) // bytes one block saves
  2. \(M \mathrel{+}= 2sbh\) // RMSNorm 1 input
  3. \(M \mathrel{+}= 2sbh\) // Q, K, V share one input
  4. \(M \mathrel{+}= 4sbh + 4sbg\) // Flash: Q, O and K, V
  5. \(M \mathrel{+}= 0\) // output proj.: input is O
  6. \(M \mathrel{+}= 2sbh\) // RMSNorm 2 input
  7. \(M \mathrel{+}= 2sbh\) // gate and up share one input
  8. \(M \mathrel{+}= 6sbI\) // gate, up, \(\operatorname{SiLU}(\text{gate})\)
  9. \(M \mathrel{+}= 2sbI\) // input of down
  10. \(A \leftarrow L \cdot M\) // each block keeps its own
  11. \(A \mathrel{+}= 8sbV\) // logits (38.5)
  12. \(\text{total} \leftarrow 16N + A\) // (38.1) + (38.6)
Figure 2. Formula (38.3) computed line by line. The top bar is one block; its colors are the three terms \(12h\), \(4g\) and \(8I\). The middle bar stacks \(L\) copies of it plus the logits; the bottom bar is the model states of mixed-precision AdamW. The middle and bottom bars share one scale. Configurations are the real ones from each model’s config.json; FlashAttention, BF16.
How to read Figure 2

Press Play, or Step through the 12 lines. The right panel shows each line with the numbers put in. For Llama 3 8B with \(s = 4096\) and \(b = 1\), the block grows 33.6 MB at a time for the \(h\)-wide tensors, then jumps by 352 MB at line 8: the FFN is the biggest part. It ends at 0.688 GB, which is \(41h\) bytes per token. Line 10 copies it 32 times (22.0 GB), line 11 adds 4.2 GB of logits, and line 12 sets it next to 128.5 GB of model states. Now move \(s\) or \(b\): the top two bars scale with them, the bottom bar does not. At \(s = 32{,}768\) the activations of one sequence (210 GB) are larger than the model states.

For Llama 3 8B (\(h = 4096\), \(g = 8 \times 128 = 1024\), \(I = 14336\)), the bracket is \(49{,}152 + 4{,}096 + 114{,}688 = 167{,}936 = 41h\). So one block stores about 41 bytes per number of the residual stream, and two thirds of it is the FFN. We checked this by hand-counting with PyTorch: a hook on the tensors that autograd saves gives 41.2 bytes per \(sbh\) for this block with a fused RMSNorm (the extra 0.2 are the small FP32 numbers). [Certain for this implementation] A plain, unfused RMSNorm keeps one more copy per norm and gives 45; the code below shows it.

Korthikanti et al. did the same count for a GPT-style block (GELU FFN with \(4h\) inner size, dropout, 16-bit storage) and found \(sbh(34 + 5as/h)\) bytes per layer, where \(a\) is the number of heads. [Certain] The constant differs (34 vs 41) because SwiGLU is wider and has three projections, while dropout masks are gone. The method is the same: walk through the block and add up what is saved.

The attention scores: the term that grows like \(s^2\)

Without FlashAttention, “eager” attention builds the full score matrix: \(a\) heads, each \(s \times s\). The common Hugging Face pattern computes the softmax in FP32 and casts the result to BF16; both copies are saved:

\[ M_{\text{scores}} = (4 + 2)\, a\, s^2\, b = 6\, a\, s^2 b \ \text{bytes per layer} \tag{38.4} \]

We measured exactly \(6\) bytes per score with saved-tensor hooks for this pattern. [Certain for this implementation] For Llama 3 8B at \(s = 4096\): \(6 \times 32 \times 4096^2 = 3.22\) GB per layer, almost five times the 0.69 GB of formula (38.3). FlashAttention never stores the score matrix; it recomputes it block by block in the backward pass, so the \(s^2\) term disappears (session 39 explains the same idea as “selective recomputation”).

The logits: a big tensor at the very end

After the last block, the output layer produces one score per vocabulary entry for every token: \(s \cdot b \cdot V\) numbers. With a large vocabulary this is huge. A common pattern is logits = lm_head(x).float() followed by cross-entropy. The FP32 logits (4 bytes) stay alive because the model returns them, and the log-softmax output (4 bytes) is saved for the backward pass:

\[ M_{\text{logits}} \approx 8\, s\, b\, V \ \text{bytes} \tag{38.5} \]

For Llama 3 8B (\(V = 128{,}256\)) and one sequence of 4,096 tokens, \(sbV = 525\) million, so 4.2 GB: as much as six transformer blocks. In the backward pass the FP32 gradient of the logits adds another \(4sbV\) for a moment. [Likely] For small models with big vocabularies this part can dominate: Cut Cross-Entropy (Wijmans et al., 2024) reports that the loss computation of Gemma 2 (2B) can take 24 GB, and that computing it piece by piece, without ever writing the full logits to GPU memory, brings it to 1 MB. [Certain]

The total, and how it grows

\[ M_{\text{act}} \approx L\, M_{\text{layer}} + [\text{no Flash}]\; L \cdot 6 a s^2 b + 8\, s b V \tag{38.6} \]
  • Batch \(b\): linear. Twice the batch, twice the activations. Model states do not change.
  • Sequence length \(s\): linear with FlashAttention, quadratic without it.
  • Depth \(L\): linear. Each block keeps its own saved tensors until the backward pass reaches it.
  • Width \(h\): linear (through \(h\), \(g\) and \(I\)), while the parameters grow like \(h^2\). So for big models the states dominate at short context, and activations dominate at long context.
Worked example: Llama 3 8B, one sequence of 4,096 tokens

One block: \(4096 \times 1 \times 167{,}936 = 0.688\) GB. Thirty-two blocks: 22.0 GB. Logits (38.5): 4.2 GB. Total activations about 26.3 GB, on top of 128.5 GB of model states.

Without FlashAttention, add \(32 \times 3.22 = 103.1\) GB of scores: 129 GB of activations for one sequence. With block checkpointing (keep only each block’s input, \(2sbh\) = 34 MB per block, and recompute one block at a time), the blocks cost \(32 \times 0.034 + 0.69 = 1.76\) GB instead of 22.0 GB.

Figure 3 puts all of this together. Choose a model, a training method and a GPU, and see which part of the bill breaks the budget.

weightsgradientsFP32 masteroptimizer momentsLoRA adapters (all 16 B)block activationslogits
  1. \(M \leftarrow b_w N\) // weights
  2. \(M \mathrel{+}= b_g N\) // gradients
  3. \(M \mathrel{+}= 4N\) if FP32 master // master copy
  4. \(M \mathrel{+}= b_{mv} N\) // Adam \(m\) and \(v\)
  5. \(M \mathrel{+}= 16\,P_{\text{LoRA}}\) // adapters (38.7)
  6. for \(l = 1..L\): \(M \mathrel{+}= M_{\text{layer}}\) or \(2sbh\) // (38.3) / (38.8)
  7. if eager: \(M \mathrel{+}= 6as^2b\) // (38.4)
  8. if checkpointing: \(M \mathrel{+}= M_{\text{layer}}\) // recompute
  9. \(M \mathrel{+}= 4sbh + 8sbV\) // final norm, logits (38.5)
  10. return \(M\); compare with the GPU
Figure 3. The memory bill of one training step on one GPU, with no sharding, built by Algorithm 1 one line at a time. Model states use formula (38.2), block activations (38.3) or the checkpointed version (38.8), scores (38.4) when FlashAttention is off, and logits (38.5) unless the loss is chunked. The dashed line is the GPU size. Configurations are the real ones from each model’s config.json.
How to read Figure 3

The figure opens on the finished bill. Press Play to build it again: each line of the program adds one colored part, the right panel shows the line with your numbers, and the banner tells you at which line the total passes the GPU size. Start with the default: Llama 2 7B, AdamW, 24 GB. The model states alone (108 GB) are four and a half times the card. Switch through the methods: only LoRA and QLoRA bring the states below 24 GB. Now pick LoRA: states are about 14.1 GB, and with \(s = 2048\) the block activations (10.1 GB) and logits (0.5 GB) push the total just over 24 GB. Tick “Checkpoint every block”: it fits. Then raise the sequence length to 32,768 and untick FlashAttention to see the quadratic term take over. Finally pick Llama 3 70B with AdamW on 80 GB: 1.13 TB of states, so you need at least 15 such GPUs just to store them (with sharding, session 45).

Why full fine-tuning of a 7B model does not fit on one 24 GB GPU

Many people own or rent a 24 GB GPU (RTX 3090, RTX 4090, L4, A10). [Certain] A 7B model in BF16 is 13.5 GB, so it fits for inference. Can we fine-tune it? Go down the table above:

Worked example: Llama 2 7B (6.74B parameters) on 24 GB

AdamW, mixed precision: \(16 \times 6.74 \times 10^9 = 107.8\) GB. Over four times the card, before one activation.

8-bit AdamW: 67.4 GB. Pure BF16 AdamW: 53.9 GB. Still far too much.

Even plain SGD with no optimizer state at all: weights 13.5 GB + gradients 13.5 GB = 27.0 GB. It does not fit even with zero activations. Full fine-tuning needs a gradient for every weight, and the gradients alone are as big as the model.

LoRA, rank 16 on all seven linear layers of every block: the adapters have \(39{,}976{,}960\) parameters (0.59% of the model, formula 38.7). Frozen BF16 weights 13.48 GB + adapters with full AdamW states \(16 \times 0.040 = 0.64\) GB = 14.1 GB. Activations at \(s = 2048\), \(b = 1\): 10.1 GB for the blocks plus 0.5 GB of logits, so 24.7 GB in total: just over. With block checkpointing, the blocks drop to 0.9 GB and the total is 15.5 GB: it fits, with room for \(b = 4\) (19.8 GB).

QLoRA: the frozen linear weights in 4-bit NF4 with double quantization take about 4.127 bits each, \(6.48 \times 10^9 \times 4.127 / 8 = 3.3\) GB, plus 0.5 GB for the BF16 embedding and output layers. [Likely for the exact layout]

The number of LoRA parameters comes from adding two thin matrices \(A \in \R^{d_{\text{in}} \times r}\) and \(B \in \R^{r \times d_{\text{out}}}\) to each adapted weight:

\[ P_{\text{LoRA}} = L \cdot r \sum_{\text{adapted } W} \big(d_{\text{in}} + d_{\text{out}}\big) \tag{38.7} \]

For Llama 2 7B with \(r = 16\): \(32 \times 16 \times \big(4 \times 8192 + 3 \times 15104\big) = 39{,}976{,}960\). The point is not the exact count. It is that LoRA removes \(b_g\) and \(b_o\) from almost all parameters, and leaves only \(b_w = 2\) (or about 0.5 with QLoRA). The activations stay, because the backward pass still has to flow through every frozen layer to reach the adapters. [Likely]

There is research that pushes full-parameter training into 24 GB: GaLore projects gradients to low rank and, with 8-bit states and per-layer updates, shows pre-training a 7B model on a 24 GB RTX 4090. [Certain] But the standard answer in an interview is: states are 16 bytes per parameter, so 7B needs about 112 GB before activations; use LoRA or QLoRA on one card, or shard the states across several GPUs.

Checkpointing changes formula (38.3) into “one input per block, plus one block being recomputed”:

\[ M_{\text{act}}^{\text{ckpt}} \approx L \cdot 2\, s b h + M_{\text{layer}} + 8\, s b V \tag{38.8} \]

The cost is about one extra forward pass, around 33% more compute (session 39).

One step over time: where is the peak?

Formulas (38.1) to (38.8) give sizes. But a GPU runs out of memory at one moment: the peak. Memory changes a lot during one step:

  1. Forward: each block adds its activations. Memory climbs steadily, then jumps at the logits.
  2. Start of backward: the gradient of the logits appears. Usually this is the highest point.
  3. Backward through the blocks: each block frees its activations and creates its weight gradients. If the activations of a block are larger than its gradients, memory falls; otherwise it rises.
  4. Optimizer step: reads all gradients, updates the states. Then zero_grad(set_to_none=True) frees the gradients.

So, with \(A\) the block activations, \(G = b_g N\) the gradients and \(S = (b_w + b_o) N\) the persistent part, the peak is roughly

\[ M_{\text{peak}} \approx S + \max\big(A + M_{\text{logits}} + 4 s b V,\ \ G\big) \tag{38.9} \]

The first term inside the max is the start of the backward pass; the second is the end of it, when all gradients exist and the activations are gone. Long sequences make the first term win; short sequences with a big model make the second win.

There is a trap with Adam: PyTorch creates \(\mathbf{m}\) and \(\mathbf{v}\) lazily, inside the first call to optimizer.step(). So the first step runs with 8 fewer bytes per parameter. The Ultra-Scale Playbook notes that this is why training sometimes succeeds in the first step and then runs out of memory in the following ones. [Certain] Figure 4 shows a step of full fine-tuning of Llama 3.2 1B with \(s = 2048\) and \(b = 4\).

BF16 weights + FP32 masterAdam \(m, v\)gradientsblock activationslogits and their gradient
  1. \(M \leftarrow 6N\) (+ \(8N\) after step 1) // \(w\), master, \(m\), \(v\)
  2. for \(l = 1..L\): \(A \mathrel{+}= M_{\text{layer}}\) or \(2sbh\) // forward
  3. logits \(\mathrel{+}= 8sbV\) // .float(), log-softmax
  4. loss.backward(): \(\mathrel{+}= 4sbV\) // FP32 dlogits
  5. free logits; \(G \mathrel{+}= 2Vh\) // output layer
  6. for \(l = L..1\):
  7. if ckpt: \(A \mathrel{+}= M_{\text{layer}}\) // recompute
  8. \(A \mathrel{-}= M_{\text{layer}}\); \(G \mathrel{+}= 2P_{\text{layer}}\)
  9. if first step: \(M \mathrel{+}= 8N\) // create \(m\), \(v\)
  10. optimizer.step() // reads all of \(G\)
  11. zero_grad(set_to_none=True): \(G \leftarrow 0\)
Figure 4. Memory during one training step of Llama 3.2 1B (1.24B parameters, 16 blocks, vocabulary 128,256), AdamW in mixed precision, \(s = 2048\), \(b = 4\), FlashAttention. Each bar is one moment: forward through 16 blocks, the logits, the backward pass, the optimizer step. Play runs the step: bars appear as the program reaches them, and the dashed line is the highest memory seen so far. Sizes come from formulas (38.1), (38.3), (38.5) and (38.8); real traces add the overheads of the next section.
How to read Figure 4

Press Play to run the step from the start, or drag the slider. The program on the left highlights the line that runs at this moment; the right panel shows the change it makes and the sum \(M = W + \text{Adam} + G + A + \text{logits}\) with live numbers. By default the peak is 42.1 GB, at the start of the backward pass. Notice how big the red part is: for this small model with a large vocabulary, the logits (8.4 GB stored, plus 4.2 GB for their gradient, 12.6 GB together) are larger than all 16 blocks (12.1 GB). Tick “Chunked loss” and the peak falls to 29.9 GB. Tick “First step”: the bars start 9.9 GB lower and the Adam states appear only at the end, so a first step that fits does not prove that step 2 fits. Tick “Keep gradients”: 2.5 GB more for the whole step. With checkpointing and chunked loss together, the peak moves to the end of the backward pass (20.4 GB), where all gradients exist.

Where memory really goes

You add up formulas (38.2) and (38.6), the total says 70 GB, the GPU has 80 GB, and the job still crashes. The difference is in the two parts of the bill we have not counted yet: temporary buffers and waste.

Overhead you cannot avoid

  • CUDA context and kernels. Just initializing CUDA and loading the kernel libraries takes memory. The Ultra-Scale Playbook says CUDA kernels typically need 1–2 GB, and suggests checking with nvidia-smi after creating one small tensor. [Certain]
  • Library workspaces. cuBLAS and attention kernels use scratch space. Communication libraries (NCCL) allocate buffers when you train on several GPUs. [Likely]

Temporary tensors

A temporary tensor is created, used and freed within one operation or one phase. It never shows up in a formula for “what is stored”, but it does count at the peak. Common examples:

  • Upcasts. .float() on the logits creates a full FP32 copy (formula 38.5 already counts it). An eager RMSNorm that computes in FP32 creates FP32 copies of the hidden states; we measured about 51 bytes per \(sbh\) for a block with such norms, versus 41 with fused norms. [Certain for that implementation]
  • Flat buffers. Some systems copy all gradients into one contiguous buffer for communication or clipping. The ZeRO paper’s example: for a 1.5B-parameter model, a flattened FP32 buffer needs 6 GB. [Certain] PyTorch DDP groups gradients into smaller “buckets” for the same reason (session 44).
  • Optimizer temporaries. The default “foreach” implementations of Adam and AdamW work on lists of tensors at once and create intermediate lists; the fused versions avoid most of them. [Likely]
  • Recomputation. With checkpointing, one block’s activations are rebuilt during backward; this is the \(M_{\text{layer}}\) term in (38.8).

The caching allocator: allocated vs reserved

Asking the GPU driver for memory (cudaMalloc) is slow, so PyTorch uses a caching allocator: when a tensor is freed, PyTorch keeps its memory block and reuses it for later tensors. The PyTorch documentation says this unused cached memory still appears as used in nvidia-smi. [Certain] So there are two numbers:

  • torch.cuda.memory_allocated(): bytes occupied by live tensors. This is what our formulas predict.
  • torch.cuda.memory_reserved(): bytes the allocator holds, used or not. This is what the GPU sees, and it is always at least as large.

The allocator also rounds sizes up (in multiples of 512 bytes, and large blocks are managed separately from small ones). [Certain] Use max_memory_allocated() after a step to read the real peak.

Fragmentation

Fragmentation means that free memory exists, but it is cut into pieces that are each too small for the tensor you need. A tensor needs one contiguous block of addresses. If short-lived tensors (temporaries, recomputed activations) are placed between long-lived ones (stored activations, gradients), freeing the short-lived ones leaves holes. The ZeRO paper reports running out of memory with over 30% of memory still free in some extreme cases, and explains it by exactly this mix of short and long lifetimes. [Certain] Step through Figure 5.

weights (long-lived)stored activations (live until backward)temporaries (short-lived)failed requestthe request, placed
Figure 5. A toy GPU with 16 GB of memory, in 1 GB cells. Temporaries placed between activations leave holes when they are freed. A 3 GB request then fails although 4 GB are free. The last step shows the same tensors placed by lifetime.
How to read Figure 5

Watch two numbers in the readout: total free memory and the largest contiguous free block. An allocation succeeds only if the largest block is big enough. In step 6 the total is 4 GB but the largest block is 2 GB, so the 3 GB request fails. In step 7 the same tensors are placed so that the short-lived ones sit together at the end; freeing them leaves one 4 GB block, and the request succeeds. This is the idea of ZeRO’s memory defragmentation: pre-allocate contiguous space for long-lived tensors so they are not mixed with short-lived ones.

Two practical fixes in PyTorch. The option PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True lets the allocator grow a memory segment instead of creating new fixed-size ones; the documentation says this leaves fewer “slivers” of unusable memory, especially when tensor sizes change from batch to batch. [Certain] The option max_split_size_mb stops the allocator from splitting large blocks, which the documentation says can reduce fragmentation. [Certain] Also: keep tensor shapes stable (pad or pack to fixed lengths), because changing sizes every step creates new block sizes again and again.

Estimating and debugging memory: the procedure

Algorithm 1: estimate the peak memory of a training run before you start
  1. input: parameters \(N\) (trainable and frozen), config \(L, h, g, I, V, a\), micro-batch \(b\), length \(s\), method, attention kernel
  2. \(S + G \leftarrow (b_w + b_g + b_o) N\) // formula (38.2); for LoRA: \(2N\) + 16 \(P_{\text{LoRA}}\), formula (38.7)
  3. \(A \leftarrow L \cdot s b (12h + 4g + 8I)\) // formula (38.3); with checkpointing use (38.8)
  4. if eager attention: \(A \leftarrow A + L \cdot 6 a s^2 b\) // formula (38.4)
  5. \(M_{\text{logits}} \leftarrow 8 s b V\) unless the loss is chunked // formula (38.5)
  6. \(M_{\text{peak}} \leftarrow S + \max(A + M_{\text{logits}} + 4sbV,\ G)\) // formula (38.9)
  7. if data parallel with ZeRO or FSDP: divide the sharded parts of \(S + G\) by the number of GPUs
  8. add 1–2 GB for CUDA and libraries, and keep about 10% free for fragmentation and temporaries // rule of thumb [Likely]
  9. return \(M_{\text{peak}}\); compare with the GPU size
Algorithm 2: what to do when a run goes out of memory
  1. Measure first: run one or two steps with reset_peak_memory_stats(), then read max_memory_allocated() and max_memory_reserved()
  2. if it fails only from step 2 on: the Adam states (8 B per parameter) are the cause; plan for them
  3. if reserved is much larger than allocated: fragmentation; set expandable_segments:True, keep shapes fixed
  4. if the model states are too large: 8-bit optimizer, LoRA or QLoRA, or shard with ZeRO / FSDP // session 45
  5. if the activations are too large: FlashAttention, then a smaller micro-batch with gradient accumulation, then checkpointing // session 39
  6. if the peak is at the logits: chunked or fused cross-entropy; do not keep FP32 logits you do not need
  7. still too large: CPU offloading, or more GPUs with tensor / sequence parallelism

The code below does the paper estimate, then measures what autograd really saves for one Llama-style block on the CPU. It needs only PyTorch and runs in a second. The attention core is written as a small function that saves only what FlashAttention saves (Q, K, V, output, one log-sum-exp per row), so the count does not depend on which attention kernel your machine has.

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# 1) The bill on paper: formulas (38.1), (38.3), (38.5) and (38.8)
def train_memory_gb(N, L, h, g, I, V, s, b, bytes_per_param=16, ckpt=False):
    states = bytes_per_param * N
    layer  = s * b * (12 * h + 4 * g + 8 * I)          # one block, FlashAttention, BF16
    acts   = L * 2 * s * b * h + layer if ckpt else L * layer
    logits = 8 * s * b * V                             # FP32 logits + saved log-softmax
    return states / 1e9, acts / 1e9, logits / 1e9

print(train_memory_gb(8.03e9, 32, 4096, 1024, 14336, 128256, 4096, 1))
# (128.48, 22.01..., 4.20...): Llama 3 8B, one sequence of 4,096 tokens

# 2) Measure what autograd really saves for one Llama-style block
h, nh, nkv, I, s, b = 256, 8, 2, 896, 64, 2
dh = h // nh

class FlashLikeAttention(torch.autograd.Function):
    """Saves only q, k, v, the output and one log-sum-exp per row, like FlashAttention."""
    @staticmethod
    def forward(ctx, q, k, v):
        rep = q.shape[1] // k.shape[1]                 # GQA: share each K/V head
        kk, vv = k.repeat_interleave(rep, 1), v.repeat_interleave(rep, 1)
        sc = (q.float() @ kk.float().transpose(-1, -2)) / dh ** 0.5
        sc = sc.masked_fill(torch.ones(s, s, dtype=torch.bool).triu(1), float("-inf"))
        lse = torch.logsumexp(sc, -1)
        o = (torch.exp(sc - lse[..., None]) @ vv.float()).to(q.dtype)
        o = o.transpose(1, 2).contiguous()             # (b, s, heads, dh)
        ctx.save_for_backward(q, k, v, o, lse)
        return o.view(b, s, h)

class RMSNorm(nn.Module):                            # plain, unfused
    def __init__(self, d):
        super().__init__(); self.w = nn.Parameter(torch.ones(d))
    def forward(self, x):
        return self.w * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + 1e-5)

class Block(nn.Module):
    def __init__(self):
        super().__init__()
        lin = lambda i, o: nn.Linear(i, o, bias=False)
        self.n1, self.n2 = RMSNorm(h), RMSNorm(h)
        self.q, self.k, self.v, self.o = lin(h, h), lin(h, nkv * dh), lin(h, nkv * dh), lin(h, h)
        self.g, self.u, self.d = lin(h, I), lin(h, I), lin(I, h)
    def forward(self, x):
        y = self.n1(x)
        q = self.q(y).view(b, s, nh, dh).transpose(1, 2)
        k = self.k(y).view(b, s, nkv, dh).transpose(1, 2)
        v = self.v(y).view(b, s, nkv, dh).transpose(1, 2)
        x = x + self.o(FlashLikeAttention.apply(q, k, v))
        y = self.n2(x)
        return x + self.d(F.silu(self.g(y)) * self.u(y))

block = Block().to(torch.bfloat16)
params = {p.untyped_storage().data_ptr() for p in block.parameters()}
saved = {}
def pack(t):                                   # called for every tensor autograd keeps
    st = t.untyped_storage()
    if st.data_ptr() not in params:            # skip weights: they are not activations
        saved[st.data_ptr()] = st.nbytes()     # count each storage once
    return t

x = torch.randn(b, s, h, dtype=torch.bfloat16, requires_grad=True)
with torch.autograd.graph.saved_tensors_hooks(pack, lambda t: t):
    out = block(x)
print(f"saved: {sum(saved.values()) / (s * b * h):.1f} bytes per (s*b*h)")        # 45.1
print(f"formula (38.3): {(12 * h + 4 * nkv * dh + 8 * I) / h:.1f}")                # 41.0
# The gap of about 4 is the unfused RMSNorm: it keeps x * rsqrt(...) as an extra copy,
# once per norm. A fused RMSNorm kernel saves only x and one FP32 number per row.

# 3) On a GPU, measure the real peak of one step
if torch.cuda.is_available():
    torch.cuda.reset_peak_memory_stats()
    # ... one forward, backward and optimizer.step() here ...
    print(torch.cuda.max_memory_allocated() / 1e9, "GB allocated at peak")
    print(torch.cuda.max_memory_reserved() / 1e9, "GB reserved by the caching allocator")

Two more tools are worth knowing. torch.cuda.memory_summary() prints a table of allocated, reserved and freed memory. torch.cuda.memory._record_memory_history() with torch.cuda.memory._dump_snapshot() records every allocation with its stack trace; the PyTorch memory visualizer then draws a timeline like Figure 4 for your real run. [Likely] You will use them in session 41.

Where this is used

SettingWhich part of the bill dominatesTypical answer
Pretraining a large model on a clusterModel states (16–20 B per parameter): 1.13 TB for 70BShard states with ZeRO-3 / FSDP; tensor and pipeline parallelism (session 46, session 48)
Long-context training (32K+ tokens)Activations, linear in \(s\) with FlashAttentionCheckpointing, sequence or context parallelism (session 49)
Fine-tuning a 7B–8B model on one 24 GB GPUModel states (108–129 GB for full fine-tuning)LoRA or QLoRA, checkpointing, micro-batch 1–4 with accumulation (session 62)
Small models with big vocabularies (1B–3B, 128K vocab)Logits: 8.4 GB for Llama 3.2 1B at \(b = 4\), \(s = 2048\)Chunked or fused cross-entropy
Training a Video-LLM (e.g. 64 frames × 196 tokens = 12,544 visual tokens)Activations: about 67 GB for an 8B language model per sequence, without checkpointingToken compression or pooling, checkpointing, context parallelism (session 212, session 206)
Fine-tuning video encoders for action recognition or video retrievalActivations of many frames × patches; batch size for contrastive lossFrame sampling, checkpointing, gradient caching for large contrastive batches [Likely] (session 208)
Capacity planning for a training budgetAll partsAlgorithm 1 before you rent GPUs; measure one step before a long run

Common mistakes

  • Counting only the weights. “7B in BF16 is 14 GB, my GPU has 24 GB, so I can fine-tune it.” The gradients alone add another 14 GB, and Adam adds 8 bytes more per parameter.
  • Forgetting the first-step trap. Adam creates its states in the first step(). A run that survives step 1 can crash at step 2. Always run at least two steps before you trust a configuration.
  • Ignoring the logits. With a 128K vocabulary, the FP32 logits and their gradient can be larger than several transformer blocks. Do not call .float() on the full logits if a fused loss is available.
  • Reading nvidia-smi as “memory used by tensors”. It shows what the caching allocator reserved. Use memory_allocated and max_memory_allocated to compare with your estimate.
  • Using eager attention for long sequences. The \(6as^2b\) term (38.4) is five times the rest of a block at 4,096 tokens and grows four times with each doubling of \(s\). Check which attention kernel you actually run.
  • Mixing GB and GiB. A 24 GB card has 25.8 × \(10^9\) bytes; tools print different units. Pick one and keep a margin.
  • Variable shapes every step. Different sequence lengths create different block sizes and grow fragmentation. Pack or bucket sequences; use expandable_segments.
  • Thinking gradient accumulation reduces model-state memory. It reduces only activations (smaller micro-batch). The 16 bytes per parameter stay.

Interview questions

Where does the “16 bytes per parameter” for Adam come from?basic
Mixed precision keeps BF16 weights (2 bytes) for the forward and backward pass and BF16 gradients (2 bytes). The optimizer keeps an FP32 master copy of the weights (4 bytes) and Adam’s two moments \(m\) and \(v\) in FP32 (4 + 4). Total 16 bytes per parameter, which ZeRO writes as \(2\Psi + 2\Psi + 12\Psi\). With an extra FP32 gradient buffer it becomes 20.
How much memory do you need to fully fine-tune a 7B model with AdamW? Does it fit on a 24 GB GPU?basic
About \(16 \times 7 \times 10^9 = 112\) GB for model states (107.8 GB for Llama 2 7B’s 6.74B parameters), plus activations. It does not fit on 24 GB. Even plain SGD in BF16 needs weights plus gradients = 27 GB. Options: LoRA (about 14 GB of states), QLoRA (about 4 GB), or sharding across several GPUs with ZeRO / FSDP.
How does activation memory scale with batch size, sequence length, layers and hidden size?core
Linear in batch size \(b\), in the number of layers \(L\) and in the hidden size (through \(h\), \(g\), \(I\)): one block stores about \(sb(12h + 4g + 8I)\) bytes in BF16, about 41 \(sbh\) for Llama 3 8B. In sequence length it is linear with FlashAttention, and quadratic without it because each layer stores \(a\) score matrices of size \(s \times s\) (about \(6as^2b\) bytes in the common FP32-softmax pattern). Parameters grow like \(h^2\), so wide models are dominated by states at short context.
Derive the activation memory of one transformer block.core
Walk through the block and list what backward needs. For a Llama-style block in BF16: each RMSNorm stores its input (\(2sbh\)); the QKV projections store their shared input (\(2sbh\)); FlashAttention stores Q (\(2sbh\)), K and V (\(4sbg\)) and its output (\(2sbh\)), which is also the output projection’s input; the second norm and the gate/up projections store \(2sbh + 2sbh\); SwiGLU stores gate, up, SiLU(gate) and their product (\(8sbI\)). Sum: \(sb(12h + 4g + 8I)\). For a GPT-style block with dropout, Korthikanti et al. get \(sbh(34 + 5as/h)\).
Training succeeds in step 1 and runs out of memory in step 2. Why?core
PyTorch’s Adam creates \(m\) and \(v\) lazily during the first optimizer.step(). Step 1’s forward and backward ran without them; step 2 runs with 8 more bytes per parameter permanently allocated. Fragmentation from the first step can make it worse. Fix: plan for the full 16 bytes, or test two steps when tuning batch size.
Where is the memory peak in a training step?core
Usually at the start of the backward pass: all activations are stored, the logits and their FP32 gradient exist, and gradients start to appear. During backward, each block frees its activations and creates its weight gradients. If the gradients are larger than the activations (big model, short sequences, checkpointing), the peak moves to the end of backward or the optimizer step. Roughly \(S + \max(A + M_{\text{logits}} + 4sbV,\ G)\).
Your estimate says 70 GB, the GPU has 80 GB, and you still get OOM. What do you check?advanced
Measure max_memory_allocated vs max_memory_reserved. The gap is caching and fragmentation; try expandable_segments:True and fixed shapes. Then look for things the formula missed: 1–2 GB of CUDA context and kernels, FP32 upcasts (logits, norms), flat gradient buffers or DDP buckets, optimizer temporaries (foreach), recomputation buffers, eager attention instead of FlashAttention, Adam states from step 2. A memory snapshot shows which allocation is at the peak.
What is memory fragmentation, and why does it happen in training?advanced
Free memory is split into many non-contiguous blocks, so a large tensor cannot be placed even though the total free memory is enough. In training it comes from mixing lifetimes: short-lived temporaries and recomputed activations are allocated between long-lived stored activations and gradients; when they are freed, they leave holes. ZeRO reports OOM with over 30% free memory in extreme cases and fixes it by pre-allocating contiguous memory for long-lived tensors. PyTorch offers expandable segments and max_split_size_mb.
Which memory does gradient accumulation reduce, and which does checkpointing reduce?core
Both reduce only activation memory. Accumulation lets you use a smaller micro-batch \(b\) for the same effective batch, so activations shrink linearly with \(b\), at no extra compute. Checkpointing keeps only one input per block (\(2sbh\)) and recomputes the rest, about one extra forward pass (around 33% more compute). Neither changes the 16 bytes per parameter of model states; for those you need sharding, 8-bit states or LoRA.
Why can the output layer be a memory problem, and how do fused losses help?advanced
The logits have \(s \cdot b \cdot V\) entries. With \(V = 128{,}256\) and \(s = 4096\) that is 525 million per sequence; in FP32 plus the saved log-softmax it is about 4.2 GB, and the gradient adds more. For small models this can rival all transformer blocks. Chunked or fused cross-entropy computes the loss and its gradient in pieces, never materializing the full logits; Cut Cross-Entropy reports going from 24 GB to 1 MB for the loss of Gemma 2 (2B).
Why do LoRA and QLoRA still need activation memory?core
LoRA removes gradients and optimizer states for the frozen weights, so model states drop from 16 to about 2 bytes per parameter (0.5 with 4-bit QLoRA). But the adapters sit inside every block, so the gradient must flow backward through all frozen layers, and that still needs the saved inputs of norms, attention and the FFN. Activation memory is about the same as for full fine-tuning; checkpointing is often still needed.

Exercises

  1. How much memory do the model states of Llama 3.2 3B (3.21B parameters) take with mixed-precision AdamW? Does it fit on an 80 GB GPU, before activations?
  2. Llama 3 70B has 70.55B parameters. How much do the model states take with AdamW, and how much per GPU if ZeRO-3 shards them evenly across 8 GPUs?
  3. For Llama 3 8B, how does activation memory change when the sequence length goes from 2,048 to 4,096 tokens (a) with FlashAttention, (b) for the score term (38.4) without it?
  4. Compute the bracket of formula (38.3) for Llama 3.2 3B (\(h = 3072\), \(g = 1024\), \(I = 8192\)) in units of \(h\).
  5. Llama 3.2 1B with \(s = 2048\), \(b = 4\): compute the logit memory (38.5). Compare it with the block activations (12.1 GB).
  6. In Figure 5, which single change in the placement of the tensors would make the 3 GB request succeed?
Answerscheck
1. \(16 \times 3.21 \times 10^9 = 51.4\) GB: it fits on 80 GB, leaving about 28 GB (minus overhead) for activations. 2. \(16 \times 70.55 \times 10^9 = 1{,}129\) GB; divided by 8: 141 GB per GPU, still too much for 80 GB GPUs, so you need more GPUs (at least 15 for the states alone at 80 GB) or other tricks. 3. (a) It doubles (linear in \(s\)): 11.0 GB to 22.0 GB for the 32 blocks. (b) It grows four times (quadratic): 25.8 GB to 103.1 GB. 4. \(12 \times 3072 + 4 \times 1024 + 8 \times 8192 = 106{,}496 = 34.7h\). 5. \(8 \times 2048 \times 4 \times 128{,}256 = 8.4\) GB, about 70% of the 12.1 GB of all 16 blocks. 6. Put the three temporaries next to each other (for example at the end, after all activations), so that freeing them leaves one 4 GB block.

Key takeaways

  • Training memory = model states + activations + temporary buffers + overhead and fragmentation.
  • Model states are \((b_w + b_g + b_o)N\) (38.2): 16 bytes per parameter for mixed-precision AdamW (38.1), 20 with FP32 gradient accumulation. A 7B model needs about 112 GB before any activation.
  • Activations grow linearly with batch, sequence length and depth: about \(sb(12h + 4g + 8I)\) bytes per Llama-style block (38.3), plus an \(s^2\) term without FlashAttention (38.4) and a big logits term with large vocabularies (38.5).
  • Full fine-tuning of a 7B model cannot fit on 24 GB: even weights plus gradients are 27 GB. LoRA or QLoRA plus checkpointing does fit.
  • The OOM happens at the peak (38.9), usually at the start of backward; Adam states appear only after step 1.
  • Measure with max_memory_allocated and max_memory_reserved; the gap is caching and fragmentation.

Key formulas

\[ M_{\text{states}} = (b_w + b_g + b_o)N = 16N, \qquad M_{\text{layer}} = s b\,(12h + 4g + 8I), \qquad M_{\text{logits}} \approx 8 s b V \]
\[ M_{\text{peak}} \approx S + \max\big(A + M_{\text{logits}} + 4sbV,\ G\big), \qquad M^{\text{ckpt}}_{\text{act}} \approx 2Lsbh + M_{\text{layer}} \]

References for this session

  1. Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  2. Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
  3. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  4. Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  5. Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
  6. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  7. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  8. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
  9. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
  10. Dettmers et al. (2021). 8-bit Optimizers via Block-wise Quantization
  11. Chen et al. (2016). Training Deep Nets with Sublinear Memory Cost
  12. Zhao et al. (2024). GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
  13. Wijmans, Huval, Hertzberg, Koltun, Krähenbühl (2024). Cut Your Losses in Large-Vocabulary Language Models
  14. PyTorch (docs). CUDA semantics: memory management and the caching allocator