ZeRO stages 1, 2 and 3
Data parallelism keeps a full copy of the weights, the gradients and the optimizer states on every GPU, so most of the memory holds the same numbers N times. ZeRO removes this waste in three stages: stage 1 splits the optimizer states across GPUs, stage 2 also splits the gradients, and stage 3 also splits the weights. This session computes the memory per GPU and the communication cost of each stage, and shows how offloading to CPU memory or NVMe disks goes further.
What you will learn
- Why plain data parallelism wastes memory: every GPU stores the same 16 bytes per parameter
- What stage 1, stage 2 and stage 3 shard, and the memory formula for each (4Ψ + 12Ψ/N, 2Ψ + 14Ψ/N, 16Ψ/N)
- Why stages 1 and 2 cost no more communication than DDP, and why stage 3 costs 1.5 times as much
- When communication can hide behind compute, and how many tokens per GPU you need for that
- What ZeRO does not shard (activations, temporary buffers) and what ZeRO-R adds
- ZeRO-Offload and ZeRO-Infinity: which states go to CPU memory or NVMe, and why the PCIe traffic is 4Ψ bytes
- How to read a DeepSpeed ZeRO config, and how ZeRO maps to PyTorch FSDP
Why this matters
In session 44 every GPU keeps a full copy of the model. That is simple and fast, but it means that on 64 GPUs the same optimizer states are stored 64 times. ZeRO (the Zero Redundancy Optimizer) removes this copy-everything waste, one kind of state at a time.
Recall the bill from session 4: mixed-precision training with Adam needs about 16 bytes per parameter before any activation. For Llama 2 7B (6.74 billion parameters) that is 108 GB, more than one 80 GB GPU. With DDP, adding GPUs does not help at all: every GPU still needs 108 GB. ZeRO makes the memory per GPU go down as you add GPUs. With all three stages on 8 GPUs, the same model needs 13.5 GB of model states per GPU.
The ZeRO paper (Rajbhandari et al., 2019) defined three stages, and the names stuck [Certain]:
- Stage 1 shards (splits into disjoint pieces) the optimizer states.
- Stage 2 also shards the gradients.
- Stage 3 also shards the parameters (the weights).
Stages 1 and 2 cost no extra communication compared with DDP. Stage 3 costs 1.5 times as much. Later work moved states to CPU memory and NVMe disks (ZeRO-Offload, ZeRO-Infinity). These ideas are inside DeepSpeed, and PyTorch FSDP (session 46) is an implementation of stage 3. Almost every large training run today uses some form of this sharding.
Where the memory goes in data parallelism
Let \(\Psi\) be the number of parameters. In mixed-precision training with Adam, each GPU in DDP stores these model states (the ZeRO paper’s name for everything that has one number per parameter) [Certain]:
| State | Precision | Bytes | Used when |
|---|---|---|---|
| Parameters (weights) for forward and backward | FP16 or BF16 | \(2\Psi\) | forward and backward pass |
| Gradients | FP16 or BF16 | \(2\Psi\) | end of backward, optimizer step |
| FP32 master copy of the weights | FP32 | \(4\Psi\) | optimizer step only |
| Adam momentum \(\mathbf{m}\) | FP32 | \(4\Psi\) | optimizer step only |
| Adam variance \(\mathbf{v}\) | FP32 | \(4\Psi\) | optimizer step only |
The paper groups the last three as the optimizer states and writes their size as \(K\Psi\), with \(K = 12\) for mixed-precision Adam. So the memory per GPU in DDP is
- Read the symbols
- \(\Psi\)
- the number of parameters of the model
- \(2 + 2\)
- bytes per parameter for the 16-bit weights and the 16-bit gradients
- \(K\)
- bytes per parameter for the optimizer states; 12 for Adam with an FP32 master copy, \(\mathbf{m}\) and \(\mathbf{v}\)
- \(M_{\text{DDP}}\)
- model-state memory on each GPU, whatever the number of GPUs
The paper’s own example: GPT-2 with 1.5 billion parameters needs only 3 GB for its 16-bit weights, but 24 GB of model states, so it did not train on a 32 GB GPU once activations were added [Certain]. Three quarters of the bill (\(12\Psi\) of \(16\Psi\)) are optimizer states, which are used only for a short moment at the end of each step.
Now look at what DDP does with \(N\) GPUs. After the all-reduce, every GPU holds the same averaged gradient, and runs the same optimizer step on the same states. \(N - 1\) of the \(N\) copies are pure redundancy. That is the waste ZeRO removes.
The rest of the memory, called residual states in the paper, is activations, temporary buffers and fragmentation. ZeRO stages 1 to 3 do not touch activations; we come back to them in the section on ZeRO-R.
The ZeRO idea: keep one copy, split across GPUs
ZeRO keeps data parallelism: every GPU still gets its own slice of the batch and runs the full forward and backward pass. What changes is storage. Cut each kind of state into \(N\) equal shards. GPU \(r\) owns shard \(r\): it stores it, and it is the only GPU that updates it. When a GPU needs a piece it does not own, it gets it from the owner with a collective (reduce-scatter or all-gather from session 43), uses it, and throws it away.
The three stages differ only in which states are cut. Each step uses the fact that a state is not needed all the time: the optimizer states are needed only in the optimizer step, a gradient only until its shard owner has reduced it, and the weights of a layer only while that layer runs. Step through Figure 1 to see who holds what during one training step.
How to read Figure 1
Start with DDP: all 12 cells of every GPU are full at the end of the step, and one all-reduce sends \(1.5\Psi\) elements per GPU (that is \(2\Psi \cdot 3/4\) with a ring on 4 GPUs). Switch to ZeRO-1: only the purple row shrinks to one cell, yet the communication is the same \(1.5\Psi\), because an all-reduce is a reduce-scatter plus an all-gather, and ZeRO only moves the optimizer step between the two. ZeRO-2 also drops the gradient row to one cell. ZeRO-3 drops the weight row too, but must gather weights twice (forward and backward), so its total is \(2.25\Psi\): 1.5 times DDP.
Stage 1: shard the optimizer states
In stage 1 (the paper calls it \(P_{os}\)), GPU \(r\) keeps the FP32 master weights, \(\mathbf{m}\) and \(\mathbf{v}\) only for its own shard: \(\Psi / N\) parameters. The 16-bit weights and the 16-bit gradients stay full on every GPU. The training step becomes:
- input: full 16-bit weights \(\theta\), optimizer states for shard \(r\) only, a local micro-batch
- forward and backward on the local micro-batch \(\;\to\;\) full local gradient \(\mathbf{g}\) (\(2\Psi\) bytes)
- \(\mathbf{g}_r \leftarrow\) reduce-scatter\((\mathbf{g})\) \(/\, N\) // GPU \(r\) gets the averaged gradient of its shard only
- update shard \(r\): AdamW on the FP32 master \(\theta^{32}_r\) with \(\mathbf{m}_r, \mathbf{v}_r\) and \(\mathbf{g}_r\)
- \(\theta_r \leftarrow \text{to16}(\theta^{32}_r)\)
- \(\theta \leftarrow\) all-gather\((\theta_1, \dots, \theta_N)\) // every GPU gets the full updated weights
The memory per GPU is
As \(N\) grows, the second term goes to zero and the memory goes to \(4\Psi\): at most a 4 times saving compared with \(16\Psi\). The paper’s example: a 7.5B model on 64 GPUs needs 31.4 GB instead of 120 GB [Certain]. Stage 1 is the cheapest stage to turn on: it changes nothing in the forward and backward pass.
Stage 2: also shard the gradients
In stage 1, every GPU still builds the full gradient and then keeps only \(1/N\) of the averaged result. Stage 2 (\(P_{os+g}\)) stops storing what it will not use. During the backward pass, gradients are put into buckets (fixed-size groups of gradients, as in DDP). As soon as a bucket is full, it is reduced to the GPUs that own its parts, and the other GPUs free that memory. So each GPU holds only a small window of full gradients at any moment, plus its own reduced shard.
The limit is \(2\Psi\), only the 16-bit weights: up to an 8 times saving. The 7.5B model on 64 GPUs: 16.6 GB [Certain]. The extra memory for buckets is fixed (DeepSpeed’s default reduce_bucket_size is \(5 \times 10^8\) elements [Certain]), and the reduction overlaps with the rest of the backward pass, as in DDP.
Stage 2 is still cheap: the forward pass does not change, and the gradient traffic is the same as before, only split into pieces. That is why ZeRO-2 is the most common choice for models that fit with it [Likely].
Stage 3: also shard the parameters
After stage 2, the full 16-bit weights (\(2\Psi\)) are the floor. Stage 3 (\(P_{os+g+p}\)) shards them too. Now GPU \(r\) stores only its \(1/N\) of everything. But the forward pass of layer \(\ell\) needs all of layer \(\ell\)’s weights. So weights are fetched just in time, one layer (or one group of layers) at a time:
- input: shard \(r\) of the 16-bit weights, of the gradients and of the optimizer states
- for layer \(\ell = 1, \dots, L\): // forward
- \(W_\ell \leftarrow\) all-gather of the shards of layer \(\ell\) // prefetch layer \(\ell+1\) while \(\ell\) computes
- run layer \(\ell\) forward on the local micro-batch, keep its activations
- free \(W_\ell\) except the own shard
- for layer \(\ell = L, \dots, 1\): // backward
- \(W_\ell \leftarrow\) all-gather again
- compute the input gradient and the local weight gradient \(\mathbf{g}_\ell\)
- \(\mathbf{g}_{\ell, r} \leftarrow\) reduce-scatter\((\mathbf{g}_\ell) / N\); free \(W_\ell\) and \(\mathbf{g}_\ell\)
- AdamW on shard \(r\) only (FP32 master, \(\mathbf{m}_r\), \(\mathbf{v}_r\)); write the new 16-bit shard // no all-gather here: the next forward gathers
Now the saving grows without limit: double \(N\), halve the memory. The 7.5B model on 64 GPUs needs 1.88 GB of model states; a 1-trillion-parameter model on 1,024 GPUs needs 15.6 GB [Certain]. The cost: the weights of one layer must be gathered twice per step, and each GPU needs room for at least one full layer (two, with prefetching) while it runs. For Llama 2 7B, one block has 202 million parameters (\(4 \cdot 4096^2 + 3 \cdot 4096 \cdot 11008\)), so one gathered block is 0.40 GB in BF16.
Stage 3 also changes how your code sees the model. Outside the forward pass, a parameter on GPU \(r\) holds only its shard (in DeepSpeed, an empty placeholder). Reading or changing full weights outside the step (for example, to initialize, inspect or save them) needs an explicit gather, such as DeepSpeed’s GatheredParameters context [Certain].
Algorithm 2 and the formulas above hide one thing: memory is not constant during the step. Watch it happen in Figure 2. It runs a real training step on 4 GPUs with a tiny 3-layer model of \(\Psi = 12\) parameters (4 per layer). Each layer multiplies its input element by element by its weights, \(\mathbf{h}_\ell = W_\ell \odot \mathbf{h}_{\ell-1}\), and the loss is \(\tfrac12 \lVert \mathbf{h}_3 - \mathbf{y} \rVert^2\). The global batch has 4 examples, and each GPU gets \(4/N\) of them. Every cell shows the number that GPU really holds. Every collective really moves numbers between the GPU panels, and the AdamW step really runs (learning rate 0.1, \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), weight decay 0.01). Bytes are counted with the 16 bytes per parameter of formula (45.1).
When ZeRO-3 gathers a layer, the GPU writes it into a separate buffer for the full layer and keeps its own shard as well. In the same way, the gradient of the layer being computed sits in a full-size buffer until the reduce-scatter. So for a short time a GPU holds more than \(16\Psi/N\). Here \(\Psi_\ell\) is the number of parameters in one layer and \(a\) is the number of layers gathered at the same time (1, or 2 with prefetching). In this toy, which counts bytes the same way, the peak is
The last term is the full gradient of the layer, before the reduce-scatter. With \(N = 4\) and \(\Psi_\ell = 4\), this gives \(48 + 8 + 8 = 64\) bytes without prefetching and 72 bytes with it. ZeRO-2 has only the gradient term, so its peak is \(66 + 8 = 74\) bytes.
- for \(\ell = 1, \dots, L\): // forward
- Z3 \(W_\ell \leftarrow\) all-gather(shards of layer \(\ell\)) // prefetch: also \(\ell+1\)
- \(\mathbf{h}_\ell \leftarrow W_\ell \odot \mathbf{h}_{\ell-1}\) for each local example
- Z3 free \(W_\ell\) except the own shard
- \(\mathbf{e} \leftarrow \mathbf{h}_L - \mathbf{y}\); loss \(\tfrac12 \lVert \mathbf{e} \rVert^2\); \(\delta_L \leftarrow \mathbf{e}\)
- for \(\ell = L, \dots, 1\): // backward
- Z3 \(W_\ell \leftarrow\) all-gather again
- \(\mathbf{g}_\ell \leftarrow \delta_\ell \odot \mathbf{h}_{\ell-1}\); \(\delta_{\ell-1} \leftarrow \delta_\ell \odot W_\ell\)
- DDP \(\mathbf{g}_\ell \leftarrow\) all-reduce\((\mathbf{g}_\ell) / N\)
- Z2Z3 \(\mathbf{g}_{\ell,r} \leftarrow\) reduce-scatter\((\mathbf{g}_\ell)/N\); free the rest
- Z1 \(\mathbf{g}_r \leftarrow\) reduce-scatter\((\mathbf{g})/N\)
- AdamW on shard \(r\) (DDP: on all of \(\theta\))
- Z1Z2 \(\theta \leftarrow\) all-gather\((\theta_1, \dots, \theta_N)\)
How to read Figure 2
Start with ZeRO-3 and press Step. At “all-gather layer 1” the first weight row of every GPU fills up: the pale cells came from the other GPUs, and the solid cell is the GPU’s own shard. The memory bar jumps from 48 to 56 bytes, and after “free” it falls back. In the backward pass the full gradient row appears in pale orange, and the reduce-scatter shrinks it to one averaged number per GPU. The formula panel shows that number as the mean of the 4 local gradients. The peak marker stops at 64 bytes, as the peak formula says; tick Prefetch and it stops at 72. Now switch to DDP: the bar stays at 192 bytes, and every GPU runs the same AdamW update on all 12 numbers. Try ZeRO-1 and ZeRO-2: right after the optimizer step only the own weight shard is new, and the last all-gather brings in the rest. The final frame compares the weights with DDP: they are exactly equal for every method and for both \(N\). ZeRO changes where numbers live, not what is computed. The counter of sent elements ends at \(1.5\Psi = 18\) for DDP, ZeRO-1 and ZeRO-2, and at \(2.25\Psi = 27\) for ZeRO-3 (with \(N = 4\)).
Memory per GPU for each stage
Formulas (45.1) to (45.4) side by side, for \(K = 12\):
| Method | What is sharded | Bytes per GPU | Limit as \(N \to \infty\) | 7.5B model, \(N = 64\) |
|---|---|---|---|---|
| DDP | nothing | \(16\Psi\) | \(16\Psi\) | 120 GB |
| ZeRO-1 | optimizer states | \(4\Psi + 12\Psi/N\) | \(4\Psi\) (4×) | 31.4 GB |
| ZeRO-2 | + gradients | \(2\Psi + 14\Psi/N\) | \(2\Psi\) (8×) | 16.6 GB |
| ZeRO-3 | + parameters | \(16\Psi/N\) | \(\to 0\) (\(N\)×) | 1.88 GB |
The last column is from Table 1 of the ZeRO paper, and the formulas give the same numbers [Certain]. Turn the formulas around to get the largest model whose states fit into \(M\) bytes per GPU:
With \(M = 80\) GB and \(N = 8\): DDP fits 5.0B parameters, ZeRO-1 14.5B, ZeRO-2 21.3B, and ZeRO-3 40B. In real runs you must leave room for activations, buckets and fragmentation, so the true limits are lower. Try your own model in Figure 3.
How to read Figure 3
Press Play first: \(N\) doubles at each tick, and in the panel below the bars only the terms divided by \(N\) get smaller. At \(N = 1\) all four methods give the same \(16\Psi\). Pick “7.5B (paper)” and move \(N\) to 64: the bars read 120, 31.4, 16.6 and 1.88 GB, the numbers in the ZeRO paper. Now drag \(N\) up to 1,024. The ZeRO-1 bar stops near 30 GB (the blue and orange parts never shrink) and the ZeRO-2 bar stops near 15 GB (the blue part never shrinks). Only the ZeRO-3 bar keeps falling. Then pick 70B with \(N = 8\): nothing but ZeRO-3 fits, and with 140 GB of weights alone, ZeRO-2 cannot fit on any number of 80 GB GPUs.
\(\Psi = 6.738 \times 10^9\), \(N = 8\), \(K = 12\).
DDP: \(16 \times 6.738 = 107.8\) GB. Does not fit in 80 GB.
ZeRO-1: \(4 \times 6.738 + 12 \times 6.738 / 8 = 26.95 + 10.11 = 37.1\) GB.
ZeRO-2: \(2 \times 6.738 + 14 \times 6.738 / 8 = 13.48 + 11.79 = 25.3\) GB.
ZeRO-3: \(16 \times 6.738 / 8 = 13.5\) GB, plus room for one or two gathered blocks (0.40 GB each).
So ZeRO-1 already makes full fine-tuning of a 7B model possible on 8 GPUs of 80 GB, with about 40 GB left for activations; ZeRO-2 leaves about 55 GB.
Communication cost vs memory saved
Memory is only half the story. Every byte we do not store must be sent at the right moment. From session 43: with a ring on \(N\) GPUs, a reduce-scatter or an all-gather of a vector of \(\Psi\) elements makes each GPU send \(\frac{N-1}{N}\Psi\) elements, and an all-reduce (which is a reduce-scatter followed by an all-gather) sends twice that.
DDP: one all-reduce of the gradients. ZeRO-1 and ZeRO-2: a reduce-scatter of the gradients, then an all-gather of the updated weights. That is exactly the two halves of the all-reduce, with the optimizer step moved into the middle. So the volume is the same [Certain]:
ZeRO-3: an all-gather of the weights for the forward pass, another for the backward pass (the weights were freed in between), and a reduce-scatter of the gradients. No all-gather after the optimizer step, because the next forward pass does it. So
- Read the symbols
- \(V\)
- elements sent by each GPU in one training step; multiply by the bytes per element (2 for BF16) to get bytes
- \(\frac{N-1}{N}\)
- the ring factor: each GPU sends all shards but its own; close to 1 for large \(N\)
Notice that the volume per GPU does not grow with \(N\). It depends only on the model size. This is the key reason ZeRO scales: memory per GPU falls with \(N\), but communication per GPU stays about the same.
Can the communication hide behind compute?
The collectives run on their own CUDA stream and overlap with compute (prefetching in the forward pass, bucketed reduce-scatter in the backward pass). So what matters is whether communication time is shorter than compute time. With \(T_g\) tokens per GPU per step, a training cost of \(6\Psi\) FLOPs per token (session 25), an achieved speed of \(F\) FLOP/s, \(b\) bytes per element and a bandwidth of \(\beta\) bytes/s per GPU:
Both grow with \(\Psi\), so \(\Psi\) cancels when you ask “is \(t_{\text{comm}} \le t_{\text{compute}}\)?”. Communication can hide when
That is a rule about tokens per GPU, not about model size. With BF16 (\(b = 2\)), \(F = 400\) TFLOP/s and 8 GPUs: inside one server (\(\beta = 450\) GB/s over NVLink) ZeRO-3 needs only about 780 tokens per GPU. Across servers (\(\beta = 50\) GB/s over InfiniBand) it needs about 7,000 tokens per GPU, and DDP or ZeRO-2 about 4,700.
Each GPU sends \(3 \times \tfrac{7}{8} \times 6.738 \times 10^9 \times 2\) bytes \(= 35.4\) GB per step (DDP or ZeRO-2: 23.6 GB). At 50 GB/s that is 0.71 s (DDP: 0.47 s).
With 8 sequences of 4,096 tokens per GPU (\(T_g = 32{,}768\)): \(t_{\text{compute}} = 6 \times 6.738 \times 10^9 \times 32{,}768 / (400 \times 10^{12}) = 3.31\) s. Communication hides easily.
With one sequence (\(T_g = 4{,}096\)): \(t_{\text{compute}} = 0.41\) s, less than 0.71 s. Now ZeRO-3 is communication bound and the GPUs wait 0.29 s per step (even DDP waits 0.06 s). This is the setting where ZeRO++ and hybrid sharding help (below).
How to read Figure 4
The default is the worked example: Llama 2 7B, 8 GPUs, 32,768 tokens per GPU, 50 GB/s. Everything hides. Now move the tokens down to 4,096: all bars turn red, but ZeRO-3 waits about 5 times longer (0.29 s against 0.06 s), because it sends 1.5 times as much. Move the parameter slider: both bars grow together and the red part keeps the same sign, as formula (45.9) says. Change \(N\): almost nothing happens, because \((N-1)/N\) is close to 1. Switch to NVLink and the problem almost disappears. In real runs, overlap is not perfect (the first all-gather and the last reduce-scatter cannot hide), so treat the red part as a lower bound.
Ways to pay less for stage 3
- Bigger micro-batches raise \(T_g\) and move you to the safe side of (45.9). Activation checkpointing (session 39) frees the memory for them.
- Hybrid sharding: shard inside a server (fast NVLink), replicate across servers (only gradients cross the slow network). PyTorch calls this
HYBRID_SHARD(session 46); ZeRO++ calls its version hpZ, set byzero_hpz_partition_size[Certain]. - ZeRO++ (Wang et al., 2023) also quantizes the weight all-gather and the gradient reduction. The paper reports a 4 times smaller communication volume and up to 2.16 times higher throughput on 384 GPUs [Certain].
- Do not over-shard. If ZeRO-2 fits, it sends less than ZeRO-3 and keeps the code simpler.
Beyond model states: ZeRO-R
Stages 1 to 3 (together called ZeRO-DP in the paper) handle model states. The paper adds ZeRO-R for the residual states [Certain]:
- Activations. Data parallelism already splits activations by batch, so ZeRO-DP cannot shrink them further. The paper’s GPT-2 1.5B example needs about 60 GB of activations at sequence length 1K and batch 32, and about 8 GB with activation checkpointing [Certain]. With tensor (model) parallelism, every GPU in the group stores the same checkpointed activations; ZeRO-R’s partitioned activation checkpointing (\(P_a\)) splits them across the group and can also move them to CPU (\(P_{a+cpu}\)).
- Constant-size buffers. Fusing all gradients into one FP32 buffer for an all-reduce would need 6 GB for a 1.5B model [Certain]. ZeRO uses buffers of a fixed size instead (the buckets above), so the buffer does not grow with the model.
- Memory defragmentation. Short-lived tensors (recomputed activations, gathered weights) interleave with long-lived ones and leave holes. The paper saw out-of-memory errors with over 30% of memory still free in extreme cases [Certain]. ZeRO-R moves long-lived tensors into pre-allocated contiguous memory.
Offloading to CPU or NVMe
A GPU server has much more CPU memory (often 1 to 2 TB) than GPU memory, and NVMe disks hold even more [Likely]. The catch is bandwidth: the link between CPU and GPU (PCIe) is about 32 GB/s per direction for Gen4 x16, more than 10 times slower than GPU memory. Offloading moves some states off the GPU and must keep PCIe traffic small.
ZeRO-Offload: which state goes where
ZeRO-Offload (Ren et al., 2021) treats one training step as a data-flow graph with four parts: the forward-backward computation (must stay on the GPU, it is the heavy compute), the 16-bit weights \(p_{16}\), the 16-bit gradients \(g_{16}\), and an “update” node that holds the FP32 master weights, \(\mathbf{m}\) and \(\mathbf{v}\) and runs Adam. Every edge cut between GPU and CPU costs PCIe traffic [Certain]:
- forward-backward reads \(p_{16}\) twice (forward and backward): \(4\Psi\) bytes if they are on different devices;
- forward-backward writes \(g_{16}\): \(2\Psi\) bytes; the update reads \(g_{16}\): \(2\Psi\) bytes;
- the update writes new \(p_{16}\): \(2\Psi\) bytes.
Any split must cut at least two edges, so at least \(4\Psi\) bytes per step. Keeping \(p_{16}\) on the GPU and moving \(g_{16}\) and the update to the CPU reaches this minimum and saves the most GPU memory. That is ZeRO-Offload’s choice [Certain]:
The optimizer step now runs on the CPU. To make that fast, ZeRO-Offload has a vectorized CPU Adam, reported as up to 6 times faster than the PyTorch CPU optimizer, and an optional one-step delayed parameter update (DPU): the CPU updates with the gradients of step \(i - 1\) while the GPU computes step \(i\). DPU changes the math slightly, so it is turned on only after the first few dozen steps [Certain]. With several GPUs, ZeRO-Offload runs on top of ZeRO-2: each GPU offloads only its own shard, so the CPU memory and the PCIe traffic per GPU are divided by \(N\) [Likely]. Find the best placement yourself in Figure 5.
How to read Figure 5
Start with “All on GPU”: \(16\Psi\) bytes on the GPU, no traffic. Move only the update node to the CPU: the GPU drops to \(4\Psi\) (4 times less) for \(4\Psi\) of traffic. Move the gradients too: \(2\Psi\) on the GPU (8 times less), still \(4\Psi\) of traffic. That is the ZeRO-Offload preset. Now move the weights to the CPU as well: the GPU memory for model states goes to zero, but the traffic jumps to \(6\Psi\), because the forward and backward pass read the weights twice. With the default 13B model, ZeRO-Offload needs 26 GB on the GPU, 182 GB on the CPU, and 52 GB over PCIe per step (1.63 s at 32 GB/s).
Is 1.63 s per step acceptable? It depends on the batch. With \(T_g\) tokens per step, the GPU computes for \(6\Psi T_g / F\) seconds, and the PCIe time is \(4\Psi/\beta_{\text{PCIe}}\). The ratio again does not depend on \(\Psi\). At \(F = 100\) TFLOP/s and 32 GB/s, the transfer hides behind compute once \(T_g \ge 4 \times 100 \times 10^{12} / (6 \times 32 \times 10^9) \approx 2{,}100\) tokens per step. The ZeRO-Offload paper reports training models with over 13 billion parameters on a single GPU, 10 times larger than PyTorch alone, and about 40 TFLOP/s on one V100 for a 10B model [Certain].
Stage 3 offload and ZeRO-Infinity
With stage 3, the 16-bit weights can be offloaded too (offload_param). Each layer’s weights then travel from CPU to GPU just in time, the same schedule as the all-gather of Algorithm 2. This is the “everything on CPU” corner of Figure 5: traffic grows to about \(6\Psi\) bytes per step, but the GPU keeps only a working set of a few layers.
ZeRO-Infinity (Rajbhandari et al., 2021) adds NVMe disks as a third tier for both parameters and optimizer states, and splits large layers into tiles so that even one layer does not have to fit on a GPU. It also spreads the PCIe and NVMe traffic over all GPUs and disks of a server in parallel, instead of one link at a time. The paper reports fine-tuning a trillion-parameter model on a single DGX-2 node (16 V100 GPUs) and over 25 PFLOP/s on 512 GPUs [Certain].
In DeepSpeed, the rules are [Certain]: offload_optimizer works with stages 1, 2 and 3, to "cpu" or "nvme", but NVMe only with stage 3; offload_param works only with stage 3. The optimizer step always runs on the CPU when the optimizer is offloaded, even with NVMe.
ZeRO in code
The code below is a complete simulation of DDP and ZeRO stages 1, 2 and 3 in one process. “Ranks” are list entries, the collectives are written by hand and count the elements each rank sends. It checks that all four methods give the same weights as torch.optim.AdamW on the full batch, and prints the communication volume. It runs on a CPU in about a second.
import torch torch.manual_seed(0) N = 4 # number of simulated GPUs (ranks) sent = [0] * N # elements sent by each rank def reduce_scatter(full_list): # every rank has a full vector; rank r gets the sum of chunk r chunks = [v.chunk(N) for v in full_list] for r in range(N): sent[r] += (N - 1) * chunks[r][0].numel() # ring: (N-1)/N of the vector return [sum(chunks[s][r] for s in range(N)) for r in range(N)] def all_gather(shards): # every rank has one shard; all ranks get the full vector for r in range(N): sent[r] += (N - 1) * shards[r].numel() full = torch.cat(shards) return [full.clone() for _ in range(N)] def all_reduce(full_list): # = reduce-scatter followed by all-gather return all_gather(reduce_scatter(full_list)) # a tiny model as one flat parameter vector: Linear(8,16) -> tanh -> Linear(16,4) shapes = [(16, 8), (16,), (4, 16), (4,)] sizes = [torch.Size(s).numel() for s in shapes] P = sum(sizes) # 212 numbers; pad to a multiple of N pad = (-P) % N def forward(flat, x): w1, b1, w2, b2 = [t.view(s) for t, s in zip(flat[:P].split(sizes), shapes)] return torch.tanh(x @ w1.T + b1) @ w2.T + b2 def grad_on_rank(flat, x, y): # local backward on this rank's micro-batch flat = flat.detach().requires_grad_(True) loss = torch.nn.functional.mse_loss(forward(flat, x), y) loss.backward() return flat.grad theta0 = torch.cat([torch.randn(P) * 0.3, torch.zeros(pad)]) X, Y = torch.randn(N, 8, 8), torch.randn(N, 8, 4) # 8 examples per rank lr, b1, b2, eps, wd = 1e-2, 0.9, 0.95, 1e-8, 0.1 def adamw(p, g, m, v, t): # one AdamW step on any slice (formula 4.10) p.mul_(1 - lr * wd) m.mul_(b1).add_(g, alpha=1 - b1); v.mul_(b2).addcmul_(g, g, value=1 - b2) p.addcdiv_(m / (1 - b1 ** t), (v / (1 - b2 ** t)).sqrt().add_(eps), value=-lr) def train(stage, steps=3): sent[:] = [0] * N S = (P + pad) // N # shard length if stage == 0: # DDP: everything replicated p = [theta0.clone() for _ in range(N)]; m = [torch.zeros(P + pad) for _ in range(N)]; v = [torch.zeros(P + pad) for _ in range(N)] else: # ZeRO: rank r owns the optimizer state of shard r p_shard = [theta0[r*S:(r+1)*S].clone() for r in range(N)] m = [torch.zeros(S) for _ in range(N)]; v = [torch.zeros(S) for _ in range(N)] p = all_gather(p_shard) if stage in (1, 2) else None; sent[:] = [0] * N # setup, not counted for t in range(1, steps + 1): if stage == 3: # gather for forward, free, gather again for backward p = all_gather(p_shard); p = all_gather(p_shard) g = [grad_on_rank(p[r], X[r], Y[r]) / N for r in range(N)] if stage == 0: g = all_reduce(g) for r in range(N): adamw(p[r], g[r], m[r], v[r], t) else: g_shard = reduce_scatter(g) # stage 2/3 do this bucket by bucket during backward for r in range(N): adamw(p_shard[r], g_shard[r], m[r], v[r], t) if stage in (1, 2): p = all_gather(p_shard) # stage 3 gathers at the next forward final = p[0] if stage in (0, 1, 2) else torch.cat(p_shard) return final[:P], sent[0] / steps # reference: one process, full batch, torch.optim.AdamW ref = theta0[:P].clone().requires_grad_(True) opt = torch.optim.AdamW([ref], lr=lr, betas=(b1, b2), eps=eps, weight_decay=wd) for t in range(3): opt.zero_grad() loss = sum(torch.nn.functional.mse_loss(forward(torch.cat([ref, torch.zeros(pad)]), X[r]), Y[r]) for r in range(N)) / N loss.backward(); opt.step() for stage in range(4): final, per_step = train(stage) print(f"stage {stage}: max diff vs AdamW = {(final - ref.detach()).abs().max().item():.1e}, " f"elements sent per rank per step = {per_step:.0f} = {per_step / (P + pad):.2f} x params") # stage 0: max diff vs AdamW = 3.0e-08, elements sent per rank per step = 318 = 1.50 x params # stage 1: max diff vs AdamW = 3.0e-08, elements sent per rank per step = 318 = 1.50 x params # stage 2: max diff vs AdamW = 3.0e-08, elements sent per rank per step = 318 = 1.50 x params # stage 3: max diff vs AdamW = 3.0e-08, elements sent per rank per step = 477 = 2.25 x params
Two lessons from the output. First, the four methods produce the same weights (the tiny difference is float rounding): ZeRO changes where numbers live, not the math of training. Second, the counts match formulas (45.6) and (45.7) with \(N = 4\): \(2 \times \tfrac34 = 1.5\) and \(3 \times \tfrac34 = 2.25\).
In practice you do not write this yourself. With DeepSpeed, the stage and the offload options are a few lines of the config file (keys and defaults from the DeepSpeed docs [Certain]):
{
"train_micro_batch_size_per_gpu": 4,
"bf16": { "enabled": true },
"zero_optimization": {
"stage": 3, // 0 = off (DDP), 1, 2 or 3; default 0
"overlap_comm": true, // overlap reduce-scatter with backward; default false
"contiguous_gradients": true, // copy gradients into one buffer to avoid fragmentation
"reduce_bucket_size": 5e8, // elements per gradient bucket (default 5e8)
"stage3_prefetch_bucket_size": 5e8, // how many weights to all-gather ahead
"stage3_param_persistence_threshold": 1e5,// tensors smaller than this stay unsharded
"stage3_max_live_parameters": 1e9, // cap on gathered weights held at once
"stage3_gather_16bit_weights_on_model_save": true,
"offload_optimizer": { "device": "cpu", "pin_memory": true }, // ZeRO-Offload
"offload_param": { "device": "none" } // "cpu" or "nvme": stage 3 only
}
}
(JSON has no comments; remove the // parts before use.) In PyTorch, the same three levels are FSDP strategies: FULL_SHARD is ZeRO-3, SHARD_GRAD_OP is roughly ZeRO-2, and NO_SHARD is DDP. The next session, session 46, shows how PyTorch implements stage 3 with FlatParameters.
Where this is used
| Setting | Typical choice | Why |
|---|---|---|
| Pretraining runs of tens to hundreds of billions of parameters | ZeRO-1 inside Megatron-style 3D parallelism (tensor + pipeline + data), for example BLOOM 176B [Likely] | Tensor and pipeline parallelism already split the weights; ZeRO-1 removes the optimizer-state copies at no extra communication |
| Training or fine-tuning 7B to 70B models on one or a few servers | ZeRO-3 or FSDP FULL_SHARD, often hybrid across servers | Simplest way to fit the model without changing model code |
| Fine-tuning a model that already fits with ZeRO-2 | ZeRO-2 | Same communication as DDP, 8 times less model-state memory at large \(N\) |
| Fine-tuning on one GPU or a workstation | ZeRO-Offload (optimizer to CPU), or ZeRO-Infinity to NVMe | The GPU holds only \(2\Psi\) bytes of weights; slower, but it runs |
| VLM training (for example LLaVA-style instruction tuning) | DeepSpeed ZeRO-2 or ZeRO-3 configs shipped with the training scripts [Likely] | The language model dominates the memory; the vision encoder is often frozen and adds no optimizer states |
| Video-language models (video QA, video retrieval, action recognition, key-moment detection) | ZeRO-3 plus activation checkpointing | Many frames mean long sequences, so activations, not model states, become the limit; ZeRO frees model-state memory for them |
| Hugging Face Trainer and Accelerate | A DeepSpeed JSON or an FSDP config passed on the command line | The same sharding without writing distributed code |
Common mistakes
- Expecting ZeRO to fix an out-of-memory caused by activations. ZeRO-DP shards model states only. If activations dominate (long sequences, video), use activation checkpointing or a smaller micro-batch; with tensor parallelism, ZeRO-R’s partitioned activations.
- Expecting ZeRO-1 or ZeRO-2 to keep scaling. They stop at \(4\Psi\) and \(2\Psi\). A 70B model has 140 GB of BF16 weights: no number of 80 GB GPUs makes ZeRO-2 fit.
- Using ZeRO-3 across a slow network with a tiny batch. Formula (45.9): below a few thousand tokens per GPU at 50 GB/s, the GPUs wait. Use hybrid sharding, larger micro-batches, or ZeRO-2.
- Touching parameters outside the forward pass in stage 3. They are shards or empty placeholders. Gather them first (DeepSpeed
GatheredParameters, FSDPsummon_full_params). - Saving a ZeRO-3 checkpoint and loading it as a normal model. The checkpoint is sharded. Either gather 16-bit weights on save (
stage3_gather_16bit_weights_on_model_save) or convert with DeepSpeed’szero_to_fp32.pyscript. - Computing the gradient norm on the local shard. Clipping needs the global norm; sum the squared shard norms over all ranks.
- Turning on CPU offload without enough host memory. ZeRO-Offload needs about \(14\Psi\) bytes of CPU memory in total (182 GB for 13B), and pinned memory for fast copies. Check the server, not just the GPU.
- Forgetting that gradient accumulation multiplies ZeRO-3 traffic. Every micro-batch needs its own forward and backward all-gathers, so with 8 accumulation steps the weight traffic is 8 times larger per optimizer step, while DDP can skip the all-reduce on all but the last micro-batch.
Interview questions
What does each ZeRO stage shard, and what stays replicated?basic
Give the memory per GPU of each stage. Compute it for a 7B model on 8 GPUs.core
Why can stage 1 save at most 4 times, and stage 2 at most 8 times?core
Why do ZeRO-1 and ZeRO-2 cost no extra communication compared with DDP?core
Why is ZeRO-3 1.5 times the communication of DDP?core
Does ZeRO change the result of training?basic
When is ZeRO-3 communication bound, and what do you do about it?advanced
ZeRO-3 vs tensor parallelism: both split the weights. What is the difference?advanced
What does ZeRO-Offload put on the CPU, and why that split?advanced
What memory does ZeRO not remove, and what does ZeRO-R do about it?core
How do ZeRO stages map to PyTorch FSDP?basic
FULL_SHARD is ZeRO-3 (gather per unit for forward, free, gather again for backward, reduce-scatter gradients). SHARD_GRAD_OP is close to ZeRO-2: weights are sharded between steps but kept gathered from forward to backward, so it sends \(2\Psi\). NO_SHARD is DDP. HYBRID_SHARD is ZeRO-3 inside a node and replication across nodes, similar to ZeRO++ hpZ. FSDP’s CPUOffload moves parameters and gradients to CPU and runs the optimizer there.Exercises
- A 13B model is trained with AdamW in mixed precision on 16 GPUs. Compute the model-state memory per GPU for DDP and the three ZeRO stages.
- You have 64 GPUs with 80 GB each and want to leave 30 GB per GPU for activations. What is the largest model you can train with ZeRO-2? With ZeRO-3?
- Your optimizer is SGD with momentum and an FP32 master copy (\(K = 8\)). Rewrite formulas (45.2) to (45.4). What is the largest saving of stage 1 now?
- A 30B model is trained with ZeRO-3 on 32 GPUs in BF16. How many gigabytes does each GPU send per step? How long does that take at 50 GB/s?
- For ZeRO-2 at 450 GB/s, \(F = 400\) TFLOP/s, \(N = 8\): what is the smallest \(T_g\) that lets communication hide, by formula (45.9)?
- With ZeRO-Offload on one GPU, a 6B model and PCIe at 24 GB/s: GPU memory, CPU memory, traffic and transfer time per step?
Answerscheck
Key takeaways
- DDP stores \(16\Psi\) bytes on every GPU (45.1); \(12\Psi\) of it are optimizer states, used only during the optimizer step. With \(N\) GPUs, \(N - 1\) copies are redundant.
- Stage 1 shards the optimizer states (\(4\Psi + 12\Psi/N\), at most 4×), stage 2 also the gradients (\(2\Psi + 14\Psi/N\), at most 8×), stage 3 also the weights (\(16\Psi/N\), no limit) (45.2 to 45.4).
- Stages 1 and 2 send the same \(2\Psi\) per GPU as DDP: a reduce-scatter plus an all-gather with the optimizer step in between (45.6). Stage 3 sends \(3\Psi\), 1.5× DDP (45.7).
- Communication per GPU does not grow with \(N\), and the condition for hiding it is about tokens per GPU, not model size (45.8, 45.9).
- ZeRO does not shard activations; ZeRO-R adds partitioned activation checkpoints, fixed buffers and defragmentation.
- ZeRO-Offload keeps \(2\Psi\) of weights on the GPU, puts \(14\Psi\) on the CPU, and moves the minimum \(4\Psi\) bytes over PCIe per step (45.10). ZeRO-Infinity adds NVMe. Offload trades time for memory.
Key formulas
References for this session
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Ren et al. (2021). ZeRO-Offload: Democratizing Billion-Scale Model Training
- Rajbhandari et al. (2021). ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
- Wang et al. (2023). ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
- DeepSpeed (docs). DeepSpeed configuration JSON: ZeRO optimizations and offloading
- Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- Micikevicius et al. (2017). Mixed Precision Training
- Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism