LLM & VLM courseBlog
45Track 3: Pretraining and scale · session 15 of 22

ZeRO stages 1, 2 and 3

Data parallelism keeps a full copy of the weights, the gradients and the optimizer states on every GPU, so most of the memory holds the same numbers N times. ZeRO removes this waste in three stages: stage 1 splits the optimizer states across GPUs, stage 2 also splits the gradients, and stage 3 also splits the weights. This session computes the memory per GPU and the communication cost of each stage, and shows how offloading to CPU memory or NVMe disks goes further.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 4, 38, 43 and 44 (AdamW and its 16 bytes per parameter, memory math for training, collective communication, DDP)

What you will learn

  • Why plain data parallelism wastes memory: every GPU stores the same 16 bytes per parameter
  • What stage 1, stage 2 and stage 3 shard, and the memory formula for each (4Ψ + 12Ψ/N, 2Ψ + 14Ψ/N, 16Ψ/N)
  • Why stages 1 and 2 cost no more communication than DDP, and why stage 3 costs 1.5 times as much
  • When communication can hide behind compute, and how many tokens per GPU you need for that
  • What ZeRO does not shard (activations, temporary buffers) and what ZeRO-R adds
  • ZeRO-Offload and ZeRO-Infinity: which states go to CPU memory or NVMe, and why the PCIe traffic is 4Ψ bytes
  • How to read a DeepSpeed ZeRO config, and how ZeRO maps to PyTorch FSDP

Why this matters

In session 44 every GPU keeps a full copy of the model. That is simple and fast, but it means that on 64 GPUs the same optimizer states are stored 64 times. ZeRO (the Zero Redundancy Optimizer) removes this copy-everything waste, one kind of state at a time.

Recall the bill from session 4: mixed-precision training with Adam needs about 16 bytes per parameter before any activation. For Llama 2 7B (6.74 billion parameters) that is 108 GB, more than one 80 GB GPU. With DDP, adding GPUs does not help at all: every GPU still needs 108 GB. ZeRO makes the memory per GPU go down as you add GPUs. With all three stages on 8 GPUs, the same model needs 13.5 GB of model states per GPU.

The ZeRO paper (Rajbhandari et al., 2019) defined three stages, and the names stuck [Certain]:

  • Stage 1 shards (splits into disjoint pieces) the optimizer states.
  • Stage 2 also shards the gradients.
  • Stage 3 also shards the parameters (the weights).

Stages 1 and 2 cost no extra communication compared with DDP. Stage 3 costs 1.5 times as much. Later work moved states to CPU memory and NVMe disks (ZeRO-Offload, ZeRO-Infinity). These ideas are inside DeepSpeed, and PyTorch FSDP (session 46) is an implementation of stage 3. Almost every large training run today uses some form of this sharding.

Where the memory goes in data parallelism

Let \(\Psi\) be the number of parameters. In mixed-precision training with Adam, each GPU in DDP stores these model states (the ZeRO paper’s name for everything that has one number per parameter) [Certain]:

StatePrecisionBytesUsed when
Parameters (weights) for forward and backwardFP16 or BF16\(2\Psi\)forward and backward pass
GradientsFP16 or BF16\(2\Psi\)end of backward, optimizer step
FP32 master copy of the weightsFP32\(4\Psi\)optimizer step only
Adam momentum \(\mathbf{m}\)FP32\(4\Psi\)optimizer step only
Adam variance \(\mathbf{v}\)FP32\(4\Psi\)optimizer step only

The paper groups the last three as the optimizer states and writes their size as \(K\Psi\), with \(K = 12\) for mixed-precision Adam. So the memory per GPU in DDP is

\[ M_{\text{DDP}} = (2 + 2 + K)\,\Psi = 16\,\Psi \ \text{bytes} \tag{45.1} \]
Read the symbols
\(\Psi\)
the number of parameters of the model
\(2 + 2\)
bytes per parameter for the 16-bit weights and the 16-bit gradients
\(K\)
bytes per parameter for the optimizer states; 12 for Adam with an FP32 master copy, \(\mathbf{m}\) and \(\mathbf{v}\)
\(M_{\text{DDP}}\)
model-state memory on each GPU, whatever the number of GPUs

The paper’s own example: GPT-2 with 1.5 billion parameters needs only 3 GB for its 16-bit weights, but 24 GB of model states, so it did not train on a 32 GB GPU once activations were added [Certain]. Three quarters of the bill (\(12\Psi\) of \(16\Psi\)) are optimizer states, which are used only for a short moment at the end of each step.

Now look at what DDP does with \(N\) GPUs. After the all-reduce, every GPU holds the same averaged gradient, and runs the same optimizer step on the same states. \(N - 1\) of the \(N\) copies are pure redundancy. That is the waste ZeRO removes.

The rest of the memory, called residual states in the paper, is activations, temporary buffers and fragmentation. ZeRO stages 1 to 3 do not touch activations; we come back to them in the section on ZeRO-R.

The ZeRO idea: keep one copy, split across GPUs

ZeRO keeps data parallelism: every GPU still gets its own slice of the batch and runs the full forward and backward pass. What changes is storage. Cut each kind of state into \(N\) equal shards. GPU \(r\) owns shard \(r\): it stores it, and it is the only GPU that updates it. When a GPU needs a piece it does not own, it gets it from the owner with a collective (reduce-scatter or all-gather from session 43), uses it, and throws it away.

The three stages differ only in which states are cut. Each step uses the fact that a state is not needed all the time: the optimizer states are needed only in the optimizer step, a gradient only until its shard owner has reduced it, and the weights of a layer only while that layer runs. Step through Figure 1 to see who holds what during one training step.

16-bit weights16-bit gradientsFP32 optimizer statesnot storedheld only for a moment
Figure 1. Who stores what on 4 GPUs during one training step. Each GPU has three rows (weights, gradients, optimizer states), and each row is cut into 4 shards. Choose a method with the buttons, then step through the training step. The readout counts the memory per GPU and the elements each GPU has sent so far.
How to read Figure 1

Start with DDP: all 12 cells of every GPU are full at the end of the step, and one all-reduce sends \(1.5\Psi\) elements per GPU (that is \(2\Psi \cdot 3/4\) with a ring on 4 GPUs). Switch to ZeRO-1: only the purple row shrinks to one cell, yet the communication is the same \(1.5\Psi\), because an all-reduce is a reduce-scatter plus an all-gather, and ZeRO only moves the optimizer step between the two. ZeRO-2 also drops the gradient row to one cell. ZeRO-3 drops the weight row too, but must gather weights twice (forward and backward), so its total is \(2.25\Psi\): 1.5 times DDP.

Stage 1: shard the optimizer states

In stage 1 (the paper calls it \(P_{os}\)), GPU \(r\) keeps the FP32 master weights, \(\mathbf{m}\) and \(\mathbf{v}\) only for its own shard: \(\Psi / N\) parameters. The 16-bit weights and the 16-bit gradients stay full on every GPU. The training step becomes:

Algorithm 1: one training step with ZeRO stage 1 (GPU \(r\) of \(N\))
  1. input: full 16-bit weights \(\theta\), optimizer states for shard \(r\) only, a local micro-batch
  2. forward and backward on the local micro-batch \(\;\to\;\) full local gradient \(\mathbf{g}\) (\(2\Psi\) bytes)
  3. \(\mathbf{g}_r \leftarrow\) reduce-scatter\((\mathbf{g})\) \(/\, N\) // GPU \(r\) gets the averaged gradient of its shard only
  4. update shard \(r\): AdamW on the FP32 master \(\theta^{32}_r\) with \(\mathbf{m}_r, \mathbf{v}_r\) and \(\mathbf{g}_r\)
  5. \(\theta_r \leftarrow \text{to16}(\theta^{32}_r)\)
  6. \(\theta \leftarrow\) all-gather\((\theta_1, \dots, \theta_N)\) // every GPU gets the full updated weights

The memory per GPU is

\[ M_{1} = 2\Psi + 2\Psi + \frac{K\Psi}{N} = 4\Psi + \frac{12\,\Psi}{N} \tag{45.2} \]

As \(N\) grows, the second term goes to zero and the memory goes to \(4\Psi\): at most a 4 times saving compared with \(16\Psi\). The paper’s example: a 7.5B model on 64 GPUs needs 31.4 GB instead of 120 GB [Certain]. Stage 1 is the cheapest stage to turn on: it changes nothing in the forward and backward pass.

Stage 2: also shard the gradients

In stage 1, every GPU still builds the full gradient and then keeps only \(1/N\) of the averaged result. Stage 2 (\(P_{os+g}\)) stops storing what it will not use. During the backward pass, gradients are put into buckets (fixed-size groups of gradients, as in DDP). As soon as a bucket is full, it is reduced to the GPUs that own its parts, and the other GPUs free that memory. So each GPU holds only a small window of full gradients at any moment, plus its own reduced shard.

\[ M_{2} = 2\Psi + \frac{(2 + K)\,\Psi}{N} = 2\Psi + \frac{14\,\Psi}{N} \tag{45.3} \]

The limit is \(2\Psi\), only the 16-bit weights: up to an 8 times saving. The 7.5B model on 64 GPUs: 16.6 GB [Certain]. The extra memory for buckets is fixed (DeepSpeed’s default reduce_bucket_size is \(5 \times 10^8\) elements [Certain]), and the reduction overlaps with the rest of the backward pass, as in DDP.

Stage 2 is still cheap: the forward pass does not change, and the gradient traffic is the same as before, only split into pieces. That is why ZeRO-2 is the most common choice for models that fit with it [Likely].

Stage 3: also shard the parameters

After stage 2, the full 16-bit weights (\(2\Psi\)) are the floor. Stage 3 (\(P_{os+g+p}\)) shards them too. Now GPU \(r\) stores only its \(1/N\) of everything. But the forward pass of layer \(\ell\) needs all of layer \(\ell\)’s weights. So weights are fetched just in time, one layer (or one group of layers) at a time:

Algorithm 2: one training step with ZeRO stage 3 (GPU \(r\) of \(N\))
  1. input: shard \(r\) of the 16-bit weights, of the gradients and of the optimizer states
  2. for layer \(\ell = 1, \dots, L\): // forward
  3. \(W_\ell \leftarrow\) all-gather of the shards of layer \(\ell\) // prefetch layer \(\ell+1\) while \(\ell\) computes
  4. run layer \(\ell\) forward on the local micro-batch, keep its activations
  5. free \(W_\ell\) except the own shard
  6. for layer \(\ell = L, \dots, 1\): // backward
  7. \(W_\ell \leftarrow\) all-gather again
  8. compute the input gradient and the local weight gradient \(\mathbf{g}_\ell\)
  9. \(\mathbf{g}_{\ell, r} \leftarrow\) reduce-scatter\((\mathbf{g}_\ell) / N\); free \(W_\ell\) and \(\mathbf{g}_\ell\)
  10. AdamW on shard \(r\) only (FP32 master, \(\mathbf{m}_r\), \(\mathbf{v}_r\)); write the new 16-bit shard // no all-gather here: the next forward gathers
\[ M_{3} = \frac{(2 + 2 + K)\,\Psi}{N} = \frac{16\,\Psi}{N} \tag{45.4} \]

Now the saving grows without limit: double \(N\), halve the memory. The 7.5B model on 64 GPUs needs 1.88 GB of model states; a 1-trillion-parameter model on 1,024 GPUs needs 15.6 GB [Certain]. The cost: the weights of one layer must be gathered twice per step, and each GPU needs room for at least one full layer (two, with prefetching) while it runs. For Llama 2 7B, one block has 202 million parameters (\(4 \cdot 4096^2 + 3 \cdot 4096 \cdot 11008\)), so one gathered block is 0.40 GB in BF16.

Stage 3 also changes how your code sees the model. Outside the forward pass, a parameter on GPU \(r\) holds only its shard (in DeepSpeed, an empty placeholder). Reading or changing full weights outside the step (for example, to initialize, inspect or save them) needs an explicit gather, such as DeepSpeed’s GatheredParameters context [Certain].

Algorithm 2 and the formulas above hide one thing: memory is not constant during the step. Watch it happen in Figure 2. It runs a real training step on 4 GPUs with a tiny 3-layer model of \(\Psi = 12\) parameters (4 per layer). Each layer multiplies its input element by element by its weights, \(\mathbf{h}_\ell = W_\ell \odot \mathbf{h}_{\ell-1}\), and the loss is \(\tfrac12 \lVert \mathbf{h}_3 - \mathbf{y} \rVert^2\). The global batch has 4 examples, and each GPU gets \(4/N\) of them. Every cell shows the number that GPU really holds. Every collective really moves numbers between the GPU panels, and the AdamW step really runs (learning rate 0.1, \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), weight decay 0.01). Bytes are counted with the 16 bytes per parameter of formula (45.1).

When ZeRO-3 gathers a layer, the GPU writes it into a separate buffer for the full layer and keeps its own shard as well. In the same way, the gradient of the layer being computed sits in a full-size buffer until the reduce-scatter. So for a short time a GPU holds more than \(16\Psi/N\). Here \(\Psi_\ell\) is the number of parameters in one layer and \(a\) is the number of layers gathered at the same time (1, or 2 with prefetching). In this toy, which counts bytes the same way, the peak is

\[ M_3^{\text{peak}} = \frac{16\,\Psi}{N} + 2\,a\,\Psi_\ell + 2\,\Psi_\ell \]

The last term is the full gradient of the layer, before the reduce-scatter. With \(N = 4\) and \(\Psi_\ell = 4\), this gives \(48 + 8 + 8 = 64\) bytes without prefetching and 72 bytes with it. ZeRO-2 has only the gradient term, so its peak is \(66 + 8 = 74\) bytes.

GPUs
Watch GPU
16-bit weights (own)gathered for a momentgradientsfull layer gradient, held for a momentFP32 master, \(m\), \(v\) (cell shows the master)
  1. for \(\ell = 1, \dots, L\): // forward
  2. Z3 \(W_\ell \leftarrow\) all-gather(shards of layer \(\ell\)) // prefetch: also \(\ell+1\)
  3. \(\mathbf{h}_\ell \leftarrow W_\ell \odot \mathbf{h}_{\ell-1}\) for each local example
  4. Z3 free \(W_\ell\) except the own shard
  5. \(\mathbf{e} \leftarrow \mathbf{h}_L - \mathbf{y}\); loss \(\tfrac12 \lVert \mathbf{e} \rVert^2\); \(\delta_L \leftarrow \mathbf{e}\)
  6. for \(\ell = L, \dots, 1\): // backward
  7. Z3 \(W_\ell \leftarrow\) all-gather again
  8. \(\mathbf{g}_\ell \leftarrow \delta_\ell \odot \mathbf{h}_{\ell-1}\); \(\delta_{\ell-1} \leftarrow \delta_\ell \odot W_\ell\)
  9. DDP \(\mathbf{g}_\ell \leftarrow\) all-reduce\((\mathbf{g}_\ell) / N\)
  10. Z2Z3 \(\mathbf{g}_{\ell,r} \leftarrow\) reduce-scatter\((\mathbf{g}_\ell)/N\); free the rest
  11. Z1 \(\mathbf{g}_r \leftarrow\) reduce-scatter\((\mathbf{g})/N\)
  12. AdamW on shard \(r\) (DDP: on all of \(\theta\))
  13. Z1Z2 \(\theta \leftarrow\) all-gather\((\theta_1, \dots, \theta_N)\)
Figure 2. Two real training steps of a 3-layer toy model on \(N\) GPUs. Each GPU panel shows its three layers of weights, gradients and optimizer states as cells with the numbers it really holds, and a memory bar: bytes held now, a red tick at the peak so far, and a dashed tick at the steady-state formula (45.1) to (45.4). The pseudo-code highlights the current line; the panel on the right puts the live numbers into the formula of that line. Lines that do not belong to the chosen method are greyed out.
How to read Figure 2

Start with ZeRO-3 and press Step. At “all-gather layer 1” the first weight row of every GPU fills up: the pale cells came from the other GPUs, and the solid cell is the GPU’s own shard. The memory bar jumps from 48 to 56 bytes, and after “free” it falls back. In the backward pass the full gradient row appears in pale orange, and the reduce-scatter shrinks it to one averaged number per GPU. The formula panel shows that number as the mean of the 4 local gradients. The peak marker stops at 64 bytes, as the peak formula says; tick Prefetch and it stops at 72. Now switch to DDP: the bar stays at 192 bytes, and every GPU runs the same AdamW update on all 12 numbers. Try ZeRO-1 and ZeRO-2: right after the optimizer step only the own weight shard is new, and the last all-gather brings in the rest. The final frame compares the weights with DDP: they are exactly equal for every method and for both \(N\). ZeRO changes where numbers live, not what is computed. The counter of sent elements ends at \(1.5\Psi = 18\) for DDP, ZeRO-1 and ZeRO-2, and at \(2.25\Psi = 27\) for ZeRO-3 (with \(N = 4\)).

Memory per GPU for each stage

Formulas (45.1) to (45.4) side by side, for \(K = 12\):

MethodWhat is shardedBytes per GPULimit as \(N \to \infty\)7.5B model, \(N = 64\)
DDPnothing\(16\Psi\)\(16\Psi\)120 GB
ZeRO-1optimizer states\(4\Psi + 12\Psi/N\)\(4\Psi\) (4×)31.4 GB
ZeRO-2+ gradients\(2\Psi + 14\Psi/N\)\(2\Psi\) (8×)16.6 GB
ZeRO-3+ parameters\(16\Psi/N\)\(\to 0\) (\(N\)×)1.88 GB

The last column is from Table 1 of the ZeRO paper, and the formulas give the same numbers [Certain]. Turn the formulas around to get the largest model whose states fit into \(M\) bytes per GPU:

\[ \Psi_{\max}^{(1)} = \frac{M}{4 + 12/N}, \qquad \Psi_{\max}^{(2)} = \frac{M}{2 + 14/N}, \qquad \Psi_{\max}^{(3)} = \frac{N\,M}{16} \tag{45.5} \]

With \(M = 80\) GB and \(N = 8\): DDP fits 5.0B parameters, ZeRO-1 14.5B, ZeRO-2 21.3B, and ZeRO-3 40B. In real runs you must leave room for activations, buckets and fragmentation, so the true limits are lower. Try your own model in Figure 3.

Model:
16-bit weights16-bit gradientsFP32 optimizer states
Figure 3. Model-state memory per GPU, formulas (45.1) to (45.4), for mixed-precision AdamW (\(K = 12\)). The dashed line is the GPU memory. Activations are not included. The panel under the bars writes each formula with the current \(\Psi\) and \(N\) put in; press Play to sweep \(N\) from 1 to 1,024 and watch which terms shrink.
How to read Figure 3

Press Play first: \(N\) doubles at each tick, and in the panel below the bars only the terms divided by \(N\) get smaller. At \(N = 1\) all four methods give the same \(16\Psi\). Pick “7.5B (paper)” and move \(N\) to 64: the bars read 120, 31.4, 16.6 and 1.88 GB, the numbers in the ZeRO paper. Now drag \(N\) up to 1,024. The ZeRO-1 bar stops near 30 GB (the blue and orange parts never shrink) and the ZeRO-2 bar stops near 15 GB (the blue part never shrinks). Only the ZeRO-3 bar keeps falling. Then pick 70B with \(N = 8\): nothing but ZeRO-3 fits, and with 140 GB of weights alone, ZeRO-2 cannot fit on any number of 80 GB GPUs.

Worked example: Llama 2 7B on one 8-GPU server

\(\Psi = 6.738 \times 10^9\), \(N = 8\), \(K = 12\).

DDP: \(16 \times 6.738 = 107.8\) GB. Does not fit in 80 GB.

ZeRO-1: \(4 \times 6.738 + 12 \times 6.738 / 8 = 26.95 + 10.11 = 37.1\) GB.

ZeRO-2: \(2 \times 6.738 + 14 \times 6.738 / 8 = 13.48 + 11.79 = 25.3\) GB.

ZeRO-3: \(16 \times 6.738 / 8 = 13.5\) GB, plus room for one or two gathered blocks (0.40 GB each).

So ZeRO-1 already makes full fine-tuning of a 7B model possible on 8 GPUs of 80 GB, with about 40 GB left for activations; ZeRO-2 leaves about 55 GB.

Communication cost vs memory saved

Memory is only half the story. Every byte we do not store must be sent at the right moment. From session 43: with a ring on \(N\) GPUs, a reduce-scatter or an all-gather of a vector of \(\Psi\) elements makes each GPU send \(\frac{N-1}{N}\Psi\) elements, and an all-reduce (which is a reduce-scatter followed by an all-gather) sends twice that.

DDP: one all-reduce of the gradients. ZeRO-1 and ZeRO-2: a reduce-scatter of the gradients, then an all-gather of the updated weights. That is exactly the two halves of the all-reduce, with the optimizer step moved into the middle. So the volume is the same [Certain]:

\[ V_{\text{DDP}} = V_{1} = V_{2} = 2\,\frac{N-1}{N}\,\Psi \approx 2\Psi \ \text{elements per GPU per step} \tag{45.6} \]

ZeRO-3: an all-gather of the weights for the forward pass, another for the backward pass (the weights were freed in between), and a reduce-scatter of the gradients. No all-gather after the optimizer step, because the next forward pass does it. So

\[ V_{3} = 3\,\frac{N-1}{N}\,\Psi \approx 3\Psi = 1.5 \times V_{\text{DDP}} \tag{45.7} \]
Read the symbols
\(V\)
elements sent by each GPU in one training step; multiply by the bytes per element (2 for BF16) to get bytes
\(\frac{N-1}{N}\)
the ring factor: each GPU sends all shards but its own; close to 1 for large \(N\)

Notice that the volume per GPU does not grow with \(N\). It depends only on the model size. This is the key reason ZeRO scales: memory per GPU falls with \(N\), but communication per GPU stays about the same.

Can the communication hide behind compute?

The collectives run on their own CUDA stream and overlap with compute (prefetching in the forward pass, bucketed reduce-scatter in the backward pass). So what matters is whether communication time is shorter than compute time. With \(T_g\) tokens per GPU per step, a training cost of \(6\Psi\) FLOPs per token (session 25), an achieved speed of \(F\) FLOP/s, \(b\) bytes per element and a bandwidth of \(\beta\) bytes/s per GPU:

\[ t_{\text{compute}} = \frac{6\,\Psi\,T_g}{F}, \qquad t_{\text{comm}} = \frac{b \cdot c\,\frac{N-1}{N}\,\Psi}{\beta}, \quad c = 2 \ (\text{DDP, ZeRO-1, 2}),\ c = 3 \ (\text{ZeRO-3}) \tag{45.8} \]

Both grow with \(\Psi\), so \(\Psi\) cancels when you ask “is \(t_{\text{comm}} \le t_{\text{compute}}\)?”. Communication can hide when

\[ T_g \;\ge\; \frac{c\, b\, F}{6\, \beta} \cdot \frac{N-1}{N} \tag{45.9} \]

That is a rule about tokens per GPU, not about model size. With BF16 (\(b = 2\)), \(F = 400\) TFLOP/s and 8 GPUs: inside one server (\(\beta = 450\) GB/s over NVLink) ZeRO-3 needs only about 780 tokens per GPU. Across servers (\(\beta = 50\) GB/s over InfiniBand) it needs about 7,000 tokens per GPU, and DDP or ZeRO-2 about 4,700.

Worked example: ZeRO-3 for Llama 2 7B across servers

Each GPU sends \(3 \times \tfrac{7}{8} \times 6.738 \times 10^9 \times 2\) bytes \(= 35.4\) GB per step (DDP or ZeRO-2: 23.6 GB). At 50 GB/s that is 0.71 s (DDP: 0.47 s).

With 8 sequences of 4,096 tokens per GPU (\(T_g = 32{,}768\)): \(t_{\text{compute}} = 6 \times 6.738 \times 10^9 \times 32{,}768 / (400 \times 10^{12}) = 3.31\) s. Communication hides easily.

With one sequence (\(T_g = 4{,}096\)): \(t_{\text{compute}} = 0.41\) s, less than 0.71 s. Now ZeRO-3 is communication bound and the GPUs wait 0.29 s per step (even DDP waits 0.06 s). This is the setting where ZeRO++ and hybrid sharding help (below).

Link:
compute (forward + backward)communicationexposed communication (GPU waits)
Figure 4. Compute time and communication time per step from formula (45.8), with BF16 communication and perfect overlap. When the orange bar is longer than the blue bar, the difference (red) is time the GPU waits.
How to read Figure 4

The default is the worked example: Llama 2 7B, 8 GPUs, 32,768 tokens per GPU, 50 GB/s. Everything hides. Now move the tokens down to 4,096: all bars turn red, but ZeRO-3 waits about 5 times longer (0.29 s against 0.06 s), because it sends 1.5 times as much. Move the parameter slider: both bars grow together and the red part keeps the same sign, as formula (45.9) says. Change \(N\): almost nothing happens, because \((N-1)/N\) is close to 1. Switch to NVLink and the problem almost disappears. In real runs, overlap is not perfect (the first all-gather and the last reduce-scatter cannot hide), so treat the red part as a lower bound.

Ways to pay less for stage 3

  • Bigger micro-batches raise \(T_g\) and move you to the safe side of (45.9). Activation checkpointing (session 39) frees the memory for them.
  • Hybrid sharding: shard inside a server (fast NVLink), replicate across servers (only gradients cross the slow network). PyTorch calls this HYBRID_SHARD (session 46); ZeRO++ calls its version hpZ, set by zero_hpz_partition_size [Certain].
  • ZeRO++ (Wang et al., 2023) also quantizes the weight all-gather and the gradient reduction. The paper reports a 4 times smaller communication volume and up to 2.16 times higher throughput on 384 GPUs [Certain].
  • Do not over-shard. If ZeRO-2 fits, it sends less than ZeRO-3 and keeps the code simpler.

Beyond model states: ZeRO-R

Stages 1 to 3 (together called ZeRO-DP in the paper) handle model states. The paper adds ZeRO-R for the residual states [Certain]:

  • Activations. Data parallelism already splits activations by batch, so ZeRO-DP cannot shrink them further. The paper’s GPT-2 1.5B example needs about 60 GB of activations at sequence length 1K and batch 32, and about 8 GB with activation checkpointing [Certain]. With tensor (model) parallelism, every GPU in the group stores the same checkpointed activations; ZeRO-R’s partitioned activation checkpointing (\(P_a\)) splits them across the group and can also move them to CPU (\(P_{a+cpu}\)).
  • Constant-size buffers. Fusing all gradients into one FP32 buffer for an all-reduce would need 6 GB for a 1.5B model [Certain]. ZeRO uses buffers of a fixed size instead (the buckets above), so the buffer does not grow with the model.
  • Memory defragmentation. Short-lived tensors (recomputed activations, gathered weights) interleave with long-lived ones and leave holes. The paper saw out-of-memory errors with over 30% of memory still free in extreme cases [Certain]. ZeRO-R moves long-lived tensors into pre-allocated contiguous memory.

Offloading to CPU or NVMe

A GPU server has much more CPU memory (often 1 to 2 TB) than GPU memory, and NVMe disks hold even more [Likely]. The catch is bandwidth: the link between CPU and GPU (PCIe) is about 32 GB/s per direction for Gen4 x16, more than 10 times slower than GPU memory. Offloading moves some states off the GPU and must keep PCIe traffic small.

ZeRO-Offload: which state goes where

ZeRO-Offload (Ren et al., 2021) treats one training step as a data-flow graph with four parts: the forward-backward computation (must stay on the GPU, it is the heavy compute), the 16-bit weights \(p_{16}\), the 16-bit gradients \(g_{16}\), and an “update” node that holds the FP32 master weights, \(\mathbf{m}\) and \(\mathbf{v}\) and runs Adam. Every edge cut between GPU and CPU costs PCIe traffic [Certain]:

  • forward-backward reads \(p_{16}\) twice (forward and backward): \(4\Psi\) bytes if they are on different devices;
  • forward-backward writes \(g_{16}\): \(2\Psi\) bytes; the update reads \(g_{16}\): \(2\Psi\) bytes;
  • the update writes new \(p_{16}\): \(2\Psi\) bytes.

Any split must cut at least two edges, so at least \(4\Psi\) bytes per step. Keeping \(p_{16}\) on the GPU and moving \(g_{16}\) and the update to the CPU reaches this minimum and saves the most GPU memory. That is ZeRO-Offload’s choice [Certain]:

\[ M_{\text{GPU}} = 2\Psi, \qquad M_{\text{CPU}} = (2 + K)\,\Psi = 14\,\Psi, \qquad V_{\text{PCIe}} = 4\Psi \ \text{bytes per step} \tag{45.10} \]

The optimizer step now runs on the CPU. To make that fast, ZeRO-Offload has a vectorized CPU Adam, reported as up to 6 times faster than the PyTorch CPU optimizer, and an optional one-step delayed parameter update (DPU): the CPU updates with the gradients of step \(i - 1\) while the GPU computes step \(i\). DPU changes the math slightly, so it is turned on only after the first few dozen steps [Certain]. With several GPUs, ZeRO-Offload runs on top of ZeRO-2: each GPU offloads only its own shard, so the CPU memory and the PCIe traffic per GPU are divided by \(N\) [Likely]. Find the best placement yourself in Figure 5.

Presets:
Figure 5. The ZeRO-Offload data-flow graph on one GPU. Click (or Tab to and press Enter on) the weights, gradients or update node to move it between GPU and CPU. Edges that cross the PCIe link are drawn thick and orange; their weights add up to the PCIe traffic per step.
How to read Figure 5

Start with “All on GPU”: \(16\Psi\) bytes on the GPU, no traffic. Move only the update node to the CPU: the GPU drops to \(4\Psi\) (4 times less) for \(4\Psi\) of traffic. Move the gradients too: \(2\Psi\) on the GPU (8 times less), still \(4\Psi\) of traffic. That is the ZeRO-Offload preset. Now move the weights to the CPU as well: the GPU memory for model states goes to zero, but the traffic jumps to \(6\Psi\), because the forward and backward pass read the weights twice. With the default 13B model, ZeRO-Offload needs 26 GB on the GPU, 182 GB on the CPU, and 52 GB over PCIe per step (1.63 s at 32 GB/s).

Is 1.63 s per step acceptable? It depends on the batch. With \(T_g\) tokens per step, the GPU computes for \(6\Psi T_g / F\) seconds, and the PCIe time is \(4\Psi/\beta_{\text{PCIe}}\). The ratio again does not depend on \(\Psi\). At \(F = 100\) TFLOP/s and 32 GB/s, the transfer hides behind compute once \(T_g \ge 4 \times 100 \times 10^{12} / (6 \times 32 \times 10^9) \approx 2{,}100\) tokens per step. The ZeRO-Offload paper reports training models with over 13 billion parameters on a single GPU, 10 times larger than PyTorch alone, and about 40 TFLOP/s on one V100 for a 10B model [Certain].

Stage 3 offload and ZeRO-Infinity

With stage 3, the 16-bit weights can be offloaded too (offload_param). Each layer’s weights then travel from CPU to GPU just in time, the same schedule as the all-gather of Algorithm 2. This is the “everything on CPU” corner of Figure 5: traffic grows to about \(6\Psi\) bytes per step, but the GPU keeps only a working set of a few layers.

ZeRO-Infinity (Rajbhandari et al., 2021) adds NVMe disks as a third tier for both parameters and optimizer states, and splits large layers into tiles so that even one layer does not have to fit on a GPU. It also spreads the PCIe and NVMe traffic over all GPUs and disks of a server in parallel, instead of one link at a time. The paper reports fine-tuning a trillion-parameter model on a single DGX-2 node (16 V100 GPUs) and over 25 PFLOP/s on 512 GPUs [Certain].

In DeepSpeed, the rules are [Certain]: offload_optimizer works with stages 1, 2 and 3, to "cpu" or "nvme", but NVMe only with stage 3; offload_param works only with stage 3. The optimizer step always runs on the CPU when the optimizer is offloaded, even with NVMe.

ZeRO in code

The code below is a complete simulation of DDP and ZeRO stages 1, 2 and 3 in one process. “Ranks” are list entries, the collectives are written by hand and count the elements each rank sends. It checks that all four methods give the same weights as torch.optim.AdamW on the full batch, and prints the communication volume. It runs on a CPU in about a second.

import torch
torch.manual_seed(0)
N = 4                                   # number of simulated GPUs (ranks)
sent = [0] * N                          # elements sent by each rank

def reduce_scatter(full_list):          # every rank has a full vector; rank r gets the sum of chunk r
    chunks = [v.chunk(N) for v in full_list]
    for r in range(N): sent[r] += (N - 1) * chunks[r][0].numel()   # ring: (N-1)/N of the vector
    return [sum(chunks[s][r] for s in range(N)) for r in range(N)]

def all_gather(shards):                 # every rank has one shard; all ranks get the full vector
    for r in range(N): sent[r] += (N - 1) * shards[r].numel()
    full = torch.cat(shards)
    return [full.clone() for _ in range(N)]

def all_reduce(full_list):              # = reduce-scatter followed by all-gather
    return all_gather(reduce_scatter(full_list))

# a tiny model as one flat parameter vector: Linear(8,16) -> tanh -> Linear(16,4)
shapes = [(16, 8), (16,), (4, 16), (4,)]
sizes = [torch.Size(s).numel() for s in shapes]
P = sum(sizes)                          # 212 numbers; pad to a multiple of N
pad = (-P) % N
def forward(flat, x):
    w1, b1, w2, b2 = [t.view(s) for t, s in zip(flat[:P].split(sizes), shapes)]
    return torch.tanh(x @ w1.T + b1) @ w2.T + b2

def grad_on_rank(flat, x, y):           # local backward on this rank's micro-batch
    flat = flat.detach().requires_grad_(True)
    loss = torch.nn.functional.mse_loss(forward(flat, x), y)
    loss.backward()
    return flat.grad

theta0 = torch.cat([torch.randn(P) * 0.3, torch.zeros(pad)])
X, Y = torch.randn(N, 8, 8), torch.randn(N, 8, 4)     # 8 examples per rank
lr, b1, b2, eps, wd = 1e-2, 0.9, 0.95, 1e-8, 0.1

def adamw(p, g, m, v, t):               # one AdamW step on any slice (formula 4.10)
    p.mul_(1 - lr * wd)
    m.mul_(b1).add_(g, alpha=1 - b1); v.mul_(b2).addcmul_(g, g, value=1 - b2)
    p.addcdiv_(m / (1 - b1 ** t), (v / (1 - b2 ** t)).sqrt().add_(eps), value=-lr)

def train(stage, steps=3):
    sent[:] = [0] * N
    S = (P + pad) // N                  # shard length
    if stage == 0:                      # DDP: everything replicated
        p = [theta0.clone() for _ in range(N)]; m = [torch.zeros(P + pad) for _ in range(N)]; v = [torch.zeros(P + pad) for _ in range(N)]
    else:                               # ZeRO: rank r owns the optimizer state of shard r
        p_shard = [theta0[r*S:(r+1)*S].clone() for r in range(N)]
        m = [torch.zeros(S) for _ in range(N)]; v = [torch.zeros(S) for _ in range(N)]
        p = all_gather(p_shard) if stage in (1, 2) else None; sent[:] = [0] * N   # setup, not counted
    for t in range(1, steps + 1):
        if stage == 3:                  # gather for forward, free, gather again for backward
            p = all_gather(p_shard); p = all_gather(p_shard)
        g = [grad_on_rank(p[r], X[r], Y[r]) / N for r in range(N)]
        if stage == 0:
            g = all_reduce(g)
            for r in range(N): adamw(p[r], g[r], m[r], v[r], t)
        else:
            g_shard = reduce_scatter(g)  # stage 2/3 do this bucket by bucket during backward
            for r in range(N): adamw(p_shard[r], g_shard[r], m[r], v[r], t)
            if stage in (1, 2): p = all_gather(p_shard)   # stage 3 gathers at the next forward
    final = p[0] if stage in (0, 1, 2) else torch.cat(p_shard)
    return final[:P], sent[0] / steps

# reference: one process, full batch, torch.optim.AdamW
ref = theta0[:P].clone().requires_grad_(True)
opt = torch.optim.AdamW([ref], lr=lr, betas=(b1, b2), eps=eps, weight_decay=wd)
for t in range(3):
    opt.zero_grad()
    loss = sum(torch.nn.functional.mse_loss(forward(torch.cat([ref, torch.zeros(pad)]), X[r]), Y[r]) for r in range(N)) / N
    loss.backward(); opt.step()

for stage in range(4):
    final, per_step = train(stage)
    print(f"stage {stage}: max diff vs AdamW = {(final - ref.detach()).abs().max().item():.1e}, "
          f"elements sent per rank per step = {per_step:.0f} = {per_step / (P + pad):.2f} x params")
# stage 0: max diff vs AdamW = 3.0e-08, elements sent per rank per step = 318 = 1.50 x params
# stage 1: max diff vs AdamW = 3.0e-08, elements sent per rank per step = 318 = 1.50 x params
# stage 2: max diff vs AdamW = 3.0e-08, elements sent per rank per step = 318 = 1.50 x params
# stage 3: max diff vs AdamW = 3.0e-08, elements sent per rank per step = 477 = 2.25 x params

Two lessons from the output. First, the four methods produce the same weights (the tiny difference is float rounding): ZeRO changes where numbers live, not the math of training. Second, the counts match formulas (45.6) and (45.7) with \(N = 4\): \(2 \times \tfrac34 = 1.5\) and \(3 \times \tfrac34 = 2.25\).

In practice you do not write this yourself. With DeepSpeed, the stage and the offload options are a few lines of the config file (keys and defaults from the DeepSpeed docs [Certain]):

{
  "train_micro_batch_size_per_gpu": 4,
  "bf16": { "enabled": true },
  "zero_optimization": {
    "stage": 3,                              // 0 = off (DDP), 1, 2 or 3; default 0
    "overlap_comm": true,                     // overlap reduce-scatter with backward; default false
    "contiguous_gradients": true,             // copy gradients into one buffer to avoid fragmentation
    "reduce_bucket_size": 5e8,                // elements per gradient bucket (default 5e8)
    "stage3_prefetch_bucket_size": 5e8,       // how many weights to all-gather ahead
    "stage3_param_persistence_threshold": 1e5,// tensors smaller than this stay unsharded
    "stage3_max_live_parameters": 1e9,        // cap on gathered weights held at once
    "stage3_gather_16bit_weights_on_model_save": true,
    "offload_optimizer": { "device": "cpu", "pin_memory": true },  // ZeRO-Offload
    "offload_param":     { "device": "none" }                       // "cpu" or "nvme": stage 3 only
  }
}

(JSON has no comments; remove the // parts before use.) In PyTorch, the same three levels are FSDP strategies: FULL_SHARD is ZeRO-3, SHARD_GRAD_OP is roughly ZeRO-2, and NO_SHARD is DDP. The next session, session 46, shows how PyTorch implements stage 3 with FlatParameters.

Where this is used

SettingTypical choiceWhy
Pretraining runs of tens to hundreds of billions of parametersZeRO-1 inside Megatron-style 3D parallelism (tensor + pipeline + data), for example BLOOM 176B [Likely]Tensor and pipeline parallelism already split the weights; ZeRO-1 removes the optimizer-state copies at no extra communication
Training or fine-tuning 7B to 70B models on one or a few serversZeRO-3 or FSDP FULL_SHARD, often hybrid across serversSimplest way to fit the model without changing model code
Fine-tuning a model that already fits with ZeRO-2ZeRO-2Same communication as DDP, 8 times less model-state memory at large \(N\)
Fine-tuning on one GPU or a workstationZeRO-Offload (optimizer to CPU), or ZeRO-Infinity to NVMeThe GPU holds only \(2\Psi\) bytes of weights; slower, but it runs
VLM training (for example LLaVA-style instruction tuning)DeepSpeed ZeRO-2 or ZeRO-3 configs shipped with the training scripts [Likely]The language model dominates the memory; the vision encoder is often frozen and adds no optimizer states
Video-language models (video QA, video retrieval, action recognition, key-moment detection)ZeRO-3 plus activation checkpointingMany frames mean long sequences, so activations, not model states, become the limit; ZeRO frees model-state memory for them
Hugging Face Trainer and AccelerateA DeepSpeed JSON or an FSDP config passed on the command lineThe same sharding without writing distributed code

Common mistakes

  • Expecting ZeRO to fix an out-of-memory caused by activations. ZeRO-DP shards model states only. If activations dominate (long sequences, video), use activation checkpointing or a smaller micro-batch; with tensor parallelism, ZeRO-R’s partitioned activations.
  • Expecting ZeRO-1 or ZeRO-2 to keep scaling. They stop at \(4\Psi\) and \(2\Psi\). A 70B model has 140 GB of BF16 weights: no number of 80 GB GPUs makes ZeRO-2 fit.
  • Using ZeRO-3 across a slow network with a tiny batch. Formula (45.9): below a few thousand tokens per GPU at 50 GB/s, the GPUs wait. Use hybrid sharding, larger micro-batches, or ZeRO-2.
  • Touching parameters outside the forward pass in stage 3. They are shards or empty placeholders. Gather them first (DeepSpeed GatheredParameters, FSDP summon_full_params).
  • Saving a ZeRO-3 checkpoint and loading it as a normal model. The checkpoint is sharded. Either gather 16-bit weights on save (stage3_gather_16bit_weights_on_model_save) or convert with DeepSpeed’s zero_to_fp32.py script.
  • Computing the gradient norm on the local shard. Clipping needs the global norm; sum the squared shard norms over all ranks.
  • Turning on CPU offload without enough host memory. ZeRO-Offload needs about \(14\Psi\) bytes of CPU memory in total (182 GB for 13B), and pinned memory for fast copies. Check the server, not just the GPU.
  • Forgetting that gradient accumulation multiplies ZeRO-3 traffic. Every micro-batch needs its own forward and backward all-gathers, so with 8 accumulation steps the weight traffic is 8 times larger per optimizer step, while DDP can skip the all-reduce on all but the last micro-batch.

Interview questions

What does each ZeRO stage shard, and what stays replicated?basic
Stage 1 shards the optimizer states (FP32 master weights, Adam \(\mathbf{m}\) and \(\mathbf{v}\)); the 16-bit weights and gradients stay full on every GPU. Stage 2 also shards the gradients; the 16-bit weights stay full. Stage 3 shards the weights too, so nothing is fully replicated between steps. In all stages, each GPU still processes its own slice of the batch (data parallelism), and activations are not sharded.
Give the memory per GPU of each stage. Compute it for a 7B model on 8 GPUs.core
With \(K = 12\): DDP \(16\Psi\), ZeRO-1 \(4\Psi + 12\Psi/N\), ZeRO-2 \(2\Psi + 14\Psi/N\), ZeRO-3 \(16\Psi/N\). For \(\Psi = 7 \times 10^9\), \(N = 8\): 112 GB, 38.5 GB, 26.25 GB and 14 GB. (For Llama 2 7B with 6.738B parameters: 107.8, 37.1, 25.3 and 13.5 GB.) Add activations, buckets and, for ZeRO-3, one or two gathered layers.
Why can stage 1 save at most 4 times, and stage 2 at most 8 times?core
Stage 1 still stores the full 16-bit weights and gradients, \(4\Psi\) bytes, on every GPU; only the \(12\Psi\) of optimizer states is divided by \(N\). As \(N \to \infty\) the memory goes to \(4\Psi\), and \(16/4 = 4\). Stage 2 still stores the full 16-bit weights, \(2\Psi\), so the limit is \(16/2 = 8\). Only stage 3 divides everything by \(N\).
Why do ZeRO-1 and ZeRO-2 cost no extra communication compared with DDP?core
A ring all-reduce is a reduce-scatter followed by an all-gather, \(2\Psi\) elements per GPU in total. ZeRO-1 and ZeRO-2 split it in the middle: reduce-scatter the gradients (each GPU gets the averaged shard it owns), run the optimizer on that shard, then all-gather the updated weights instead of the gradients. Same two collectives, same volume; only what is gathered changes. Stage 2 just does the reduce-scatter bucket by bucket during backward and frees the rest.
Why is ZeRO-3 1.5 times the communication of DDP?core
The weights are sharded, so each layer must be all-gathered before its forward pass (\(\Psi\) in total) and, because it was freed to save memory, all-gathered again before its backward pass (\(\Psi\)). The gradients are reduce-scattered (\(\Psi\)). No all-gather is needed after the optimizer step because the next forward pass does it. Total \(3\Psi\) against \(2\Psi\) for DDP. (Keeping the weights from forward to backward, as FSDP’s SHARD_GRAD_OP does, gets back to \(2\Psi\) at the cost of memory.)
Does ZeRO change the result of training?basic
No. Each GPU computes the same gradient on its micro-batch, the averaged gradient is the same, and AdamW is element-wise, so updating each shard separately gives the same weights as updating the whole vector. Only the order of floating-point additions in the reduction can differ, giving tiny rounding differences. The exception is ZeRO-Offload’s optional delayed parameter update, which uses gradients one step old.
When is ZeRO-3 communication bound, and what do you do about it?advanced
Compute per step is \(6\Psi T_g/F\) and communication is about \(3 \cdot 2\Psi/\beta\) bytes in BF16, so \(\Psi\) cancels and the condition is about tokens per GPU: \(T_g \ge F b c/(6\beta)\). At 400 TFLOP/s and 50 GB/s that is about 7,000 tokens per GPU (8 GPUs: \(\times 7/8\)). Below it, GPUs wait. Fixes: larger micro-batches (with activation checkpointing), hybrid sharding so weight gathers stay on NVLink (FSDP HYBRID_SHARD, ZeRO++ hpZ), quantized communication (ZeRO++ reports 4 times less volume), prefetching tuned to overlap, or ZeRO-2 if it fits.
ZeRO-3 vs tensor parallelism: both split the weights. What is the difference?advanced
Tensor parallelism splits the computation of each layer: each GPU multiplies with its slice of the matrix, and activations are exchanged (all-reduce) inside every layer, in the critical path, so it needs very fast links and stays inside a server. ZeRO-3 splits only the storage: before computing, each GPU gathers the full layer and computes it alone on its own data. Its traffic is weights and gradients, independent of batch size, and it can be prefetched and overlapped. ZeRO-3 needs no model code changes; tensor parallelism also cuts activation memory and helps when a single layer is too big or the batch per GPU is too small. Large runs combine them.
What does ZeRO-Offload put on the CPU, and why that split?advanced
It keeps the 16-bit weights and the forward and backward pass on the GPU, and moves the 16-bit gradients, the FP32 master weights, \(\mathbf{m}\), \(\mathbf{v}\) and the Adam step to the CPU. In the step’s data-flow graph, any GPU/CPU split cuts at least two edges of at least \(2\Psi\) bytes, so \(4\Psi\) bytes per step is the minimum. This split reaches it (gradients down, new weights up) and leaves only \(2\Psi\) on the GPU, an 8 times saving. Moving the weights too would cost \(6\Psi\), because forward and backward read them twice. A fast vectorized CPU Adam and optional one-step delayed updates hide the CPU time.
What memory does ZeRO not remove, and what does ZeRO-R do about it?core
Residual states: activations, temporary buffers and fragmentation. Data parallelism already splits activations by batch, so ZeRO-DP cannot shrink them. ZeRO-R adds partitioned activation checkpointing (split the checkpoints of a tensor-parallel group across its GPUs, optionally to CPU), constant-size fused buffers instead of one buffer as large as the model (6 GB in FP32 for 1.5B), and memory defragmentation by placing long-lived tensors in pre-allocated contiguous memory.
How do ZeRO stages map to PyTorch FSDP?basic
FSDP FULL_SHARD is ZeRO-3 (gather per unit for forward, free, gather again for backward, reduce-scatter gradients). SHARD_GRAD_OP is close to ZeRO-2: weights are sharded between steps but kept gathered from forward to backward, so it sends \(2\Psi\). NO_SHARD is DDP. HYBRID_SHARD is ZeRO-3 inside a node and replication across nodes, similar to ZeRO++ hpZ. FSDP’s CPUOffload moves parameters and gradients to CPU and runs the optimizer there.

Exercises

  1. A 13B model is trained with AdamW in mixed precision on 16 GPUs. Compute the model-state memory per GPU for DDP and the three ZeRO stages.
  2. You have 64 GPUs with 80 GB each and want to leave 30 GB per GPU for activations. What is the largest model you can train with ZeRO-2? With ZeRO-3?
  3. Your optimizer is SGD with momentum and an FP32 master copy (\(K = 8\)). Rewrite formulas (45.2) to (45.4). What is the largest saving of stage 1 now?
  4. A 30B model is trained with ZeRO-3 on 32 GPUs in BF16. How many gigabytes does each GPU send per step? How long does that take at 50 GB/s?
  5. For ZeRO-2 at 450 GB/s, \(F = 400\) TFLOP/s, \(N = 8\): what is the smallest \(T_g\) that lets communication hide, by formula (45.9)?
  6. With ZeRO-Offload on one GPU, a 6B model and PCIe at 24 GB/s: GPU memory, CPU memory, traffic and transfer time per step?
Answerscheck
1. \(\Psi = 13 \times 10^9\), \(N = 16\): DDP \(16 \times 13 = 208\) GB; ZeRO-1 \(52 + 156/16 = 61.75\) GB; ZeRO-2 \(26 + 182/16 = 37.4\) GB; ZeRO-3 \(208/16 = 13\) GB. 2. \(M = 50\) GB. ZeRO-2: \(\Psi = 50/(2 + 14/64) = 22.5\)B. ZeRO-3: \(\Psi = 64 \times 50/16 = 200\)B. 3. ZeRO-1: \(4\Psi + 8\Psi/N\); ZeRO-2: \(2\Psi + 10\Psi/N\); ZeRO-3: \(12\Psi/N\). Baseline is \(12\Psi\), so stage 1 saves at most \(12/4 = 3\) times. 4. \(3 \times \tfrac{31}{32} \times 30 \times 10^9 \times 2 = 174.4\) GB; \(174.4/50 = 3.49\) s, which needs at least about \(3.49 \times 400 \times 10^{12} / (6 \times 30 \times 10^9) \approx 7{,}750\) tokens per GPU at 400 TFLOP/s to hide. 5. \(c = 2\), \(b = 2\): \(2 \times 2 \times 400 \times 10^{12} / (6 \times 450 \times 10^9) \times 7/8 = 518.5\), so about 520 tokens per GPU. 6. GPU \(2 \times 6 = 12\) GB; CPU \(14 \times 6 = 84\) GB; traffic \(4 \times 6 = 24\) GB; time \(24/24 = 1.0\) s per step.

Key takeaways

  • DDP stores \(16\Psi\) bytes on every GPU (45.1); \(12\Psi\) of it are optimizer states, used only during the optimizer step. With \(N\) GPUs, \(N - 1\) copies are redundant.
  • Stage 1 shards the optimizer states (\(4\Psi + 12\Psi/N\), at most 4×), stage 2 also the gradients (\(2\Psi + 14\Psi/N\), at most 8×), stage 3 also the weights (\(16\Psi/N\), no limit) (45.2 to 45.4).
  • Stages 1 and 2 send the same \(2\Psi\) per GPU as DDP: a reduce-scatter plus an all-gather with the optimizer step in between (45.6). Stage 3 sends \(3\Psi\), 1.5× DDP (45.7).
  • Communication per GPU does not grow with \(N\), and the condition for hiding it is about tokens per GPU, not model size (45.8, 45.9).
  • ZeRO does not shard activations; ZeRO-R adds partitioned activation checkpoints, fixed buffers and defragmentation.
  • ZeRO-Offload keeps \(2\Psi\) of weights on the GPU, puts \(14\Psi\) on the CPU, and moves the minimum \(4\Psi\) bytes over PCIe per step (45.10). ZeRO-Infinity adds NVMe. Offload trades time for memory.

Key formulas

\[ M_{\text{DDP}} = 16\Psi, \quad M_1 = 4\Psi + \frac{12\Psi}{N}, \quad M_2 = 2\Psi + \frac{14\Psi}{N}, \quad M_3 = \frac{16\Psi}{N} \]
\[ V_{\text{DDP}} = V_1 = V_2 = 2\tfrac{N-1}{N}\Psi, \quad V_3 = 3\tfrac{N-1}{N}\Psi, \quad T_g \ge \frac{c\,b\,F}{6\beta}\cdot\frac{N-1}{N} \]
\[ \text{ZeRO-Offload:}\quad M_{\text{GPU}} = 2\Psi, \quad M_{\text{CPU}} = 14\Psi, \quad V_{\text{PCIe}} = 4\Psi \]

References for this session

  1. Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  2. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  3. Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  4. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  5. Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
  6. Ren et al. (2021). ZeRO-Offload: Democratizing Billion-Scale Model Training
  7. Rajbhandari et al. (2021). ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
  8. Wang et al. (2023). ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
  9. DeepSpeed (docs). DeepSpeed configuration JSON: ZeRO optimizations and offloading
  10. Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
  11. Micikevicius et al. (2017). Mixed Precision Training
  12. Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism