LLM & VLM courseBlog
44Track 3: Pretraining and scale · session 14 of 22

Data parallelism (DDP)

Data parallelism is the first and simplest way to train on many GPUs: every GPU keeps a full copy of the model, works on its own slice of the batch, and the GPUs average their gradients before each update. This session shows why that gives exactly the same update as one big GPU, how PyTorch DDP hides the averaging behind the backward pass with gradient buckets, how the data is split and seeded on each rank, and why the method stops working when the model and its optimizer no longer fit on one GPU.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 4, 39 and 43 (optimization, gradient accumulation, collective communication)

What you will learn

  • Why averaging the gradients of N equal shards gives exactly the gradient of the full batch, and why the replicas then stay identical
  • What one DDP training step does: processes and ranks, the broadcast at start-up, the DistributedSampler, the backward hooks and the optimizer step
  • How gradient buckets work in PyTorch (25 MiB cap, 1 MiB first bucket, rebuilt after the first iteration), and how they overlap the all-reduce with the backward pass
  • When the backward pass can hide the all-reduce: a rule that depends only on tokens per GPU, GPU speed and network bandwidth
  • Global batch size, the linear scaling rule, and gradient accumulation with no_sync
  • The limit of DDP: 16 bytes per parameter on every GPU, so a 7B model does not fit on an 80 GB GPU, however many GPUs you add
  • Seeds and determinism across ranks: what must be the same on every rank, what must differ, and why changing the number of GPUs changes the last bits of the result

Why this matters

Pretraining a modern LLM needs about \(10^{24}\) floating-point operations or more. One GPU would take decades. The only way to finish in weeks is to use thousands of GPUs at once. The first and simplest way to do that is data parallelism: give every GPU a full copy of the model, give each GPU a different slice of the data, and let the GPUs agree on one update at every step.

Data parallelism is in every large training run. Even when a model is too big for one GPU and is split with tensor or pipeline parallelism, the outermost layer of the setup is almost always data parallel: many groups of GPUs, each holding one copy of the (split) model, each reading different data. [Certain] So the ideas of this session, averaging gradients, overlapping communication with the backward pass, splitting the data correctly and keeping the copies identical, come back in every later session of this track.

In session 43 we built the tool: the ring all-reduce, which leaves every GPU with the sum of everyone’s data and sends about twice the buffer size per GPU, whatever the number of GPUs. In this session we use it. PyTorch’s DistributedDataParallel (DDP) is the standard implementation, and its paper (Li et al., 2020) reports near-linear scaling up to 256 GPUs when it is configured well. [Certain] We will see what “configured well” means, and where the method hits a wall: every GPU must hold the whole model, its gradients and its optimizer state. That wall is the reason for session 45 and session 46.

The idea: copy the model, split the batch, average the gradients

Start with what one GPU does in one training step (see session 4). It takes a batch of \(B\) examples, computes the loss as the mean of the per-example losses, runs the backward pass to get the gradient, and lets the optimizer update the weights.

Now take \(N\) GPUs. Some words first. Each GPU runs its own copy of the training program, called a process, and each process has a number called its rank, from 0 to \(N-1\). \(N\) is the world size. In data parallelism, each rank holds a full copy of the model, called a replica. A training step then has four parts:

  1. Split. The global batch of \(B\) examples is cut into \(N\) shards (or local batches) of \(b = B/N\) examples. Rank \(r\) gets shard \(r\).
  2. Compute locally. Each rank runs the forward and backward pass on its own shard and gets a local gradient \(\mathbf{g}_r\).
  3. Average. The ranks run an all-reduce on the gradients and divide by \(N\). Now every rank holds the same averaged gradient.
  4. Update. Each rank runs the same optimizer step on the same gradient. The replicas change in the same way and stay identical.

Why is step 3 correct? Write \(\ell_i(\mathbf{w})\) for the loss of example \(i\) with weights \(\mathbf{w}\). The loss of the global batch is the mean over all \(B\) examples. Group the examples by shard:

\[ L(\mathbf{w}) = \frac{1}{B}\sum_{i=1}^{B} \ell_i(\mathbf{w}) = \frac{1}{N}\sum_{r=0}^{N-1} \underbrace{\frac{1}{b}\sum_{i \in S_r} \ell_i(\mathbf{w})}_{L_r(\mathbf{w})} \tag{44.1} \]

The gradient is linear, so the gradient of a mean is the mean of the gradients:

\[ \nabla L(\mathbf{w}) = \frac{1}{N}\sum_{r=0}^{N-1} \nabla L_r(\mathbf{w}) = \frac{1}{N}\sum_{r=0}^{N-1} \mathbf{g}_r \tag{44.2} \]
Read the symbols
\(B\), \(b\), \(N\)
global batch size, local batch size per rank, and number of ranks, with \(B = N b\)
\(S_r\)
the set of examples in the shard of rank \(r\); the shards do not overlap and together cover the batch
\(L_r\)
the local loss on rank \(r\): the mean loss over its own \(b\) examples, exactly what a normal training loop computes
\(\mathbf{g}_r\)
the local gradient on rank \(r\), what loss.backward() produces before any communication
\(\frac{1}{N}\sum_r\)
an all-reduce with sum, followed by a division by \(N\)

So the averaged gradient is exactly the gradient that one GPU would compute on the whole batch of \(B\) examples (up to floating-point rounding, which we discuss at the end). Data parallelism does not approximate anything. It is the same algorithm as large-batch training on one huge GPU. [Certain]

Two conditions hide in (44.1). First, the shards must have equal size. If rank 0 has 10 examples and rank 1 has 6, the plain average of the two local means gives the examples of rank 1 too much weight. The correct formula weights each local gradient by its shard size:

\[ \nabla L(\mathbf{w}) = \frac{\sum_r b_r\, \mathbf{g}_r}{\sum_r b_r} \tag{44.3} \]

DDP always divides by \(N\), so it assumes equal \(b_r\). This matters for LLMs when the loss is a mean over tokens and different ranks have different numbers of non-padding tokens; we return to it in the mistakes section. Second, all replicas must start from the same weights. If they start equal and get the same gradient and the same optimizer step, they stay equal after every step, by induction. If they start different, they stay different forever, because they all receive the same correction.

Figure 1 runs this on a toy model with one weight. Step through it, then try the three bugs.

weight of the replicalocal gradientaveraged gradient (after all-reduce)
Figure 1. One DDP step on four ranks for the model \(\hat y = w x\) with loss \(\tfrac12 (w x - y)^2\), mean over the two examples of each rank, and plain SGD with learning rate 0.1. The bottom row is one GPU that sees all eight examples. The readout also shows where each replica is after 20 steps.
How to read Figure 1

Each row is one rank with its weight \(w\), its two examples \((x, y)\), and its local gradient. In step 1 the ranks start from different random weights. Step 2 broadcasts rank 0’s \(w = 0.5\) to all ranks. Step 3 gives four different local gradients: −2.75, −5.5, −14 and −4.75. Step 4 is the all-reduce: the sum −27, divided by 4, is −6.75, exactly the gradient the single GPU gets on all eight examples. Step 5 updates every replica to \(0.5 - 0.1 \cdot (-6.75) = 1.175\), the same as the single GPU. Now pick a bug. Without the broadcast, the four weights get the same corrections but keep their different offsets forever. Without the division, the step is four times too large and the training blows up (here the effective learning rate 0.4 times the curvature 6 is above 2). Without the all-reduce, each replica fits only its own data and the copies drift apart.

Worked example: the numbers of Figure 1

Rank 0 has the examples \((1, 2)\) and \((2, 3)\). With \(w = 0.5\) its local gradient is the mean of \((w x - y)\,x\): \(\tfrac12[(0.5 - 2)\cdot 1 + (1 - 3)\cdot 2] = \tfrac12(-1.5 - 4) = -2.75\). The other ranks get −5.5, −14 and −4.75. The all-reduce returns the sum −27 to every rank; divided by \(N = 4\) it is −6.75.

One GPU with all eight examples computes \(\tfrac18 \sum (w x - y)\,x = -54/8 = -6.75\). The same number, as (44.2) promised. After one SGD step every replica holds \(w = 1.175\); after 20 steps all four hold 1.625, which is the least-squares answer \(\sum xy / \sum x^2 = 78/48\).

One DDP training step in PyTorch

PyTorch’s DDP turns the four parts above into a wrapper around your model. You still write a normal training loop; the communication happens inside backward(). Here is what happens, in order.

Start-up. A launcher such as torchrun starts one process per GPU and gives each one environment variables: RANK (global rank), LOCAL_RANK (the GPU index inside the server), WORLD_SIZE, and MASTER_ADDR and MASTER_PORT (where rank 0 listens, so the processes can find each other). [Certain] Each process calls init_process_group("nccl") and picks its GPU with LOCAL_RANK.

Wrapping. When you call DDP(model), it does two things. It broadcasts the parameters and buffers of rank 0 to all other ranks, so every replica starts from the same weights even if each process initialized its model differently. [Certain] And it creates a Reducer: an object that puts the parameters into gradient buckets (next section) and registers one autograd hook per parameter. A hook is a function that autograd calls when the gradient of that parameter is ready. [Certain]

Forward. A normal forward pass on the local shard. If the model has buffers (for example the running statistics of BatchNorm), DDP broadcasts rank 0’s buffers to all ranks at the start of each forward pass by default (broadcast_buffers=True). [Certain] Transformers with LayerNorm or RMSNorm have no such statistics, so this costs almost nothing for LLMs.

Backward. As gradients become ready, the hooks fill the buckets. When a bucket is full, the Reducer starts an asynchronous all-reduce for it and the backward pass keeps going. At the end of the backward pass, the Reducer waits for all buckets and writes the averaged gradients into param.grad. [Certain]

Optimizer step. The optimizer is a normal local optimizer. It does not know about the other ranks. It does not need to: every rank has the same weights, the same gradients and the same optimizer state, so it computes the same update. DDP never sends weights after start-up; only gradients. [Certain]

Algorithm 1: training with DDP, the program that runs on rank \(r\) of \(N\)
  1. init_process_group("nccl"); select GPU LOCAL_RANK
  2. build the model; \(\text{model} \leftarrow \text{DDP}(\text{model})\) // broadcast rank 0's weights, build buckets, register hooks
  3. build the optimizer on model.parameters() // a normal local optimizer
  4. \(\text{sampler} \leftarrow\) DistributedSampler(data, num_replicas=N, rank=r, seed=s) // same seed \(s\) on every rank
  5. for each epoch \(e\): sampler.set_epoch(e)
  6.   for each local batch \((x, y)\) of size \(b\) from the sampler:
  7.     \(\text{loss} \leftarrow\) mean loss of \(\text{model}(x)\) against \(y\) // the local loss \(L_r\)
  8.     loss.backward() // hooks all-reduce the buckets while the backward pass runs
  9.     // here every rank holds .grad \(= \frac{1}{N} \sum_r \mathbf{g}_r\)
  10.     optional: clip the gradient norm // same gradients, so the same clipping on every rank
  11.     optimizer.step(); optimizer.zero_grad()
  12.   only rank 0 logs and saves checkpoints // but every rank must still join every collective

Why not DataParallel?

PyTorch also has an older wrapper, nn.DataParallel. It uses one process with one thread per GPU. In every forward pass it copies the model from GPU 0 to the other GPUs, splits the input, runs the copies, and gathers the outputs back on GPU 0, where the loss is computed. This has three problems: the Python threads compete for the global interpreter lock (GIL), GPU 0 does extra work and holds extra memory, and it works only inside one machine. The PyTorch documentation recommends DDP instead, even on a single machine. [Certain]

Splitting the data: the DistributedSampler

Each rank must see different examples in a step, and together the ranks should cover the dataset once per epoch. PyTorch’s DistributedSampler does this with a simple rule. At the start of an epoch, every rank builds the same random permutation of all \(n\) indices, using a generator seeded with seed + epoch. Then rank \(r\) takes every \(N\)-th index, starting at position \(r\). [Certain, checked in the PyTorch 2.8 source]

If \(n\) is not a multiple of \(N\), there are two options. By default (drop_last=False), the sampler pads the list by repeating indices from its start, so a few examples are seen twice in that epoch. With drop_last=True, it cuts the tail instead, so a few examples are not seen in that epoch. Either way every rank gets exactly \(\lceil n/N \rceil\) or \(\lfloor n/N \rfloor\) indices, so all ranks run the same number of steps. [Certain] This is important: if one rank ran one step more, it would wait in an all-reduce that nobody else joins, and the job would hang.

Two rules follow, and they are about seeds. The sampler seed must be the same on every rank, because each rank builds the permutation by itself and they must agree on it. And you must call sampler.set_epoch(epoch) before each epoch, or the permutation is the same in every epoch. Figure 2 shows both.

index given to a rankpadding: an index repeated to make the split evenexample seen once this epochseen twice or never
Figure 2. The DistributedSampler rule: one shared permutation per epoch, rank \(r\) takes positions \(r, r + N, r + 2N, \dots\). The permutation here comes from a small seeded generator in the page, not from torch.randperm, so the exact order differs from PyTorch; the rule and the counts are the same.
How to read Figure 2

The top rows show the indices each rank reads in this epoch, in order. The bottom strip shows every example of the dataset and how many times it is read in total. With \(n = 10\) and \(N = 4\), each rank gets 3 indices; 12 slots for 10 examples, so 2 indices are padding and 2 examples are read twice. Tick drop_last: each rank gets 2, and 2 examples are skipped. Move the epoch slider: the permutation changes. Untick “call set_epoch”: the epoch slider no longer changes anything, so the model sees the data in the same order every epoch. Untick “same sampler seed”: each rank now uses its own permutation; several examples are read twice and others never, even with no padding at all.

LLM pretraining usually does not use a map-style dataset with a sampler. The data is a long stream of pre-tokenized documents, cut into fixed-length sequences (see session 34). The same rule applies, though: one global order that every rank can compute from a shared seed and the step number, and rank \(r\) reads its own disjoint part of it. The data loader state (the position in the stream) must be saved in the checkpoint so that a restarted job does not repeat or skip data. [Likely]

Gradient bucketing and overlap with the backward pass

The simplest DDP would wait for the whole backward pass and then run one all-reduce over all gradients. That wastes time: the network is idle during the backward pass, and the GPUs are idle during the all-reduce. But the gradients do not all appear at the end. The backward pass goes from the last layer to the first, so the gradients of the last layer are ready long before those of the first. DDP uses this. [Certain]

The Reducer groups the parameters into buckets. A bucket is one flat buffer that holds the gradients of several parameters next to each other. When the last gradient of a bucket arrives, the whole bucket is all-reduced as one message, asynchronously, on a separate CUDA stream, while the backward pass continues on the compute stream. [Certain] Why buckets and not one all-reduce per parameter? A transformer has hundreds of parameter tensors, many tiny (biases, norm weights). Each all-reduce pays a fixed latency of \(2(N-1)\alpha\) (formula 43.4), and small messages use the links badly. Buckets turn hundreds of small messages into a few large ones. [Certain]

Figure 3 runs one whole DDP step, with real numbers, on a tiny network. It has four layers, each with one weight \(w_l\) and one bias \(b_l\), so 8 parameters in total: \(h_l = \tanh(w_l h_{l-1} + b_l)\) for \(l = 1, 2, 3\), and \(\hat y = w_4 h_3 + b_4\), with the loss \(\tfrac12(\hat y - y)^2\) averaged over the shard. Eight examples are split over the ranks. The page really runs the forward and backward pass on every rank, fills the buckets, averages them and updates the weights. The times are made-up units that look like a real step: 1 unit per layer for the forward pass, 2 units per layer for the backward pass (it costs about twice the forward), and 1 unit for the optimizer step. Each bucket’s all-reduce takes the ring time of formula (43.4), \(2(N-1)\alpha + 2\frac{N-1}{N}\cdot\frac{S}{\beta}\), with \(\alpha = 0.1\) unit and \(S/\beta\) set by the network slider (the time to send one layer’s gradients once).

forward / weightsbackward / local gradientall-reduce / averaged gradientoptimizer stepGPU idle, waiting for the network
  1. for \(l = 1, \dots, 4\): \(h_l \leftarrow \tanh(w_l h_{l-1} + b_l)\) // forward, own shard
  2. \(L_r \leftarrow \frac{1}{b} \sum_i \tfrac12 (\hat y_i - y_i)^2\) // local loss (44.1)
  3. for \(l = 4, \dots, 1\): \(g_r \leftarrow \partial L_r / \partial (w_l, b_l)\) // backward
  4. hook: copy \(g_r\) into its bucket
  5. if bucket full: launch async all_reduce // network stream
  6. wait for all buckets; \(\bar g \leftarrow \text{bucket} / N\) // (44.2)
  7. \(w \leftarrow w - \eta\, \bar g,\ \ b \leftarrow b - \eta\, \bar g\) // same on every rank
Figure 3. One DDP step, computed live. Top: each rank (row) with its shard of the 8 examples and its copy of the four layers; each layer box shows the weight and bias, and then the gradient (orange: local, green: after the all-reduce). Middle: the buckets, filled in backward order (layer 4 first). Bottom: the timeline of one step. All ranks run in lockstep, so one compute row stands for all of them; the network row is the separate communication stream. Learning rate \(\eta = 0.1\). The time units are a toy cost model, not a measurement.
How to read Figure 3

Press Play, or use Step to move one event at a time; the box under the code shows the line being run with the current numbers. With 4 ranks, 1 layer per bucket and network 1: the forward pass ends at time 4, and each rank has a different local loss (0.397, 0.488, 0.358, 0.537). The backward pass starts at layer 4. At time 6 its gradients are ready: the four local gradients of \(w_4\) are 0.383, 0.541, 0.371 and 0.502. Bucket 0 is full, so its all-reduce starts at once, while the backward pass goes on with layer 3. It takes \(2 \cdot 3 \cdot 0.1 + 1.5 \cdot 1 = 2.1\) units. At 8.1 every rank holds the same \(\bar g = (0.383 + 0.541 + 0.371 + 0.502)/4 = 0.449\), exactly the gradient one GPU computes on all 8 examples. The backward pass ends at 12, the last bucket at 14.4, so the GPUs wait 2.4 units (red), then all apply the same update (\(w_4\): 0.8 → 0.755). The step takes 15.4 units; untick overlap and it takes 21.4. Move the network slider to 3: now the network is the bottleneck (27.4 against 33.4). Choose one bucket: nothing can start before the backward pass ends, so overlap gives no gain (19.6 at network 1).

Two things in Figure 3 are worth saying in words. First, the all-reduce of a bucket cannot start before its last gradient is ready, and the buckets go through the network one after another, in the same order on every rank. So when the network is slower than the backward pass, buckets queue up and the GPU waits at the end. Second, the bucket that holds the first layers is ready only when the backward pass ends. Its all-reduce is the tail, which no overlap can hide.

The details, as of PyTorch 2.8 (checked in the source):

  • Size. The cap is bucket_cap_mb = 25, which is 25 MiB (\(25 \cdot 2^{20}\) bytes). A bucket is closed when it reaches the cap, so it can be larger than the cap when one big tensor is added last. [Certain]
  • Order. At construction, parameters are put into buckets in roughly the reverse order of model.parameters(), because that is roughly the order in which the backward pass produces their gradients. [Certain]
  • The first iteration. With the default settings, the first iteration uses one single bucket for all gradients, because the real order is not known yet. During that iteration DDP records the order in which gradients actually became ready and then rebuilds the buckets in that order. [Certain]
  • A small first bucket. In the rebuilt layout, the first bucket has a cap of only 1 MiB, so that the first all-reduce starts as early as possible. [Certain]
  • Same order on every rank. Buckets are all-reduced in bucket index order, on every rank. If ranks disagreed on the order, collectives would be matched with the wrong partners: wrong results or a hang. [Certain]

Figure 4 shows the real layout for a GPT-2 small model (124,439,808 parameters, FP32 gradients). We checked it with PyTorch: after the first iteration, DDP makes 13 buckets: a first bucket of 9.0 MiB (the final norm plus one 9 MiB weight matrix, which already passes the 1 MiB cap), eleven buckets of about 27 MiB, roughly one transformer block each, and a last bucket of 168.3 MiB that holds the token embedding table. GPT-2 ties the embedding with the output layer, so the embedding gradient is complete only at the very end of the backward pass, when the input embedding has been processed.

Figure 4. The 148 parameter tensors of GPT-2 small in the order their gradients become ready (left: final norm and block 11; right: block 0 and the embeddings), cut into buckets by PyTorch’s rule. Each colored box is one bucket; its width is its size. Press Play to run the backward pass over them. The bottom timeline is the step on one of 8 GPUs: the compute stream (forward, then backward) and the network stream (one block per bucket all-reduce, in bucket order, each starting when its bucket is full and the previous one is done). Cost model, an assumption and not a measurement: an H100 at 40% of peak (\(F = 3.96 \times 10^{14}\) FLOP/s), forward \(2\Psi T/F\), backward \(4 \cdot\) (parameters of the tensor) \(\cdot\, T/F\) per weight tensor, with the tied output layer computed first and the embedding lookups taken as free; attention-score FLOPs are ignored. All-reduce times use the ring model (43.4) with \(\alpha = 20\) µs.
How to read Figure 4

The top strip shows the parameters: the wide gray block on the right is the 147 MiB embedding table; each transformer block is about 27 MiB. The colored boxes below are buckets. At the default cap of 25 MiB there are 13. The readout gives the number of buckets, the total time all of them would take on the network, the part that is pure latency, and the tail: the all-reduce of the last bucket, which can only start when the backward pass ends and so can never be hidden. Lower the cap to 1 MiB: 50 buckets, more latency, but the tail stays large because the embedding table is one tensor and cannot be split. Raise it to 500 MiB: only 2 buckets, and most of the communication waits until the end. Tick “first iteration”: everything is one bucket, nothing overlaps.

Now press Play at the defaults (cap 25 MiB, InfiniBand, \(T = 8{,}192\) tokens per GPU). The forward pass takes 5.1 ms and the backward pass 10.2 ms. The first buckets start their all-reduce early, but on 50 GB/s all 13 together need 21.1 ms on the network, more than the backward pass, so they queue up: the step ends at 29.6 ms, with 14.2 ms of exposed communication (red). Without overlap it would be 36.4 ms. Switch to NVLink: the network needs only 5.6 ms, everything except the tail hides under the backward pass, and the step takes 16.3 ms, of which 1.0 ms is the tail. Back on InfiniBand, move \(T\) up to 32,768: the backward pass grows to 40.9 ms and covers all buckets except the last one; only the 6.5 ms tail is left. This is rule (44.5) at work. Note the tail: the last bucket holds the tied embedding table, 147 MiB, which is ready only at the very end.

The time of one step with overlap is then about

\[ T_{\text{step}} \approx T_{\text{fwd}} + \max\!\big(T_{\text{bwd}},\ T_{\text{comm}}\big) + T_{\text{tail}} \tag{44.4} \]
Read the symbols
\(T_{\text{fwd}}\), \(T_{\text{bwd}}\)
time of the forward and backward pass on the local batch
\(T_{\text{comm}}\)
total time of all bucket all-reduces, from formula (43.4) for each bucket
\(T_{\text{tail}}\)
the communication that cannot overlap: mainly the all-reduce of the last bucket

This is the same formula as (43.8) in the last session, which also has an interactive timeline of the overlap. The new question here is: when is \(T_{\text{comm}}\) smaller than \(T_{\text{bwd}}\)? The answer is a short calculation.

When can the backward pass hide the all-reduce?

For a transformer with \(\Psi\) parameters, the backward pass costs about \(4\Psi\) FLOPs per token (twice the forward pass of \(2\Psi\), see session 25). With \(T\) tokens per GPU per step and a GPU that really delivers \(F\) FLOP/s, the backward pass takes \(4\Psi T / F\). The all-reduce of BF16 gradients (\(2\Psi\) bytes) sends \(2 \cdot \frac{N-1}{N} \cdot 2\Psi \approx 4\Psi\) bytes per GPU, so it takes about \(4\Psi/\beta\). The communication fits under the backward pass when

\[ \frac{4\Psi T}{F} \gt \frac{4\Psi}{\beta} \quad\Longleftrightarrow\quad T \gt \frac{F}{\beta} \tag{44.5} \]
Read the symbols
\(T\)
tokens processed by one GPU in one step (local batch size times sequence length)
\(F\)
the FLOP/s one GPU really reaches (peak times utilization)
\(\beta\)
the bandwidth per GPU of the slowest link of the ring, in bytes per second
\(\Psi\)
the number of parameters; it cancels

The model size cancels. Both the work and the bytes grow with \(\Psi\), so what matters is how many tokens each GPU processes per byte of gradient it sends.

Worked example: is DDP communication-bound?

An H100 has 989 TFLOP/s of dense BF16 compute; assume training reaches 40% of it, so \(F \approx 3.96 \times 10^{14}\) FLOP/s. Across servers, each GPU has one 50 GB/s InfiniBand port. Then \(F/\beta = 3.96 \times 10^{14} / 5 \times 10^{10} \approx 7{,}900\) tokens. With sequences of 4,096 tokens, each GPU needs a local batch of at least 2 sequences per step for the all-reduce to hide under the backward pass. Inside one server over NVLink (450 GB/s), \(F/\beta \approx 880\) tokens: almost any batch is enough.

For Llama 2 7B (6.74 billion parameters), the BF16 gradients are 13.5 GB. On 64 GPUs across servers in a flat ring at 50 GB/s, one all-reduce takes about \(2 \cdot \tfrac{63}{64} \cdot 13.5 / 50 = 0.53\) s. With \(T = 8{,}192\) tokens per GPU, the backward pass takes \(4 \cdot 6.74\times10^9 \cdot 8192 / 3.96\times10^{14} = 0.56\) s. Just enough. Halve the local batch and the step becomes communication-bound. (And as the next sections show, plain DDP cannot even hold this model in 80 GB.)

Rule (44.5) explains a real tension. Many GPUs and a fixed global batch mean fewer tokens per GPU, and at some point the all-reduce can no longer be hidden. This is one reason large runs use a very large global batch, and one reason the fastest links (NVLink) are used for the most communication-heavy parallelism, while data parallelism, which communicates once per step, goes across servers. [Likely]

Options that change the buckets

  • gradient_as_bucket_view=True: by default DDP keeps the gradients in param.grad and a copy in the bucket buffers. With this flag, param.grad becomes a view into the bucket, which saves memory equal to the total gradient size and avoids a copy. [Certain]
  • find_unused_parameters=True: a bucket is all-reduced only when all its gradients are ready. If some parameter is not used in a forward pass (a branch that was skipped, a frozen head), its gradient never arrives and the backward pass waits forever. This flag makes DDP walk the autograd graph after each forward pass and mark unused parameters as ready. It costs time, so use it only when needed. [Certain]
  • static_graph=True: tells DDP that the set of used parameters and the graph do not change between iterations. DDP can then optimize, and it supports cases such as activation checkpointing that runs a block twice (see session 39). [Certain]
  • Communication hooks: register_comm_hook replaces the all-reduce of each bucket with your own function, for example one that casts the gradients to BF16 before sending them and back afterwards, which halves the bytes. [Certain]

Global batch, learning rate and gradient accumulation

With DDP the global batch is what the optimizer sees:

\[ B = N \cdot b \cdot k \tag{44.6} \]
Read the symbols
\(N\)
number of data-parallel ranks
\(b\)
micro-batch size: examples (or sequences) per rank per forward and backward pass
\(k\)
gradient accumulation steps: forward and backward passes per optimizer step

So adding GPUs while keeping \(b\) and \(k\) fixed makes the global batch larger, and a larger batch changes the training. With a larger batch, the gradient estimate is less noisy, so you can take larger steps. For SGD, Goyal et al. (2017) proposed the linear scaling rule: when the batch grows by a factor \(c\), multiply the learning rate by \(c\), with a short warmup at the start. With it they trained ResNet-50 on ImageNet with a batch of 8,192 on 256 GPUs in one hour, with the same accuracy as a small batch. [Certain]

\[ \eta(B) = \eta_0 \cdot \frac{B}{B_0} \tag{44.7} \]

The rule fails beyond some batch size. McCandlish et al. (2018) call it the critical batch size: below it, doubling the batch roughly halves the number of steps; above it, a larger batch gives almost no fewer steps, so the extra GPUs are wasted. [Certain] For Adam, the learning rate should grow more slowly with the batch; analysis of the training as a stochastic differential equation suggests a square-root rule, \(\eta \propto \sqrt{B}\). [Likely] In LLM practice, the global batch is a tuned hyperparameter, set in tokens (often a few million per step) and sometimes increased during training (see session 36). The number of GPUs is then chosen to fit the batch, not the other way round.

Gradient accumulation without extra communication: no_sync

If the global batch must stay fixed but memory allows only a small micro-batch, you accumulate gradients over \(k\) micro-batches before one optimizer step (session 39). A naive DDP loop would all-reduce after every micro-batch: \(k\) times the communication for the same result, because the sum of averages equals the average of sums. DDP has a context manager for this. Inside with model.no_sync():, the backward pass only accumulates into the local .grad; the first backward pass outside it all-reduces the accumulated gradients. [Certain] The forward pass must also be inside the context, or the gradients are still synchronized. [Certain]

Algorithm 2: one optimizer step with \(k\) micro-batches per rank
  1. for \(j = 1, \dots, k-1\):
  2.   with model.no_sync(): \(\big(\text{loss}(\text{model}(x_j), y_j) / k\big)\).backward() // local accumulation, no all-reduce
  3. \(\big(\text{loss}(\text{model}(x_k), y_k) / k\big)\).backward() // buckets all-reduce the accumulated sums now
  4. // .grad \(= \frac{1}{N} \sum_r \frac{1}{k} \sum_j \mathbf{g}_{r,j}\): the gradient of the global batch \(B = N b k\)
  5. optimizer.step(); optimizer.zero_grad()

The division by \(k\) makes the accumulated gradient a mean over micro-batches, just as DDP’s division by \(N\) makes it a mean over ranks. With accumulation, the all-reduce overlaps only with the backward pass of the last micro-batch, but it happens once per \(k\) micro-batches, so its share of the step time is about \(k\) times smaller.

The code

The first listing checks equation (44.2) in one process: four “GPUs” are four slices of a batch. It also shows the two arithmetic bugs: forgetting to divide by \(N\), and averaging unequal shards.

import torch, torch.nn as nn

torch.manual_seed(0)
model = nn.Sequential(nn.Linear(8, 16), nn.Tanh(), nn.Linear(16, 1))
X, Y = torch.randn(32, 8), torch.randn(32, 1)
loss_fn = nn.MSELoss()                      # mean over the batch

def grads(xb, yb):
    model.zero_grad()
    loss_fn(model(xb), yb).backward()
    return torch.cat([p.grad.flatten() for p in model.parameters()])

g_full = grads(X, Y)                        # one GPU, global batch of 32
N = 4                                       # four "GPUs", 8 samples each
g_local = [grads(xb, yb) for xb, yb in zip(X.chunk(N), Y.chunk(N))]
g_sum = torch.stack(g_local).sum(0)         # what all_reduce(SUM) returns
g_avg = g_sum / N                           # what DDP stores in .grad

print(torch.allclose(g_avg, g_full, atol=1e-6))     # True
print(round((g_sum.norm() / g_full.norm()).item(), 3))  # 4.0: forgot to divide by N
print(round((g_local[0] - g_full).norm().item() / g_full.norm().item(), 2))  # 0.92: one shard alone is a noisy estimate

# uneven shards: 10 + 10 + 6 + 6 samples; the plain average of local means is wrong
sizes = [10, 10, 6, 6]
parts = list(zip(X.split(sizes), Y.split(sizes)))
g_bad = torch.stack([grads(xb, yb) for xb, yb in parts]).mean(0)
g_ok = sum(n * grads(xb, yb) for n, (xb, yb) in zip(sizes, parts)) / sum(sizes)
print(torch.allclose(g_bad, g_full, atol=1e-6), torch.allclose(g_ok, g_full, atol=1e-6))  # False True

The second listing is real DDP on two CPU processes with the Gloo backend (change to "nccl" and move the model and data to cuda:LOCAL_RANK on GPUs). It uses a DistributedSampler, set_epoch, and no_sync for accumulation. At the end the ranks all-gather a checksum of their weights to show that the replicas are identical, and the main process checks that the result equals one process training on the same global batches.

import os, torch, torch.nn as nn, torch.distributed as dist, torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import TensorDataset, DataLoader, DistributedSampler

def make_model():
    torch.manual_seed(0)                       # same seed everywhere (DDP also broadcasts rank 0)
    return nn.Sequential(nn.Linear(8, 16), nn.Tanh(), nn.Linear(16, 1))

def make_data():
    g = torch.Generator().manual_seed(1)
    return TensorDataset(torch.randn(64, 8, generator=g), torch.randn(64, 1, generator=g))

def worker(rank, world, accum):
    os.environ["MASTER_ADDR"], os.environ["MASTER_PORT"] = "127.0.0.1", "29516"
    dist.init_process_group("gloo", rank=rank, world_size=world)   # "nccl" on GPUs
    torch.manual_seed(1234 + rank)            # per-rank seed: dropout, augmentation
    model = DDP(make_model())                  # broadcasts weights and buffers from rank 0
    opt = torch.optim.SGD(model.parameters(), lr=0.1)
    sampler = DistributedSampler(make_data(), num_replicas=world, rank=rank, shuffle=True, seed=0)
    loader = DataLoader(make_data(), batch_size=4, sampler=sampler)  # 4 per rank per micro-step
    for epoch in range(2):
        sampler.set_epoch(epoch)               # new shuffle each epoch, same on all ranks
        for i, (x, y) in enumerate(loader):
            last = (i + 1) % accum == 0
            if not last:
                with model.no_sync():          # accumulate locally, no all-reduce
                    (nn.functional.mse_loss(model(x), y) / accum).backward()
                continue
            (nn.functional.mse_loss(model(x), y) / accum).backward()   # all-reduce happens here
            opt.step(); opt.zero_grad()
    flat = torch.cat([p.detach().flatten() for p in model.parameters()])
    check = torch.tensor([flat.sum().item()], dtype=torch.float64)
    every = [torch.zeros(1, dtype=torch.float64) for _ in range(world)]
    dist.all_gather(every, check)              # compare a checksum across ranks
    if rank == 0:
        print("accum", accum, "checksums", [round(c.item(), 6) for c in every])
        torch.save(flat, f"/tmp/ddp_w_{accum}.pt")
    dist.destroy_process_group()

def single_gpu_reference(world, accum):
    model, data = make_model(), make_data()
    opt = torch.optim.SGD(model.parameters(), lr=0.1)
    for epoch in range(2):
        g = torch.Generator(); g.manual_seed(0 + epoch)       # what DistributedSampler does
        order = torch.randperm(64, generator=g).tolist()
        step = 4 * world * accum                               # global batch
        for s in range(0, 64, step):
            x, y = data[order[s:s + step]]
            nn.functional.mse_loss(model(x), y).backward()     # one big batch, plain mean
            opt.step(); opt.zero_grad()
    return torch.cat([p.detach().flatten() for p in model.parameters()])

if __name__ == "__main__":
    for accum in (1, 2):
        mp.spawn(worker, args=(2, accum), nprocs=2)
        ddp_w = torch.load(f"/tmp/ddp_w_{accum}.pt")
        ref = single_gpu_reference(2, accum)
        print("same as one GPU with batch", 8 * accum, ":", torch.allclose(ddp_w, ref, atol=1e-5))
# accum 1 checksums [-1.428242, -1.428242]
# same as one GPU with batch 8 : True
# accum 2 checksums [-1.643059, -1.643059]
# same as one GPU with batch 16 : True

The printed values are from PyTorch 2.8 on CPU. Two ranks with 4 examples each and \(k = 2\) give exactly the same weights as one process with batches of 16, which is (44.2) and (44.6) at work.

The limit: every GPU holds everything

DDP divides the data, not the model. Each rank keeps a full replica, which means a full copy of everything the training step needs. With the usual mixed-precision recipe and Adam (see session 38):

\[ M_{\text{DDP}} = \underbrace{2\Psi}_{\text{BF16 weights}} + \underbrace{2\Psi}_{\text{BF16 grads}} + \underbrace{4\Psi + 4\Psi + 4\Psi}_{\text{FP32 master, } m,\ v} + M_{\text{act}} = 16\,\Psi + M_{\text{act}} \tag{44.8} \]
Read the symbols
\(\Psi\)
the number of parameters
\(m\), \(v\)
Adam’s first and second moment estimates, one FP32 number each per parameter
\(M_{\text{act}}\)
the activations saved for the backward pass; they depend on the micro-batch, not on \(N\)

(The standard PyTorch AMP recipe keeps FP32 weights and FP32 gradients instead: \(4 + 4 + 4 + 4\) is also 16 bytes per parameter.) The important point is what is missing from (44.8): \(N\). Adding GPUs makes training faster, but it does not lower the memory per GPU by one byte. The \(N\) copies of the optimizer state are identical, so \(N-1\) of them are pure redundancy. On top of (44.8), DDP’s bucket buffers hold another copy of the gradients unless you set gradient_as_bucket_view=True. [Certain]

Worked example: Llama 2 7B with DDP

Llama 2 7B has 6,738,415,616 parameters (32 layers, hidden size 4,096, MLP size 11,008, vocabulary 32,000, untied embeddings). With (44.8): \(16 \times 6.74 \times 10^{9} = 107.8\) GB per GPU before any activation; with the extra bucket copy of BF16 gradients, 121.3 GB. An 80 GB H100 cannot hold it, and neither can 1,024 of them. The largest model whose training state fits in 80 GB with DDP is \(80 / 16 = 5\) billion parameters, with no room left for activations.

BF16 weightsBF16 gradients (+ bucket copy)FP32 master weights and Adam m, v
Figure 5. Training-state memory per GPU from (44.8), without activations. The top bar is DDP. The bottom bar is a preview of full sharding (ZeRO stage 3 or FSDP, next sessions), where each GPU keeps only \(1/N\) of each part. The dashed line is the GPU’s memory.
How to read Figure 5

Start at Llama 2 7B on 8 GPUs of 80 GB: the DDP bar is 107.8 GB and crosses the dashed line, so it does not fit, while the sharded bar is 13.5 GB. Now move the GPU slider to 1,024: the DDP bar does not move at all. Only the sharded bar shrinks. Untick gradient_as_bucket_view to see the extra gradient copy that DDP keeps by default. Then go down to 1B parameters: 16 GB, so DDP fits easily, and since it is the simplest and has the least communication, it is the right choice there.

This is the wall. DDP is the best choice when the training state fits on one GPU with room for activations: it is simple, robust, and moves the fewest bytes (one all-reduce of the gradients per step). For larger models, the next session removes the redundancy step by step. ZeRO stage 1 shards the optimizer state, stage 2 also the gradients, and stage 3 also the weights. Stages 1 and 2 even keep DDP’s communication volume, because an all-reduce is a reduce-scatter plus an all-gather (43.4). [Certain]

\[ V_{\text{DDP}} = 2 \cdot \frac{N-1}{N} \cdot 2\Psi \ \text{bytes sent per GPU per step (BF16 gradients)} \tag{44.9} \]

Seeds and determinism across ranks

A data-parallel job has \(N\) copies of every random number generator. Some of them must agree across ranks, and some must not. Getting this wrong does not crash the job. It silently changes the training, which is why it is a favorite interview topic.

Source of randomnessAcross ranks it must beWhy
Weight initializationthe samereplicas must start equal; DDP’s start-up broadcast enforces this for parameters and buffers, but not for tensors that are plain attributes of a module
Data order (sampler seed)the same, and set_epoch every epocheach rank computes the shared permutation by itself; different seeds mean overlapping shards and skipped examples (Figure 2)
Dropout, stochastic depth, token masking, data augmentationdifferent (for example seed + rank)with the same seed, every rank draws the same masks and crop positions; the noise is then correlated across the global batch instead of independent [Likely]
DataLoader workersdifferent per rank and per workerPyTorch derives each worker’s seed from the main process seed, so a per-rank main seed gives per-rank workers [Likely]
Gradient noise from the hardware (atomic adds, kernel choice)does not matter for synclocal gradients may differ in the last bits, but after the all-reduce every rank has the same bits, so the replicas do not drift

The last row needs a closer look. A ring all-reduce computes each chunk of the sum in one place, then copies the finished chunk to everyone (Figure 2 of session 43). So all ranks receive bitwise the same result and the replicas stay bitwise identical. [Likely] But the result depends on the order of the additions, and floating-point addition is not associative: \((a + b) + c\) can differ from \(a + (b + c)\) in the last bit. Different \(N\), a different NCCL algorithm (ring or tree), or a different bucket layout changes the order. So a run on 16 GPUs is not bitwise equal to the same run on 8 GPUs, even with the same global batch and the same seeds. [Certain] Figure 6 shows the size of the effect.

Figure 6. One gradient entry averaged over a global batch of 1,024 examples (seeded random values of very different sizes). With \(N\) ranks, each rank sums its 1,024/\(N\) values in order, divides by its batch and by \(N\), and the ring adds the \(N\) results starting at rank 1. Every operation is rounded to the chosen format. The bars show the error of each result against the exact (64-bit) answer.
How to read Figure 6

Each bar is the same mathematical quantity computed with a different number of ranks; the highlighted bar is the \(N\) of the slider. In FP32, the results differ by a few units in the last place (ULP): for example the results for \(N = 1\) and \(N = 2\) differ by 6 ULP. That is a relative difference of about \(10^{-7}\), harmless for training but enough to make two runs diverge in their last digits after many steps. Switch to BF16 arithmetic: the errors grow to about 1 to 3%, which is why gradients are usually reduced in FP32, or at least accumulated in FP32. [Likely]

What can you control? Within one configuration (same \(N\), same software versions, same hardware, deterministic kernels via torch.use_deterministic_algorithms(True)), runs are usually reproducible. [Likely] Across configurations, aim for statistical reproducibility (the same loss curve within noise) rather than bitwise equality. For resuming a job, save every rank’s RNG states, the sampler epoch and position, and the optimizer state, or the restarted run will see different data and different dropout masks. A cheap health check during long runs is the one in the second listing: all-gather a checksum of the weights every few thousand steps and alert if the replicas disagree. [Likely]

Where this is used

SettingHow data parallelism is usedNote
Pretraining and fine-tuning models up to a few billion parametersplain DDP with bucketed all-reducethe training state (16 bytes per parameter) fits on one GPU
Large LLM pretraining (Llama-class)the outer “data-parallel” dimension; inside it, ZeRO/FSDP, tensor and pipeline parallelismcombined in session 49
Video classification and action recognitionDDP, each rank decodes and augments its own clipsvideo decoding is the bottleneck, so each rank also runs several DataLoader workers; 3D CNNs with BatchNorm often use SyncBatchNorm because the per-GPU batch is small [Likely]
Contrastive video–text retrieval models (CLIP-style)DDP for the gradients, plus an all-gather of embeddings so each clip sees the texts of the whole global batch as negativesthe all-gather must pass gradients back (torch.distributed.nn.all_gather) [Likely]
Vision-language model fine-tuning with a frozen vision encoderDDP with find_unused_parameters or static_graph when some parameters get no gradientotherwise the buckets wait forever
Video fingerprinting, key-moment detection, building a Video-RAG indexdata-parallel inference: DistributedSampler shards the videos, no DDP wrapper (no gradients)results are all-gathered or written per rank and merged
Distributed evaluationDistributedSampler with padding, then all-reduce of countsremove the padded duplicates, or the metric is slightly wrong

Common mistakes

  • Forgetting sampler.set_epoch(epoch). Every epoch then uses the same order. Training still works, but worse, and nothing warns you.
  • Different sampler seeds on different ranks. For example, seeding everything with seed + rank and passing that to the sampler. Ranks then use different permutations, so shards overlap and some data is never seen. Use the same sampler seed and a per-rank seed only for dropout and augmentation.
  • A collective inside if rank == 0:. Logging that calls all_reduce or a checkpoint function that gathers state only on rank 0 makes rank 0 wait for partners that never come: a hang, not an error.
  • Unequal work across ranks. Without padding or drop_last, or with a hand-written data split, one rank may run one step more and hang. DDP’s join() context manager handles uneven inputs on purpose. [Certain]
  • Mean over tokens per rank with unequal token counts. If each rank averages its loss over its own number of non-padding tokens and the counts differ, DDP’s division by \(N\) weights the ranks wrongly, as in (44.3). Fix: all-reduce the token count and divide the summed loss by the global count (and multiply by \(N\) to cancel DDP’s division), or use packed sequences of equal length.
  • Unused parameters without find_unused_parameters. A skipped branch or a frozen layer inside the DDP module leaves a bucket incomplete. Recent PyTorch raises an error at the next iteration; older setups simply hung. [Likely]
  • Accumulating without no_sync, or with the forward pass outside it. The result is correct but the communication is \(k\) times larger.
  • Changing \(N\) but not the learning rate or batch. Doubling the GPUs with a fixed micro-batch doubles the global batch (44.6). Either keep the global batch fixed (fewer accumulation steps) or retune the learning rate.

Interview questions

What does DDP do in one training step?basic
Each GPU holds a full replica and processes its own shard of the global batch. During the backward pass, gradients are grouped into buckets and each bucket is all-reduced (summed, divided by \(N\)) asynchronously as soon as it is complete. After the backward pass all ranks hold the same averaged gradient, and each runs the same local optimizer step, so the replicas stay identical. Weights are broadcast only once, at construction.
Why is DDP mathematically equivalent to training on one GPU with the global batch?basic
The global loss is the mean over all \(B\) examples, which equals the mean over ranks of the local mean losses when each rank has \(b = B/N\) examples (44.1). Gradients are linear, so the full-batch gradient is the average of the local gradients (44.2), which is exactly what the all-reduce plus division computes. With equal starting weights and the same deterministic optimizer, every step is the same as one large-batch step. Only floating-point summation order differs.
Why does DDP use gradient buckets? How big are they?core
Two reasons. First, overlap: gradients of later layers are ready early in the backward pass, so a bucket can be all-reduced while earlier layers are still being differentiated. Second, efficiency: one all-reduce per tiny tensor would pay the latency \(2(N-1)\alpha\) hundreds of times. The default cap is 25 MiB; after the first iteration DDP rebuilds buckets in the observed gradient-ready order, with a 1 MiB first bucket so that communication starts early. Smaller buckets give more overlap but more latency; one huge bucket overlaps nothing.
Does adding GPUs with DDP let you train a bigger model?basic
No. Every rank holds the full weights, gradients and optimizer state, about 16 bytes per parameter with mixed-precision Adam, plus activations. That does not depend on \(N\). A 7B model needs about 108 GB per GPU and does not fit on 80 GB, with 8 or with 1,024 GPUs. To train bigger models you shard the state (ZeRO, FSDP) or split the model (tensor and pipeline parallelism).
How much does DDP communicate per step, and when is it hidden?core
One all-reduce of the gradients: with a ring, each GPU sends \(2\frac{N-1}{N}\) times the gradient size, about \(4\Psi\) bytes for BF16 gradients (44.9). The backward pass costs about \(4\Psi T\) FLOPs for \(T\) tokens per GPU. So the all-reduce fits under the backward pass when \(T \gt F/\beta\) (44.5): for an H100 at 40% utilization over 50 GB/s InfiniBand that is about 7,900 tokens per GPU per step, independent of model size. Plus there is always a tail: the last bucket.
What is no_sync, and why does the forward pass have to be inside it?core
It is a context manager that disables the gradient all-reduce, so gradients accumulate locally over several micro-batches; the first backward pass outside it all-reduces the accumulated sum. It saves \(k-1\) of every \(k\) all-reduces in gradient accumulation with the same result. DDP prepares the reducer for synchronization during the forward pass, so if the forward runs outside the context and only the backward inside, the gradients are still synchronized.
Why must you call sampler.set_epoch, and why must the sampler seed be the same on all ranks?core
The DistributedSampler shuffles with a generator seeded with seed + epoch; each rank computes the permutation independently and takes every \(N\)-th index starting at its rank. Without set_epoch the epoch stays 0 and every epoch has the same order. If ranks used different seeds, they would index into different permutations, so shards would overlap and some examples would never be seen. The seeds for dropout and augmentation, in contrast, should differ by rank.
DDP versus DataParallel?basic
DataParallel is one process with a thread per GPU: it replicates the model from GPU 0 every forward pass, scatters inputs, gathers outputs on GPU 0, suffers from Python GIL contention, puts extra load on GPU 0, and works only within a machine. DDP is one process per GPU, broadcasts once, all-reduces bucketed gradients overlapped with backward, and works across machines. PyTorch recommends DDP even on one machine.
Two runs with the same seeds, one on 8 GPUs and one on 16 with the same global batch, give slightly different loss curves. Bug?advanced
Not necessarily. The math is identical, but the order of floating-point additions changes: each rank sums a different set of examples, the all-reduce adds a different number of partial sums in a different order, and NCCL may choose a different algorithm. Floating-point addition is not associative, so the gradients differ in the last bits, and training is chaotic enough that small differences grow over steps. Check that the curves agree within noise; demand bitwise equality only for the same configuration with deterministic kernels. If the difference is large, look for real bugs: global batch, learning-rate scaling, sampler seeds, loss normalization.
A model with a frozen vision tower hangs in the second backward pass under DDP. Why, and what are the fixes?advanced
DDP all-reduces a bucket only when every parameter in it has produced a gradient. Parameters that do not get gradients (unused branch, or parameters inside the DDP module that are not part of the loss) leave their bucket incomplete, so ranks wait (or PyTorch raises an error about unused parameters). Fixes: set requires_grad=False on frozen parameters before wrapping (then DDP ignores them), keep frozen modules outside the DDP wrapper, use find_unused_parameters=True for truly dynamic graphs (costs a graph traversal per step), or static_graph=True if the set of unused parameters is fixed.
How do you scale the learning rate when you go from 8 to 64 GPUs?core
First decide whether the global batch changes. If you keep it fixed (fewer accumulation steps or smaller micro-batches per GPU), keep the learning rate. If the global batch grows by 8 times, the linear scaling rule (Goyal et al.) says multiply the learning rate by 8 with a warmup, which works for SGD up to a point; for Adam a square-root rule is often closer. Beyond the critical batch size, a larger batch gives little speedup, so the extra GPUs waste compute. In LLM training the global batch in tokens is tuned, and the GPU count is chosen around it.

Exercises

  1. In Figure 1, compute rank 2’s local gradient at \(w = 0.5\) by hand (examples \((2, 5)\) and \((4, 7)\)).
  2. A job uses \(N = 32\) GPUs, a micro-batch of 4 sequences of 4,096 tokens, and 8 accumulation steps. How many tokens are in one global batch?
  3. A rank has 3,000 non-padding tokens and another has 1,000. Each computes its mean token loss, and DDP averages the two gradients. By what factor is a token on the second rank over-weighted compared with the correct global mean?
  4. Using (44.8), what is the largest model you can train with plain DDP and Adam on a 141 GB GPU, ignoring activations?
  5. With \(F = 4 \times 10^{14}\) FLOP/s and \(\beta = 25\) GB/s, how many tokens per GPU per step are needed to hide the all-reduce (44.5)?
  6. The DistributedSampler gets \(n = 1{,}000\) examples and \(N = 64\) ranks. How many indices does each rank get with and without drop_last, and how many duplicates or skipped examples are there?
Answerscheck
1. \(\tfrac12[(1 - 5)\cdot 2 + (2 - 7)\cdot 4] = \tfrac12(-8 - 20) = -14\). 2. \(32 \times 4 \times 8 = 1{,}024\) sequences, \(\times 4{,}096 = 4{,}194{,}304\) tokens, about 4.2M. 3. Correct weight per token: \(1/4{,}000\). DDP gives each rank weight \(1/2\), so a token on rank 2 gets \(\tfrac12 \cdot \tfrac1{1000} = 1/2{,}000\) (2 times too much) and a token on rank 1 gets \(\tfrac12 \cdot \tfrac1{3000} = 1/6{,}000\) (\(2/3\) of the correct weight). 4. \(141/16 \approx 8.8\) billion parameters. 5. \(T \gt 4\times10^{14}/2.5\times10^{10} = 16{,}000\) tokens. 6. Without drop_last: \(\lceil 1000/64 \rceil = 16\) each, \(64 \cdot 16 = 1{,}024\) slots, so 24 padded duplicates. With drop_last: \(\lfloor 1000/64 \rfloor = 15\) each, 960 used, 40 examples skipped this epoch.

Key takeaways

  • Data parallelism: a full replica on every rank, a different shard of the batch on each, one all-reduce of the gradients per step. With equal shards, the averaged gradient is exactly the full-batch gradient (44.2).
  • Replicas stay identical because they start equal (DDP broadcasts rank 0 at construction) and apply the same update. DDP never sends weights after that.
  • DDP all-reduces gradients in buckets (25 MiB cap, 1 MiB first bucket, rebuilt after the first iteration in the real order), overlapped with the backward pass. The last bucket is the tail that cannot be hidden (44.4).
  • The all-reduce hides under the backward pass when the tokens per GPU per step exceed \(F/\beta\) (44.5), whatever the model size.
  • Global batch \(B = N b k\) (44.6). More GPUs change the batch, and then the learning rate; no_sync makes accumulation cost one all-reduce per optimizer step.
  • The wall: 16 bytes per parameter on every GPU, independent of \(N\) (44.8). A 7B model needs about 108 GB. ZeRO and FSDP remove this redundancy.
  • Seeds: same for initialization and the sampler (and call set_epoch), different for dropout and augmentation. Changing \(N\) changes summation order, so bitwise equality holds only within one configuration.

Key formulas

\[ \nabla L = \frac{1}{N}\sum_{r=0}^{N-1} \mathbf{g}_r, \qquad B = N\, b\, k, \qquad \eta(B) = \eta_0 \frac{B}{B_0} \]
\[ M_{\text{DDP}} = 16\,\Psi + M_{\text{act}}, \qquad V_{\text{DDP}} = 2\,\frac{N-1}{N}\cdot 2\Psi, \qquad \text{hidden if } T \gt \frac{F}{\beta} \]

References for this session

  1. Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
  2. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  3. Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
  4. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  5. PyTorch (docs). Distributed Data Parallel (design notes: construction, buckets, backward hooks)
  6. PyTorch (docs). torch.nn.parallel.DistributedDataParallel API reference
  7. PyTorch (docs). torch.utils.data.distributed.DistributedSampler
  8. PyTorch (docs). Reproducibility (randomness and deterministic algorithms)
  9. Goyal et al. (2017). Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
  10. McCandlish, Kaplan, Amodei et al. (2018). An Empirical Model of Large-Batch Training
  11. Malladi, Lyu, Panigrahi and Arora (2022). On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
  12. Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  13. Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel