Data parallelism (DDP)
Data parallelism is the first and simplest way to train on many GPUs: every GPU keeps a full copy of the model, works on its own slice of the batch, and the GPUs average their gradients before each update. This session shows why that gives exactly the same update as one big GPU, how PyTorch DDP hides the averaging behind the backward pass with gradient buckets, how the data is split and seeded on each rank, and why the method stops working when the model and its optimizer no longer fit on one GPU.
What you will learn
- Why averaging the gradients of N equal shards gives exactly the gradient of the full batch, and why the replicas then stay identical
- What one DDP training step does: processes and ranks, the broadcast at start-up, the DistributedSampler, the backward hooks and the optimizer step
- How gradient buckets work in PyTorch (25 MiB cap, 1 MiB first bucket, rebuilt after the first iteration), and how they overlap the all-reduce with the backward pass
- When the backward pass can hide the all-reduce: a rule that depends only on tokens per GPU, GPU speed and network bandwidth
- Global batch size, the linear scaling rule, and gradient accumulation with no_sync
- The limit of DDP: 16 bytes per parameter on every GPU, so a 7B model does not fit on an 80 GB GPU, however many GPUs you add
- Seeds and determinism across ranks: what must be the same on every rank, what must differ, and why changing the number of GPUs changes the last bits of the result
Why this matters
Pretraining a modern LLM needs about \(10^{24}\) floating-point operations or more. One GPU would take decades. The only way to finish in weeks is to use thousands of GPUs at once. The first and simplest way to do that is data parallelism: give every GPU a full copy of the model, give each GPU a different slice of the data, and let the GPUs agree on one update at every step.
Data parallelism is in every large training run. Even when a model is too big for one GPU and is split with tensor or pipeline parallelism, the outermost layer of the setup is almost always data parallel: many groups of GPUs, each holding one copy of the (split) model, each reading different data. [Certain] So the ideas of this session, averaging gradients, overlapping communication with the backward pass, splitting the data correctly and keeping the copies identical, come back in every later session of this track.
In session 43 we built the tool: the ring all-reduce, which leaves every GPU with the sum of everyone’s data and sends about twice the buffer size per GPU, whatever the number of GPUs. In this session we use it. PyTorch’s DistributedDataParallel (DDP) is the standard implementation, and its paper (Li et al., 2020) reports near-linear scaling up to 256 GPUs when it is configured well. [Certain] We will see what “configured well” means, and where the method hits a wall: every GPU must hold the whole model, its gradients and its optimizer state. That wall is the reason for session 45 and session 46.
The idea: copy the model, split the batch, average the gradients
Start with what one GPU does in one training step (see session 4). It takes a batch of \(B\) examples, computes the loss as the mean of the per-example losses, runs the backward pass to get the gradient, and lets the optimizer update the weights.
Now take \(N\) GPUs. Some words first. Each GPU runs its own copy of the training program, called a process, and each process has a number called its rank, from 0 to \(N-1\). \(N\) is the world size. In data parallelism, each rank holds a full copy of the model, called a replica. A training step then has four parts:
- Split. The global batch of \(B\) examples is cut into \(N\) shards (or local batches) of \(b = B/N\) examples. Rank \(r\) gets shard \(r\).
- Compute locally. Each rank runs the forward and backward pass on its own shard and gets a local gradient \(\mathbf{g}_r\).
- Average. The ranks run an all-reduce on the gradients and divide by \(N\). Now every rank holds the same averaged gradient.
- Update. Each rank runs the same optimizer step on the same gradient. The replicas change in the same way and stay identical.
Why is step 3 correct? Write \(\ell_i(\mathbf{w})\) for the loss of example \(i\) with weights \(\mathbf{w}\). The loss of the global batch is the mean over all \(B\) examples. Group the examples by shard:
The gradient is linear, so the gradient of a mean is the mean of the gradients:
- Read the symbols
- \(B\), \(b\), \(N\)
- global batch size, local batch size per rank, and number of ranks, with \(B = N b\)
- \(S_r\)
- the set of examples in the shard of rank \(r\); the shards do not overlap and together cover the batch
- \(L_r\)
- the local loss on rank \(r\): the mean loss over its own \(b\) examples, exactly what a normal training loop computes
- \(\mathbf{g}_r\)
- the local gradient on rank \(r\), what
loss.backward()produces before any communication - \(\frac{1}{N}\sum_r\)
- an all-reduce with sum, followed by a division by \(N\)
So the averaged gradient is exactly the gradient that one GPU would compute on the whole batch of \(B\) examples (up to floating-point rounding, which we discuss at the end). Data parallelism does not approximate anything. It is the same algorithm as large-batch training on one huge GPU. [Certain]
Two conditions hide in (44.1). First, the shards must have equal size. If rank 0 has 10 examples and rank 1 has 6, the plain average of the two local means gives the examples of rank 1 too much weight. The correct formula weights each local gradient by its shard size:
DDP always divides by \(N\), so it assumes equal \(b_r\). This matters for LLMs when the loss is a mean over tokens and different ranks have different numbers of non-padding tokens; we return to it in the mistakes section. Second, all replicas must start from the same weights. If they start equal and get the same gradient and the same optimizer step, they stay equal after every step, by induction. If they start different, they stay different forever, because they all receive the same correction.
Figure 1 runs this on a toy model with one weight. Step through it, then try the three bugs.
How to read Figure 1
Each row is one rank with its weight \(w\), its two examples \((x, y)\), and its local gradient. In step 1 the ranks start from different random weights. Step 2 broadcasts rank 0’s \(w = 0.5\) to all ranks. Step 3 gives four different local gradients: −2.75, −5.5, −14 and −4.75. Step 4 is the all-reduce: the sum −27, divided by 4, is −6.75, exactly the gradient the single GPU gets on all eight examples. Step 5 updates every replica to \(0.5 - 0.1 \cdot (-6.75) = 1.175\), the same as the single GPU. Now pick a bug. Without the broadcast, the four weights get the same corrections but keep their different offsets forever. Without the division, the step is four times too large and the training blows up (here the effective learning rate 0.4 times the curvature 6 is above 2). Without the all-reduce, each replica fits only its own data and the copies drift apart.
Rank 0 has the examples \((1, 2)\) and \((2, 3)\). With \(w = 0.5\) its local gradient is the mean of \((w x - y)\,x\): \(\tfrac12[(0.5 - 2)\cdot 1 + (1 - 3)\cdot 2] = \tfrac12(-1.5 - 4) = -2.75\). The other ranks get −5.5, −14 and −4.75. The all-reduce returns the sum −27 to every rank; divided by \(N = 4\) it is −6.75.
One GPU with all eight examples computes \(\tfrac18 \sum (w x - y)\,x = -54/8 = -6.75\). The same number, as (44.2) promised. After one SGD step every replica holds \(w = 1.175\); after 20 steps all four hold 1.625, which is the least-squares answer \(\sum xy / \sum x^2 = 78/48\).
One DDP training step in PyTorch
PyTorch’s DDP turns the four parts above into a wrapper around your model. You still write a normal training loop; the communication happens inside backward(). Here is what happens, in order.
Start-up. A launcher such as torchrun starts one process per GPU and gives each one environment variables: RANK (global rank), LOCAL_RANK (the GPU index inside the server), WORLD_SIZE, and MASTER_ADDR and MASTER_PORT (where rank 0 listens, so the processes can find each other). [Certain] Each process calls init_process_group("nccl") and picks its GPU with LOCAL_RANK.
Wrapping. When you call DDP(model), it does two things. It broadcasts the parameters and buffers of rank 0 to all other ranks, so every replica starts from the same weights even if each process initialized its model differently. [Certain] And it creates a Reducer: an object that puts the parameters into gradient buckets (next section) and registers one autograd hook per parameter. A hook is a function that autograd calls when the gradient of that parameter is ready. [Certain]
Forward. A normal forward pass on the local shard. If the model has buffers (for example the running statistics of BatchNorm), DDP broadcasts rank 0’s buffers to all ranks at the start of each forward pass by default (broadcast_buffers=True). [Certain] Transformers with LayerNorm or RMSNorm have no such statistics, so this costs almost nothing for LLMs.
Backward. As gradients become ready, the hooks fill the buckets. When a bucket is full, the Reducer starts an asynchronous all-reduce for it and the backward pass keeps going. At the end of the backward pass, the Reducer waits for all buckets and writes the averaged gradients into param.grad. [Certain]
Optimizer step. The optimizer is a normal local optimizer. It does not know about the other ranks. It does not need to: every rank has the same weights, the same gradients and the same optimizer state, so it computes the same update. DDP never sends weights after start-up; only gradients. [Certain]
init_process_group("nccl"); select GPULOCAL_RANK- build the model; \(\text{model} \leftarrow \text{DDP}(\text{model})\) // broadcast rank 0's weights, build buckets, register hooks
- build the optimizer on
model.parameters()// a normal local optimizer - \(\text{sampler} \leftarrow\)
DistributedSampler(data, num_replicas=N, rank=r, seed=s)// same seed \(s\) on every rank - for each epoch \(e\):
sampler.set_epoch(e) - for each local batch \((x, y)\) of size \(b\) from the sampler:
- \(\text{loss} \leftarrow\) mean loss of \(\text{model}(x)\) against \(y\) // the local loss \(L_r\)
-
loss.backward()// hooks all-reduce the buckets while the backward pass runs - // here every rank holds
.grad\(= \frac{1}{N} \sum_r \mathbf{g}_r\) - optional: clip the gradient norm // same gradients, so the same clipping on every rank
-
optimizer.step();optimizer.zero_grad() - only rank 0 logs and saves checkpoints // but every rank must still join every collective
Why not DataParallel?
PyTorch also has an older wrapper, nn.DataParallel. It uses one process with one thread per GPU. In every forward pass it copies the model from GPU 0 to the other GPUs, splits the input, runs the copies, and gathers the outputs back on GPU 0, where the loss is computed. This has three problems: the Python threads compete for the global interpreter lock (GIL), GPU 0 does extra work and holds extra memory, and it works only inside one machine. The PyTorch documentation recommends DDP instead, even on a single machine. [Certain]
Splitting the data: the DistributedSampler
Each rank must see different examples in a step, and together the ranks should cover the dataset once per epoch. PyTorch’s DistributedSampler does this with a simple rule. At the start of an epoch, every rank builds the same random permutation of all \(n\) indices, using a generator seeded with seed + epoch. Then rank \(r\) takes every \(N\)-th index, starting at position \(r\). [Certain, checked in the PyTorch 2.8 source]
If \(n\) is not a multiple of \(N\), there are two options. By default (drop_last=False), the sampler pads the list by repeating indices from its start, so a few examples are seen twice in that epoch. With drop_last=True, it cuts the tail instead, so a few examples are not seen in that epoch. Either way every rank gets exactly \(\lceil n/N \rceil\) or \(\lfloor n/N \rfloor\) indices, so all ranks run the same number of steps. [Certain] This is important: if one rank ran one step more, it would wait in an all-reduce that nobody else joins, and the job would hang.
Two rules follow, and they are about seeds. The sampler seed must be the same on every rank, because each rank builds the permutation by itself and they must agree on it. And you must call sampler.set_epoch(epoch) before each epoch, or the permutation is the same in every epoch. Figure 2 shows both.
torch.randperm, so the exact order differs from PyTorch; the rule and the counts are the same.How to read Figure 2
The top rows show the indices each rank reads in this epoch, in order. The bottom strip shows every example of the dataset and how many times it is read in total. With \(n = 10\) and \(N = 4\), each rank gets 3 indices; 12 slots for 10 examples, so 2 indices are padding and 2 examples are read twice. Tick drop_last: each rank gets 2, and 2 examples are skipped. Move the epoch slider: the permutation changes. Untick “call set_epoch”: the epoch slider no longer changes anything, so the model sees the data in the same order every epoch. Untick “same sampler seed”: each rank now uses its own permutation; several examples are read twice and others never, even with no padding at all.
LLM pretraining usually does not use a map-style dataset with a sampler. The data is a long stream of pre-tokenized documents, cut into fixed-length sequences (see session 34). The same rule applies, though: one global order that every rank can compute from a shared seed and the step number, and rank \(r\) reads its own disjoint part of it. The data loader state (the position in the stream) must be saved in the checkpoint so that a restarted job does not repeat or skip data. [Likely]
Gradient bucketing and overlap with the backward pass
The simplest DDP would wait for the whole backward pass and then run one all-reduce over all gradients. That wastes time: the network is idle during the backward pass, and the GPUs are idle during the all-reduce. But the gradients do not all appear at the end. The backward pass goes from the last layer to the first, so the gradients of the last layer are ready long before those of the first. DDP uses this. [Certain]
The Reducer groups the parameters into buckets. A bucket is one flat buffer that holds the gradients of several parameters next to each other. When the last gradient of a bucket arrives, the whole bucket is all-reduced as one message, asynchronously, on a separate CUDA stream, while the backward pass continues on the compute stream. [Certain] Why buckets and not one all-reduce per parameter? A transformer has hundreds of parameter tensors, many tiny (biases, norm weights). Each all-reduce pays a fixed latency of \(2(N-1)\alpha\) (formula 43.4), and small messages use the links badly. Buckets turn hundreds of small messages into a few large ones. [Certain]
Figure 3 runs one whole DDP step, with real numbers, on a tiny network. It has four layers, each with one weight \(w_l\) and one bias \(b_l\), so 8 parameters in total: \(h_l = \tanh(w_l h_{l-1} + b_l)\) for \(l = 1, 2, 3\), and \(\hat y = w_4 h_3 + b_4\), with the loss \(\tfrac12(\hat y - y)^2\) averaged over the shard. Eight examples are split over the ranks. The page really runs the forward and backward pass on every rank, fills the buckets, averages them and updates the weights. The times are made-up units that look like a real step: 1 unit per layer for the forward pass, 2 units per layer for the backward pass (it costs about twice the forward), and 1 unit for the optimizer step. Each bucket’s all-reduce takes the ring time of formula (43.4), \(2(N-1)\alpha + 2\frac{N-1}{N}\cdot\frac{S}{\beta}\), with \(\alpha = 0.1\) unit and \(S/\beta\) set by the network slider (the time to send one layer’s gradients once).
- for \(l = 1, \dots, 4\): \(h_l \leftarrow \tanh(w_l h_{l-1} + b_l)\) // forward, own shard
- \(L_r \leftarrow \frac{1}{b} \sum_i \tfrac12 (\hat y_i - y_i)^2\) // local loss (44.1)
- for \(l = 4, \dots, 1\): \(g_r \leftarrow \partial L_r / \partial (w_l, b_l)\) // backward
- hook: copy \(g_r\) into its bucket
- if bucket full: launch async all_reduce // network stream
- wait for all buckets; \(\bar g \leftarrow \text{bucket} / N\) // (44.2)
- \(w \leftarrow w - \eta\, \bar g,\ \ b \leftarrow b - \eta\, \bar g\) // same on every rank
How to read Figure 3
Press Play, or use Step to move one event at a time; the box under the code shows the line being run with the current numbers. With 4 ranks, 1 layer per bucket and network 1: the forward pass ends at time 4, and each rank has a different local loss (0.397, 0.488, 0.358, 0.537). The backward pass starts at layer 4. At time 6 its gradients are ready: the four local gradients of \(w_4\) are 0.383, 0.541, 0.371 and 0.502. Bucket 0 is full, so its all-reduce starts at once, while the backward pass goes on with layer 3. It takes \(2 \cdot 3 \cdot 0.1 + 1.5 \cdot 1 = 2.1\) units. At 8.1 every rank holds the same \(\bar g = (0.383 + 0.541 + 0.371 + 0.502)/4 = 0.449\), exactly the gradient one GPU computes on all 8 examples. The backward pass ends at 12, the last bucket at 14.4, so the GPUs wait 2.4 units (red), then all apply the same update (\(w_4\): 0.8 → 0.755). The step takes 15.4 units; untick overlap and it takes 21.4. Move the network slider to 3: now the network is the bottleneck (27.4 against 33.4). Choose one bucket: nothing can start before the backward pass ends, so overlap gives no gain (19.6 at network 1).
Two things in Figure 3 are worth saying in words. First, the all-reduce of a bucket cannot start before its last gradient is ready, and the buckets go through the network one after another, in the same order on every rank. So when the network is slower than the backward pass, buckets queue up and the GPU waits at the end. Second, the bucket that holds the first layers is ready only when the backward pass ends. Its all-reduce is the tail, which no overlap can hide.
The details, as of PyTorch 2.8 (checked in the source):
- Size. The cap is
bucket_cap_mb = 25, which is 25 MiB (\(25 \cdot 2^{20}\) bytes). A bucket is closed when it reaches the cap, so it can be larger than the cap when one big tensor is added last. [Certain] - Order. At construction, parameters are put into buckets in roughly the reverse order of
model.parameters(), because that is roughly the order in which the backward pass produces their gradients. [Certain] - The first iteration. With the default settings, the first iteration uses one single bucket for all gradients, because the real order is not known yet. During that iteration DDP records the order in which gradients actually became ready and then rebuilds the buckets in that order. [Certain]
- A small first bucket. In the rebuilt layout, the first bucket has a cap of only 1 MiB, so that the first all-reduce starts as early as possible. [Certain]
- Same order on every rank. Buckets are all-reduced in bucket index order, on every rank. If ranks disagreed on the order, collectives would be matched with the wrong partners: wrong results or a hang. [Certain]
Figure 4 shows the real layout for a GPT-2 small model (124,439,808 parameters, FP32 gradients). We checked it with PyTorch: after the first iteration, DDP makes 13 buckets: a first bucket of 9.0 MiB (the final norm plus one 9 MiB weight matrix, which already passes the 1 MiB cap), eleven buckets of about 27 MiB, roughly one transformer block each, and a last bucket of 168.3 MiB that holds the token embedding table. GPT-2 ties the embedding with the output layer, so the embedding gradient is complete only at the very end of the backward pass, when the input embedding has been processed.
How to read Figure 4
The top strip shows the parameters: the wide gray block on the right is the 147 MiB embedding table; each transformer block is about 27 MiB. The colored boxes below are buckets. At the default cap of 25 MiB there are 13. The readout gives the number of buckets, the total time all of them would take on the network, the part that is pure latency, and the tail: the all-reduce of the last bucket, which can only start when the backward pass ends and so can never be hidden. Lower the cap to 1 MiB: 50 buckets, more latency, but the tail stays large because the embedding table is one tensor and cannot be split. Raise it to 500 MiB: only 2 buckets, and most of the communication waits until the end. Tick “first iteration”: everything is one bucket, nothing overlaps.
Now press Play at the defaults (cap 25 MiB, InfiniBand, \(T = 8{,}192\) tokens per GPU). The forward pass takes 5.1 ms and the backward pass 10.2 ms. The first buckets start their all-reduce early, but on 50 GB/s all 13 together need 21.1 ms on the network, more than the backward pass, so they queue up: the step ends at 29.6 ms, with 14.2 ms of exposed communication (red). Without overlap it would be 36.4 ms. Switch to NVLink: the network needs only 5.6 ms, everything except the tail hides under the backward pass, and the step takes 16.3 ms, of which 1.0 ms is the tail. Back on InfiniBand, move \(T\) up to 32,768: the backward pass grows to 40.9 ms and covers all buckets except the last one; only the 6.5 ms tail is left. This is rule (44.5) at work. Note the tail: the last bucket holds the tied embedding table, 147 MiB, which is ready only at the very end.
The time of one step with overlap is then about
- Read the symbols
- \(T_{\text{fwd}}\), \(T_{\text{bwd}}\)
- time of the forward and backward pass on the local batch
- \(T_{\text{comm}}\)
- total time of all bucket all-reduces, from formula (43.4) for each bucket
- \(T_{\text{tail}}\)
- the communication that cannot overlap: mainly the all-reduce of the last bucket
This is the same formula as (43.8) in the last session, which also has an interactive timeline of the overlap. The new question here is: when is \(T_{\text{comm}}\) smaller than \(T_{\text{bwd}}\)? The answer is a short calculation.
When can the backward pass hide the all-reduce?
For a transformer with \(\Psi\) parameters, the backward pass costs about \(4\Psi\) FLOPs per token (twice the forward pass of \(2\Psi\), see session 25). With \(T\) tokens per GPU per step and a GPU that really delivers \(F\) FLOP/s, the backward pass takes \(4\Psi T / F\). The all-reduce of BF16 gradients (\(2\Psi\) bytes) sends \(2 \cdot \frac{N-1}{N} \cdot 2\Psi \approx 4\Psi\) bytes per GPU, so it takes about \(4\Psi/\beta\). The communication fits under the backward pass when
- Read the symbols
- \(T\)
- tokens processed by one GPU in one step (local batch size times sequence length)
- \(F\)
- the FLOP/s one GPU really reaches (peak times utilization)
- \(\beta\)
- the bandwidth per GPU of the slowest link of the ring, in bytes per second
- \(\Psi\)
- the number of parameters; it cancels
The model size cancels. Both the work and the bytes grow with \(\Psi\), so what matters is how many tokens each GPU processes per byte of gradient it sends.
An H100 has 989 TFLOP/s of dense BF16 compute; assume training reaches 40% of it, so \(F \approx 3.96 \times 10^{14}\) FLOP/s. Across servers, each GPU has one 50 GB/s InfiniBand port. Then \(F/\beta = 3.96 \times 10^{14} / 5 \times 10^{10} \approx 7{,}900\) tokens. With sequences of 4,096 tokens, each GPU needs a local batch of at least 2 sequences per step for the all-reduce to hide under the backward pass. Inside one server over NVLink (450 GB/s), \(F/\beta \approx 880\) tokens: almost any batch is enough.
For Llama 2 7B (6.74 billion parameters), the BF16 gradients are 13.5 GB. On 64 GPUs across servers in a flat ring at 50 GB/s, one all-reduce takes about \(2 \cdot \tfrac{63}{64} \cdot 13.5 / 50 = 0.53\) s. With \(T = 8{,}192\) tokens per GPU, the backward pass takes \(4 \cdot 6.74\times10^9 \cdot 8192 / 3.96\times10^{14} = 0.56\) s. Just enough. Halve the local batch and the step becomes communication-bound. (And as the next sections show, plain DDP cannot even hold this model in 80 GB.)
Rule (44.5) explains a real tension. Many GPUs and a fixed global batch mean fewer tokens per GPU, and at some point the all-reduce can no longer be hidden. This is one reason large runs use a very large global batch, and one reason the fastest links (NVLink) are used for the most communication-heavy parallelism, while data parallelism, which communicates once per step, goes across servers. [Likely]
Options that change the buckets
gradient_as_bucket_view=True: by default DDP keeps the gradients inparam.gradand a copy in the bucket buffers. With this flag,param.gradbecomes a view into the bucket, which saves memory equal to the total gradient size and avoids a copy. [Certain]find_unused_parameters=True: a bucket is all-reduced only when all its gradients are ready. If some parameter is not used in a forward pass (a branch that was skipped, a frozen head), its gradient never arrives and the backward pass waits forever. This flag makes DDP walk the autograd graph after each forward pass and mark unused parameters as ready. It costs time, so use it only when needed. [Certain]static_graph=True: tells DDP that the set of used parameters and the graph do not change between iterations. DDP can then optimize, and it supports cases such as activation checkpointing that runs a block twice (see session 39). [Certain]- Communication hooks:
register_comm_hookreplaces the all-reduce of each bucket with your own function, for example one that casts the gradients to BF16 before sending them and back afterwards, which halves the bytes. [Certain]
Global batch, learning rate and gradient accumulation
With DDP the global batch is what the optimizer sees:
- Read the symbols
- \(N\)
- number of data-parallel ranks
- \(b\)
- micro-batch size: examples (or sequences) per rank per forward and backward pass
- \(k\)
- gradient accumulation steps: forward and backward passes per optimizer step
So adding GPUs while keeping \(b\) and \(k\) fixed makes the global batch larger, and a larger batch changes the training. With a larger batch, the gradient estimate is less noisy, so you can take larger steps. For SGD, Goyal et al. (2017) proposed the linear scaling rule: when the batch grows by a factor \(c\), multiply the learning rate by \(c\), with a short warmup at the start. With it they trained ResNet-50 on ImageNet with a batch of 8,192 on 256 GPUs in one hour, with the same accuracy as a small batch. [Certain]
The rule fails beyond some batch size. McCandlish et al. (2018) call it the critical batch size: below it, doubling the batch roughly halves the number of steps; above it, a larger batch gives almost no fewer steps, so the extra GPUs are wasted. [Certain] For Adam, the learning rate should grow more slowly with the batch; analysis of the training as a stochastic differential equation suggests a square-root rule, \(\eta \propto \sqrt{B}\). [Likely] In LLM practice, the global batch is a tuned hyperparameter, set in tokens (often a few million per step) and sometimes increased during training (see session 36). The number of GPUs is then chosen to fit the batch, not the other way round.
Gradient accumulation without extra communication: no_sync
If the global batch must stay fixed but memory allows only a small micro-batch, you accumulate gradients over \(k\) micro-batches before one optimizer step (session 39). A naive DDP loop would all-reduce after every micro-batch: \(k\) times the communication for the same result, because the sum of averages equals the average of sums. DDP has a context manager for this. Inside with model.no_sync():, the backward pass only accumulates into the local .grad; the first backward pass outside it all-reduces the accumulated gradients. [Certain] The forward pass must also be inside the context, or the gradients are still synchronized. [Certain]
- for \(j = 1, \dots, k-1\):
- with
model.no_sync(): \(\big(\text{loss}(\text{model}(x_j), y_j) / k\big)\).backward()// local accumulation, no all-reduce - \(\big(\text{loss}(\text{model}(x_k), y_k) / k\big)\)
.backward()// buckets all-reduce the accumulated sums now - //
.grad\(= \frac{1}{N} \sum_r \frac{1}{k} \sum_j \mathbf{g}_{r,j}\): the gradient of the global batch \(B = N b k\) optimizer.step();optimizer.zero_grad()
The division by \(k\) makes the accumulated gradient a mean over micro-batches, just as DDP’s division by \(N\) makes it a mean over ranks. With accumulation, the all-reduce overlaps only with the backward pass of the last micro-batch, but it happens once per \(k\) micro-batches, so its share of the step time is about \(k\) times smaller.
The code
The first listing checks equation (44.2) in one process: four “GPUs” are four slices of a batch. It also shows the two arithmetic bugs: forgetting to divide by \(N\), and averaging unequal shards.
import torch, torch.nn as nn torch.manual_seed(0) model = nn.Sequential(nn.Linear(8, 16), nn.Tanh(), nn.Linear(16, 1)) X, Y = torch.randn(32, 8), torch.randn(32, 1) loss_fn = nn.MSELoss() # mean over the batch def grads(xb, yb): model.zero_grad() loss_fn(model(xb), yb).backward() return torch.cat([p.grad.flatten() for p in model.parameters()]) g_full = grads(X, Y) # one GPU, global batch of 32 N = 4 # four "GPUs", 8 samples each g_local = [grads(xb, yb) for xb, yb in zip(X.chunk(N), Y.chunk(N))] g_sum = torch.stack(g_local).sum(0) # what all_reduce(SUM) returns g_avg = g_sum / N # what DDP stores in .grad print(torch.allclose(g_avg, g_full, atol=1e-6)) # True print(round((g_sum.norm() / g_full.norm()).item(), 3)) # 4.0: forgot to divide by N print(round((g_local[0] - g_full).norm().item() / g_full.norm().item(), 2)) # 0.92: one shard alone is a noisy estimate # uneven shards: 10 + 10 + 6 + 6 samples; the plain average of local means is wrong sizes = [10, 10, 6, 6] parts = list(zip(X.split(sizes), Y.split(sizes))) g_bad = torch.stack([grads(xb, yb) for xb, yb in parts]).mean(0) g_ok = sum(n * grads(xb, yb) for n, (xb, yb) in zip(sizes, parts)) / sum(sizes) print(torch.allclose(g_bad, g_full, atol=1e-6), torch.allclose(g_ok, g_full, atol=1e-6)) # False True
The second listing is real DDP on two CPU processes with the Gloo backend (change to "nccl" and move the model and data to cuda:LOCAL_RANK on GPUs). It uses a DistributedSampler, set_epoch, and no_sync for accumulation. At the end the ranks all-gather a checksum of their weights to show that the replicas are identical, and the main process checks that the result equals one process training on the same global batches.
import os, torch, torch.nn as nn, torch.distributed as dist, torch.multiprocessing as mp from torch.nn.parallel import DistributedDataParallel as DDP from torch.utils.data import TensorDataset, DataLoader, DistributedSampler def make_model(): torch.manual_seed(0) # same seed everywhere (DDP also broadcasts rank 0) return nn.Sequential(nn.Linear(8, 16), nn.Tanh(), nn.Linear(16, 1)) def make_data(): g = torch.Generator().manual_seed(1) return TensorDataset(torch.randn(64, 8, generator=g), torch.randn(64, 1, generator=g)) def worker(rank, world, accum): os.environ["MASTER_ADDR"], os.environ["MASTER_PORT"] = "127.0.0.1", "29516" dist.init_process_group("gloo", rank=rank, world_size=world) # "nccl" on GPUs torch.manual_seed(1234 + rank) # per-rank seed: dropout, augmentation model = DDP(make_model()) # broadcasts weights and buffers from rank 0 opt = torch.optim.SGD(model.parameters(), lr=0.1) sampler = DistributedSampler(make_data(), num_replicas=world, rank=rank, shuffle=True, seed=0) loader = DataLoader(make_data(), batch_size=4, sampler=sampler) # 4 per rank per micro-step for epoch in range(2): sampler.set_epoch(epoch) # new shuffle each epoch, same on all ranks for i, (x, y) in enumerate(loader): last = (i + 1) % accum == 0 if not last: with model.no_sync(): # accumulate locally, no all-reduce (nn.functional.mse_loss(model(x), y) / accum).backward() continue (nn.functional.mse_loss(model(x), y) / accum).backward() # all-reduce happens here opt.step(); opt.zero_grad() flat = torch.cat([p.detach().flatten() for p in model.parameters()]) check = torch.tensor([flat.sum().item()], dtype=torch.float64) every = [torch.zeros(1, dtype=torch.float64) for _ in range(world)] dist.all_gather(every, check) # compare a checksum across ranks if rank == 0: print("accum", accum, "checksums", [round(c.item(), 6) for c in every]) torch.save(flat, f"/tmp/ddp_w_{accum}.pt") dist.destroy_process_group() def single_gpu_reference(world, accum): model, data = make_model(), make_data() opt = torch.optim.SGD(model.parameters(), lr=0.1) for epoch in range(2): g = torch.Generator(); g.manual_seed(0 + epoch) # what DistributedSampler does order = torch.randperm(64, generator=g).tolist() step = 4 * world * accum # global batch for s in range(0, 64, step): x, y = data[order[s:s + step]] nn.functional.mse_loss(model(x), y).backward() # one big batch, plain mean opt.step(); opt.zero_grad() return torch.cat([p.detach().flatten() for p in model.parameters()]) if __name__ == "__main__": for accum in (1, 2): mp.spawn(worker, args=(2, accum), nprocs=2) ddp_w = torch.load(f"/tmp/ddp_w_{accum}.pt") ref = single_gpu_reference(2, accum) print("same as one GPU with batch", 8 * accum, ":", torch.allclose(ddp_w, ref, atol=1e-5)) # accum 1 checksums [-1.428242, -1.428242] # same as one GPU with batch 8 : True # accum 2 checksums [-1.643059, -1.643059] # same as one GPU with batch 16 : True
The printed values are from PyTorch 2.8 on CPU. Two ranks with 4 examples each and \(k = 2\) give exactly the same weights as one process with batches of 16, which is (44.2) and (44.6) at work.
The limit: every GPU holds everything
DDP divides the data, not the model. Each rank keeps a full replica, which means a full copy of everything the training step needs. With the usual mixed-precision recipe and Adam (see session 38):
- Read the symbols
- \(\Psi\)
- the number of parameters
- \(m\), \(v\)
- Adam’s first and second moment estimates, one FP32 number each per parameter
- \(M_{\text{act}}\)
- the activations saved for the backward pass; they depend on the micro-batch, not on \(N\)
(The standard PyTorch AMP recipe keeps FP32 weights and FP32 gradients instead: \(4 + 4 + 4 + 4\) is also 16 bytes per parameter.) The important point is what is missing from (44.8): \(N\). Adding GPUs makes training faster, but it does not lower the memory per GPU by one byte. The \(N\) copies of the optimizer state are identical, so \(N-1\) of them are pure redundancy. On top of (44.8), DDP’s bucket buffers hold another copy of the gradients unless you set gradient_as_bucket_view=True. [Certain]
Llama 2 7B has 6,738,415,616 parameters (32 layers, hidden size 4,096, MLP size 11,008, vocabulary 32,000, untied embeddings). With (44.8): \(16 \times 6.74 \times 10^{9} = 107.8\) GB per GPU before any activation; with the extra bucket copy of BF16 gradients, 121.3 GB. An 80 GB H100 cannot hold it, and neither can 1,024 of them. The largest model whose training state fits in 80 GB with DDP is \(80 / 16 = 5\) billion parameters, with no room left for activations.
How to read Figure 5
Start at Llama 2 7B on 8 GPUs of 80 GB: the DDP bar is 107.8 GB and crosses the dashed line, so it does not fit, while the sharded bar is 13.5 GB. Now move the GPU slider to 1,024: the DDP bar does not move at all. Only the sharded bar shrinks. Untick gradient_as_bucket_view to see the extra gradient copy that DDP keeps by default. Then go down to 1B parameters: 16 GB, so DDP fits easily, and since it is the simplest and has the least communication, it is the right choice there.
This is the wall. DDP is the best choice when the training state fits on one GPU with room for activations: it is simple, robust, and moves the fewest bytes (one all-reduce of the gradients per step). For larger models, the next session removes the redundancy step by step. ZeRO stage 1 shards the optimizer state, stage 2 also the gradients, and stage 3 also the weights. Stages 1 and 2 even keep DDP’s communication volume, because an all-reduce is a reduce-scatter plus an all-gather (43.4). [Certain]
Seeds and determinism across ranks
A data-parallel job has \(N\) copies of every random number generator. Some of them must agree across ranks, and some must not. Getting this wrong does not crash the job. It silently changes the training, which is why it is a favorite interview topic.
| Source of randomness | Across ranks it must be | Why |
|---|---|---|
| Weight initialization | the same | replicas must start equal; DDP’s start-up broadcast enforces this for parameters and buffers, but not for tensors that are plain attributes of a module |
| Data order (sampler seed) | the same, and set_epoch every epoch | each rank computes the shared permutation by itself; different seeds mean overlapping shards and skipped examples (Figure 2) |
| Dropout, stochastic depth, token masking, data augmentation | different (for example seed + rank) | with the same seed, every rank draws the same masks and crop positions; the noise is then correlated across the global batch instead of independent [Likely] |
| DataLoader workers | different per rank and per worker | PyTorch derives each worker’s seed from the main process seed, so a per-rank main seed gives per-rank workers [Likely] |
| Gradient noise from the hardware (atomic adds, kernel choice) | does not matter for sync | local gradients may differ in the last bits, but after the all-reduce every rank has the same bits, so the replicas do not drift |
The last row needs a closer look. A ring all-reduce computes each chunk of the sum in one place, then copies the finished chunk to everyone (Figure 2 of session 43). So all ranks receive bitwise the same result and the replicas stay bitwise identical. [Likely] But the result depends on the order of the additions, and floating-point addition is not associative: \((a + b) + c\) can differ from \(a + (b + c)\) in the last bit. Different \(N\), a different NCCL algorithm (ring or tree), or a different bucket layout changes the order. So a run on 16 GPUs is not bitwise equal to the same run on 8 GPUs, even with the same global batch and the same seeds. [Certain] Figure 6 shows the size of the effect.
How to read Figure 6
Each bar is the same mathematical quantity computed with a different number of ranks; the highlighted bar is the \(N\) of the slider. In FP32, the results differ by a few units in the last place (ULP): for example the results for \(N = 1\) and \(N = 2\) differ by 6 ULP. That is a relative difference of about \(10^{-7}\), harmless for training but enough to make two runs diverge in their last digits after many steps. Switch to BF16 arithmetic: the errors grow to about 1 to 3%, which is why gradients are usually reduced in FP32, or at least accumulated in FP32. [Likely]
What can you control? Within one configuration (same \(N\), same software versions, same hardware, deterministic kernels via torch.use_deterministic_algorithms(True)), runs are usually reproducible. [Likely] Across configurations, aim for statistical reproducibility (the same loss curve within noise) rather than bitwise equality. For resuming a job, save every rank’s RNG states, the sampler epoch and position, and the optimizer state, or the restarted run will see different data and different dropout masks. A cheap health check during long runs is the one in the second listing: all-gather a checksum of the weights every few thousand steps and alert if the replicas disagree. [Likely]
Where this is used
| Setting | How data parallelism is used | Note |
|---|---|---|
| Pretraining and fine-tuning models up to a few billion parameters | plain DDP with bucketed all-reduce | the training state (16 bytes per parameter) fits on one GPU |
| Large LLM pretraining (Llama-class) | the outer “data-parallel” dimension; inside it, ZeRO/FSDP, tensor and pipeline parallelism | combined in session 49 |
| Video classification and action recognition | DDP, each rank decodes and augments its own clips | video decoding is the bottleneck, so each rank also runs several DataLoader workers; 3D CNNs with BatchNorm often use SyncBatchNorm because the per-GPU batch is small [Likely] |
| Contrastive video–text retrieval models (CLIP-style) | DDP for the gradients, plus an all-gather of embeddings so each clip sees the texts of the whole global batch as negatives | the all-gather must pass gradients back (torch.distributed.nn.all_gather) [Likely] |
| Vision-language model fine-tuning with a frozen vision encoder | DDP with find_unused_parameters or static_graph when some parameters get no gradient | otherwise the buckets wait forever |
| Video fingerprinting, key-moment detection, building a Video-RAG index | data-parallel inference: DistributedSampler shards the videos, no DDP wrapper (no gradients) | results are all-gathered or written per rank and merged |
| Distributed evaluation | DistributedSampler with padding, then all-reduce of counts | remove the padded duplicates, or the metric is slightly wrong |
Common mistakes
- Forgetting
sampler.set_epoch(epoch). Every epoch then uses the same order. Training still works, but worse, and nothing warns you. - Different sampler seeds on different ranks. For example, seeding everything with
seed + rankand passing that to the sampler. Ranks then use different permutations, so shards overlap and some data is never seen. Use the same sampler seed and a per-rank seed only for dropout and augmentation. - A collective inside
if rank == 0:. Logging that callsall_reduceor a checkpoint function that gathers state only on rank 0 makes rank 0 wait for partners that never come: a hang, not an error. - Unequal work across ranks. Without padding or
drop_last, or with a hand-written data split, one rank may run one step more and hang. DDP’sjoin()context manager handles uneven inputs on purpose. [Certain] - Mean over tokens per rank with unequal token counts. If each rank averages its loss over its own number of non-padding tokens and the counts differ, DDP’s division by \(N\) weights the ranks wrongly, as in (44.3). Fix: all-reduce the token count and divide the summed loss by the global count (and multiply by \(N\) to cancel DDP’s division), or use packed sequences of equal length.
- Unused parameters without
find_unused_parameters. A skipped branch or a frozen layer inside the DDP module leaves a bucket incomplete. Recent PyTorch raises an error at the next iteration; older setups simply hung. [Likely] - Accumulating without
no_sync, or with the forward pass outside it. The result is correct but the communication is \(k\) times larger. - Changing \(N\) but not the learning rate or batch. Doubling the GPUs with a fixed micro-batch doubles the global batch (44.6). Either keep the global batch fixed (fewer accumulation steps) or retune the learning rate.
Interview questions
What does DDP do in one training step?basic
Why is DDP mathematically equivalent to training on one GPU with the global batch?basic
Why does DDP use gradient buckets? How big are they?core
Does adding GPUs with DDP let you train a bigger model?basic
How much does DDP communicate per step, and when is it hidden?core
What is no_sync, and why does the forward pass have to be inside it?core
Why must you call sampler.set_epoch, and why must the sampler seed be the same on all ranks?core
DDP versus DataParallel?basic
Two runs with the same seeds, one on 8 GPUs and one on 16 with the same global batch, give slightly different loss curves. Bug?advanced
A model with a frozen vision tower hangs in the second backward pass under DDP. Why, and what are the fixes?advanced
requires_grad=False on frozen parameters before wrapping (then DDP ignores them), keep frozen modules outside the DDP wrapper, use find_unused_parameters=True for truly dynamic graphs (costs a graph traversal per step), or static_graph=True if the set of unused parameters is fixed.How do you scale the learning rate when you go from 8 to 64 GPUs?core
Exercises
- In Figure 1, compute rank 2’s local gradient at \(w = 0.5\) by hand (examples \((2, 5)\) and \((4, 7)\)).
- A job uses \(N = 32\) GPUs, a micro-batch of 4 sequences of 4,096 tokens, and 8 accumulation steps. How many tokens are in one global batch?
- A rank has 3,000 non-padding tokens and another has 1,000. Each computes its mean token loss, and DDP averages the two gradients. By what factor is a token on the second rank over-weighted compared with the correct global mean?
- Using (44.8), what is the largest model you can train with plain DDP and Adam on a 141 GB GPU, ignoring activations?
- With \(F = 4 \times 10^{14}\) FLOP/s and \(\beta = 25\) GB/s, how many tokens per GPU per step are needed to hide the all-reduce (44.5)?
- The DistributedSampler gets \(n = 1{,}000\) examples and \(N = 64\) ranks. How many indices does each rank get with and without
drop_last, and how many duplicates or skipped examples are there?
Answerscheck
Key takeaways
- Data parallelism: a full replica on every rank, a different shard of the batch on each, one all-reduce of the gradients per step. With equal shards, the averaged gradient is exactly the full-batch gradient (44.2).
- Replicas stay identical because they start equal (DDP broadcasts rank 0 at construction) and apply the same update. DDP never sends weights after that.
- DDP all-reduces gradients in buckets (25 MiB cap, 1 MiB first bucket, rebuilt after the first iteration in the real order), overlapped with the backward pass. The last bucket is the tail that cannot be hidden (44.4).
- The all-reduce hides under the backward pass when the tokens per GPU per step exceed \(F/\beta\) (44.5), whatever the model size.
- Global batch \(B = N b k\) (44.6). More GPUs change the batch, and then the learning rate;
no_syncmakes accumulation cost one all-reduce per optimizer step. - The wall: 16 bytes per parameter on every GPU, independent of \(N\) (44.8). A 7B model needs about 108 GB. ZeRO and FSDP remove this redundancy.
- Seeds: same for initialization and the sampler (and call set_epoch), different for dropout and augmentation. Changing \(N\) changes summation order, so bitwise equality holds only within one configuration.
Key formulas
References for this session
- Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- PyTorch (docs). Distributed Data Parallel (design notes: construction, buckets, backward hooks)
- PyTorch (docs). torch.nn.parallel.DistributedDataParallel API reference
- PyTorch (docs). torch.utils.data.distributed.DistributedSampler
- PyTorch (docs). Reproducibility (randomness and deterministic algorithms)
- Goyal et al. (2017). Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- McCandlish, Kaplan, Amodei et al. (2018). An Empirical Model of Large-Batch Training
- Malladi, Lyu, Panigrahi and Arora (2022). On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel