DDP vs FSDP2 and reading a profile
In this lab you train the same small GPT three ways: in one process, with DDP, and with FSDP2, using 2 or 4 processes on a CPU in place of GPUs. You check that all runs give the same losses, then measure memory per rank, the collectives in one step and the step time. You capture a profiler trace, read it, and find the top three time sinks. Finally you add activation checkpointing to FSDP2 and measure what it saves and what it costs.
What you will learn
- How to wrap the same model with DDP and with FSDP2 (fully_shard on each block, then on the root), and why the order of wrapping matters
- How to prove that a distributed run is correct: the loss curve must match a single-process run with the same global batch
- How to measure memory per rank, count the collectives of one step, and turn them into bytes with the ring formulas
- Why FSDP2 sends 1.5 times the bytes of DDP in full precision, and the same bytes with BF16 gathers and FP32 reductions
- How to capture a torch.profiler trace, the difference between self time and total time, and how to find the top 3 time sinks
- How to combine activation checkpointing with FSDP2, and what it saves (91% of activation memory here) and costs (32% more matrix multiplies)
- What transfers from a CPU simulation to real GPUs (memory, collective counts, loss parity) and what does not (step time)
Why this matters
The last few sessions were theory. Session 44 copies the model to every GPU. Session 45, session 46 and session 47 shard the model instead, so each GPU keeps only a slice. In this lab you stop reading and start measuring. You train one small GPT with DDP and with FSDP2, on the same data, and you check every claim from those sessions with numbers from your own runs.
Then you learn the skill that separates people who use distributed training from people who can fix it: reading a profile. You capture a trace of one training step with torch.profiler, open it, and find where the time goes. Last, you add activation checkpointing (recompute activations in the backward pass instead of storing them, session 39) on top of FSDP2 and measure both sides of the trade.
- Interviews. “How much memory does each GPU need with DDP and with FSDP?”, “How much more does FSDP communicate?”, and “Your step got 30% slower after you switched to FSDP. How do you find out why?” are common questions for training-infrastructure roles. Each one is answered with a measured number below.
- Real systems. torchtitan, PyTorch’s reference code for LLM pretraining, uses exactly the pieces of this lab:
fully_shardon each transformer block, activation checkpointing applied before sharding, and BF16 parameters with FP32 gradient reduction. [Certain: torchtitan v0.2.0 source] - Debugging. The checks here (losses equal to a single-process run, collectives counted per step, a memory timeline, a top-3 list from the profiler) take minutes and find most setup bugs before you pay for a large run.
Setup and plan
You do not need GPUs. PyTorch can run its distributed code on a CPU: we start 2 or 4 processes on one machine, and each process plays the role of one GPU (one rank). They talk through gloo, PyTorch’s CPU communication backend, instead of NCCL. Every measured number on this page comes from such runs on a laptop CPU with PyTorch 2.8.0, 4 threads per process. [Certain: measured]
Be clear about what this simulation can and cannot tell you:
| Quantity | Transfers to GPUs? | Why |
|---|---|---|
| Loss curve, equal to one process | yes | the math is the same; only the device changes |
| Bytes of model states per rank | yes | same tensors, same sharding |
| Number and size of collectives | yes | DDP and FSDP2 issue the same calls on any backend |
| Activation memory, and what checkpointing saves | yes, almost | GPU kernels may keep slightly different tensors (for example FlashAttention) |
| Step time, overlap, top time sinks | no | CPU math is much slower, gloo is not NCCL, and there are no CUDA streams to overlap on |
The model is the GPT of session 30, a little wider, with characters of Tiny Shakespeare as tokens. The output head is not tied to the embedding, so every parameter belongs to exactly one place.
| Setting | Value | Meaning |
|---|---|---|
| \(|V|, T\) | 65, 128 | vocabulary (characters) and context length |
| \(d, L, h\) | 256, 6, 8 | width, number of blocks, attention heads |
| block | RMSNorm, attention (SDPA), RMSNorm, GELU MLP with \(4d\) | no biases; Pre-LN |
| \(\Psi\) | 4,787,968 | parameters: 6 blocks of 786,944 plus 66,304 in embeddings, final norm and head |
| global batch | 16 windows × 128 tokens | split evenly: 8 windows per rank with \(W = 2\), 4 with \(W = 4\) |
| optimizer | AdamW, lr \(10^{-3}\), weight decay 0.1, 30 steps | everything in FP32, so 16 bytes per parameter of model states |
The plan: (1) one model and one data stream for all runs, (2) the DDP version, (3) the FSDP2 version, (4) check that all runs agree, (5) memory, (6) communication and step time, (7) read a profile, (8) add activation checkpointing.
Step 1: one model, one stream of data
A fair comparison needs the same training problem in every run. So every run builds the model with torch.manual_seed(0) (same initial weights on every rank) and draws the global batch of 16 windows from one generator with seed 1234. Rank \(r\) then takes its own slice: windows \(r \cdot b\) to \((r+1) \cdot b - 1\), where \(b = 16 / W\).
Why does this make the runs comparable? The loss is a mean over all tokens, and all ranks hold the same number of tokens. So the gradient of the global batch is exactly the average of the gradients that the ranks compute on their slices:
- Read the symbols
- \(\theta\)
- all the weights of the model
- \(W\)
- world size: the number of ranks (simulated GPUs)
- \(\mathcal{L}_r\)
- the mean cross-entropy over the \(b \cdot T\) tokens of rank \(r\)
- \(\ell\)
- the loss of one token
DDP averages gradients with an all-reduce; FSDP2 averages them with a reduce-scatter. Both compute the right side of (52.1). So a single process that trains on all 16 windows must produce the same losses, step by step, up to floating-point rounding. This is the strongest test you have: a wrong mask, a missing average, a rank that reads the wrong data or a parameter that never syncs all show up as a different loss curve.
Step 2: the DDP version
DDP (DistributedDataParallel) keeps a full copy of the model on every rank. During the backward pass it groups gradients into buckets and starts an all-reduce for each bucket as soon as all its gradients are ready, so communication overlaps with the rest of the backward pass (session 44). We wrap the model and add a communication hook: a function that DDP calls for every bucket. Ours does what the default does (divide by \(W\), all-reduce) and also records the bucket size.
model = DDP(model) # on GPUs: DDP(model, device_ids=[rank]) def hook(state, bucket): buf = bucket.buffer(); COMM.append(("all_reduce", buf.numel() * buf.element_size())) return dist.all_reduce(buf.div_(W), async_op=True).get_future().then(lambda f: f.value()[0]) model.register_comm_hook(None, hook) opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.1)
What we measured in every step: two all-reduces. The first bucket held 279,040 values (1.12 MB): the output head, the final norm and the down projection of the last block, the first gradients that backward produces. The second held the other 4,508,928 values (18.04 MB). Together that is all 4,787,968 gradients, 19.15 MB in FP32. [Certain: measured] DDP makes its first bucket small on purpose, so that communication starts early; the later buckets are capped at 25 MB by default (bucket_cap_mb). [Certain: PyTorch 2.8 source]
- input: full model copy, full AdamW states, this rank’s slice \((x_r, y_r)\)
- forward on \(x_r\); store activations // no communication
- backward; when every gradient of a bucket is ready, start all-reduce of that bucket // runs while backward continues
- at the end of backward, wait for the last all-reduce // the exposed part
- every rank now has the same averaged gradient (52.1); full AdamW step on the full model
Step 3: the FSDP2 version
FSDP2 is the fully_shard API (session 47). You call it on each transformer block, then on the whole model (the root). Each call makes one parameter group: its parameters are all-gathered together before the module runs and their gradients are reduce-scattered together after its backward. The root call takes everything not already in a group, here the embeddings, the final norm and the head. Each parameter becomes a DTensor (a tensor that knows it is split across ranks) sharded on dimension 0: with \(W = 2\), rank 0 holds the first half of the rows of every weight matrix.
mesh = init_device_mesh("cpu", (W,)) # "cuda" on GPUs for block in model.blocks: fully_shard(block, mesh=mesh) # one group per block: bottom-up fully_shard(model, mesh=mesh) # the root: embeddings, final norm, head opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.1) # AFTER sharding
Three details matter. Order: shard the blocks first, then the root; the docs say to call fully_shard bottom-up. [Certain] Optimizer last: build it after sharding, so it holds the sharded DTensors; an optimizer built before would update tensors that no longer exist. Reshard after forward: by default (reshard_after_forward=None) each block frees its gathered weights after its forward and gathers them again in backward, while the root keeps its weights, because backward starts with the root’s head. [Certain: PyTorch 2.8 docstring]
What we measured. Rank 0 holds 9.58 MB of the 19.15 MB of weights. The split is not perfectly even: the embedding and the head have 65 rows, so rank 0 gets 33 rows and rank 1 gets 32, and the root’s all-gather is padded from 66,304 to 66,816 values. In one step FSDP2 issued 13 all-gathers (1 for the root, 6 for the blocks in forward, 6 again in backward) and 7 reduce-scatters (6 blocks and the root). [Certain: measured]
- input: shards of all weights and AdamW states (1/W of each); this rank’s slice \((x_r, y_r)\)
- all-gather the root group; it stays gathered for the whole step
- for block \(i = 1, \dots, L\): all-gather block \(i\) (prefetched during block \(i-1\)); forward; free its full weights
- compute the loss; backward through the head
- for block \(i = L, \dots, 1\): all-gather block \(i\) again (prefetch \(i-1\)); compute its gradients; reduce-scatter them so each rank keeps the averaged gradient of its own shard; free full weights and gradients
- reduce-scatter the root’s gradients; AdamW step on the local shards only // no communication
Step 4: check that every run trains the same model
Run all versions for 30 steps and compare the global loss (the mean of the rank losses) with the single-process run. This is the first thing to do after any change to a distributed setup.
| Run | Loss at step 0 | Loss at step 29 | Largest difference from one process, steps 0 to 11 | Largest difference, all 30 steps |
|---|---|---|---|---|
| one process, 16 windows | 4.1330 | 3.3593 | 0 | 0 |
| DDP, \(W = 2\) | 4.1330 | 3.3594 | \(3.6 \times 10^{-7}\) | \(1.3 \times 10^{-5}\) |
| DDP, \(W = 4\) | 4.1330 | 3.3594 | \(3.0 \times 10^{-7}\) | \(1.6 \times 10^{-5}\) |
| FSDP2, \(W = 2\) | 4.1330 | 3.3593 | \(2.7 \times 10^{-6}\) | \(2.7 \times 10^{-4}\) |
| FSDP2, \(W = 4\) | 4.1330 | 3.3593 | \(4.0 \times 10^{-6}\) | \(4.3 \times 10^{-4}\) |
| FSDP2 + checkpointing, \(W = 2\) | bit-for-bit the same losses as FSDP2 without checkpointing | |||
[Certain: measured] How to read this table. At the start, the differences are \(10^{-7}\) to \(10^{-6}\): the size of FP32 rounding. They are not zero because the ranks add their gradients in a different order than one process does, and floating-point addition is not associative. They grow over 30 steps because AdamW divides by \(\sqrt{v}\), which is tiny early in training, so it magnifies tiny gradient differences. FSDP2 drifts more than DDP, probably because it also runs the AdamW update on shards through DTensor, a different sequence of operations. [Likely]
One more check is hidden in the data: at step 12 every run has a loss spike to 6.34 and recovers by step 13. Because every run spikes at the same step by the same amount, the spike is a property of the training recipe (we used no warmup, session 36), not of the parallelism. A bug would not reproduce so exactly. Checkpointing changes nothing at all, as it should: the recomputed forward is exactly the same computation.
Step 5: memory per rank
To measure memory on a CPU we start torch.profiler with profile_memory=True before building the model. Then every tensor allocation and free is recorded with a running total of live bytes, so we get the exact memory timeline of each rank. (On a GPU you would use torch.cuda.max_memory_allocated() or the memory snapshot tool.) Figure 1 shows the last of four steps on rank 0.
How to read Figure 1
Start with FSDP2 at 2 processes. The line starts at 28.76 MB: rank 0’s half of the weights (9.58 MB) and of the AdamW states (19.15 MB). The gradients were freed by zero_grad(set_to_none=True). The forward pass adds 117 MB of activations, the largest part by far. The peak (155 MB) comes early in the backward pass, when the activations are still there and a block is gathered with its gradients. Now click DDP: it starts at 76.64 MB (full weights, full AdamW states, and a 19.15 MB bucket buffer that DDP keeps for the whole run) and peaks at 196 MB. Switch to 4 processes: the DDP start does not move (DDP never shards), but every peak falls, because each rank now has 4 windows instead of 8, so half the activations. Finally click “FSDP2 + AC every block”: the forward pass adds only 10 MB.
The measured numbers, all for rank 0, in MB:
| Run (\(W\) processes) | Start of step | End of forward | Peak | Peak vs DDP |
|---|---|---|---|---|
| one process (16 windows) | 57.49 | 291.31 | 297.12 | |
| DDP, 2 | 76.64 | 193.55 | 196.49 | 100% |
| FSDP2, 2 | 28.76 | 145.94 | 155.18 | 79% |
| FSDP2, no reshard, 2 | 28.76 | 164.83 | 168.24 | 86% |
| DDP, 4 | 76.64 | 135.10 | 137.13 | 100% |
| FSDP2, 4 | 14.40 | 73.13 | 81.46 | 59% |
| FSDP2, no reshard, 4 | 14.40 | 92.01 | 95.30 | 69% |
[Certain: measured] Three formulas explain every row. First, the model states: weights, gradients and the two AdamW moments, 4 + 4 + 8 = 16 bytes per parameter in FP32 (session 38):
DDP’s extra \(4\Psi\) is its bucket buffer: by default (gradient_as_bucket_view=False) gradients are copied into the buckets, so both exist. With gradient_as_bucket_view=True the gradients live inside the buckets and DDP needs \(16\Psi\). [Certain: the 19.15 MB is measured; the flag is in the DDP API]
Second, the activations. We counted the tensors that autograd saves for the backward pass in one block: 18 vectors of size \(d\) per token (the MLP’s \(4d\) hidden layer twice, \(q, k, v\), and the inputs and outputs of the norms, attention and residual adds), plus 10 small numbers per token. [Certain: measured]
- Read the symbols
- \(n\)
- tokens per rank: \(b \cdot T\) (1,024 with \(W = 2\))
- \(c\)
- values saved per token per block, in units of \(d\); about 18 for this block (18.04 measured)
- \(p\)
- bytes per value: 4 in this FP32 lab, 2 with BF16 activations
- \(A_{\text{AC}}\)
- the activations with checkpointing of every block (Step 8)
Third, the peak of FSDP2: its states, the activations, the root kept gathered (weights and gradients), and in the backward pass one block’s gathered weights, its gradients and the prefetched next block:
- Read the symbols
- \(\psi_{\text{root}}, \psi_{\text{block}}\)
- parameters in the root group (66,304) and in one block (786,944)
- \(3\,\psi_{\text{block}}\)
- current block’s weights + its gradients + the next block’s weights (backward prefetch)
States. \(16 \times 4{,}787{,}968 / 2 = 38.30\) MB. At the start of a step the gradient shards are freed, so we expect \(12\Psi/W = 28.73\) MB; we measured 28.76 MB.
Activations. \(n = 8 \times 128 = 1{,}024\) tokens: \(A = 1{,}024 \times 6 \times 18.04 \times 256 \times 4 = 113.5\) MB in the blocks, plus 3.4 MB for the embeddings, final norm, head and loss: 116.9 MB. The forward pass added \(145.94 - 28.76 = 117.2\) MB (0.27 MB of it is the gathered root).
Peak. Formula (52.4): \(38.30 + 116.9 + 4 \times (2 \times 66{,}304 + 3 \times 786{,}944)/10^6 = 38.30 + 116.9 + 9.97 = 165.2\) MB. Measured: 155.18 MB. The formula is 6% high because at the peak, early in backward, most gradient shards do not exist yet. It is an upper estimate, which is what you want for planning.
DDP. \(20\Psi = 95.76\) MB of states (76.64 MB at the start of the step plus 19.15 MB of gradients that appear during backward: 95.79 MB measured at the end of backward) and the same 117 MB of activations.
Two lessons from the table. FSDP2 helps more as \(W\) grows: its states fall as \(1/W\), DDP’s do not. At this size, activations dominate: with 117 MB of activations against 38 MB of states, sharding alone cuts the peak by only 21% at \(W = 2\). Real LLM runs are often the same at long context, which is why Step 8 matters.
Step 6: communication and step time
Each recorded collective has a size: the full tensor in bytes. With the ring algorithms of session 43, an all-gather or reduce-scatter of \(N\) bytes over \(W\) ranks sends \(N(W-1)/W\) bytes from each rank, and an all-reduce sends twice that. Adding up one step:
- Read the symbols
- \(V\)
- bytes sent by each rank in one training step
- \(b_p\)
- bytes per value of the gathered weights (4 here; 2 with BF16
param_dtype) - \(b_g\)
- bytes per value of the reduced gradients (4 here; torchtitan also reduces in FP32)
- \(2\Psi b_p\)
- FSDP2 gathers every block twice: in forward and again in backward
| Run | Collectives per step | Full-tensor bytes | Sent per rank, ring | Formula (52.5) |
|---|---|---|---|---|
| DDP, \(W = 2\) | 2 all-reduce | 19.15 MB | 19.15 MB | 19.15 MB |
| FSDP2, \(W = 2\) | 13 all-gather, 7 reduce-scatter | 38.04 + 19.15 MB | 28.60 MB (1.49×) | 28.73 MB |
| FSDP2, no reshard, \(W = 2\) | 7 all-gather, 7 reduce-scatter | 19.15 + 19.15 MB | 19.15 MB (1.00×) | 19.15 MB |
| DDP, \(W = 4\) | 2 all-reduce | 19.15 MB | 28.73 MB | 28.73 MB |
| FSDP2, \(W = 4\) | 13 all-gather, 7 reduce-scatter | 38.05 + 19.16 MB | 42.90 MB (1.49×) | 43.09 MB |
[Certain: measured counts and sizes; computed send volumes] The measured FSDP2 volume is slightly below the formula because the root is gathered only once. So in full precision, FSDP2 sends 1.5 times the bytes of DDP, as session 46 derived. But look at (52.5) with the torchtitan defaults, BF16 gathers (\(b_p = 2\)) and FP32 reduction (\(b_g = 4\)): \(V_{\text{FSDP2}} = 8\Psi(W-1)/W\), exactly the same as DDP all-reducing FP32 gradients. For Llama 2 7B on 8 GPUs both are 47.2 GB per step per GPU. [Certain: computed] The “1.5×” is true only when both use the same dtype.
Step time
Here is the measured step time: for each run, the median over steps 5 to 29 (the slowest rank counts). We repeated each configuration up to 5 times. Read it with care: other programs were running on the same machine, and the same configuration varied by up to 3× between runs. The fastest run is the least disturbed one, so compare that column. [Certain: measured]
| Run (processes) | Fastest run | Median of runs | Slowest run | Runs |
|---|---|---|---|---|
| one process, 16 windows | 76.2 ms | 121.9 ms | 176.4 ms | 5 |
| DDP, 2 | 73.8 ms | 100.2 ms | 198.4 ms | 5 |
| FSDP2, 2 | 100.1 ms | 131.6 ms | 316.8 ms | 5 |
| FSDP2, no reshard, 2 | 91.8 ms | 98.5 ms | 203.0 ms | 5 |
| DDP + AC every block, 2 | 95.8 ms | 102.3 ms | 129.8 ms | 5 |
| FSDP2 + AC every other block, 2 | 109.8 ms | 116.3 ms | 136.2 ms | 3 |
| FSDP2 + AC every block, 2 | 119.3 ms | 129.9 ms | 318.9 ms | 5 |
| DDP, 4 | 82.7 ms | 87.6 ms | 100.8 ms | 5 |
| FSDP2, 4 | 121.3 ms | 124.5 ms | 175.8 ms | 5 |
| FSDP2, no reshard, 4 | 92.0 ms | 110.4 ms | 148.3 ms | 5 |
| FSDP2 + AC every block, 4 | 136.3 ms | 141.6 ms | 188.2 ms | 5 |
On this CPU, FSDP2 is slower than DDP (fastest runs: 100.1 against 73.8 ms with 2 processes, +36%; 121.3 against 82.7 ms with 4, +47%), and the gap grows with \(W\). Checkpointing every block adds 19% to FSDP2 and 30% to DDP. That is a property of the simulation, not of FSDP2. With gloo on a CPU, every collective is a memory copy done by CPU threads that compete with the math for the same cores, and there are no CUDA streams on which transfers can run beside compute. On GPUs, FSDP2 runs its all-gathers and reduce-scatters on separate streams and prefetches the next block, so most of the transfer hides behind compute. The useful model is:
The first all-gather cannot hide (nothing computes before it), and the optimizer must wait for the last reduce-scatter. Everything in between can hide if each transfer is shorter than the compute it overlaps. Figure 2 plays this out for 6 blocks.
How to read Figure 2
At the default ratio 0.3 with BF16 gradients, DDP exposes only its last all-reduce (0.6 units) and FSDP2 exposes three transfers of 0.3: the first all-gather, the first all-gather of the backward pass and the last reduce-scatter (0.9 units). Both are within 5% of pure compute. Drag the slider to 1.5: now one all-gather takes longer than one forward block, prefetching can no longer keep up, and FSDP2 falls far behind DDP. “No reshard” skips the backward all-gathers; in this model it even beats DDP at large ratios, because its reduce-scatter moves half the bytes of DDP’s all-reduce. It pays with the memory of Step 5. What sets the ratio on real hardware? The bytes of one block over the link bandwidth, against the FLOPs of one block over the GPU speed. More tokens per GPU raise the compute and lower the ratio; slower links (across nodes) raise it.
Step 7: capture and read a profile
A profiler records every operation with its start time and duration. torch.profiler does it for PyTorch: CPU-side operators (aten::mm and so on), your own labels from record_function, and on GPUs the CUDA kernels. Two ideas are needed to read its output:
- Total time of an event: from its start to its end, including everything it called.
- Self time: total time minus the time of its children. Self time is where the time was really spent; summing self times never counts anything twice.
Do not profile step 0: the first steps allocate memory and warm up caches. A schedule skips some steps, warms up, then records a few:
from torch.profiler import profile, ProfilerActivity, record_function, schedule acts = [ProfilerActivity.CPU] + ([ProfilerActivity.CUDA] if torch.cuda.is_available() else []) with profile(activities=acts, schedule=schedule(wait=5, warmup=2, active=3), record_shapes=True) as prof: for step in range(30): with record_function("## forward ##"): loss = model(x, y) with record_function("## backward ##"): loss.backward() with record_function("## optimizer ##"): opt.step(); opt.zero_grad(set_to_none=True) prof.step() # tells the schedule a step ended print(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=8)) # self_cuda_time_total on GPUs prof.export_chrome_trace("trace.json") # open in ui.perfetto.dev or chrome://tracing
For FSDP2 with \(W = 2\), the table starts like this (3 recorded steps; real output, shortened):
Name Self CPU % Self CPU # of Calls aten::mm 29.22% 104.933ms 225 aten::_scaled_dot_product_flash_attention_.. 5.07% 18.212ms 18 aten::copy_ 4.10% 14.737ms 1419 aten::gelu_backward 3.51% 12.594ms 18 aten::mul 3.28% 11.775ms 390
The table is a good start, but it groups by operator name, so the same cost can be split over many rows (FSDP2 makes one row per block for its hooks). The trace keeps the time order, which tells you when each cost happens and what waits for what. Figure 3 is the real trace of one step, step 6 of the run, on rank 0.
torch.profiler on rank 0 (CPU, gloo). Row 2: top-level operators longer than 0.3 ms (in backward, the autograd node of each operator), colored by kind. Row 3: FSDP2’s hooks, whose self time is mostly waiting for an all-gather or a reduce-scatter. Rows 4 and 5: the collectives, which gloo runs on its own threads. The top-3 line groups the self time (52.7) of all operators on the main thread in this step.How to read Figure 3
In DDP, look at the two collectives in the backward pass: the small first bucket starts right after the head’s gradient, the big one near the end. The backward phase has 9.4 ms of self time outside any operator: the autograd engine and, mostly, the wait for that last all-reduce. That is DDP’s exposed communication. Switch to FSDP2: before each block, a purple FSDP2 hook bar waits for the block’s all-gather; in backward, red hooks wait for the reduce-scatters. With 4 processes the two FSDP2 groups climb to places 2 and 3 of the top 3: on this CPU, communication now costs more than all non-matmul math. With checkpointing, the forward ops appear twice: once in forward, once again inside the backward pass.
The top 3 time sinks in each run, from Figure 3 (one step, self time on the main thread):
| Run | 1st | 2nd | 3rd |
|---|---|---|---|
| DDP, 2 (106.4 ms) | aten::mm 51.4 ms (48.3%) | other element-wise ops 14.9 ms | attention 10.2 ms |
| FSDP2, 2 (138.8 ms) | aten::mm 51.6 ms (37.2%) | other element-wise ops 17.4 ms | FSDP2 all-gather waits 16.1 ms |
| FSDP2 + AC, 2 (159.6 ms) | aten::mm 64.8 ms (40.6%) | other element-wise ops 21.8 ms | attention 15.1 ms |
| FSDP2, 4 (162.1 ms) | aten::mm 39.8 ms (24.5%) | FSDP2 all-gather waits 32.1 ms | FSDP2 reduce-scatter waits 29.7 ms |
[Certain: measured] What each line says. (1) The matrix multiplies are the real work; a healthy run has them on top. (2) In FSDP2, waits for communication show up as self time of FSDP2’s hooks; at \(W = 4\) they are 38% of the step, so on this machine the run is communication-bound. (3) With checkpointing, aten::mm gets 26% more time: 99 calls per step instead of 75 (Step 8). One surprise: AdamW math took 13.8 ms per step with FSDP2 against 9.2 ms with DDP, although each rank updates only half the values. With tensors this small, the fixed cost of each DTensor operation is larger than the math. [Certain: measured] On a real model the shards are large and this cost becomes small.
- profile with a schedule (skip at least 3 steps, warm up 2, record 3); label forward, backward and optimizer with
record_function - print
key_averages()sorted by self time (CPU on a CPU run, CUDA on a GPU run); note the top rows and their call counts - group rows that are one cost: all hooks of FSDP2, all attention kernels, all optimizer kernels
- open the trace; for each group, check when it happens and whether the compute stream is idle at that time // idle = exposed
- compare the sum of the top 3 with the step time; if a large part of the step is in no group, look for gaps (data loading, Python, CPU-GPU syncs)
- change one thing (bucket size, prefetch, checkpointing, batch), profile again, compare the same groups
On a GPU the same method works, with three more things to look for in the trace [Likely: general practice]. The NCCL kernels run on their own stream; you want them to sit under compute kernels, not under gaps. A call like loss.item() or torch.cuda.synchronize() makes the CPU wait for the GPU and shows up as a long cudaStreamSynchronize. And if the GPU row has many small gaps between short kernels, the CPU cannot launch kernels fast enough, which is where torch.compile and CUDA graphs help (session 50).
Step 8: activation checkpointing with FSDP2
Activation checkpointing (AC) keeps only the input of each wrapped block in the forward pass and throws away everything inside. In the backward pass it runs the block’s forward again to rebuild the activations, then computes the gradients. You trade compute for memory. With FSDP2 there is one rule: apply checkpointing first, then fully_shard. torchtitan does it in this order: checkpointing, then torch.compile, then FSDP. [Certain: torchtitan v0.2.0 source]
from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import checkpoint_wrapper for i, block in enumerate(model.blocks): # 1) checkpointing ... model.blocks[i] = checkpoint_wrapper(block, preserve_rng_state=False) for block in model.blocks: # 2) ... then FSDP2 on the wrapped blocks fully_shard(block, mesh=mesh) fully_shard(model, mesh=mesh)
Why this order? The checkpoint wrapper must sit inside the FSDP2 group, so that when backward recomputes the block, FSDP2’s pre-backward hook has already gathered the block’s weights. preserve_rng_state=False skips saving the random-number state; that is safe here because the block has no dropout. torchtitan uses the same wrapper and the same flag. [Certain: torchtitan v0.2.0 source]
Formula (52.3) predicts what checkpointing saves: per block, \(c\,d = 18d\) values per token become \(d\). Measured, rank 0, \(W = 2\):
| FSDP2, \(W = 2\) | Added in forward | Peak | Matmul calls per step | Step time (median of runs) |
|---|---|---|---|---|
| no checkpointing | 117.2 MB | 155.18 MB | 75 | 100.1 ms |
| every other block (3 of 6) | 63.6 MB | 105.25 MB | 87 | 109.8 ms |
| every block | 10.0 MB | 71.09 MB | 99 | 119.3 ms |
[Certain: measured] Checkpointing every block removes 91% of the activations (117.2 → 10.0 MB) and cuts the peak by 54%. The 10.0 MB are the six block inputs (6.3 MB), the tensors outside the blocks (3.4 MB) and the gathered root (0.3 MB). The peak (71 MB) is now set by the backward pass, where one block’s activations are rebuilt (18.9 MB) while its weights and gradients are gathered.
The cost. Each block’s forward runs twice. If backward costs twice the forward, compute goes from \(1 + 2 = 3\) to \(1 + 1 + 2 = 4\) forward units:
Our step has 75 matrix multiplies (4 per block in forward, 8 per block in backward, plus 3 for the head); checkpointing adds the 24 forward ones again, exactly 32% more. The measured aten::mm time grew by 26% (51.6 → 64.8 ms). The step time grew less than that, because communication and other overheads did not grow. Checkpointing every other block is the middle point; it is also torchtitan’s default (selective_ac_option = "2"). torchtitan also offers op-level selective checkpointing, which saves the outputs of expensive operations such as matrix multiplies (by default every second one) and recomputes only the cheap ones. [Certain: torchtitan v0.2.0 source and paper]
How does this scale to a real model? Figure 4 applies (52.2), (52.3) and (52.4) to this lab model and to Llama 2 7B.
How to read Figure 4
Choose Llama 2 7B, FSDP2, 8 GPUs, 4,096 tokens: 13.48 GB of states, 2.26 GB gathered, 19.33 GB of activations, 35.07 GB in total. Click DDP: 154 GB, far above the 80 GB line, so DDP cannot train this model at all. Back to FSDP2, turn on “AC every block”: activations fall to 1.68 GB and the total to 17.42 GB. That leaves room for 8 times more tokens: at 32,768 tokens per GPU the total is 29.16 GB. Now slide the GPUs to 64 for the lab model: the states almost vanish, but the gathered block and the activations do not shrink with \(W\). Last, compare the communication line for DDP and FSDP2 on the 7B model: equal, because of BF16 gathers and FP32 reduction (52.5).
States. \(16 \times 6.738 \times 10^9 / 8 = 13.48\) GB. DDP would need \(20 \times 6.738 \times 10^9 = 134.8\) GB of states alone.
Gathered. \(2 \times (2 \times 262.1\text{M} + 3 \times 202.4\text{M}) = 2.26\) GB in BF16.
Activations. \(4{,}096 \times 32 \times 18 \times 4{,}096 \times 2 = 19.33\) GB without checkpointing. With checkpointing of every block: \(4{,}096 \times 32 \times 4{,}096 \times 2 = 1.07\) GB of inputs plus \(4{,}096 \times 18 \times 4{,}096 \times 2 = 0.60\) GB for one rebuilt block: 1.68 GB.
Total. 35.07 GB without and 17.42 GB with checkpointing, plus temporary buffers. The value \(c = 18\) comes from our GELU block; Llama’s SwiGLU MLP and its attention kernels save a somewhat different set of tensors, so treat the activation numbers as estimates. [Likely]
The full program
One file runs every version. On a CPU it starts \(W\) processes itself; on GPUs, start it with torchrun. The modes are single, ddp, ddpac, fsdp2, fsdp2norsh, fsdp2ac (every block) and fsdp2ac2 (every other block). We ran the CPU path; the GPU path uses the standard calls but we did not run it for this page.
# lab.py: the same GPT with DDP or FSDP2. CPU: python lab.py fsdp2 2 GPU: torchrun --nproc_per_node 2 lab.py fsdp2 import os, sys, time, statistics, torch, torch.nn as nn, torch.nn.functional as F import torch.distributed as dist, torch.multiprocessing as mp from torch.profiler import profile, ProfilerActivity, record_function V, T, D, L, H, GLOBAL_B, STEPS = 65, 128, 256, 6, 8, 16, 30 class Block(nn.Module): def __init__(s): super().__init__() s.n1, s.n2 = nn.RMSNorm(D), nn.RMSNorm(D) s.qkv, s.o = nn.Linear(D, 3 * D, bias=False), nn.Linear(D, D, bias=False) s.up, s.down = nn.Linear(D, 4 * D, bias=False), nn.Linear(4 * D, D, bias=False) def forward(s, x): B, T_, _ = x.shape q, k, v = s.qkv(s.n1(x)).split(D, dim=-1) q, k, v = (t.view(B, T_, H, D // H).transpose(1, 2) for t in (q, k, v)) a = F.scaled_dot_product_attention(q, k, v, is_causal=True) x = x + s.o(a.transpose(1, 2).reshape(B, T_, D)) return x + s.down(F.gelu(s.up(s.n2(x)))) class GPT(nn.Module): def __init__(s): super().__init__() s.tok, s.pos = nn.Embedding(V, D), nn.Embedding(T, D) s.blocks = nn.ModuleList(Block() for _ in range(L)) s.norm, s.head = nn.RMSNorm(D), nn.Linear(D, V, bias=False) # head not tied for p in s.parameters(): if p.dim() == 2: nn.init.normal_(p, std=0.02) def forward(s, idx, y): x = s.tok(idx) + s.pos(torch.arange(idx.shape[1], device=idx.device)) for b in s.blocks: x = b(x) return F.cross_entropy(s.head(s.norm(x)).view(-1, V), y.view(-1)) COMM = [] # (collective, bytes of the full tensor) def counted(name, fn, key, pos): # FSDP2 calls these with keyword arguments def w(*a, **k): t = k[key] if key in k else a[pos] COMM.append((name, t.numel() * t.element_size())); return fn(*a, **k) return w def main(rank, W, mode, dev): dist.init_process_group("nccl" if dev == "cuda" else "gloo", rank=rank, world_size=W) if dev == "cuda": torch.cuda.set_device(rank) else: torch.set_num_threads(4) text = open("input.txt").read(); stoi = {c: i for i, c in enumerate(sorted(set(text)))} data = torch.tensor([stoi[c] for c in text]) # Tiny Shakespeare, 65 characters torch.manual_seed(0) # same init on every rank model = GPT().to(dev) if "ac" in mode: # 1) activation checkpointing first ... from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import checkpoint_wrapper for i, b in enumerate(model.blocks): if mode.endswith("ac") or i % 2 == 0: # "...ac": every block, "...ac2": every other model.blocks[i] = checkpoint_wrapper(b, preserve_rng_state=False) if mode.startswith("ddp"): # 2a) ... then DDP from torch.nn.parallel import DistributedDataParallel as DDP model = DDP(model, device_ids=[rank] if dev == "cuda" else None) def hook(state, bucket): # same as the default all-reduce hook, plus counting buf = bucket.buffer(); COMM.append(("all_reduce", buf.numel() * buf.element_size())) return dist.all_reduce(buf.div_(W), async_op=True).get_future().then(lambda f: f.value()[0]) model.register_comm_hook(None, hook) elif mode.startswith("fsdp2"): # 2b) ... or FSDP2: blocks first, then the root from torch.distributed.fsdp import fully_shard from torch.distributed.device_mesh import init_device_mesh mesh = init_device_mesh(dev, (W,)) for b in model.blocks: fully_shard(b, mesh=mesh, reshard_after_forward="norsh" not in mode) fully_shard(model, mesh=mesh) dist.all_gather_into_tensor = counted("all_gather", dist.all_gather_into_tensor, "output_tensor", 0) dist.reduce_scatter_tensor = counted("reduce_scatter", dist.reduce_scatter_tensor, "input", 1) opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.1) # after wrapping! g, bs = torch.Generator().manual_seed(1234), GLOBAL_B // W losses, times = [], [] prof = profile(activities=[ProfilerActivity.CPU] + ([ProfilerActivity.CUDA] if dev == "cuda" else []), schedule=torch.profiler.schedule(wait=5, warmup=2, active=3), record_shapes=True) with prof: for step in range(STEPS): ix = torch.randint(len(data) - T - 1, (GLOBAL_B,), generator=g) # the global batch ... ix = ix[rank * bs:(rank + 1) * bs] # ... and this rank's slice x = torch.stack([data[i:i + T] for i in ix]).to(dev) y = torch.stack([data[i + 1:i + T + 1] for i in ix]).to(dev) n0 = len(COMM) if dev == "cuda": torch.cuda.synchronize() t0 = time.perf_counter() with record_function("## forward ##"): loss = model(x, y) with record_function("## backward ##"): loss.backward() with record_function("## optimizer ##"): opt.step(); opt.zero_grad(set_to_none=True) if dev == "cuda": torch.cuda.synchronize() times.append(time.perf_counter() - t0) l = loss.detach().clone(); dist.all_reduce(l); losses.append(l.item() / W) # not counted prof.step() local = lambda t: t.to_local() if hasattr(t, "to_local") else t p_mb = sum(local(p).numel() * 4 for p in model.parameters()) / 1e6 o_mb = sum(local(v).numel() * 4 for s in opt.state.values() for v in s.values() if v.dim() > 0) / 1e6 step_comm = COMM[n0:] if rank == 0: print(f"{mode} W={W}: loss {losses[0]:.4f} -> {losses[-1]:.4f}, step {1e3 * statistics.median(times[5:]):.1f} ms") print(f"model states on rank 0: params {p_mb:.2f} + grads {p_mb:.2f} + Adam {o_mb:.2f} MB") kinds = sorted(set(k for k, _ in step_comm)) print("collectives per step:", {k: (sum(1 for c, _ in step_comm if c == k), round(sum(b for c, b in step_comm if c == k) / 1e6, 2)) for k in kinds}) print(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=8)) prof.export_chrome_trace(f"trace_{mode}_{W}.json") # open in ui.perfetto.dev print("losses:", " ".join(f"{v:.6f}" for v in losses)) dist.destroy_process_group() if __name__ == "__main__": mode = sys.argv[1] # single | ddp | fsdp2 | fsdp2norsh | fsdp2ac | fsdp2ac2 | ddpac if "RANK" in os.environ: # started by torchrun (GPUs) main(int(os.environ["RANK"]), int(os.environ["WORLD_SIZE"]), mode, "cuda") else: # CPU simulation: W processes on one machine W = int(sys.argv[2]); os.environ.setdefault("MASTER_ADDR", "127.0.0.1"); os.environ.setdefault("MASTER_PORT", "29655") mp.spawn(main, args=(W, mode, "cpu"), nprocs=W)
curl -O https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
python lab.py single 1 ; python lab.py ddp 2 ; python lab.py fsdp2 2 ; python lab.py fsdp2ac 2
torchrun --nproc_per_node 2 lab.py fsdp2 # the same on 2 GPUs
On our machine python lab.py fsdp2 2 prints loss 4.1330 -> 3.3593, params 9.58 + grads 9.58 + Adam 19.15 MB and collectives per step: {'all_gather': (13, 38.04), 'reduce_scatter': (7, 19.15)}, followed by the profiler table. [Certain: measured]
Where this is used
| System or task | What it shares with this lab |
|---|---|
| torchtitan (Llama 3 and other pretraining recipes) | fully_shard on the token embedding, each block, and the final norm with the output head; the last group is not resharded after forward; BF16 parameters, FP32 reduction; checkpointing before sharding [Certain: v0.2.0 source] |
| Fine-tuning a 7B to 70B model on one node | FSDP2 plus activation checkpointing is the usual way to fit full fine-tuning on 8 GPUs; the calculator of Figure 4 is the first estimate |
| Small models on many GPUs | When the model fits, DDP (or FSDP2 without resharding) is simpler and exposes less communication, as Figure 2 shows |
| Vision-language model training | The vision encoder and the language model are wrapped as separate groups; a frozen encoder is easy with FSDP2 because each parameter is its own DTensor (Track 7) |
| Video-language pretraining, video classification | Video clips give many tokens per sample, so activations dominate memory even more than here; checkpointing and small micro-batches are standard (Track 8) |
| Video retrieval and Video-RAG embedding models | Contrastive training wants large global batches; data parallelism over many GPUs with gathered embeddings, and the same profiling method to find the slow collective |
| Performance debugging of any training job | Algorithm 3: profile with a schedule, group self time, check overlap in the trace, change one thing at a time |
Common mistakes
- Creating the optimizer before
fully_shard. The optimizer then holds the old full parameters, which FSDP2 has replaced by DTensors. Training seems to run, but nothing is updated. Build the optimizer after wrapping. - Calling
fully_shardonly on the root. Then the whole model is one group, all of it is gathered at once, and the peak memory is the full model again. Shard each block first, then the root. - Wrapping in the wrong order. Checkpointing must be applied before
fully_shard(andtorch.compilebetween them, as torchtitan does). Applying it after puts the recompute outside FSDP2’s hooks. - Different data or seeds on different ranks without meaning to. If ranks build the model with different seeds, DDP broadcasts rank 0’s weights at the start [Certain], but FSDP2 shards whatever each rank has, so the shards come from different models [Likely]. Seed the model the same everywhere and split the data on purpose.
- Comparing step time from a CPU simulation with GPU runs. The simulation is correct for memory, collective counts and losses. Its timing says nothing about NCCL overlap.
- Profiling the first step. Step 0 includes allocations, autotuning and lazy initialization. Use a schedule with
waitandwarmup. - Reading total time instead of self time.
## backward ##has a total time of the whole backward pass; summing total times counts the same work many times. Rank by self time, then group. - Expecting bit-identical losses. Differences of \(10^{-7}\) at the start, growing to \(10^{-4}\), are normal (Step 4). A difference of \(10^{-2}\) at step 1 is a bug.
Interview questions
A model has \(\Psi\) parameters, trained with AdamW in mixed precision on \(W\) GPUs. Give the memory for model states per GPU with DDP and with FSDP.basic
gradient_as_bucket_view=True. We measured 76.64 MB at the start of a step and 95.79 MB after backward for \(\Psi\) = 4.79M, that is \(16\Psi\) and \(20\Psi\). FSDP shards all of it: \(16\Psi/W\), plus the gathered root and one or two blocks in flight. Adding GPUs helps FSDP; it never helps DDP.How do you check that a new distributed training setup is correct?basic
How much more does FSDP communicate than DDP? Is it always 1.5×?core
reshard_after_forward=False, the backward all-gather disappears and FSDP sends the same elements as DDP.What does reshard_after_forward do, and when would you set it to False?core
In what order do you apply activation checkpointing, torch.compile, fully_shard and the optimizer?core
fully_shard bottom-up (blocks, then the root), then build the optimizer. This is torchtitan’s order. Checkpointing must be inside the FSDP2 group so that the recomputed forward finds its weights gathered by FSDP2’s pre-backward hook. The optimizer must see the sharded DTensor parameters, so it comes last.What is the difference between self time and total time in a profile? Why did DDP’s ## backward ## label have 9 ms of self time?core
## backward ## has children for every backward operator; its self time is the time inside backward when no operator was running. In our DDP trace that was 9.4 ms of a 106 ms step: the autograd engine and, mostly, the wait for the last bucket’s all-reduce. That is DDP’s exposed communication, and it is invisible if you only read operator rows.You switched a job from DDP to FSDP2 and the step got 30% slower. How do you find out why?core
set_modules_to_forward_prefetch, or torch.compile); resharding groups that are needed right away (set reshard_after_forward=False for the last groups). Change one thing at a time and profile again.What does activation checkpointing save and cost? Give numbers.core
Why are FSDP2 losses not bit-identical to single-GPU losses, and how could you get closer?advanced
torch.use_deterministic_algorithms(True)), and the same number of ranks; bit-identical results across different world sizes are not a realistic goal. What matters is that the drift stays at rounding level and does not grow into a different curve.Estimate the memory per GPU for full fine-tuning of Llama 2 7B on 8 × 80 GB GPUs with FSDP2, 4,096 tokens per GPU. Does it fit? What if you use DDP?advanced
What can a CPU simulation of distributed training tell you, and what can it not?advanced
Exercises
- Compute the model states per rank for the lab model (\(\Psi\) = 4,787,968, FP32 AdamW) with DDP and with FSDP2 at \(W = 4\). Compare with the start-of-step memory in Figure 1.
- With formula (52.5), how many bytes does each rank send per step at \(W = 8\) for DDP and for FSDP2 (all FP32)?
- DDP does not shard. Why is its peak at \(W = 4\) (137 MB) much lower than at \(W = 2\) (196 MB)?
- Predict the activations added in the forward pass when only blocks 0, 2 and 4 are checkpointed, from the measured 18.9 MB per full block, 1.05 MB per block input and 3.4 MB outside the blocks. Compare with Figure 1.
- In Figure 3 (FSDP2, 2), click a reduce-scatter on a communication thread. How many values does gloo move in it, and how many bytes would NCCL’s ring reduce-scatter send per rank for the same gradient?
- Llama 2 7B, 16 GPUs, FSDP2, checkpointing of every block, 8,192 tokens per GPU. Estimate the memory per GPU with Figure 4’s formulas.
Answerscheck
Key takeaways
- Test a distributed setup against one process with the same global batch: the losses must agree to rounding (52.1). Ours agreed to \(10^{-7}\) at first and \(10^{-4}\) after 30 steps.
- DDP keeps \(16\Psi\) (plus \(4\Psi\) of buckets) on every rank; FSDP2 keeps \(16\Psi/W\) plus the gathered root and about three blocks in flight (52.2, 52.4).
- Activations often dominate: 117 MB against 38 MB of states in this lab. More ranks with a fixed global batch shrink activations per rank for both DDP and FSDP2.
- FSDP2 sends \(3\Psi\) elements per step against DDP’s \(2\Psi\) (1.49× measured), but with BF16 gathers and FP32 reduction the bytes are equal (52.5).
- Read profiles by self time (52.7), group related rows, then use the trace to see what is exposed. In DDP the exposed wait hides in the self time of the backward label; in FSDP2 it is in its hooks.
- Apply checkpointing before
fully_shard. Checkpointing every block cut activations by 91% and the peak by 54%, for 32% more matmuls (52.8). - A CPU simulation is exact for memory, collectives and losses, and says nothing about GPU step time.
Key formulas
References for this session
- PyTorch (docs). Getting Started with Fully Sharded Data Parallel (FSDP2)
- PyTorch (code). torchtitan: reference large-scale training with FSDP2
- PyTorch (docs). Getting Started with Fully Sharded Data Parallel (FSDP2)
- PyTorch (code). torchtitan: reference large-scale training with FSDP2
- Liang, Liu, Wright, Constable, Gu, Huang et al. (2024). TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining
- Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Chen et al. (2016). Training Deep Nets with Sublinear Memory Cost
- Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
- PyTorch (docs). PyTorch Profiler recipe (torch.profiler: schedule, key_averages, export_chrome_trace)
- PyTorch (docs). torch.distributed.fsdp.fully_shard (FSDP2) API reference: fully_shard, FSDPModule, MixedPrecisionPolicy, CPUOffloadPolicy
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters