LLM & VLM courseBlog
52Track 3: Pretraining and scale · session 22 of 22

DDP vs FSDP2 and reading a profile

In this lab you train the same small GPT three ways: in one process, with DDP, and with FSDP2, using 2 or 4 processes on a CPU in place of GPUs. You check that all runs give the same losses, then measure memory per rank, the collectives in one step and the step time. You capture a profiler trace, read it, and find the top three time sinks. Finally you add activation checkpointing to FSDP2 and measure what it saves and what it costs.

Time: 3 to 4 hours (mostly hands-on; every run takes less than a minute on a laptop CPU)Level: Intermediate · hands-on labBefore this: Sessions 39 and 43 to 47 (checkpointing, collectives, DDP, ZeRO, FSDP1, FSDP2), and the GPT from session 30

What you will learn

  • How to wrap the same model with DDP and with FSDP2 (fully_shard on each block, then on the root), and why the order of wrapping matters
  • How to prove that a distributed run is correct: the loss curve must match a single-process run with the same global batch
  • How to measure memory per rank, count the collectives of one step, and turn them into bytes with the ring formulas
  • Why FSDP2 sends 1.5 times the bytes of DDP in full precision, and the same bytes with BF16 gathers and FP32 reductions
  • How to capture a torch.profiler trace, the difference between self time and total time, and how to find the top 3 time sinks
  • How to combine activation checkpointing with FSDP2, and what it saves (91% of activation memory here) and costs (32% more matrix multiplies)
  • What transfers from a CPU simulation to real GPUs (memory, collective counts, loss parity) and what does not (step time)

Why this matters

The last few sessions were theory. Session 44 copies the model to every GPU. Session 45, session 46 and session 47 shard the model instead, so each GPU keeps only a slice. In this lab you stop reading and start measuring. You train one small GPT with DDP and with FSDP2, on the same data, and you check every claim from those sessions with numbers from your own runs.

Then you learn the skill that separates people who use distributed training from people who can fix it: reading a profile. You capture a trace of one training step with torch.profiler, open it, and find where the time goes. Last, you add activation checkpointing (recompute activations in the backward pass instead of storing them, session 39) on top of FSDP2 and measure both sides of the trade.

  • Interviews. “How much memory does each GPU need with DDP and with FSDP?”, “How much more does FSDP communicate?”, and “Your step got 30% slower after you switched to FSDP. How do you find out why?” are common questions for training-infrastructure roles. Each one is answered with a measured number below.
  • Real systems. torchtitan, PyTorch’s reference code for LLM pretraining, uses exactly the pieces of this lab: fully_shard on each transformer block, activation checkpointing applied before sharding, and BF16 parameters with FP32 gradient reduction. [Certain: torchtitan v0.2.0 source]
  • Debugging. The checks here (losses equal to a single-process run, collectives counted per step, a memory timeline, a top-3 list from the profiler) take minutes and find most setup bugs before you pay for a large run.

Setup and plan

You do not need GPUs. PyTorch can run its distributed code on a CPU: we start 2 or 4 processes on one machine, and each process plays the role of one GPU (one rank). They talk through gloo, PyTorch’s CPU communication backend, instead of NCCL. Every measured number on this page comes from such runs on a laptop CPU with PyTorch 2.8.0, 4 threads per process. [Certain: measured]

Be clear about what this simulation can and cannot tell you:

QuantityTransfers to GPUs?Why
Loss curve, equal to one processyesthe math is the same; only the device changes
Bytes of model states per rankyessame tensors, same sharding
Number and size of collectivesyesDDP and FSDP2 issue the same calls on any backend
Activation memory, and what checkpointing savesyes, almostGPU kernels may keep slightly different tensors (for example FlashAttention)
Step time, overlap, top time sinksnoCPU math is much slower, gloo is not NCCL, and there are no CUDA streams to overlap on

The model is the GPT of session 30, a little wider, with characters of Tiny Shakespeare as tokens. The output head is not tied to the embedding, so every parameter belongs to exactly one place.

SettingValueMeaning
\(|V|, T\)65, 128vocabulary (characters) and context length
\(d, L, h\)256, 6, 8width, number of blocks, attention heads
blockRMSNorm, attention (SDPA), RMSNorm, GELU MLP with \(4d\)no biases; Pre-LN
\(\Psi\)4,787,968parameters: 6 blocks of 786,944 plus 66,304 in embeddings, final norm and head
global batch16 windows × 128 tokenssplit evenly: 8 windows per rank with \(W = 2\), 4 with \(W = 4\)
optimizerAdamW, lr \(10^{-3}\), weight decay 0.1, 30 stepseverything in FP32, so 16 bytes per parameter of model states

The plan: (1) one model and one data stream for all runs, (2) the DDP version, (3) the FSDP2 version, (4) check that all runs agree, (5) memory, (6) communication and step time, (7) read a profile, (8) add activation checkpointing.

Step 1: one model, one stream of data

A fair comparison needs the same training problem in every run. So every run builds the model with torch.manual_seed(0) (same initial weights on every rank) and draws the global batch of 16 windows from one generator with seed 1234. Rank \(r\) then takes its own slice: windows \(r \cdot b\) to \((r+1) \cdot b - 1\), where \(b = 16 / W\).

Why does this make the runs comparable? The loss is a mean over all tokens, and all ranks hold the same number of tokens. So the gradient of the global batch is exactly the average of the gradients that the ranks compute on their slices:

\[ \nabla_\theta \mathcal{L}_{\text{global}} = \frac{1}{W} \sum_{r=1}^{W} \nabla_\theta \mathcal{L}_r, \qquad \mathcal{L}_r = \frac{1}{b\,T} \sum_{\text{tokens of rank } r} \ell \tag{52.1} \]
Read the symbols
\(\theta\)
all the weights of the model
\(W\)
world size: the number of ranks (simulated GPUs)
\(\mathcal{L}_r\)
the mean cross-entropy over the \(b \cdot T\) tokens of rank \(r\)
\(\ell\)
the loss of one token

DDP averages gradients with an all-reduce; FSDP2 averages them with a reduce-scatter. Both compute the right side of (52.1). So a single process that trains on all 16 windows must produce the same losses, step by step, up to floating-point rounding. This is the strongest test you have: a wrong mask, a missing average, a rank that reads the wrong data or a parameter that never syncs all show up as a different loss curve.

Step 2: the DDP version

DDP (DistributedDataParallel) keeps a full copy of the model on every rank. During the backward pass it groups gradients into buckets and starts an all-reduce for each bucket as soon as all its gradients are ready, so communication overlaps with the rest of the backward pass (session 44). We wrap the model and add a communication hook: a function that DDP calls for every bucket. Ours does what the default does (divide by \(W\), all-reduce) and also records the bucket size.

model = DDP(model)                       # on GPUs: DDP(model, device_ids=[rank])
def hook(state, bucket):
    buf = bucket.buffer(); COMM.append(("all_reduce", buf.numel() * buf.element_size()))
    return dist.all_reduce(buf.div_(W), async_op=True).get_future().then(lambda f: f.value()[0])
model.register_comm_hook(None, hook)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.1)

What we measured in every step: two all-reduces. The first bucket held 279,040 values (1.12 MB): the output head, the final norm and the down projection of the last block, the first gradients that backward produces. The second held the other 4,508,928 values (18.04 MB). Together that is all 4,787,968 gradients, 19.15 MB in FP32. [Certain: measured] DDP makes its first bucket small on purpose, so that communication starts early; the later buckets are capped at 25 MB by default (bucket_cap_mb). [Certain: PyTorch 2.8 source]

Algorithm 1: one DDP training step, seen from rank \(r\)
  1. input: full model copy, full AdamW states, this rank’s slice \((x_r, y_r)\)
  2. forward on \(x_r\); store activations // no communication
  3. backward; when every gradient of a bucket is ready, start all-reduce of that bucket // runs while backward continues
  4. at the end of backward, wait for the last all-reduce // the exposed part
  5. every rank now has the same averaged gradient (52.1); full AdamW step on the full model

Step 3: the FSDP2 version

FSDP2 is the fully_shard API (session 47). You call it on each transformer block, then on the whole model (the root). Each call makes one parameter group: its parameters are all-gathered together before the module runs and their gradients are reduce-scattered together after its backward. The root call takes everything not already in a group, here the embeddings, the final norm and the head. Each parameter becomes a DTensor (a tensor that knows it is split across ranks) sharded on dimension 0: with \(W = 2\), rank 0 holds the first half of the rows of every weight matrix.

mesh = init_device_mesh("cpu", (W,))     # "cuda" on GPUs
for block in model.blocks:
    fully_shard(block, mesh=mesh)        # one group per block: bottom-up
fully_shard(model, mesh=mesh)            # the root: embeddings, final norm, head
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.1)   # AFTER sharding

Three details matter. Order: shard the blocks first, then the root; the docs say to call fully_shard bottom-up. [Certain] Optimizer last: build it after sharding, so it holds the sharded DTensors; an optimizer built before would update tensors that no longer exist. Reshard after forward: by default (reshard_after_forward=None) each block frees its gathered weights after its forward and gathers them again in backward, while the root keeps its weights, because backward starts with the root’s head. [Certain: PyTorch 2.8 docstring]

What we measured. Rank 0 holds 9.58 MB of the 19.15 MB of weights. The split is not perfectly even: the embedding and the head have 65 rows, so rank 0 gets 33 rows and rank 1 gets 32, and the root’s all-gather is padded from 66,304 to 66,816 values. In one step FSDP2 issued 13 all-gathers (1 for the root, 6 for the blocks in forward, 6 again in backward) and 7 reduce-scatters (6 blocks and the root). [Certain: measured]

Algorithm 2: one FSDP2 training step, seen from rank \(r\)
  1. input: shards of all weights and AdamW states (1/W of each); this rank’s slice \((x_r, y_r)\)
  2. all-gather the root group; it stays gathered for the whole step
  3. for block \(i = 1, \dots, L\): all-gather block \(i\) (prefetched during block \(i-1\)); forward; free its full weights
  4. compute the loss; backward through the head
  5. for block \(i = L, \dots, 1\): all-gather block \(i\) again (prefetch \(i-1\)); compute its gradients; reduce-scatter them so each rank keeps the averaged gradient of its own shard; free full weights and gradients
  6. reduce-scatter the root’s gradients; AdamW step on the local shards only // no communication

Step 4: check that every run trains the same model

Run all versions for 30 steps and compare the global loss (the mean of the rank losses) with the single-process run. This is the first thing to do after any change to a distributed setup.

RunLoss at step 0Loss at step 29Largest difference from one process, steps 0 to 11Largest difference, all 30 steps
one process, 16 windows4.13303.359300
DDP, \(W = 2\)4.13303.3594\(3.6 \times 10^{-7}\)\(1.3 \times 10^{-5}\)
DDP, \(W = 4\)4.13303.3594\(3.0 \times 10^{-7}\)\(1.6 \times 10^{-5}\)
FSDP2, \(W = 2\)4.13303.3593\(2.7 \times 10^{-6}\)\(2.7 \times 10^{-4}\)
FSDP2, \(W = 4\)4.13303.3593\(4.0 \times 10^{-6}\)\(4.3 \times 10^{-4}\)
FSDP2 + checkpointing, \(W = 2\)bit-for-bit the same losses as FSDP2 without checkpointing

[Certain: measured] How to read this table. At the start, the differences are \(10^{-7}\) to \(10^{-6}\): the size of FP32 rounding. They are not zero because the ranks add their gradients in a different order than one process does, and floating-point addition is not associative. They grow over 30 steps because AdamW divides by \(\sqrt{v}\), which is tiny early in training, so it magnifies tiny gradient differences. FSDP2 drifts more than DDP, probably because it also runs the AdamW update on shards through DTensor, a different sequence of operations. [Likely]

One more check is hidden in the data: at step 12 every run has a loss spike to 6.34 and recovers by step 13. Because every run spikes at the same step by the same amount, the spike is a property of the training recipe (we used no warmup, session 36), not of the parallelism. A bug would not reproduce so exactly. Checkpointing changes nothing at all, as it should: the recomputed forward is exactly the same computation.

Step 5: memory per rank

To measure memory on a CPU we start torch.profiler with profile_memory=True before building the model. Then every tensor allocation and free is recorded with a running total of live bytes, so we get the exact memory timeline of each rank. (On a GPU you would use torch.cuda.max_memory_allocated() or the memory snapshot tool.) Figure 1 shows the last of four steps on rank 0.

Processes (simulated GPUs):
Figure 1. Measured live tensor memory of rank 0 during one training step (the fourth), from the profiler’s memory events. Solid line: the chosen configuration. Dashed line: DDP with the same number of processes. Shading: forward, backward and optimizer phases of the chosen run. AC is activation checkpointing (Step 8). Global batch 16 in every run.
How to read Figure 1

Start with FSDP2 at 2 processes. The line starts at 28.76 MB: rank 0’s half of the weights (9.58 MB) and of the AdamW states (19.15 MB). The gradients were freed by zero_grad(set_to_none=True). The forward pass adds 117 MB of activations, the largest part by far. The peak (155 MB) comes early in the backward pass, when the activations are still there and a block is gathered with its gradients. Now click DDP: it starts at 76.64 MB (full weights, full AdamW states, and a 19.15 MB bucket buffer that DDP keeps for the whole run) and peaks at 196 MB. Switch to 4 processes: the DDP start does not move (DDP never shards), but every peak falls, because each rank now has 4 windows instead of 8, so half the activations. Finally click “FSDP2 + AC every block”: the forward pass adds only 10 MB.

The measured numbers, all for rank 0, in MB:

Run (\(W\) processes)Start of stepEnd of forwardPeakPeak vs DDP
one process (16 windows)57.49291.31297.12
DDP, 276.64193.55196.49100%
FSDP2, 228.76145.94155.1879%
FSDP2, no reshard, 228.76164.83168.2486%
DDP, 476.64135.10137.13100%
FSDP2, 414.4073.1381.4659%
FSDP2, no reshard, 414.4092.0195.3069%

[Certain: measured] Three formulas explain every row. First, the model states: weights, gradients and the two AdamW moments, 4 + 4 + 8 = 16 bytes per parameter in FP32 (session 38):

\[ M^{\text{DDP}}_{\text{states}} = 16\,\Psi + \underbrace{4\,\Psi}_{\text{buckets}} = 20\,\Psi, \qquad M^{\text{FSDP2}}_{\text{states}} = \frac{16\,\Psi}{W} \tag{52.2} \]

DDP’s extra \(4\Psi\) is its bucket buffer: by default (gradient_as_bucket_view=False) gradients are copied into the buckets, so both exist. With gradient_as_bucket_view=True the gradients live inside the buckets and DDP needs \(16\Psi\). [Certain: the 19.15 MB is measured; the flag is in the DDP API]

Second, the activations. We counted the tensors that autograd saves for the backward pass in one block: 18 vectors of size \(d\) per token (the MLP’s \(4d\) hidden layer twice, \(q, k, v\), and the inputs and outputs of the norms, attention and residual adds), plus 10 small numbers per token. [Certain: measured]

\[ A = n\, L\, c\, d\, p, \qquad A_{\text{AC}} = \underbrace{n\, L\, d\, p}_{\text{block inputs}} + \underbrace{n\, c\, d\, p}_{\text{one block, recomputed}} \tag{52.3} \]
Read the symbols
\(n\)
tokens per rank: \(b \cdot T\) (1,024 with \(W = 2\))
\(c\)
values saved per token per block, in units of \(d\); about 18 for this block (18.04 measured)
\(p\)
bytes per value: 4 in this FP32 lab, 2 with BF16 activations
\(A_{\text{AC}}\)
the activations with checkpointing of every block (Step 8)

Third, the peak of FSDP2: its states, the activations, the root kept gathered (weights and gradients), and in the backward pass one block’s gathered weights, its gradients and the prefetched next block:

\[ M^{\text{FSDP2}}_{\text{peak}} \approx \frac{16\,\Psi}{W} + A + p\,\big(2\,\psi_{\text{root}} + 3\,\psi_{\text{block}}\big) \tag{52.4} \]
Read the symbols
\(\psi_{\text{root}}, \psi_{\text{block}}\)
parameters in the root group (66,304) and in one block (786,944)
\(3\,\psi_{\text{block}}\)
current block’s weights + its gradients + the next block’s weights (backward prefetch)
Worked example: FSDP2 with \(W = 2\), measured vs formula

States. \(16 \times 4{,}787{,}968 / 2 = 38.30\) MB. At the start of a step the gradient shards are freed, so we expect \(12\Psi/W = 28.73\) MB; we measured 28.76 MB.

Activations. \(n = 8 \times 128 = 1{,}024\) tokens: \(A = 1{,}024 \times 6 \times 18.04 \times 256 \times 4 = 113.5\) MB in the blocks, plus 3.4 MB for the embeddings, final norm, head and loss: 116.9 MB. The forward pass added \(145.94 - 28.76 = 117.2\) MB (0.27 MB of it is the gathered root).

Peak. Formula (52.4): \(38.30 + 116.9 + 4 \times (2 \times 66{,}304 + 3 \times 786{,}944)/10^6 = 38.30 + 116.9 + 9.97 = 165.2\) MB. Measured: 155.18 MB. The formula is 6% high because at the peak, early in backward, most gradient shards do not exist yet. It is an upper estimate, which is what you want for planning.

DDP. \(20\Psi = 95.76\) MB of states (76.64 MB at the start of the step plus 19.15 MB of gradients that appear during backward: 95.79 MB measured at the end of backward) and the same 117 MB of activations.

Two lessons from the table. FSDP2 helps more as \(W\) grows: its states fall as \(1/W\), DDP’s do not. At this size, activations dominate: with 117 MB of activations against 38 MB of states, sharding alone cuts the peak by only 21% at \(W = 2\). Real LLM runs are often the same at long context, which is why Step 8 matters.

Step 6: communication and step time

Each recorded collective has a size: the full tensor in bytes. With the ring algorithms of session 43, an all-gather or reduce-scatter of \(N\) bytes over \(W\) ranks sends \(N(W-1)/W\) bytes from each rank, and an all-reduce sends twice that. Adding up one step:

\[ V_{\text{DDP}} = 2\,\Psi\, b_g\,\frac{W-1}{W}, \qquad V_{\text{FSDP2}} = \big(2\,\Psi\, b_p + \Psi\, b_g\big)\frac{W-1}{W}, \qquad V_{\text{no reshard}} = \big(\Psi\, b_p + \Psi\, b_g\big)\frac{W-1}{W} \tag{52.5} \]
Read the symbols
\(V\)
bytes sent by each rank in one training step
\(b_p\)
bytes per value of the gathered weights (4 here; 2 with BF16 param_dtype)
\(b_g\)
bytes per value of the reduced gradients (4 here; torchtitan also reduces in FP32)
\(2\Psi b_p\)
FSDP2 gathers every block twice: in forward and again in backward
RunCollectives per stepFull-tensor bytesSent per rank, ringFormula (52.5)
DDP, \(W = 2\)2 all-reduce19.15 MB19.15 MB19.15 MB
FSDP2, \(W = 2\)13 all-gather, 7 reduce-scatter38.04 + 19.15 MB28.60 MB (1.49×)28.73 MB
FSDP2, no reshard, \(W = 2\)7 all-gather, 7 reduce-scatter19.15 + 19.15 MB19.15 MB (1.00×)19.15 MB
DDP, \(W = 4\)2 all-reduce19.15 MB28.73 MB28.73 MB
FSDP2, \(W = 4\)13 all-gather, 7 reduce-scatter38.05 + 19.16 MB42.90 MB (1.49×)43.09 MB

[Certain: measured counts and sizes; computed send volumes] The measured FSDP2 volume is slightly below the formula because the root is gathered only once. So in full precision, FSDP2 sends 1.5 times the bytes of DDP, as session 46 derived. But look at (52.5) with the torchtitan defaults, BF16 gathers (\(b_p = 2\)) and FP32 reduction (\(b_g = 4\)): \(V_{\text{FSDP2}} = 8\Psi(W-1)/W\), exactly the same as DDP all-reducing FP32 gradients. For Llama 2 7B on 8 GPUs both are 47.2 GB per step per GPU. [Certain: computed] The “1.5×” is true only when both use the same dtype.

Step time

Here is the measured step time: for each run, the median over steps 5 to 29 (the slowest rank counts). We repeated each configuration up to 5 times. Read it with care: other programs were running on the same machine, and the same configuration varied by up to 3× between runs. The fastest run is the least disturbed one, so compare that column. [Certain: measured]

Run (processes)Fastest runMedian of runsSlowest runRuns
one process, 16 windows76.2 ms121.9 ms176.4 ms5
DDP, 273.8 ms100.2 ms198.4 ms5
FSDP2, 2100.1 ms131.6 ms316.8 ms5
FSDP2, no reshard, 291.8 ms98.5 ms203.0 ms5
DDP + AC every block, 295.8 ms102.3 ms129.8 ms5
FSDP2 + AC every other block, 2109.8 ms116.3 ms136.2 ms3
FSDP2 + AC every block, 2119.3 ms129.9 ms318.9 ms5
DDP, 482.7 ms87.6 ms100.8 ms5
FSDP2, 4121.3 ms124.5 ms175.8 ms5
FSDP2, no reshard, 492.0 ms110.4 ms148.3 ms5
FSDP2 + AC every block, 4136.3 ms141.6 ms188.2 ms5

On this CPU, FSDP2 is slower than DDP (fastest runs: 100.1 against 73.8 ms with 2 processes, +36%; 121.3 against 82.7 ms with 4, +47%), and the gap grows with \(W\). Checkpointing every block adds 19% to FSDP2 and 30% to DDP. That is a property of the simulation, not of FSDP2. With gloo on a CPU, every collective is a memory copy done by CPU threads that compete with the math for the same cores, and there are no CUDA streams on which transfers can run beside compute. On GPUs, FSDP2 runs its all-gathers and reduce-scatters on separate streams and prefetches the next block, so most of the transfer hides behind compute. The useful model is:

\[ T_{\text{step}} = T_{\text{comp}} + T_{\text{exposed}}, \qquad T_{\text{exposed}} \ge t_{\text{first all-gather}} + t_{\text{last reduce-scatter}} \tag{52.6} \]

The first all-gather cannot hide (nothing computes before it), and the optimizer must wait for the last reduce-scatter. Everything in between can hide if each transfer is shorter than the compute it overlaps. Figure 2 plays this out for 6 blocks.

Gradient reduction in:
forward of a block (1 unit)backward of a block (2 units)all-gatherreduce-scatterDDP all-reduce
Figure 2. A model of one GPU training step with one compute stream and one communication stream, for 6 blocks with one bucket per block. FSDP2 prefetches one block ahead in both passes. An all-reduce moves twice the bytes of an all-gather; FP32 gradients double the gradient traffic. The dashed line is the end of the step; the green line is pure compute (18 units). This is a model, not a measurement.
How to read Figure 2

At the default ratio 0.3 with BF16 gradients, DDP exposes only its last all-reduce (0.6 units) and FSDP2 exposes three transfers of 0.3: the first all-gather, the first all-gather of the backward pass and the last reduce-scatter (0.9 units). Both are within 5% of pure compute. Drag the slider to 1.5: now one all-gather takes longer than one forward block, prefetching can no longer keep up, and FSDP2 falls far behind DDP. “No reshard” skips the backward all-gathers; in this model it even beats DDP at large ratios, because its reduce-scatter moves half the bytes of DDP’s all-reduce. It pays with the memory of Step 5. What sets the ratio on real hardware? The bytes of one block over the link bandwidth, against the FLOPs of one block over the GPU speed. More tokens per GPU raise the compute and lower the ratio; slower links (across nodes) raise it.

Step 7: capture and read a profile

A profiler records every operation with its start time and duration. torch.profiler does it for PyTorch: CPU-side operators (aten::mm and so on), your own labels from record_function, and on GPUs the CUDA kernels. Two ideas are needed to read its output:

  • Total time of an event: from its start to its end, including everything it called.
  • Self time: total time minus the time of its children. Self time is where the time was really spent; summing self times never counts anything twice.
\[ t^{\text{self}}_e = t^{\text{total}}_e - \sum_{c \,\in\, \text{children}(e)} t^{\text{total}}_c, \qquad \text{share}_g = \frac{\sum_{e \in g} t^{\text{self}}_e}{T_{\text{step}}} \tag{52.7} \]

Do not profile step 0: the first steps allocate memory and warm up caches. A schedule skips some steps, warms up, then records a few:

from torch.profiler import profile, ProfilerActivity, record_function, schedule
acts = [ProfilerActivity.CPU] + ([ProfilerActivity.CUDA] if torch.cuda.is_available() else [])
with profile(activities=acts, schedule=schedule(wait=5, warmup=2, active=3), record_shapes=True) as prof:
    for step in range(30):
        with record_function("## forward ##"):   loss = model(x, y)
        with record_function("## backward ##"):  loss.backward()
        with record_function("## optimizer ##"): opt.step(); opt.zero_grad(set_to_none=True)
        prof.step()                                  # tells the schedule a step ended
print(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=8))   # self_cuda_time_total on GPUs
prof.export_chrome_trace("trace.json")            # open in ui.perfetto.dev or chrome://tracing

For FSDP2 with \(W = 2\), the table starts like this (3 recorded steps; real output, shortened):

Name                                          Self CPU %   Self CPU   # of Calls
aten::mm                                         29.22%    104.933ms        225
aten::_scaled_dot_product_flash_attention_..      5.07%     18.212ms         18
aten::copy_                                       4.10%     14.737ms       1419
aten::gelu_backward                               3.51%     12.594ms         18
aten::mul                                         3.28%     11.775ms        390

The table is a good start, but it groups by operator name, so the same cost can be split over many rows (FSDP2 makes one row per block for its hooks). The trace keeps the time order, which tells you when each cost happens and what waits for what. Figure 3 is the real trace of one step, step 6 of the run, on rank 0.

aten::mmattentionGELU / all-gathercopiesFSDP2 all-gather hookFSDP2 reduce hook / all-reduceAdamW mathother
Figure 3. One measured training step from torch.profiler on rank 0 (CPU, gloo). Row 2: top-level operators longer than 0.3 ms (in backward, the autograd node of each operator), colored by kind. Row 3: FSDP2’s hooks, whose self time is mostly waiting for an all-gather or a reduce-scatter. Rows 4 and 5: the collectives, which gloo runs on its own threads. The top-3 line groups the self time (52.7) of all operators on the main thread in this step.
How to read Figure 3

In DDP, look at the two collectives in the backward pass: the small first bucket starts right after the head’s gradient, the big one near the end. The backward phase has 9.4 ms of self time outside any operator: the autograd engine and, mostly, the wait for that last all-reduce. That is DDP’s exposed communication. Switch to FSDP2: before each block, a purple FSDP2 hook bar waits for the block’s all-gather; in backward, red hooks wait for the reduce-scatters. With 4 processes the two FSDP2 groups climb to places 2 and 3 of the top 3: on this CPU, communication now costs more than all non-matmul math. With checkpointing, the forward ops appear twice: once in forward, once again inside the backward pass.

The top 3 time sinks in each run, from Figure 3 (one step, self time on the main thread):

Run1st2nd3rd
DDP, 2 (106.4 ms)aten::mm 51.4 ms (48.3%)other element-wise ops 14.9 msattention 10.2 ms
FSDP2, 2 (138.8 ms)aten::mm 51.6 ms (37.2%)other element-wise ops 17.4 msFSDP2 all-gather waits 16.1 ms
FSDP2 + AC, 2 (159.6 ms)aten::mm 64.8 ms (40.6%)other element-wise ops 21.8 msattention 15.1 ms
FSDP2, 4 (162.1 ms)aten::mm 39.8 ms (24.5%)FSDP2 all-gather waits 32.1 msFSDP2 reduce-scatter waits 29.7 ms

[Certain: measured] What each line says. (1) The matrix multiplies are the real work; a healthy run has them on top. (2) In FSDP2, waits for communication show up as self time of FSDP2’s hooks; at \(W = 4\) they are 38% of the step, so on this machine the run is communication-bound. (3) With checkpointing, aten::mm gets 26% more time: 99 calls per step instead of 75 (Step 8). One surprise: AdamW math took 13.8 ms per step with FSDP2 against 9.2 ms with DDP, although each rank updates only half the values. With tensors this small, the fixed cost of each DTensor operation is larger than the math. [Certain: measured] On a real model the shards are large and this cost becomes small.

Algorithm 3: find the top 3 time sinks of a training step
  1. profile with a schedule (skip at least 3 steps, warm up 2, record 3); label forward, backward and optimizer with record_function
  2. print key_averages() sorted by self time (CPU on a CPU run, CUDA on a GPU run); note the top rows and their call counts
  3. group rows that are one cost: all hooks of FSDP2, all attention kernels, all optimizer kernels
  4. open the trace; for each group, check when it happens and whether the compute stream is idle at that time // idle = exposed
  5. compare the sum of the top 3 with the step time; if a large part of the step is in no group, look for gaps (data loading, Python, CPU-GPU syncs)
  6. change one thing (bucket size, prefetch, checkpointing, batch), profile again, compare the same groups

On a GPU the same method works, with three more things to look for in the trace [Likely: general practice]. The NCCL kernels run on their own stream; you want them to sit under compute kernels, not under gaps. A call like loss.item() or torch.cuda.synchronize() makes the CPU wait for the GPU and shows up as a long cudaStreamSynchronize. And if the GPU row has many small gaps between short kernels, the CPU cannot launch kernels fast enough, which is where torch.compile and CUDA graphs help (session 50).

Step 8: activation checkpointing with FSDP2

Activation checkpointing (AC) keeps only the input of each wrapped block in the forward pass and throws away everything inside. In the backward pass it runs the block’s forward again to rebuild the activations, then computes the gradients. You trade compute for memory. With FSDP2 there is one rule: apply checkpointing first, then fully_shard. torchtitan does it in this order: checkpointing, then torch.compile, then FSDP. [Certain: torchtitan v0.2.0 source]

from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import checkpoint_wrapper
for i, block in enumerate(model.blocks):           # 1) checkpointing ...
    model.blocks[i] = checkpoint_wrapper(block, preserve_rng_state=False)
for block in model.blocks:                         # 2) ... then FSDP2 on the wrapped blocks
    fully_shard(block, mesh=mesh)
fully_shard(model, mesh=mesh)

Why this order? The checkpoint wrapper must sit inside the FSDP2 group, so that when backward recomputes the block, FSDP2’s pre-backward hook has already gathered the block’s weights. preserve_rng_state=False skips saving the random-number state; that is safe here because the block has no dropout. torchtitan uses the same wrapper and the same flag. [Certain: torchtitan v0.2.0 source]

Formula (52.3) predicts what checkpointing saves: per block, \(c\,d = 18d\) values per token become \(d\). Measured, rank 0, \(W = 2\):

FSDP2, \(W = 2\)Added in forwardPeakMatmul calls per stepStep time (median of runs)
no checkpointing117.2 MB155.18 MB75100.1 ms
every other block (3 of 6)63.6 MB105.25 MB87109.8 ms
every block10.0 MB71.09 MB99119.3 ms

[Certain: measured] Checkpointing every block removes 91% of the activations (117.2 → 10.0 MB) and cuts the peak by 54%. The 10.0 MB are the six block inputs (6.3 MB), the tensors outside the blocks (3.4 MB) and the gathered root (0.3 MB). The peak (71 MB) is now set by the backward pass, where one block’s activations are rebuilt (18.9 MB) while its weights and gradients are gathered.

The cost. Each block’s forward runs twice. If backward costs twice the forward, compute goes from \(1 + 2 = 3\) to \(1 + 1 + 2 = 4\) forward units:

\[ \frac{T^{\text{AC}}_{\text{comp}}}{T_{\text{comp}}} \approx \frac{1 + 1 + 2}{1 + 2} = \frac{4}{3}, \qquad \text{matmuls: } \frac{75 + 24}{75} = 1.32 \tag{52.8} \]

Our step has 75 matrix multiplies (4 per block in forward, 8 per block in backward, plus 3 for the head); checkpointing adds the 24 forward ones again, exactly 32% more. The measured aten::mm time grew by 26% (51.6 → 64.8 ms). The step time grew less than that, because communication and other overheads did not grow. Checkpointing every other block is the middle point; it is also torchtitan’s default (selective_ac_option = "2"). torchtitan also offers op-level selective checkpointing, which saves the outputs of expensive operations such as matrix multiplies (by default every second one) and recomputes only the cheap ones. [Certain: torchtitan v0.2.0 source and paper]

How does this scale to a real model? Figure 4 applies (52.2), (52.3) and (52.4) to this lab model and to Llama 2 7B.

Figure 4. Estimated memory per GPU from formulas (52.2) to (52.4), with \(c = 18\) values per token per block, and communication per step from (52.5). Lab model: everything FP32. Llama 2 7B (\(\Psi\) = 6.74B, 32 blocks of 202.4M, root 262.1M): 16 bytes per parameter of states, BF16 activations and gathered weights, FP32 gradient reduction. Activations outside the blocks, temporary buffers and fragmentation are not included.
How to read Figure 4

Choose Llama 2 7B, FSDP2, 8 GPUs, 4,096 tokens: 13.48 GB of states, 2.26 GB gathered, 19.33 GB of activations, 35.07 GB in total. Click DDP: 154 GB, far above the 80 GB line, so DDP cannot train this model at all. Back to FSDP2, turn on “AC every block”: activations fall to 1.68 GB and the total to 17.42 GB. That leaves room for 8 times more tokens: at 32,768 tokens per GPU the total is 29.16 GB. Now slide the GPUs to 64 for the lab model: the states almost vanish, but the gathered block and the activations do not shrink with \(W\). Last, compare the communication line for DDP and FSDP2 on the 7B model: equal, because of BF16 gathers and FP32 reduction (52.5).

Worked example: Llama 2 7B, 8 GPUs, 4,096 tokens per GPU

States. \(16 \times 6.738 \times 10^9 / 8 = 13.48\) GB. DDP would need \(20 \times 6.738 \times 10^9 = 134.8\) GB of states alone.

Gathered. \(2 \times (2 \times 262.1\text{M} + 3 \times 202.4\text{M}) = 2.26\) GB in BF16.

Activations. \(4{,}096 \times 32 \times 18 \times 4{,}096 \times 2 = 19.33\) GB without checkpointing. With checkpointing of every block: \(4{,}096 \times 32 \times 4{,}096 \times 2 = 1.07\) GB of inputs plus \(4{,}096 \times 18 \times 4{,}096 \times 2 = 0.60\) GB for one rebuilt block: 1.68 GB.

Total. 35.07 GB without and 17.42 GB with checkpointing, plus temporary buffers. The value \(c = 18\) comes from our GELU block; Llama’s SwiGLU MLP and its attention kernels save a somewhat different set of tensors, so treat the activation numbers as estimates. [Likely]

The full program

One file runs every version. On a CPU it starts \(W\) processes itself; on GPUs, start it with torchrun. The modes are single, ddp, ddpac, fsdp2, fsdp2norsh, fsdp2ac (every block) and fsdp2ac2 (every other block). We ran the CPU path; the GPU path uses the standard calls but we did not run it for this page.

# lab.py: the same GPT with DDP or FSDP2. CPU: python lab.py fsdp2 2   GPU: torchrun --nproc_per_node 2 lab.py fsdp2
import os, sys, time, statistics, torch, torch.nn as nn, torch.nn.functional as F
import torch.distributed as dist, torch.multiprocessing as mp
from torch.profiler import profile, ProfilerActivity, record_function

V, T, D, L, H, GLOBAL_B, STEPS = 65, 128, 256, 6, 8, 16, 30

class Block(nn.Module):
    def __init__(s):
        super().__init__()
        s.n1, s.n2 = nn.RMSNorm(D), nn.RMSNorm(D)
        s.qkv, s.o = nn.Linear(D, 3 * D, bias=False), nn.Linear(D, D, bias=False)
        s.up, s.down = nn.Linear(D, 4 * D, bias=False), nn.Linear(4 * D, D, bias=False)
    def forward(s, x):
        B, T_, _ = x.shape
        q, k, v = s.qkv(s.n1(x)).split(D, dim=-1)
        q, k, v = (t.view(B, T_, H, D // H).transpose(1, 2) for t in (q, k, v))
        a = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        x = x + s.o(a.transpose(1, 2).reshape(B, T_, D))
        return x + s.down(F.gelu(s.up(s.n2(x))))

class GPT(nn.Module):
    def __init__(s):
        super().__init__()
        s.tok, s.pos = nn.Embedding(V, D), nn.Embedding(T, D)
        s.blocks = nn.ModuleList(Block() for _ in range(L))
        s.norm, s.head = nn.RMSNorm(D), nn.Linear(D, V, bias=False)   # head not tied
        for p in s.parameters():
            if p.dim() == 2: nn.init.normal_(p, std=0.02)
    def forward(s, idx, y):
        x = s.tok(idx) + s.pos(torch.arange(idx.shape[1], device=idx.device))
        for b in s.blocks: x = b(x)
        return F.cross_entropy(s.head(s.norm(x)).view(-1, V), y.view(-1))

COMM = []                                    # (collective, bytes of the full tensor)
def counted(name, fn, key, pos):           # FSDP2 calls these with keyword arguments
    def w(*a, **k):
        t = k[key] if key in k else a[pos]
        COMM.append((name, t.numel() * t.element_size())); return fn(*a, **k)
    return w

def main(rank, W, mode, dev):
    dist.init_process_group("nccl" if dev == "cuda" else "gloo", rank=rank, world_size=W)
    if dev == "cuda": torch.cuda.set_device(rank)
    else: torch.set_num_threads(4)
    text = open("input.txt").read(); stoi = {c: i for i, c in enumerate(sorted(set(text)))}
    data = torch.tensor([stoi[c] for c in text])         # Tiny Shakespeare, 65 characters
    torch.manual_seed(0)                     # same init on every rank
    model = GPT().to(dev)
    if "ac" in mode:                         # 1) activation checkpointing first ...
        from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import checkpoint_wrapper
        for i, b in enumerate(model.blocks):
            if mode.endswith("ac") or i % 2 == 0:       # "...ac": every block, "...ac2": every other
                model.blocks[i] = checkpoint_wrapper(b, preserve_rng_state=False)
    if mode.startswith("ddp"):               # 2a) ... then DDP
        from torch.nn.parallel import DistributedDataParallel as DDP
        model = DDP(model, device_ids=[rank] if dev == "cuda" else None)
        def hook(state, bucket):             # same as the default all-reduce hook, plus counting
            buf = bucket.buffer(); COMM.append(("all_reduce", buf.numel() * buf.element_size()))
            return dist.all_reduce(buf.div_(W), async_op=True).get_future().then(lambda f: f.value()[0])
        model.register_comm_hook(None, hook)
    elif mode.startswith("fsdp2"):           # 2b) ... or FSDP2: blocks first, then the root
        from torch.distributed.fsdp import fully_shard
        from torch.distributed.device_mesh import init_device_mesh
        mesh = init_device_mesh(dev, (W,))
        for b in model.blocks: fully_shard(b, mesh=mesh, reshard_after_forward="norsh" not in mode)
        fully_shard(model, mesh=mesh)
        dist.all_gather_into_tensor = counted("all_gather", dist.all_gather_into_tensor, "output_tensor", 0)
        dist.reduce_scatter_tensor = counted("reduce_scatter", dist.reduce_scatter_tensor, "input", 1)
    opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.1)   # after wrapping!
    g, bs = torch.Generator().manual_seed(1234), GLOBAL_B // W
    losses, times = [], []
    prof = profile(activities=[ProfilerActivity.CPU] + ([ProfilerActivity.CUDA] if dev == "cuda" else []),
                   schedule=torch.profiler.schedule(wait=5, warmup=2, active=3), record_shapes=True)
    with prof:
        for step in range(STEPS):
            ix = torch.randint(len(data) - T - 1, (GLOBAL_B,), generator=g)   # the global batch ...
            ix = ix[rank * bs:(rank + 1) * bs]                                # ... and this rank's slice
            x = torch.stack([data[i:i + T] for i in ix]).to(dev)
            y = torch.stack([data[i + 1:i + T + 1] for i in ix]).to(dev)
            n0 = len(COMM)
            if dev == "cuda": torch.cuda.synchronize()
            t0 = time.perf_counter()
            with record_function("## forward ##"): loss = model(x, y)
            with record_function("## backward ##"): loss.backward()
            with record_function("## optimizer ##"): opt.step(); opt.zero_grad(set_to_none=True)
            if dev == "cuda": torch.cuda.synchronize()
            times.append(time.perf_counter() - t0)
            l = loss.detach().clone(); dist.all_reduce(l); losses.append(l.item() / W)  # not counted
            prof.step()
    local = lambda t: t.to_local() if hasattr(t, "to_local") else t
    p_mb = sum(local(p).numel() * 4 for p in model.parameters()) / 1e6
    o_mb = sum(local(v).numel() * 4 for s in opt.state.values() for v in s.values() if v.dim() > 0) / 1e6
    step_comm = COMM[n0:]
    if rank == 0:
        print(f"{mode} W={W}: loss {losses[0]:.4f} -> {losses[-1]:.4f}, step {1e3 * statistics.median(times[5:]):.1f} ms")
        print(f"model states on rank 0: params {p_mb:.2f} + grads {p_mb:.2f} + Adam {o_mb:.2f} MB")
        kinds = sorted(set(k for k, _ in step_comm))
        print("collectives per step:", {k: (sum(1 for c, _ in step_comm if c == k),
              round(sum(b for c, b in step_comm if c == k) / 1e6, 2)) for k in kinds})
        print(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=8))
        prof.export_chrome_trace(f"trace_{mode}_{W}.json")       # open in ui.perfetto.dev
        print("losses:", " ".join(f"{v:.6f}" for v in losses))
    dist.destroy_process_group()

if __name__ == "__main__":
    mode = sys.argv[1]                       # single | ddp | fsdp2 | fsdp2norsh | fsdp2ac | fsdp2ac2 | ddpac
    if "RANK" in os.environ:                 # started by torchrun (GPUs)
        main(int(os.environ["RANK"]), int(os.environ["WORLD_SIZE"]), mode, "cuda")
    else:                                    # CPU simulation: W processes on one machine
        W = int(sys.argv[2]); os.environ.setdefault("MASTER_ADDR", "127.0.0.1"); os.environ.setdefault("MASTER_PORT", "29655")
        mp.spawn(main, args=(W, mode, "cpu"), nprocs=W)
curl -O https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
python lab.py single 1 ; python lab.py ddp 2 ; python lab.py fsdp2 2 ; python lab.py fsdp2ac 2
torchrun --nproc_per_node 2 lab.py fsdp2                 # the same on 2 GPUs

On our machine python lab.py fsdp2 2 prints loss 4.1330 -> 3.3593, params 9.58 + grads 9.58 + Adam 19.15 MB and collectives per step: {'all_gather': (13, 38.04), 'reduce_scatter': (7, 19.15)}, followed by the profiler table. [Certain: measured]

Where this is used

System or taskWhat it shares with this lab
torchtitan (Llama 3 and other pretraining recipes)fully_shard on the token embedding, each block, and the final norm with the output head; the last group is not resharded after forward; BF16 parameters, FP32 reduction; checkpointing before sharding [Certain: v0.2.0 source]
Fine-tuning a 7B to 70B model on one nodeFSDP2 plus activation checkpointing is the usual way to fit full fine-tuning on 8 GPUs; the calculator of Figure 4 is the first estimate
Small models on many GPUsWhen the model fits, DDP (or FSDP2 without resharding) is simpler and exposes less communication, as Figure 2 shows
Vision-language model trainingThe vision encoder and the language model are wrapped as separate groups; a frozen encoder is easy with FSDP2 because each parameter is its own DTensor (Track 7)
Video-language pretraining, video classificationVideo clips give many tokens per sample, so activations dominate memory even more than here; checkpointing and small micro-batches are standard (Track 8)
Video retrieval and Video-RAG embedding modelsContrastive training wants large global batches; data parallelism over many GPUs with gathered embeddings, and the same profiling method to find the slow collective
Performance debugging of any training jobAlgorithm 3: profile with a schedule, group self time, check overlap in the trace, change one thing at a time

Common mistakes

  • Creating the optimizer before fully_shard. The optimizer then holds the old full parameters, which FSDP2 has replaced by DTensors. Training seems to run, but nothing is updated. Build the optimizer after wrapping.
  • Calling fully_shard only on the root. Then the whole model is one group, all of it is gathered at once, and the peak memory is the full model again. Shard each block first, then the root.
  • Wrapping in the wrong order. Checkpointing must be applied before fully_shard (and torch.compile between them, as torchtitan does). Applying it after puts the recompute outside FSDP2’s hooks.
  • Different data or seeds on different ranks without meaning to. If ranks build the model with different seeds, DDP broadcasts rank 0’s weights at the start [Certain], but FSDP2 shards whatever each rank has, so the shards come from different models [Likely]. Seed the model the same everywhere and split the data on purpose.
  • Comparing step time from a CPU simulation with GPU runs. The simulation is correct for memory, collective counts and losses. Its timing says nothing about NCCL overlap.
  • Profiling the first step. Step 0 includes allocations, autotuning and lazy initialization. Use a schedule with wait and warmup.
  • Reading total time instead of self time. ## backward ## has a total time of the whole backward pass; summing total times counts the same work many times. Rank by self time, then group.
  • Expecting bit-identical losses. Differences of \(10^{-7}\) at the start, growing to \(10^{-4}\), are normal (Step 4). A difference of \(10^{-2}\) at step 1 is a bug.

Interview questions

A model has \(\Psi\) parameters, trained with AdamW in mixed precision on \(W\) GPUs. Give the memory for model states per GPU with DDP and with FSDP.basic
16 bytes per parameter: FP32 weights and gradients (4 + 4) and the two Adam moments (8). DDP keeps all of it on every GPU: \(16\Psi\), plus \(4\Psi\) for its gradient buckets unless gradient_as_bucket_view=True. We measured 76.64 MB at the start of a step and 95.79 MB after backward for \(\Psi\) = 4.79M, that is \(16\Psi\) and \(20\Psi\). FSDP shards all of it: \(16\Psi/W\), plus the gathered root and one or two blocks in flight. Adding GPUs helps FSDP; it never helps DDP.
How do you check that a new distributed training setup is correct?basic
Run a single process with the same global batch and the same seeds, and compare the loss at every step. With equal tokens per rank, the average of rank gradients equals the global gradient (52.1), so the curves must agree to rounding: we saw \(10^{-7}\) at the first steps, drifting to \(10^{-4}\) after 30 AdamW steps. Also count the collectives per step (2 all-reduces for DDP in our model; 13 all-gathers and 7 reduce-scatters for FSDP2 with 6 blocks), and check that each rank holds \(1/W\) of the parameters.
How much more does FSDP communicate than DDP? Is it always 1.5×?core
Per step, DDP all-reduces the gradients once: \(2\Psi(W-1)/W\) elements. FSDP all-gathers the weights twice (forward and backward) and reduce-scatters the gradients once: \(3\Psi(W-1)/W\). In the same dtype that is 1.5×; we measured 28.60 MB against 19.15 MB at \(W = 2\) (1.49×, the root is gathered once). But the dtypes usually differ: FSDP2 gathers BF16 weights and reduces in FP32 (torchtitan’s defaults), so it sends \(8\Psi(W-1)/W\) bytes, the same as DDP all-reducing FP32 gradients. With reshard_after_forward=False, the backward all-gather disappears and FSDP sends the same elements as DDP.
What does reshard_after_forward do, and when would you set it to False?core
True: free a group’s gathered weights after its forward and gather them again in backward (like FULL_SHARD). False: keep them until backward (like SHARD_GRAD_OP), which saves one all-gather per group but keeps the full weights in memory between forward and backward. The default (None) is True for inner groups and False for the root, because backward starts at the root. Set False when memory allows and communication is exposed, for the last groups before the loss (torchtitan does this for the final norm and head), and with pipeline parallelism, where each micro-batch would otherwise gather again. In our lab it raised the peak from 155 to 168 MB and reduced the gathers from 13 to 7.
In what order do you apply activation checkpointing, torch.compile, fully_shard and the optimizer?core
Checkpointing first (wrap each block), then compile each block, then fully_shard bottom-up (blocks, then the root), then build the optimizer. This is torchtitan’s order. Checkpointing must be inside the FSDP2 group so that the recomputed forward finds its weights gathered by FSDP2’s pre-backward hook. The optimizer must see the sharded DTensor parameters, so it comes last.
What is the difference between self time and total time in a profile? Why did DDP’s ## backward ## label have 9 ms of self time?core
Total time runs from an event’s start to its end, including its children; self time subtracts the children (52.7). Ranking by self time shows where time is really spent and never counts twice. A label like ## backward ## has children for every backward operator; its self time is the time inside backward when no operator was running. In our DDP trace that was 9.4 ms of a 106 ms step: the autograd engine and, mostly, the wait for the last bucket’s all-reduce. That is DDP’s exposed communication, and it is invisible if you only read operator rows.
You switched a job from DDP to FSDP2 and the step got 30% slower. How do you find out why?core
Profile a few steady-state steps of both. Compare the top self-time groups: if matmul time is the same, the extra time is communication or overhead. In the trace, check whether the NCCL all-gathers and reduce-scatters overlap compute or sit in gaps. Common causes: blocks so small that each all-gather is longer than the block’s compute (the ratio of Figure 2 above 1); slow links across nodes (use HSDP: shard inside a node, replicate across nodes); a CPU-bound launch loop (explicit prefetching with set_modules_to_forward_prefetch, or torch.compile); resharding groups that are needed right away (set reshard_after_forward=False for the last groups). Change one thing at a time and profile again.
What does activation checkpointing save and cost? Give numbers.core
It stores only each checkpointed block’s input (\(d\) values per token instead of about \(18d\) in our block) and recomputes the block’s forward in backward. In our FSDP2 run it cut activations from 117.2 to 10.0 MB (−91%) and the peak from 155 to 71 MB (−54%). The cost is one extra forward per block: compute goes from 3 to 4 forward units, about +33% (52.8); matmul calls went from 75 to 99 (+32%) and measured matmul time grew by 26%. Checkpointing every other block, torchtitan’s default, gives about half the saving for half the cost; op-level selective checkpointing keeps expensive matmul outputs and recomputes cheap element-wise ops.
Why are FSDP2 losses not bit-identical to single-GPU losses, and how could you get closer?advanced
Floating-point addition is not associative: reducing gradients across ranks adds them in a different order, and FSDP2 also runs the optimizer on shards through DTensor, a different sequence of kernels. Each step differs by about \(10^{-7}\); AdamW then divides by \(\sqrt{v}\), which is tiny early on, and magnifies the differences (we saw \(2.7 \times 10^{-4}\) after 30 steps). To get closer: FP32 reductions, deterministic algorithms (torch.use_deterministic_algorithms(True)), and the same number of ranks; bit-identical results across different world sizes are not a realistic goal. What matters is that the drift stays at rounding level and does not grow into a different curve.
Estimate the memory per GPU for full fine-tuning of Llama 2 7B on 8 × 80 GB GPUs with FSDP2, 4,096 tokens per GPU. Does it fit? What if you use DDP?advanced
States: \(16 \times 6.74\text{B} / 8 = 13.48\) GB. Gathered root and blocks in BF16: \(2 \times (2 \times 262\text{M} + 3 \times 202\text{M}) = 2.26\) GB. Activations at about \(18d\) values per token per block in BF16: \(4096 \times 32 \times 18 \times 4096 \times 2 = 19.3\) GB. Total about 35 GB plus temporary buffers: it fits. With checkpointing of every block, activations drop to about 1.7 GB, total about 17.4 GB, so you can raise tokens per GPU. DDP needs \(16\Psi = 108\) GB (135 GB with separate buckets) for states alone: it does not fit on any 80 GB GPU.
What can a CPU simulation of distributed training tell you, and what can it not?advanced
It is exact for the program’s logic: loss parity with one process, the bytes each rank holds, the number, order and size of collectives, and how activation memory changes with checkpointing. It cannot tell you GPU step time or overlap: CPU math is far slower, gloo’s collectives run as CPU copies that compete with compute (and gloo even runs reduce-scatter as an all-reduce), and there are no CUDA streams. In our runs FSDP2 was 36% (2 processes) to 47% (4 processes) slower than DDP on the CPU, comparing the fastest runs; that number does not transfer to GPUs with NCCL and prefetching.

Exercises

  1. Compute the model states per rank for the lab model (\(\Psi\) = 4,787,968, FP32 AdamW) with DDP and with FSDP2 at \(W = 4\). Compare with the start-of-step memory in Figure 1.
  2. With formula (52.5), how many bytes does each rank send per step at \(W = 8\) for DDP and for FSDP2 (all FP32)?
  3. DDP does not shard. Why is its peak at \(W = 4\) (137 MB) much lower than at \(W = 2\) (196 MB)?
  4. Predict the activations added in the forward pass when only blocks 0, 2 and 4 are checkpointed, from the measured 18.9 MB per full block, 1.05 MB per block input and 3.4 MB outside the blocks. Compare with Figure 1.
  5. In Figure 3 (FSDP2, 2), click a reduce-scatter on a communication thread. How many values does gloo move in it, and how many bytes would NCCL’s ring reduce-scatter send per rank for the same gradient?
  6. Llama 2 7B, 16 GPUs, FSDP2, checkpointing of every block, 8,192 tokens per GPU. Estimate the memory per GPU with Figure 4’s formulas.
Answerscheck
1. DDP: \(16\Psi = 76.61\) MB (\(20\Psi = 95.76\) MB with the buckets during backward). FSDP2: \(16\Psi/4 = 19.15\) MB; without gradients at the start of the step, \(12\Psi/4 = 14.36\) MB, and Figure 1 shows 14.40 MB. 2. DDP: \(2 \times 4{,}787{,}968 \times 4 \times 7/8 = 33.52\) MB. FSDP2: \(3 \times 4{,}787{,}968 \times 4 \times 7/8 = 50.27\) MB. 3. The global batch is fixed at 16, so each rank has 4 windows instead of 8 and stores half the activations (58 MB instead of 117 MB); the states stay at 76.64 MB. 4. \(3 \times 18.9 + 3 \times 1.05 + 3.4 = 63.3\) MB; Figure 1 shows \(92.34 - 28.76 = 63.6\) MB. 5. gloo runs an all-reduce of the whole block, 786,944 values (3.15 MB in FP32). A ring reduce-scatter sends \(3.15 \times 1/2 = 1.57\) MB per rank at \(W = 2\); a ring all-reduce sends twice that, 3.15 MB. 6. States \(16 \times 6.738 / 16 = 6.74\) GB; gathered 2.26 GB; activations \(8{,}192 \times 32 \times 4{,}096 \times 2 = 2.15\) GB plus \(8{,}192 \times 18 \times 4{,}096 \times 2 = 1.21\) GB, so 3.36 GB; total 12.36 GB.

Key takeaways

  • Test a distributed setup against one process with the same global batch: the losses must agree to rounding (52.1). Ours agreed to \(10^{-7}\) at first and \(10^{-4}\) after 30 steps.
  • DDP keeps \(16\Psi\) (plus \(4\Psi\) of buckets) on every rank; FSDP2 keeps \(16\Psi/W\) plus the gathered root and about three blocks in flight (52.2, 52.4).
  • Activations often dominate: 117 MB against 38 MB of states in this lab. More ranks with a fixed global batch shrink activations per rank for both DDP and FSDP2.
  • FSDP2 sends \(3\Psi\) elements per step against DDP’s \(2\Psi\) (1.49× measured), but with BF16 gathers and FP32 reduction the bytes are equal (52.5).
  • Read profiles by self time (52.7), group related rows, then use the trace to see what is exposed. In DDP the exposed wait hides in the self time of the backward label; in FSDP2 it is in its hooks.
  • Apply checkpointing before fully_shard. Checkpointing every block cut activations by 91% and the peak by 54%, for 32% more matmuls (52.8).
  • A CPU simulation is exact for memory, collectives and losses, and says nothing about GPU step time.

Key formulas

\[ M^{\text{DDP}}_{\text{states}} = 20\Psi, \quad M^{\text{FSDP2}}_{\text{peak}} \approx \frac{16\Psi}{W} + n L c d p + p(2\psi_{\text{root}} + 3\psi_{\text{block}}), \quad A_{\text{AC}} = nLdp + ncdp \]
\[ V_{\text{DDP}} = 2\Psi b_g \frac{W-1}{W}, \quad V_{\text{FSDP2}} = (2\Psi b_p + \Psi b_g)\frac{W-1}{W}, \quad t^{\text{self}}_e = t^{\text{total}}_e - \sum_{c} t^{\text{total}}_c, \quad \frac{T^{\text{AC}}_{\text{comp}}}{T_{\text{comp}}} \approx \frac{4}{3} \]

References for this session

  1. PyTorch (docs). Getting Started with Fully Sharded Data Parallel (FSDP2)
  2. PyTorch (code). torchtitan: reference large-scale training with FSDP2
  3. PyTorch (docs). Getting Started with Fully Sharded Data Parallel (FSDP2)
  4. PyTorch (code). torchtitan: reference large-scale training with FSDP2
  5. Liang, Liu, Wright, Constable, Gu, Huang et al. (2024). TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining
  6. Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
  7. Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
  8. Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
  9. Chen et al. (2016). Training Deep Nets with Sublinear Memory Cost
  10. Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models
  11. PyTorch (docs). PyTorch Profiler recipe (torch.profiler: schedule, key_averages, export_chrome_trace)
  12. PyTorch (docs). torch.distributed.fsdp.fully_shard (FSDP2) API reference: fully_shard, FSDPModule, MixedPrecisionPolicy, CPUOffloadPolicy
  13. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters