FSDP (version 1)
DDP keeps a full copy of the model, its gradients and its optimizer states on every GPU, so a 7B model needs about 108 GB per GPU before any activation. FSDP is PyTorch's built-in way to shard all of this across GPUs: each GPU keeps only a slice, and gathers the full weights of one block just before it needs them. This session shows how FSDP (version 1) does it: the FlatParameter, the all-gather and reduce-scatter schedule, wrapping, prefetching, sharding strategies, mixed precision and CPU offload.
What you will learn
- How FSDP turns ZeRO stage 3 into a PyTorch wrapper: FSDP units, and the FlatParameter that flattens, pads and shards each unit
- The schedule of one training step: all-gather, compute, free in the forward pass; all-gather, gradients, reduce-scatter in the backward pass
- Why full sharding sends 1.5 times the bytes of DDP, and why the root unit is not gathered twice
- How the wrapping policy sets peak memory and the number of collectives, with real numbers for Llama 2 7B
- Backward prefetching (BACKWARD_PRE, BACKWARD_POST), forward prefetching and the rate limiter, and how they hide communication
- The sharding strategies FULL_SHARD, SHARD_GRAD_OP, NO_SHARD, HYBRID_SHARD and _HYBRID_SHARD_ZERO2, and when to use each
- MixedPrecision (param, reduce and buffer dtypes), CPU offload, state dicts, and the bugs people hit most often
Why this matters
In session 44 every GPU holds the full model, the full gradients and the full optimizer states. For mixed-precision AdamW that is 16 bytes per parameter. Llama 2 7B has 6.74 billion parameters, so DDP needs about 108 GB per GPU before a single activation is stored. An 80 GB GPU cannot do it, however many GPUs you buy.
Session 45 showed the cure: do not replicate, shard. Each of \(F\) GPUs keeps only \(1/F\) of the weights, gradients and optimizer states, and the GPUs pass weights to each other just in time. FSDP (Fully Sharded Data Parallel) is PyTorch’s own implementation of this idea (ZeRO stage 3). On 8 GPUs, the same 7B model needs about 13.5 GB of model states per GPU instead of 108 GB.
This session is about the first version of FSDP: the FullyShardedDataParallel wrapper described by Zhao et al. (2023). Its main data structure is the FlatParameter. The next session, session 47, replaces it with per-parameter sharding. You need to know version 1 well for three reasons. A lot of training code (Hugging Face Accelerate, older Lightning and many research repositories) still uses it [Likely]. Its ideas (units, all-gather and reduce-scatter, prefetching) are the same in FSDP2. And interviewers love to ask about it.
How well does it work? In the FSDP paper, a 175B GPT model trained on 128 to 512 A100 GPUs reached more than 173 TFLOPS per GPU with batch size 1 and 186 with batch size 2, about 55% and 60% of the A100’s 312 TFLOPS BF16 peak, and scaled almost linearly in TFLOPS from 128 to 512 GPUs. [Certain] For small models (611M and 2.28B parameters) it was as fast as DDP, and it trained an 11B T5 model where DDP ran out of memory. [Certain]
From ZeRO-3 to FSDP: the idea in one picture
Let \(\Psi\) be the number of parameters, and \(F\) the sharding factor: the number of GPUs (ranks) that share one copy of the model. With mixed-precision AdamW, the model states cost 16 bytes per parameter (session 38). DDP keeps all of them on every GPU. FSDP with full sharding keeps \(1/F\) of them:
- Read the symbols
- \(\Psi\)
- number of parameters of the model (6.74 billion for Llama 2 7B)
- \(F\)
- sharding factor: how many GPUs split one copy of the model. With full sharding, \(F\) is the number of GPUs.
- 16
- bytes per parameter: FP32 weights (4), FP32 gradients (4), Adam \(m\) and \(v\) (4 + 4)
- \(M_{\text{unsharded}}\)
- the extra memory for the few layers whose full weights are gathered right now. FSDP keeps this small by gathering only one block at a time.
But a GPU cannot multiply by weights it does not have. So before a layer runs, the GPUs do an all-gather: every GPU sends its slice to everyone, and every GPU ends up with the full weights of that layer. After the layer runs, the full weights are thrown away again. In the backward pass, each GPU computes gradients for the full layer from its own data, and a reduce-scatter sums the gradients over GPUs and leaves each GPU with only the summed gradient of its own slice (session 43). Each GPU then runs the optimizer step on its own slice only.
Two engineering questions decide whether this is fast: in what pieces do we gather the weights, and when do we start each transfer so that it hides behind computation. The rest of the session answers both.
FSDP units and the FlatParameter
FSDP does not gather one tensor at a time. A transformer block has about ten weight tensors (attention projections, MLP matrices, norm weights), and one collective per tensor would be slow: every collective has a fixed launch cost, and small messages use the network badly. In the paper’s measurements, with the same total amount of data, all-gather time began to rise quickly once each all-gather moved fewer than about 33 million elements. [Certain] NCCL’s fastest all-gather also wants the same input size on every rank.
So FSDP groups modules into FSDP units. You choose them: usually one unit per transformer block, plus a root unit (the outermost wrapper) that takes all leftover parameters, such as the embedding and the output head. For each unit, FSDP builds one FlatParameter:
- Flatten every parameter of the unit into a 1-D vector.
- Concatenate them into one long 1-D vector of \(\psi\) elements.
- Pad with zeros on the right so the length is divisible by \(F\).
- Chunk it into \(F\) equal pieces. Rank \(r\) keeps piece \(r\) and frees the rest.
- Read the symbols
- \(\psi\)
- number of elements in all parameters of one unit together
- \(\tilde\psi\)
- the padded length, the smallest multiple of \(F\) that is at least \(\psi\)
- \(\lceil x \rceil\)
- round up to the next whole number
The original parameters (for example weight of shape 3 × 4) become views into the FlatParameter: they share its memory. When the FlatParameter is unsharded, FSDP points each original parameter at its slice of the full vector, so normal PyTorch code runs unchanged. The gradient of the FlatParameter has the same layout. This layout is exactly what all-gather writes and reduce-scatter reads, so neither needs an extra copy. [Certain] Figure 1 builds a FlatParameter for small units.
How to read Figure 1
Start with Linear(4, 3) and \(F = 16\): 12 weight numbers plus 3 bias numbers make 15, so FSDP adds 1 zero and each rank holds exactly one number. This is the example in the FSDP paper. Now pick \(F = 3\) for the two-layer unit: 26 numbers, 1 padding, 9 per rank. Notice that a shard does not respect parameter borders. A rank may hold half of a weight matrix and a piece of a bias. That is why, in FSDP1, “the part of weight on rank 2” is not a clean tensor, and why per-parameter tricks (freezing one tensor, a different learning rate for one tensor) are awkward. FSDP2 fixes exactly this.
- input: the parameters \(p_1, \dots, p_k\) of one unit, sharding factor \(F\), this rank’s index \(r\)
- \(\text{flat} \leftarrow \text{concat}(\text{flatten}(p_1), \dots, \text{flatten}(p_k))\) // length \(\psi\)
- append \(\tilde\psi - \psi\) zeros to flat // formula (46.2), at most \(F - 1\) zeros
- \(\text{shard}_r \leftarrow \text{flat}[\,r\tilde\psi/F : (r+1)\tilde\psi/F\,]\); free the rest of flat
- remember each \(p_i\)’s offset and shape, so it can later be a view into the unsharded flat vector
- the optimizer receives \(\text{shard}_r\) (one 1-D tensor) instead of \(p_1, \dots, p_k\)
One training step: all-gather, compute, free
With full sharding, every non-root unit goes through the same life in each training step:
- Forward: all-gather the unit’s FlatParameter, run the unit’s forward, free the full weights (keep only the local shard).
- Backward: all-gather the unit again (it was freed), compute the gradients for the full unit, reduce-scatter the gradients so each rank keeps the summed gradient of its own shard, free the full weights and full gradients.
- Optimizer step: each rank updates only its own shard, with its own shard of the Adam states. No communication.
The root unit is special. It is gathered at the start of the forward pass and kept until the backward pass is done, because the backward pass begins right where the forward pass ended: freeing it and gathering it again at once would be pure waste. [Certain] This saves one all-gather per step.
- input: units \(U_0\) (root), \(U_1, \dots, U_N\); local shards; this rank’s micro-batch \(x_r\)
- all-gather \(U_0\) // root stays unsharded for the whole step
- for \(i = 1, \dots, N\): all-gather \(U_i\); run forward of \(U_i\); free \(U_i\)’s full weights // activations are kept
- finish the root’s forward (for example the output head) and compute the loss on \(x_r\)
- start backward; the root’s full gradient buffer is allocated when the head’s gradient is computed
- for \(i = N, \dots, 1\): all-gather \(U_i\); compute gradients of \(U_i\); reduce-scatter them (average over ranks); free full weights and full gradients
- finish the root’s backward; reduce-scatter the root’s gradients; free the root’s full weights
- optimizer step on the local shards only // sharded Adam states, no communication
Write \(\theta_i^{(r)}\) for the shard of unit \(i\) that rank \(r\) owns, and \(s = \tilde\psi_i / F\) for the shard size from (46.2). The three operations of one unit are then three short formulas:
- Read the symbols
- \(\theta_i\)
- the full (unsharded, padded) FlatParameter of unit \(i\), only alive between its all-gather and its free
- \(g_i^{(q)}\)
- the full gradient of unit \(i\) that rank \(q\) computed from its own micro-batch; different on every rank
- \([\,a : b\,]\)
- the elements from position \(a\) up to (not including) \(b\): the slice that rank \(r\) owns
- \(\hat g_i^{(r)}\)
- the averaged gradient of rank \(r\)’s slice: exactly the slice of the full-batch gradient
- \(\eta\)
- the learning rate (plain SGD here; AdamW works the same way, with the shard of \(m\) and \(v\))
Figure 2 runs Algorithm 2 for real, in your browser, on a model small enough to show every number. Three units, each a Linear(2, 2) (4 weights and 2 biases, so \(\psi = 6\)); the first two use tanh. Each GPU gets one example \(\mathbf{x}_r\) with target \(\mathbf{y}_r\) and the loss \(L_r = \tfrac12 \lVert \mathbf{h}_3 - \mathbf{y}_r \rVert^2\). The algorithm box highlights the line that runs; the box next to it shows that line’s formula with the live numbers of the GPU you pick.
- for \(i = 1, 2, 3\) do // forward
- all-gather \(\theta_i\) from all shards // (46.3)
- \(\mathbf{h}_i \leftarrow \phi(W_i \mathbf{h}_{i-1} + \mathbf{b}_i)\) on the local example
- free \(\theta_i\), keep \(\theta_i^{(r)}\) // SHARD_GRAD_OP: keep \(\theta_i\)
- \(L_r \leftarrow \tfrac12 \lVert \mathbf{h}_3 - \mathbf{y}_r \rVert^2\)
- for \(i = 3, 2, 1\) do // backward
- all-gather \(\theta_i\) again // SHARD_GRAD_OP: skipped
- \(g_i^{(r)} \leftarrow \partial L_r / \partial \theta_i\) // full gradient, local data
- reduce-scatter \(\to \hat g_i^{(r)}\); free \(\theta_i\) and \(g_i^{(r)}\) // (46.3)
- \(\theta_i^{(r)} \leftarrow \theta_i^{(r)} - \eta\, \hat g_i^{(r)}\) for every unit // no communication
How to read Figure 2
Press Step and follow the highlighted line. With \(F = 4\), \(\psi = 6\) is padded to \(\tilde\psi = 8\), so every GPU owns 2 numbers of each unit, and the last 2 numbers (owned by GPU 3) are padding. Each all-gather fills the row with the full 8 numbers on every GPU; the forward then gives a different \(\mathbf{h}\) on each GPU, because each has its own example (GPU 0: \(\mathbf{h}_1 = (0.336, 0.422)\), loss 0.509). In the backward pass each GPU computes a different full gradient (orange), and the reduce-scatter averages them and keeps only the GPU’s own slice (green). For U3 the averaged weight gradient starts \(0.072, -0.202\) (GPU 0’s slice). After the local step the mean loss over the 4 examples falls from 0.342 to 0.148, and the gathered new weights equal single-GPU SGD exactly. Watch the memory curve: with FULL_SHARD it rises by 8 for one unit and drops back in the forward pass, and the peak is 26 numbers while U1’s gradient is computed. Switch to SHARD_GRAD_OP: 3 fewer all-gathers (36 numbers sent per GPU instead of 54), but all three gathered units stay until the backward, and the peak grows to 38.
A toy with 6-number units makes the shards and the padding look large. In a real model, each unit has hundreds of millions of numbers, the padding is at most \(F - 1\) of them, and the AdamW states (\(m\), \(v\)) are sharded too. Figure 3 shows the same step at that scale. The model has a root and three blocks of 100 million parameters each, on 4 GPUs, with BF16 for the gathered weights and gradients.
How to read Figure 3
Watch the memory bar. In the forward pass it rises by 0.2 GB for one block at a time and falls back. In the backward pass a block needs its weights and its full gradients, so it costs 0.4 GB while it runs. The root is the exception: its full weights stay from the first step to the last, and its full gradient buffer lives through the whole backward pass, because the root’s gradient is only complete when the embedding (the first layer) has its gradient. Count the collectives at the end: 7 all-gathers and 4 reduce-scatters, not 8 and 4, because the root is gathered once.
How much does it communicate?
Count elements sent per GPU, with the ring algorithms from session 43. An all-gather or a reduce-scatter of \(\Psi\) elements over \(F\) ranks sends \(\Psi (F-1)/F\) per GPU, and an all-reduce sends twice that. DDP does one all-reduce of the gradients. Full sharding does one all-gather in the forward pass, one in the backward pass, and one reduce-scatter:
This is the 1.5× from the FSDP paper and from ZeRO stage 3. [Certain] The root skips one all-gather, so the real number is slightly below 1.5×. In Figure 3: 11 collectives of 150 MB each (\(100\text{M} \times 3/4 \times 2\) bytes) make 1.65 GB, while formula (46.4) for 400M parameters gives 1.8 GB. With BF16 communication (next sections), each element is 2 bytes.
Peak memory and the wrapping policy
How you cut the model into units decides two things. Let the units have \(\psi_1, \dots, \psi_N\) elements, with \(\sum_i \psi_i = \Psi\). Every rank always keeps all its shards, and it has to hold the full weights of about one unit at a time. So the paper writes the parameter memory as [Certain]
The first term is fixed (\(\Psi / F\)). Only the second term depends on the wrapping: smaller units give a smaller peak, but more and smaller collectives. That is FSDP’s memory–throughput trade-off, and you set it with the auto_wrap_policy:
transformer_auto_wrap_policywithtransformer_layer_cls={LlamaDecoderLayer}(or your block class): one unit per transformer block. This is the standard choice for LLMs. Everything not inside a block (embedding, final norm, output head) falls into the root.size_based_auto_wrap_policy: wrap any submodule with at leastmin_num_paramsparameters (default \(10^8\)). [Certain] Simple, but its cuts may not follow the block structure.- No policy: the whole model is one unit. Then all-gathering “one unit” means all-gathering the whole model, and FSDP saves memory only on gradients and optimizer states: the peak is the full model again.
Figure 4 applies the same simple memory model to Llama 2 7B (32 blocks of 202.4M parameters; embedding and output head of 131.1M each, not tied).
How to read Figure 4
Start at 8 GPUs, per block: 13.48 GB sharded plus 2.26 GB in flight. Switch to “No wrapping”: the in-flight part jumps to 27 GB, because the whole model is gathered with its gradients. Now slide to 64 GPUs. The sharded part falls to 1.68 GB, but the in-flight part does not shrink at all: it depends on unit size, not on \(F\). At large scale, the wrapping is what decides your peak. Also notice the root: with per-block wrapping, the root (embedding + head, 262M) is bigger than a block (202M). Wrapping the embedding and the head as their own units drops the peak by about 1 GB [Likely]. The cost is the collective count shown in the readout.
Parameters: each block has \(4d^2 = 67{,}108{,}864\) attention weights, \(3 d\, d_{ff} = 135{,}266{,}304\) MLP weights and \(2d = 8{,}192\) norm weights: 202,383,360. The embedding and the output head have \(32{,}000 \times 4{,}096 = 131{,}072{,}000\) each. Total: \(32 \times 202{,}383{,}360 + 2 \times 131{,}072{,}000 + 4{,}096 = 6{,}738{,}415{,}616\).
DDP: \(16 \times 6.738 \times 10^9 = 107.8\) GB per GPU. FSDP, \(F = 8\): \(107.8 / 8 = 13.48\) GB of sharded states. Per-block wrapping with BACKWARD_PRE adds \(2 \times (2 \times 262.1\text{M} + 3 \times 202.4\text{M}) = 2.26\) GB, so about 15.7 GB, plus activations. That leaves most of an 80 GB GPU for activations.
Communication per step and per GPU, formula (46.4) in BF16: \(3 \times 6.738 \times 10^9 \times 7/8 \times 2\) bytes \(= 35.4\) GB, against 23.6 GB for DDP’s all-reduce.
Hiding the communication: prefetching and the rate limiter
An all-gather takes time. If the GPU waits for each one, the step gets slower. FSDP runs all collectives on a separate CUDA stream (a queue of GPU work), so they can run while the compute stream is busy. But all of FSDP’s all-gathers and reduce-scatters go through one NCCL stream, so they run one after another, in the order the CPU issued them. [Certain] The order is everything.
Forward pass. The CPU usually runs ahead of the GPU, so the all-gather for block \(i+1\) is issued while block \(i\) still computes, and it overlaps naturally. If the CPU is slow (many small kernels), forward_prefetch=True issues the next all-gather explicitly before the current block’s forward. It uses the order of the previous step, so it is only for models whose execution order does not change. [Certain]
Backward pass. Here the natural order is bad. After block \(i\) computes its gradients, FSDP issues its reduce-scatter. Only then does autograd reach block \(i-1\) and ask for its all-gather. On a single NCCL stream the all-gather waits for the reduce-scatter, and the compute waits for the all-gather: two transfers in a row, fully exposed. The backward_prefetch option changes the issue order: [Certain]
BACKWARD_PRE(the default): issue the all-gather of the next block before computing the current block’s gradients. Most overlap. At the peak, a rank holds the current weights, the next weights and the current gradients.BACKWARD_POST: issue the next all-gather after the current gradients, but before the current reduce-scatter. Less overlap, less memory (the current weights are freed first).None: no prefetching. Not recommended: it can be much slower.
FSDP knows which block comes “next” in the backward pass by recording the forward order and reversing it, freshly each step. [Certain] In the paper, backward prefetching made a 175B model about 18% faster on 128 to 512 GPUs. [Certain]
The rate limiter. If the CPU runs too far ahead, it issues many all-gathers at once. Their output buffers are allocated on the communication stream, and the PyTorch caching allocator cannot reuse those blocks for the compute stream until the GPU is done with them. Memory fills up, and the allocator falls back to slow cudaFree and cudaMalloc calls (“malloc retries”). limit_all_gathers=True (the default) blocks the CPU so that at most two units are gathered at a time: the one computing and the one prefetched. [Certain] In the paper it gave up to 5× speedups on T5-11B but cost about 5% on a DeepViT model, so check torch.cuda.memory_stats()["num_alloc_retries"] before you change it. [Certain]
A simple model of step time: if \(T_{\text{comp}}\) is the pure compute time, the step takes
The first all-gather and the last reduce-scatter can never be hidden; everything in between can be, if prefetching is on and each transfer is shorter than the compute it overlaps. Figure 5 simulates it for 4 blocks and plays it like a clock. The algorithm box shows the issue order of the chosen mode; the lines of the operations that run at the current moment light up. The bottom lane counts how many full-size buffers (gathered weights \(W_i\) or full gradients \(G_i\), each the size of one block) the GPU holds at that moment.
How to read Figure 5
Press Play and watch the red clock line. At the default (transfer 0.6), no prefetch gives 17.4 time units: every backward block waits for a reduce-scatter and then an all-gather. BACKWARD_POST gives 15.6: only the all-gather waits. BACKWARD_PRE gives 13.8: only the first all-gather of each pass and the final reduce-scatter are exposed. Now look at the bottom lane. Without prefetch the GPU never holds more than 2 buffers; with either prefetch mode it reaches 3 in the backward pass (for BACKWARD_PRE: \(W_i\), the prefetched \(W_{i-1}\) and \(G_i\)). That is the memory price of overlap. Set the transfer time to 0.3 and untick the rate limiter: all four forward all-gathers run back to back and the GPU holds 4 gathered blocks at once; with the limiter it is never more than 2 in the forward pass. Raise the transfer time above 1.0 and even BACKWARD_PRE cannot hide everything, because two transfers (a reduce-scatter and the next all-gather) must fit inside one block’s backward compute of 2. That is the regime of small batches and slow networks, where HYBRID_SHARD (next section) helps.
Sharding strategies
Full sharding is not always the best choice. The sharding_strategy argument picks how much is sharded, and over how many GPUs: [Certain]
| Strategy | What is sharded | Collectives per step | Like |
|---|---|---|---|
FULL_SHARD (default) | Weights, gradients, optimizer states over all \(W\) GPUs. Weights are freed after forward and gathered again for backward. | 2 all-gathers + 1 reduce-scatter per unit | ZeRO-3 |
SHARD_GRAD_OP | Gradients and optimizer states. Weights are sharded outside the step, but the gathered weights are kept from the forward pass until the end of the backward pass. | 1 all-gather + 1 reduce-scatter per unit | ZeRO-2 (roughly) |
NO_SHARD | Nothing. Gradients are all-reduced. | 1 all-reduce | DDP |
HYBRID_SHARD | FULL_SHARD inside each group of \(F\) GPUs (usually one node), replicated across groups | inside node: as FULL_SHARD; across nodes: 1 all-reduce of the gradient shard | ZeRO-3 + DDP |
_HYBRID_SHARD_ZERO2 | SHARD_GRAD_OP inside each node, replicated across nodes | inside node: as SHARD_GRAD_OP; across nodes: 1 all-reduce | ZeRO-2 + DDP |
SHARD_GRAD_OP trades memory for speed: it saves the second all-gather (one third of the communication) but keeps the full BF16 weights in memory during the whole step, \(2\Psi\) bytes. For Llama 2 7B that is 13.5 GB extra per GPU.
Hybrid sharding
Inside a node, GPUs talk over NVLink at hundreds of GB/s; between nodes, the network is usually several times slower (session 42) [Likely]. Hybrid sharding uses this. With \(W\) GPUs in \(H = W/G\) nodes of \(G\) GPUs each, it shards over the \(G\) GPUs of one node (\(F = G\)) and keeps one full copy per node. The gradient sum is split into two steps, which is exact because addition can be grouped any way:
- Read the symbols
- \(\mathbf{g}_r\)
- the gradient computed on rank \(r\) from its own data
- \(S_i\)
- the sharding group \(i\): the \(G\) GPUs of node \(i\)
- \(H\)
- number of sharding groups (nodes)
The inner sum is a reduce-scatter inside the node. The outer sum is an all-reduce among the GPUs that hold the same shard, one per node (the “replication group”). Only that all-reduce crosses nodes, and it moves only a \(1/G\) slice of the gradients. With a ring over the \(H\) nodes, the bytes that leave each GPU through the network per step are
The FSDP paper gives the same picture: hybrid sharding cuts cross-host traffic by roughly a factor \(G\) compared to full replication or full sharding. [Certain] The price is memory: each GPU holds \(16\Psi/G\) bytes of states, not \(16\Psi/W\). Hybrid sharding is a good fit for “medium” models that do not fit with DDP but do fit when sharded over one node. Figure 6 compares all five strategies.
How to read Figure 6
Defaults: a 7B model (6.74B) on 64 GPUs. FULL_SHARD needs only 1.68 GB of states but sends 39.8 GB per step, all of it over a group that spans 8 nodes. HYBRID_SHARD needs 13.5 GB and sends 35.4 GB, but only 2.95 GB of it leaves the node. NO_SHARD (DDP) needs about 108 GB: it does not fit. Push the model to 70B: now HYBRID_SHARD needs 140 GB per GPU and no longer fits, so you must shard across nodes, or use larger sharding groups, or add tensor parallelism (session 48).
Mixed precision and CPU offload
MixedPrecision
FSDP has its own mixed-precision option, MixedPrecision, with three dtypes (all default to None, meaning “keep FP32”): [Certain]
param_dtype: the dtype of the gathered weights, used in forward and backward. The sharded weights stay in FP32 for the optimizer step, and checkpoints are saved in FP32.reduce_dtype: the dtype of the gradient reduce-scatter (or all-reduce). If it isNonebutparam_dtypeis set, it followsparam_dtype. Set it to FP32 if you want the gradient sum in full precision.buffer_dtype: the dtype of buffers (for example rotary-embedding tables). FSDP does not shard buffers.
The key trick: FSDP casts the local FP32 shard to BF16 before the all-gather. So the all-gather moves half the bytes, and the gathered copy costs half the memory. The FP32 copy of the full unit never exists. The paper writes the peak parameter memory as [Certain]
- Read the symbols
- \(K_{\text{full}}\)
- bytes per element in full precision (4 for FP32)
- \(K_{\text{low}}\)
- bytes per element in low precision (2 for BF16 or FP16)
Example, Llama 2 7B with per-block wrapping on 8 GPUs, where the largest unit is the 262M-parameter root: \(4 \times 6.738\text{B}/8 + 4 \times 262\text{M} = 4.42\) GB becomes \(3.37 + 0.52 = 3.89\) GB. FSDP’s cast happens once per unit per gather, unlike torch.autocast, which casts at each operation. [Certain] With FP16, use ShardedGradScaler, not the normal GradScaler: the gradients are sharded, so the check for infinities must look at all ranks. [Certain] BF16 does not need a scaler (session 37).
CPU offload
CPUOffload(offload_params=True) keeps the sharded parameters on the CPU when they are not in use. It offloads the gradients too, so the optimizer step runs on the CPU. [Certain] Before each all-gather, the shard is copied to the GPU over PCIe; after each reduce-scatter, the gradient shard is copied back. This frees nearly all of the \(16\Psi/F\) on the GPU (for Llama 2 7B on 8 GPUs: 13.5 GB per GPU, so 108 GB of CPU RAM per node), but PCIe is much slower than GPU memory, and an Adam step on CPU is slow. Use it when nothing else fits (for example, fine-tuning a big model on few GPUs), not for large pretraining runs. [Likely]
Using FSDP1 in PyTorch
First, the mechanics without any distributed library: a FlatParameter for one unit on 4 simulated ranks, all-gather and reduce-scatter written by hand, and a check that one sharded SGD step equals one normal SGD step on the full batch.
import torch torch.manual_seed(0) F = 4 # sharding factor = number of ranks # --- 1) build a FlatParameter for one unit: flatten, concat, pad, chunk (Algorithm 1) --- unit = torch.nn.Linear(4, 3) # weight 3x4 = 12, bias 3 -> 15 numbers params = [unit.weight, unit.bias] numels = [p.numel() for p in params] flat = torch.cat([p.detach().reshape(-1) for p in params]) pad = (-flat.numel()) % F # formula (46.2): at most F - 1 flat = torch.cat([flat, flat.new_zeros(pad)]) shards = list(flat.chunk(F)) # rank r keeps shards[r] only print("numel", sum(numels), "pad", pad, "shard size", shards[0].numel()) # 15 1 4 def all_gather(shards): # every rank gets the full flat tensor return torch.cat(shards) def reduce_scatter(full_grads): # sum over ranks, rank r keeps chunk r return list(torch.stack(full_grads).sum(0).chunk(F)) def views(full): # original shapes are views into the flat tensor return full[:numels[0]].view(3, 4), full[numels[0]:numels[0] + numels[1]] # --- 2) one FSDP step with SGD (Algorithm 2), data split over F ranks --- X, Y = torch.randn(8, 4), torch.randn(8, 3) lr, grads = 0.1, [] for r in range(F): full = all_gather(shards).requires_grad_(True) # unshard w, b = views(full) x, y = X.chunk(F)[r], Y.chunk(F)[r] loss = ((x @ w.T + b - y) ** 2).mean() / F # /F: average over ranks loss.backward() # gradient of the whole flat tensor grads.append(full.grad) # (then the full weights are freed) g_shards = reduce_scatter(grads) # each rank: summed gradient of its own chunk shards = [s - lr * g for s, g in zip(shards, g_shards)] # local optimizer step on the shard # --- 3) reference: plain single-device SGD on the whole batch --- ref = torch.nn.Linear(4, 3); ref.load_state_dict(unit.state_dict()) ((ref(X) - Y) ** 2).mean().backward() with torch.no_grad(): for p in ref.parameters(): p -= lr * p.grad w, b = views(all_gather(shards)) print("max diff:", max((w - ref.weight).abs().max().item(), (b - ref.bias).abs().max().item())) # about 3e-8 print("padding stays zero:", all_gather(shards)[-pad:].abs().max().item()) # 0.0
Now the real thing. This script runs PyTorch’s FSDP1 on 4 CPU processes with the gloo backend, so it works on a laptop. It wraps each block as a unit, trains 3 AdamW steps with each rank on a quarter of the batch, and compares the gathered weights with single-process AdamW on the full batch. We ran it with PyTorch 2.8: it prints 4 units, local shard sizes [53, 540, 540, 540] (the root has \(144 + 68 = 212\) parameters, each block 2,160), matching keys, and a maximum difference of about \(10^{-7}\).
import os, functools, torch, torch.nn as nn import torch.distributed as dist, torch.multiprocessing as mp from torch.distributed.fsdp import (FullyShardedDataParallel as FSDP, ShardingStrategy, StateDictType, FullStateDictConfig) from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy class Block(nn.Module): # one "transformer block" (an MLP block here) def __init__(s, d): super().__init__(); s.norm = nn.LayerNorm(d); s.up = nn.Linear(d, 4 * d); s.down = nn.Linear(4 * d, d) def forward(s, x): return x + s.down(torch.relu(s.up(s.norm(x)))) def make_model(): torch.manual_seed(0) # same init on every rank and for the reference return nn.Sequential(nn.Linear(8, 16), Block(16), Block(16), Block(16), nn.Linear(16, 4)) def worker(rank, W): os.environ.update(MASTER_ADDR="127.0.0.1", MASTER_PORT="29541") dist.init_process_group("gloo", rank=rank, world_size=W) # CPU backend; use "nccl" on GPUs policy = functools.partial(transformer_auto_wrap_policy, transformer_layer_cls={Block}) model = FSDP(make_model(), auto_wrap_policy=policy, device_id=torch.device("cpu"), sharding_strategy=ShardingStrategy.FULL_SHARD) units = [m for m in model.modules() if isinstance(m, FSDP)] if rank == 0: print("FSDP units:", len(units)) print("local shard numels:", [u._flat_param.numel() for u in units]) # private attribute, for teaching only opt = torch.optim.AdamW(model.parameters(), lr=1e-2) # created AFTER wrapping: sees only shards ref = make_model(); ropt = torch.optim.AdamW(ref.parameters(), lr=1e-2) torch.manual_seed(1); X, Y = torch.randn(16, 8), torch.randn(16, 4) for step in range(3): x, y = X.chunk(W)[rank], Y.chunk(W)[rank] # each rank: its own quarter loss = nn.functional.mse_loss(model(x), y) loss.backward(); opt.step(); opt.zero_grad() # gradients are averaged over ranks rl = nn.functional.mse_loss(ref(X), Y); rl.backward(); ropt.step(); ropt.zero_grad() cfg = FullStateDictConfig(rank0_only=True) with FSDP.state_dict_type(model, StateDictType.FULL_STATE_DICT, cfg): sd = model.state_dict() # gathers full, original-shaped weights on rank 0 if rank == 0: diff = max((sd[k] - v).abs().max().item() for k, v in ref.state_dict().items()) print("keys match:", sorted(sd) == sorted(ref.state_dict()), " max diff vs 1-GPU AdamW:", f"{diff:.1e}") dist.destroy_process_group() if __name__ == "__main__": mp.spawn(worker, args=(4,), nprocs=4) # 4 ranks = 4 "GPUs"
Change FULL_SHARD to SHARD_GRAD_OP and the result is the same. With NO_SHARD the local sizes become [212, 2160, 2160, 2160]: nothing is sharded. With MixedPrecision(param_dtype=torch.bfloat16, reduce_dtype=torch.float32) (and the loss computed on model(x).float()), the difference grows to about \(2 \times 10^{-2}\) after 3 steps: that is BF16 rounding in forward and backward, not a bug. On a GPU cluster, a typical FSDP1 setup for a Llama-style model looks like this (launched with torchrun --nproc_per_node=8; not run here, it needs GPUs):
from torch.distributed.fsdp import (FullyShardedDataParallel as FSDP, ShardingStrategy, MixedPrecision, BackwardPrefetch, CPUOffload) from transformers.models.llama.modeling_llama import LlamaDecoderLayer torch.cuda.set_device(local_rank) model = FSDP(model, auto_wrap_policy=functools.partial(transformer_auto_wrap_policy, transformer_layer_cls={LlamaDecoderLayer}), sharding_strategy=ShardingStrategy.FULL_SHARD, # or HYBRID_SHARD across nodes mixed_precision=MixedPrecision(param_dtype=torch.bfloat16, reduce_dtype=torch.float32, buffer_dtype=torch.bfloat16), backward_prefetch=BackwardPrefetch.BACKWARD_PRE, # the default limit_all_gathers=True, # the rate limiter (default) cpu_offload=CPUOffload(offload_params=False), # True only if nothing else fits device_id=torch.cuda.current_device(), # shard on GPU, fast init sync_module_states=True, # broadcast rank 0's weights (e.g. loaded checkpoint) use_orig_params=True) # needed for torch.compile and per-parameter groups optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1) # after wrapping! ... model.clip_grad_norm_(1.0) # FSDP's own: computes the norm over all shards
State dicts, gradient accumulation and other details
- Saving.
StateDictType.FULL_STATE_DICTgathers normal, original-shaped weights (userank0_only=Trueandoffload_to_cpu=Trueon GPUs to avoid running out of memory).SHARDED_STATE_DICTsaves each rank’s piece with the original names, and can be loaded with a different number of GPUs.LOCAL_STATE_DICTsaves the raw flat shards and only works with the same setup. [Certain] Big runs use sharded checkpoints (session 51). - Gradient accumulation (session 39). Without anything special, FSDP still reduce-scatters after every micro-batch and adds the sharded gradients. Inside
model.no_sync()it skips the communication but keeps unsharded gradients, which costs much more memory. [Certain] - Initialization of huge models. Build the model on the
metadevice (no memory), and let FSDP materialize and shard it one unit at a time withparam_init_fn, or load weights on rank 0 only and usesync_module_states=True. [Certain] - Activation checkpointing combines well with FSDP: apply it to the same blocks you wrap. FSDP shards the model states; checkpointing cuts the activations. The paper’s 175B run used both. [Certain]
Where this is used
| Setting | Typical FSDP1 setup | Why |
|---|---|---|
| Pretraining 1B–70B models on a cluster | FULL_SHARD or HYBRID_SHARD, per-block wrap, BF16 params, BACKWARD_PRE, activation checkpointing | Model states no longer limit the model size; hybrid keeps slow cross-node traffic small |
| Full fine-tuning of a 7B–13B model on one 8-GPU node | FULL_SHARD over 8 GPUs, often through Hugging Face Accelerate or Trainer [Likely] | 108 GB of states for 7B becomes 13.5 GB per GPU |
| Fine-tuning a big model on very few GPUs | FULL_SHARD + CPUOffload (slow) or QLoRA instead | Trades speed for memory when nothing else fits |
| Vision-language models (vision encoder + LLM) | Wrap the vision blocks and the LLM blocks as separate units; often freeze the vision tower with use_orig_params=True [Likely] | Units of very different sizes; frozen and trainable tensors must not share one FlatParameter without use_orig_params |
| Video encoders and video-language models (video classification, action recognition, Video-RAG retrieval encoders) | FSDP for the model states, plus activation checkpointing, because long frame sequences make activations huge [Likely] | FSDP alone does not shrink activations; the two tools solve different halves of the memory bill |
| Recommendation models (DHEN in the paper) | Dense part with FSDP, huge sparse embedding tables with a different sharding | The paper trained 550M dense parameters with FSDP next to 768B sparse ones [Certain] |
Common mistakes
- Creating the optimizer before wrapping. The optimizer then holds the original, full parameters, which FSDP has replaced. Always wrap first, then build the optimizer from
model.parameters(). - No auto-wrap policy. The whole model becomes one unit, so the forward pass gathers the full model: the peak memory is the full model again (Figure 4, “No wrapping”).
- Using
torch.nn.utils.clip_grad_norm_on sharded gradients. Each rank would compute the norm of its own shard only. Usemodel.clip_grad_norm_(), which combines the norms of all ranks. [Certain] - Calling a submodule’s forward directly (for example
model.lm_head(h)outside the wrapped forward) when it belongs to an FSDP unit whose weights are not gathered at that moment. You get an error about a size-0 or 1-D tensor. Only run code through the wrapped module’s forward, or usesummon_full_params. - Mixing frozen and trainable parameters in one unit without
use_orig_params=True. A FlatParameter has onerequires_grad, so this fails. Common in LoRA and VLM fine-tuning (session 47 makes it easy). - Saving with FULL_STATE_DICT on every rank. Each rank gathers the full model in GPU memory and may run out. Use
rank0_only=True, offload_to_cpu=True, or a sharded state dict. - Using
no_sync()for accumulation to “save communication” and then running out of memory: it keeps full, unsharded gradients for every unit. - Using the plain
GradScalerwith FP16. UseShardedGradScaler, or switch to BF16.
Interview questions
What does FSDP do in the forward and backward pass of one unit?basic
What is a FlatParameter, and why does FSDP1 use it?core
How much does FSDP communicate compared to DDP?core
FULL_SHARD vs SHARD_GRAD_OP: what is the trade-off?core
When would you use HYBRID_SHARD? What exactly does it communicate?advanced
Why wrap each transformer block as its own FSDP unit?core
Explain BACKWARD_PRE vs BACKWARD_POST.advanced
What is the rate limiter (limit_all_gathers) and why does it exist?advanced
How does FSDP’s MixedPrecision save memory and bandwidth compared to autocast?core
Why is FSDP1 awkward for LoRA, frozen layers and torch.compile, and what fixes it?advanced
A 7B model on 8 × 80 GB GPUs runs out of memory with FSDP. What do you check?core
Exercises
- A unit has parameters of 1,000, 300 and 7 elements, and \(F = 8\). How much padding does FSDP add, and how large is each shard?
- A 13B model is trained with FULL_SHARD on 32 GPUs. How many GB of model states (16 bytes per parameter) does each GPU hold? How many GB does each GPU send per step in BF16, by formula (46.4)?
- Same model and GPUs, but SHARD_GRAD_OP. What are the memory (states plus the kept BF16 weights) and the communication now?
- In Figure 3, why are there 7 all-gathers and not 8? What would you get with 10 non-root blocks?
- With 128 GPUs in nodes of 8, and a 7B (6.74B) model, compute the cross-node bytes per GPU per step in BF16 for HYBRID_SHARD by formula (46.8).
- In Figure 5, with transfer time 0.6 and BACKWARD_PRE, the exposed time is 1.8. Name the three transfers that cannot be hidden.
Answerscheck
Key takeaways
- FSDP is PyTorch’s ZeRO-3: each of \(F\) GPUs keeps \(1/F\) of the weights, gradients and optimizer states (46.1). A 7B model drops from 108 GB to 13.5 GB of states per GPU on 8 GPUs.
- FSDP1 groups parameters into units and stores each unit as a FlatParameter: flatten, concatenate, pad to a multiple of \(F\), split into equal shards (46.2).
- Forward: all-gather, compute, free. Backward: all-gather, gradients, reduce-scatter, then a local step on the shard (46.3). That is 1.5× the communication of DDP (46.4); the root unit is gathered once.
- Peak memory is the shards plus the largest gathered unit (46.5): wrap per transformer block, and watch the root.
- BACKWARD_PRE prefetching and the rate limiter hide most communication; the first gather and the last reduce-scatter always stay exposed (46.6).
- FULL_SHARD, SHARD_GRAD_OP, NO_SHARD, HYBRID_SHARD and _HYBRID_SHARD_ZERO2 trade memory against communication; hybrid keeps cross-node traffic to \(1/G\) of the gradients (46.7, 46.8).
- MixedPrecision gathers in BF16 from FP32 shards (46.9); CPU offload frees GPU memory at a large speed cost.
Key formulas
References for this session
- Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- PyTorch (docs). Getting Started with Fully Sharded Data Parallel (FSDP2)
- Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- PyTorch (docs). Getting Started with Fully Sharded Data Parallel (FSDP2)
- PyTorch (docs). FullyShardedDataParallel (FSDP1) API reference: ShardingStrategy, BackwardPrefetch, MixedPrecision, CPUOffload
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Micikevicius et al. (2017). Mixed Precision Training
- PyTorch (docs). Advanced Model Training with Fully Sharded Data Parallel (FSDP1 tutorial)