LLM & VLM courseBlog
46Track 3: Pretraining and scale · session 16 of 22

FSDP (version 1)

DDP keeps a full copy of the model, its gradients and its optimizer states on every GPU, so a 7B model needs about 108 GB per GPU before any activation. FSDP is PyTorch's built-in way to shard all of this across GPUs: each GPU keeps only a slice, and gathers the full weights of one block just before it needs them. This session shows how FSDP (version 1) does it: the FlatParameter, the all-gather and reduce-scatter schedule, wrapping, prefetching, sharding strategies, mixed precision and CPU offload.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 38, 43, 44 and 45 (memory math for training, collective communication, DDP, ZeRO)

What you will learn

  • How FSDP turns ZeRO stage 3 into a PyTorch wrapper: FSDP units, and the FlatParameter that flattens, pads and shards each unit
  • The schedule of one training step: all-gather, compute, free in the forward pass; all-gather, gradients, reduce-scatter in the backward pass
  • Why full sharding sends 1.5 times the bytes of DDP, and why the root unit is not gathered twice
  • How the wrapping policy sets peak memory and the number of collectives, with real numbers for Llama 2 7B
  • Backward prefetching (BACKWARD_PRE, BACKWARD_POST), forward prefetching and the rate limiter, and how they hide communication
  • The sharding strategies FULL_SHARD, SHARD_GRAD_OP, NO_SHARD, HYBRID_SHARD and _HYBRID_SHARD_ZERO2, and when to use each
  • MixedPrecision (param, reduce and buffer dtypes), CPU offload, state dicts, and the bugs people hit most often

Why this matters

In session 44 every GPU holds the full model, the full gradients and the full optimizer states. For mixed-precision AdamW that is 16 bytes per parameter. Llama 2 7B has 6.74 billion parameters, so DDP needs about 108 GB per GPU before a single activation is stored. An 80 GB GPU cannot do it, however many GPUs you buy.

Session 45 showed the cure: do not replicate, shard. Each of \(F\) GPUs keeps only \(1/F\) of the weights, gradients and optimizer states, and the GPUs pass weights to each other just in time. FSDP (Fully Sharded Data Parallel) is PyTorch’s own implementation of this idea (ZeRO stage 3). On 8 GPUs, the same 7B model needs about 13.5 GB of model states per GPU instead of 108 GB.

This session is about the first version of FSDP: the FullyShardedDataParallel wrapper described by Zhao et al. (2023). Its main data structure is the FlatParameter. The next session, session 47, replaces it with per-parameter sharding. You need to know version 1 well for three reasons. A lot of training code (Hugging Face Accelerate, older Lightning and many research repositories) still uses it [Likely]. Its ideas (units, all-gather and reduce-scatter, prefetching) are the same in FSDP2. And interviewers love to ask about it.

How well does it work? In the FSDP paper, a 175B GPT model trained on 128 to 512 A100 GPUs reached more than 173 TFLOPS per GPU with batch size 1 and 186 with batch size 2, about 55% and 60% of the A100’s 312 TFLOPS BF16 peak, and scaled almost linearly in TFLOPS from 128 to 512 GPUs. [Certain] For small models (611M and 2.28B parameters) it was as fast as DDP, and it trained an 11B T5 model where DDP ran out of memory. [Certain]

From ZeRO-3 to FSDP: the idea in one picture

Let \(\Psi\) be the number of parameters, and \(F\) the sharding factor: the number of GPUs (ranks) that share one copy of the model. With mixed-precision AdamW, the model states cost 16 bytes per parameter (session 38). DDP keeps all of them on every GPU. FSDP with full sharding keeps \(1/F\) of them:

\[ M_{\text{DDP}} = 16\,\Psi \ \text{bytes per GPU}, \qquad M_{\text{FSDP}} \approx \frac{16\,\Psi}{F} + M_{\text{unsharded}} \tag{46.1} \]
Read the symbols
\(\Psi\)
number of parameters of the model (6.74 billion for Llama 2 7B)
\(F\)
sharding factor: how many GPUs split one copy of the model. With full sharding, \(F\) is the number of GPUs.
16
bytes per parameter: FP32 weights (4), FP32 gradients (4), Adam \(m\) and \(v\) (4 + 4)
\(M_{\text{unsharded}}\)
the extra memory for the few layers whose full weights are gathered right now. FSDP keeps this small by gathering only one block at a time.

But a GPU cannot multiply by weights it does not have. So before a layer runs, the GPUs do an all-gather: every GPU sends its slice to everyone, and every GPU ends up with the full weights of that layer. After the layer runs, the full weights are thrown away again. In the backward pass, each GPU computes gradients for the full layer from its own data, and a reduce-scatter sums the gradients over GPUs and leaves each GPU with only the summed gradient of its own slice (session 43). Each GPU then runs the optimizer step on its own slice only.

Two engineering questions decide whether this is fast: in what pieces do we gather the weights, and when do we start each transfer so that it hides behind computation. The rest of the session answers both.

FSDP units and the FlatParameter

FSDP does not gather one tensor at a time. A transformer block has about ten weight tensors (attention projections, MLP matrices, norm weights), and one collective per tensor would be slow: every collective has a fixed launch cost, and small messages use the network badly. In the paper’s measurements, with the same total amount of data, all-gather time began to rise quickly once each all-gather moved fewer than about 33 million elements. [Certain] NCCL’s fastest all-gather also wants the same input size on every rank.

So FSDP groups modules into FSDP units. You choose them: usually one unit per transformer block, plus a root unit (the outermost wrapper) that takes all leftover parameters, such as the embedding and the output head. For each unit, FSDP builds one FlatParameter:

  1. Flatten every parameter of the unit into a 1-D vector.
  2. Concatenate them into one long 1-D vector of \(\psi\) elements.
  3. Pad with zeros on the right so the length is divisible by \(F\).
  4. Chunk it into \(F\) equal pieces. Rank \(r\) keeps piece \(r\) and frees the rest.
\[ \tilde\psi = F \left\lceil \frac{\psi}{F} \right\rceil, \qquad \text{shard size} = \frac{\tilde\psi}{F}, \qquad \text{padding} = \tilde\psi - \psi \le F - 1 \tag{46.2} \]
Read the symbols
\(\psi\)
number of elements in all parameters of one unit together
\(\tilde\psi\)
the padded length, the smallest multiple of \(F\) that is at least \(\psi\)
\(\lceil x \rceil\)
round up to the next whole number

The original parameters (for example weight of shape 3 × 4) become views into the FlatParameter: they share its memory. When the FlatParameter is unsharded, FSDP points each original parameter at its slice of the full vector, so normal PyTorch code runs unchanged. The gradient of the FlatParameter has the same layout. This layout is exactly what all-gather writes and reduce-scatter reads, so neither needs an extra copy. [Certain] Figure 1 builds a FlatParameter for small units.

Ranks \(F\):
Figure 1. Building a FlatParameter. Top: the original parameters, each in its own color. Middle: flattened and concatenated into one vector, with gray zero padding at the end. Bottom: the vector cut into \(F\) equal shards, one per rank. Click (or Tab to and press Enter on) a rank to see what it stores.
How to read Figure 1

Start with Linear(4, 3) and \(F = 16\): 12 weight numbers plus 3 bias numbers make 15, so FSDP adds 1 zero and each rank holds exactly one number. This is the example in the FSDP paper. Now pick \(F = 3\) for the two-layer unit: 26 numbers, 1 padding, 9 per rank. Notice that a shard does not respect parameter borders. A rank may hold half of a weight matrix and a piece of a bias. That is why, in FSDP1, “the part of weight on rank 2” is not a clean tensor, and why per-parameter tricks (freezing one tensor, a different learning rate for one tensor) are awkward. FSDP2 fixes exactly this.

Algorithm 1: build and shard the FlatParameter of one FSDP unit
  1. input: the parameters \(p_1, \dots, p_k\) of one unit, sharding factor \(F\), this rank’s index \(r\)
  2. \(\text{flat} \leftarrow \text{concat}(\text{flatten}(p_1), \dots, \text{flatten}(p_k))\) // length \(\psi\)
  3. append \(\tilde\psi - \psi\) zeros to flat // formula (46.2), at most \(F - 1\) zeros
  4. \(\text{shard}_r \leftarrow \text{flat}[\,r\tilde\psi/F : (r+1)\tilde\psi/F\,]\); free the rest of flat
  5. remember each \(p_i\)’s offset and shape, so it can later be a view into the unsharded flat vector
  6. the optimizer receives \(\text{shard}_r\) (one 1-D tensor) instead of \(p_1, \dots, p_k\)

One training step: all-gather, compute, free

With full sharding, every non-root unit goes through the same life in each training step:

  • Forward: all-gather the unit’s FlatParameter, run the unit’s forward, free the full weights (keep only the local shard).
  • Backward: all-gather the unit again (it was freed), compute the gradients for the full unit, reduce-scatter the gradients so each rank keeps the summed gradient of its own shard, free the full weights and full gradients.
  • Optimizer step: each rank updates only its own shard, with its own shard of the Adam states. No communication.

The root unit is special. It is gathered at the start of the forward pass and kept until the backward pass is done, because the backward pass begins right where the forward pass ended: freeing it and gathering it again at once would be pure waste. [Certain] This saves one all-gather per step.

Algorithm 2: one FSDP training step with FULL_SHARD, seen from rank \(r\)
  1. input: units \(U_0\) (root), \(U_1, \dots, U_N\); local shards; this rank’s micro-batch \(x_r\)
  2. all-gather \(U_0\) // root stays unsharded for the whole step
  3. for \(i = 1, \dots, N\): all-gather \(U_i\); run forward of \(U_i\); free \(U_i\)’s full weights // activations are kept
  4. finish the root’s forward (for example the output head) and compute the loss on \(x_r\)
  5. start backward; the root’s full gradient buffer is allocated when the head’s gradient is computed
  6. for \(i = N, \dots, 1\): all-gather \(U_i\); compute gradients of \(U_i\); reduce-scatter them (average over ranks); free full weights and full gradients
  7. finish the root’s backward; reduce-scatter the root’s gradients; free the root’s full weights
  8. optimizer step on the local shards only // sharded Adam states, no communication

Write \(\theta_i^{(r)}\) for the shard of unit \(i\) that rank \(r\) owns, and \(s = \tilde\psi_i / F\) for the shard size from (46.2). The three operations of one unit are then three short formulas:

\[ \begin{aligned} \text{all-gather:}\quad & \theta_i = \big[\, \theta_i^{(0)} \,\big|\, \theta_i^{(1)} \,\big|\, \cdots \,\big|\, \theta_i^{(F-1)} \,\big] \\ \text{reduce-scatter:}\quad & \hat g_i^{(r)} = \frac{1}{F} \sum_{q=0}^{F-1} g_i^{(q)}\big[\, r s : (r+1) s \,\big] \\ \text{local step:}\quad & \theta_i^{(r)} \leftarrow \theta_i^{(r)} - \eta\, \hat g_i^{(r)} \end{aligned} \tag{46.3} \]
Read the symbols
\(\theta_i\)
the full (unsharded, padded) FlatParameter of unit \(i\), only alive between its all-gather and its free
\(g_i^{(q)}\)
the full gradient of unit \(i\) that rank \(q\) computed from its own micro-batch; different on every rank
\([\,a : b\,]\)
the elements from position \(a\) up to (not including) \(b\): the slice that rank \(r\) owns
\(\hat g_i^{(r)}\)
the averaged gradient of rank \(r\)’s slice: exactly the slice of the full-batch gradient
\(\eta\)
the learning rate (plain SGD here; AdamW works the same way, with the shard of \(m\) and \(v\))

Figure 2 runs Algorithm 2 for real, in your browser, on a model small enough to show every number. Three units, each a Linear(2, 2) (4 weights and 2 biases, so \(\psi = 6\)); the first two use tanh. Each GPU gets one example \(\mathbf{x}_r\) with target \(\mathbf{y}_r\) and the loss \(L_r = \tfrac12 \lVert \mathbf{h}_3 - \mathbf{y}_r \rVert^2\). The algorithm box highlights the line that runs; the box next to it shows that line’s formula with the live numbers of the GPU you pick.

GPUs \(F\):
Formulas for:
own weight shardgathered weightszero paddingfull gradientgradient shard after reduce-scatter
  1. for \(i = 1, 2, 3\) do // forward
  2. all-gather \(\theta_i\) from all shards // (46.3)
  3. \(\mathbf{h}_i \leftarrow \phi(W_i \mathbf{h}_{i-1} + \mathbf{b}_i)\) on the local example
  4. free \(\theta_i\), keep \(\theta_i^{(r)}\) // SHARD_GRAD_OP: keep \(\theta_i\)
  5. \(L_r \leftarrow \tfrac12 \lVert \mathbf{h}_3 - \mathbf{y}_r \rVert^2\)
  6. for \(i = 3, 2, 1\) do // backward
  7. all-gather \(\theta_i\) again // SHARD_GRAD_OP: skipped
  8. \(g_i^{(r)} \leftarrow \partial L_r / \partial \theta_i\) // full gradient, local data
  9. reduce-scatter \(\to \hat g_i^{(r)}\); free \(\theta_i\) and \(g_i^{(r)}\) // (46.3)
  10. \(\theta_i^{(r)} \leftarrow \theta_i^{(r)} - \eta\, \hat g_i^{(r)}\) for every unit // no communication
Figure 2. One FSDP training step computed live on \(F\) simulated GPUs (rows) and three units (columns), with SGD and \(\eta = 0.5\). Each unit is a padded FlatParameter of \(\tilde\psi\) numbers; each GPU owns \(s = \tilde\psi/F\) of them. Bottom: how many numbers (weights and gradients, not activations) one GPU holds after each step of the algorithm; it is the same on every GPU. The faint dashed curve is the other strategy. At the end, the figure compares the result with plain SGD on one GPU with the whole batch.
How to read Figure 2

Press Step and follow the highlighted line. With \(F = 4\), \(\psi = 6\) is padded to \(\tilde\psi = 8\), so every GPU owns 2 numbers of each unit, and the last 2 numbers (owned by GPU 3) are padding. Each all-gather fills the row with the full 8 numbers on every GPU; the forward then gives a different \(\mathbf{h}\) on each GPU, because each has its own example (GPU 0: \(\mathbf{h}_1 = (0.336, 0.422)\), loss 0.509). In the backward pass each GPU computes a different full gradient (orange), and the reduce-scatter averages them and keeps only the GPU’s own slice (green). For U3 the averaged weight gradient starts \(0.072, -0.202\) (GPU 0’s slice). After the local step the mean loss over the 4 examples falls from 0.342 to 0.148, and the gathered new weights equal single-GPU SGD exactly. Watch the memory curve: with FULL_SHARD it rises by 8 for one unit and drops back in the forward pass, and the peak is 26 numbers while U1’s gradient is computed. Switch to SHARD_GRAD_OP: 3 fewer all-gathers (36 numbers sent per GPU instead of 54), but all three gathered units stay until the backward, and the peak grows to 38.

A toy with 6-number units makes the shards and the padding look large. In a real model, each unit has hundreds of millions of numbers, the padding is at most \(F - 1\) of them, and the AdamW states (\(m\), \(v\)) are sharded too. Figure 3 shows the same step at that scale. The model has a root and three blocks of 100 million parameters each, on 4 GPUs, with BF16 for the gathered weights and gradients.

local shard only (1/4)full weights gathered (BF16)full gradients (BF16)gradient shard after reduce-scatter
Figure 3. One FULL_SHARD training step as seen by GPU 0 of 4. Each column is one FSDP unit (100M parameters each). The bar under the units is GPU 0’s parameter-related memory: the fixed 1.6 GB of sharded model states plus whatever is gathered right now. Activations are not shown.
How to read Figure 3

Watch the memory bar. In the forward pass it rises by 0.2 GB for one block at a time and falls back. In the backward pass a block needs its weights and its full gradients, so it costs 0.4 GB while it runs. The root is the exception: its full weights stay from the first step to the last, and its full gradient buffer lives through the whole backward pass, because the root’s gradient is only complete when the embedding (the first layer) has its gradient. Count the collectives at the end: 7 all-gathers and 4 reduce-scatters, not 8 and 4, because the root is gathered once.

How much does it communicate?

Count elements sent per GPU, with the ring algorithms from session 43. An all-gather or a reduce-scatter of \(\Psi\) elements over \(F\) ranks sends \(\Psi (F-1)/F\) per GPU, and an all-reduce sends twice that. DDP does one all-reduce of the gradients. Full sharding does one all-gather in the forward pass, one in the backward pass, and one reduce-scatter:

\[ V_{\text{DDP}} = 2\,\Psi\,\frac{F-1}{F}, \qquad V_{\text{FULL\_SHARD}} = 3\,\Psi\,\frac{F-1}{F} = 1.5 \times V_{\text{DDP}} \tag{46.4} \]

This is the 1.5× from the FSDP paper and from ZeRO stage 3. [Certain] The root skips one all-gather, so the real number is slightly below 1.5×. In Figure 3: 11 collectives of 150 MB each (\(100\text{M} \times 3/4 \times 2\) bytes) make 1.65 GB, while formula (46.4) for 400M parameters gives 1.8 GB. With BF16 communication (next sections), each element is 2 bytes.

Peak memory and the wrapping policy

How you cut the model into units decides two things. Let the units have \(\psi_1, \dots, \psi_N\) elements, with \(\sum_i \psi_i = \Psi\). Every rank always keeps all its shards, and it has to hold the full weights of about one unit at a time. So the paper writes the parameter memory as [Certain]

\[ M_{\text{param}} = O\!\left( \sum_{i=1}^{N} \frac{\psi_i}{F} + \max_{i} \psi_i \right), \qquad \text{collectives per step} = O(N) \tag{46.5} \]

The first term is fixed (\(\Psi / F\)). Only the second term depends on the wrapping: smaller units give a smaller peak, but more and smaller collectives. That is FSDP’s memory–throughput trade-off, and you set it with the auto_wrap_policy:

  • transformer_auto_wrap_policy with transformer_layer_cls={LlamaDecoderLayer} (or your block class): one unit per transformer block. This is the standard choice for LLMs. Everything not inside a block (embedding, final norm, output head) falls into the root.
  • size_based_auto_wrap_policy: wrap any submodule with at least min_num_params parameters (default \(10^8\)). [Certain] Simple, but its cuts may not follow the block structure.
  • No policy: the whole model is one unit. Then all-gathering “one unit” means all-gathering the whole model, and FSDP saves memory only on gradients and optimizer states: the peak is the full model again.

Figure 4 applies the same simple memory model to Llama 2 7B (32 blocks of 202.4M parameters; embedding and output head of 131.1M each, not tied).

Backward prefetch:
sharded model states \(16\Psi/F\)root: full weights + full gradientslargest other unit(s), in flight
Figure 4. Peak parameter-related memory per GPU for Llama 2 7B with FULL_SHARD and BF16 gathering, for four ways to wrap the model. The model: sharded states \(16\Psi/F\), plus \(2\ \text{bytes} \times (\text{root weights}\) \(+\ \text{root gradients}\) \(+\ k \times \text{largest other unit})\), where \(k = 3\) for BACKWARD_PRE and \(k = 2\) otherwise. Activations, temporary buffers and fragmentation are not included.
How to read Figure 4

Start at 8 GPUs, per block: 13.48 GB sharded plus 2.26 GB in flight. Switch to “No wrapping”: the in-flight part jumps to 27 GB, because the whole model is gathered with its gradients. Now slide to 64 GPUs. The sharded part falls to 1.68 GB, but the in-flight part does not shrink at all: it depends on unit size, not on \(F\). At large scale, the wrapping is what decides your peak. Also notice the root: with per-block wrapping, the root (embedding + head, 262M) is bigger than a block (202M). Wrapping the embedding and the head as their own units drops the peak by about 1 GB [Likely]. The cost is the collective count shown in the readout.

Worked example: Llama 2 7B on 8 GPUs

Parameters: each block has \(4d^2 = 67{,}108{,}864\) attention weights, \(3 d\, d_{ff} = 135{,}266{,}304\) MLP weights and \(2d = 8{,}192\) norm weights: 202,383,360. The embedding and the output head have \(32{,}000 \times 4{,}096 = 131{,}072{,}000\) each. Total: \(32 \times 202{,}383{,}360 + 2 \times 131{,}072{,}000 + 4{,}096 = 6{,}738{,}415{,}616\).

DDP: \(16 \times 6.738 \times 10^9 = 107.8\) GB per GPU. FSDP, \(F = 8\): \(107.8 / 8 = 13.48\) GB of sharded states. Per-block wrapping with BACKWARD_PRE adds \(2 \times (2 \times 262.1\text{M} + 3 \times 202.4\text{M}) = 2.26\) GB, so about 15.7 GB, plus activations. That leaves most of an 80 GB GPU for activations.

Communication per step and per GPU, formula (46.4) in BF16: \(3 \times 6.738 \times 10^9 \times 7/8 \times 2\) bytes \(= 35.4\) GB, against 23.6 GB for DDP’s all-reduce.

Hiding the communication: prefetching and the rate limiter

An all-gather takes time. If the GPU waits for each one, the step gets slower. FSDP runs all collectives on a separate CUDA stream (a queue of GPU work), so they can run while the compute stream is busy. But all of FSDP’s all-gathers and reduce-scatters go through one NCCL stream, so they run one after another, in the order the CPU issued them. [Certain] The order is everything.

Forward pass. The CPU usually runs ahead of the GPU, so the all-gather for block \(i+1\) is issued while block \(i\) still computes, and it overlaps naturally. If the CPU is slow (many small kernels), forward_prefetch=True issues the next all-gather explicitly before the current block’s forward. It uses the order of the previous step, so it is only for models whose execution order does not change. [Certain]

Backward pass. Here the natural order is bad. After block \(i\) computes its gradients, FSDP issues its reduce-scatter. Only then does autograd reach block \(i-1\) and ask for its all-gather. On a single NCCL stream the all-gather waits for the reduce-scatter, and the compute waits for the all-gather: two transfers in a row, fully exposed. The backward_prefetch option changes the issue order: [Certain]

  • BACKWARD_PRE (the default): issue the all-gather of the next block before computing the current block’s gradients. Most overlap. At the peak, a rank holds the current weights, the next weights and the current gradients.
  • BACKWARD_POST: issue the next all-gather after the current gradients, but before the current reduce-scatter. Less overlap, less memory (the current weights are freed first).
  • None: no prefetching. Not recommended: it can be much slower.

FSDP knows which block comes “next” in the backward pass by recording the forward order and reversing it, freshly each step. [Certain] In the paper, backward prefetching made a 175B model about 18% faster on 128 to 512 GPUs. [Certain]

The rate limiter. If the CPU runs too far ahead, it issues many all-gathers at once. Their output buffers are allocated on the communication stream, and the PyTorch caching allocator cannot reuse those blocks for the compute stream until the GPU is done with them. Memory fills up, and the allocator falls back to slow cudaFree and cudaMalloc calls (“malloc retries”). limit_all_gathers=True (the default) blocks the CPU so that at most two units are gathered at a time: the one computing and the one prefetched. [Certain] In the paper it gave up to 5× speedups on T5-11B but cost about 5% on a DeepViT model, so check torch.cuda.memory_stats()["num_alloc_retries"] before you change it. [Certain]

A simple model of step time: if \(T_{\text{comp}}\) is the pure compute time, the step takes

\[ T_{\text{step}} = T_{\text{comp}} + T_{\text{exposed}}, \qquad T_{\text{exposed}} \ge \underbrace{t_{\text{AG}}}_{\text{first gather}} + \underbrace{t_{\text{RS}}}_{\text{last reduce-scatter}} \tag{46.6} \]

The first all-gather and the last reduce-scatter can never be hidden; everything in between can be, if prefetching is on and each transfer is shorter than the compute it overlaps. Figure 5 simulates it for 4 blocks and plays it like a clock. The algorithm box shows the issue order of the chosen mode; the lines of the operations that run at the current moment light up. The bottom lane counts how many full-size buffers (gathered weights \(W_i\) or full gradients \(G_i\), each the size of one block) the GPU holds at that moment.

forward compute (1 per block)backward compute (2 per block)all-gatherreduce-scatter
    Figure 5. A toy timeline of one step with 4 FSDP units, played as a clock. Forward compute takes 1 time unit per block, backward 2. All collectives share one NCCL stream (middle lane), in the order FSDP issues them. Pure compute is 12 time units; the readout shows how much communication is still exposed. Bottom lane: full-size buffers held by the GPU; \(W_i\) lives from the start of its all-gather to the end of the compute that uses it, \(G_i\) from the start of block \(i\)’s backward to the end of its reduce-scatter.
    How to read Figure 5

    Press Play and watch the red clock line. At the default (transfer 0.6), no prefetch gives 17.4 time units: every backward block waits for a reduce-scatter and then an all-gather. BACKWARD_POST gives 15.6: only the all-gather waits. BACKWARD_PRE gives 13.8: only the first all-gather of each pass and the final reduce-scatter are exposed. Now look at the bottom lane. Without prefetch the GPU never holds more than 2 buffers; with either prefetch mode it reaches 3 in the backward pass (for BACKWARD_PRE: \(W_i\), the prefetched \(W_{i-1}\) and \(G_i\)). That is the memory price of overlap. Set the transfer time to 0.3 and untick the rate limiter: all four forward all-gathers run back to back and the GPU holds 4 gathered blocks at once; with the limiter it is never more than 2 in the forward pass. Raise the transfer time above 1.0 and even BACKWARD_PRE cannot hide everything, because two transfers (a reduce-scatter and the next all-gather) must fit inside one block’s backward compute of 2. That is the regime of small batches and slow networks, where HYBRID_SHARD (next section) helps.

    Sharding strategies

    Full sharding is not always the best choice. The sharding_strategy argument picks how much is sharded, and over how many GPUs: [Certain]

    StrategyWhat is shardedCollectives per stepLike
    FULL_SHARD (default)Weights, gradients, optimizer states over all \(W\) GPUs. Weights are freed after forward and gathered again for backward.2 all-gathers + 1 reduce-scatter per unitZeRO-3
    SHARD_GRAD_OPGradients and optimizer states. Weights are sharded outside the step, but the gathered weights are kept from the forward pass until the end of the backward pass.1 all-gather + 1 reduce-scatter per unitZeRO-2 (roughly)
    NO_SHARDNothing. Gradients are all-reduced.1 all-reduceDDP
    HYBRID_SHARDFULL_SHARD inside each group of \(F\) GPUs (usually one node), replicated across groupsinside node: as FULL_SHARD; across nodes: 1 all-reduce of the gradient shardZeRO-3 + DDP
    _HYBRID_SHARD_ZERO2SHARD_GRAD_OP inside each node, replicated across nodesinside node: as SHARD_GRAD_OP; across nodes: 1 all-reduceZeRO-2 + DDP

    SHARD_GRAD_OP trades memory for speed: it saves the second all-gather (one third of the communication) but keeps the full BF16 weights in memory during the whole step, \(2\Psi\) bytes. For Llama 2 7B that is 13.5 GB extra per GPU.

    Hybrid sharding

    Inside a node, GPUs talk over NVLink at hundreds of GB/s; between nodes, the network is usually several times slower (session 42) [Likely]. Hybrid sharding uses this. With \(W\) GPUs in \(H = W/G\) nodes of \(G\) GPUs each, it shards over the \(G\) GPUs of one node (\(F = G\)) and keeps one full copy per node. The gradient sum is split into two steps, which is exact because addition can be grouped any way:

    \[ \sum_{r=1}^{W} \mathbf{g}_r = \sum_{i=1}^{H} \Big( \sum_{r \in S_i} \mathbf{g}_r \Big) \tag{46.7} \]
    Read the symbols
    \(\mathbf{g}_r\)
    the gradient computed on rank \(r\) from its own data
    \(S_i\)
    the sharding group \(i\): the \(G\) GPUs of node \(i\)
    \(H\)
    number of sharding groups (nodes)

    The inner sum is a reduce-scatter inside the node. The outer sum is an all-reduce among the GPUs that hold the same shard, one per node (the “replication group”). Only that all-reduce crosses nodes, and it moves only a \(1/G\) slice of the gradients. With a ring over the \(H\) nodes, the bytes that leave each GPU through the network per step are

    \[ V_{\text{cross-node}}^{\text{HYBRID}} = 2\,\frac{\Psi}{G}\,\frac{H-1}{H} \qquad \text{vs.} \qquad V^{\text{FULL\_SHARD}} = 3\,\Psi\,\frac{W-1}{W} \ \text{(over the whole cluster)} \tag{46.8} \]

    The FSDP paper gives the same picture: hybrid sharding cuts cross-host traffic by roughly a factor \(G\) compared to full replication or full sharding. [Certain] The price is memory: each GPU holds \(16\Psi/G\) bytes of states, not \(16\Psi/W\). Hybrid sharding is a good fit for “medium” models that do not fit with DDP but do fit when sharded over one node. Figure 6 compares all five strategies.

    GPUs per node \(G = 8\). Highlight:
    Figure 6. Per-GPU model-state memory (left bars) and per-step communication volume in BF16 (right bars) for the five strategies. For the hybrid ones, communication is split into traffic inside the node and traffic between nodes (darker). Memory counts 16 bytes per sharded parameter plus the kept BF16 weights for the ZeRO-2-like strategies; activations are not included. The dashed line is 80 GB.
    How to read Figure 6

    Defaults: a 7B model (6.74B) on 64 GPUs. FULL_SHARD needs only 1.68 GB of states but sends 39.8 GB per step, all of it over a group that spans 8 nodes. HYBRID_SHARD needs 13.5 GB and sends 35.4 GB, but only 2.95 GB of it leaves the node. NO_SHARD (DDP) needs about 108 GB: it does not fit. Push the model to 70B: now HYBRID_SHARD needs 140 GB per GPU and no longer fits, so you must shard across nodes, or use larger sharding groups, or add tensor parallelism (session 48).

    Mixed precision and CPU offload

    MixedPrecision

    FSDP has its own mixed-precision option, MixedPrecision, with three dtypes (all default to None, meaning “keep FP32”): [Certain]

    • param_dtype: the dtype of the gathered weights, used in forward and backward. The sharded weights stay in FP32 for the optimizer step, and checkpoints are saved in FP32.
    • reduce_dtype: the dtype of the gradient reduce-scatter (or all-reduce). If it is None but param_dtype is set, it follows param_dtype. Set it to FP32 if you want the gradient sum in full precision.
    • buffer_dtype: the dtype of buffers (for example rotary-embedding tables). FSDP does not shard buffers.

    The key trick: FSDP casts the local FP32 shard to BF16 before the all-gather. So the all-gather moves half the bytes, and the gathered copy costs half the memory. The FP32 copy of the full unit never exists. The paper writes the peak parameter memory as [Certain]

    \[ K_{\text{full}} \frac{\Psi}{F} + K_{\text{full}} \max_i \psi_i \quad \longrightarrow \quad K_{\text{full}} \frac{\Psi}{F} + K_{\text{low}} \max_i \psi_i \tag{46.9} \]
    Read the symbols
    \(K_{\text{full}}\)
    bytes per element in full precision (4 for FP32)
    \(K_{\text{low}}\)
    bytes per element in low precision (2 for BF16 or FP16)

    Example, Llama 2 7B with per-block wrapping on 8 GPUs, where the largest unit is the 262M-parameter root: \(4 \times 6.738\text{B}/8 + 4 \times 262\text{M} = 4.42\) GB becomes \(3.37 + 0.52 = 3.89\) GB. FSDP’s cast happens once per unit per gather, unlike torch.autocast, which casts at each operation. [Certain] With FP16, use ShardedGradScaler, not the normal GradScaler: the gradients are sharded, so the check for infinities must look at all ranks. [Certain] BF16 does not need a scaler (session 37).

    CPU offload

    CPUOffload(offload_params=True) keeps the sharded parameters on the CPU when they are not in use. It offloads the gradients too, so the optimizer step runs on the CPU. [Certain] Before each all-gather, the shard is copied to the GPU over PCIe; after each reduce-scatter, the gradient shard is copied back. This frees nearly all of the \(16\Psi/F\) on the GPU (for Llama 2 7B on 8 GPUs: 13.5 GB per GPU, so 108 GB of CPU RAM per node), but PCIe is much slower than GPU memory, and an Adam step on CPU is slow. Use it when nothing else fits (for example, fine-tuning a big model on few GPUs), not for large pretraining runs. [Likely]

    Using FSDP1 in PyTorch

    First, the mechanics without any distributed library: a FlatParameter for one unit on 4 simulated ranks, all-gather and reduce-scatter written by hand, and a check that one sharded SGD step equals one normal SGD step on the full batch.

    import torch
    torch.manual_seed(0)
    F = 4                                                    # sharding factor = number of ranks
    
    # --- 1) build a FlatParameter for one unit: flatten, concat, pad, chunk (Algorithm 1) ---
    unit = torch.nn.Linear(4, 3)                             # weight 3x4 = 12, bias 3 -> 15 numbers
    params = [unit.weight, unit.bias]
    numels = [p.numel() for p in params]
    flat = torch.cat([p.detach().reshape(-1) for p in params])
    pad = (-flat.numel()) % F                                # formula (46.2): at most F - 1
    flat = torch.cat([flat, flat.new_zeros(pad)])
    shards = list(flat.chunk(F))                             # rank r keeps shards[r] only
    print("numel", sum(numels), "pad", pad, "shard size", shards[0].numel())   # 15 1 4
    
    def all_gather(shards):                                  # every rank gets the full flat tensor
        return torch.cat(shards)
    def reduce_scatter(full_grads):                          # sum over ranks, rank r keeps chunk r
        return list(torch.stack(full_grads).sum(0).chunk(F))
    def views(full):                                         # original shapes are views into the flat tensor
        return full[:numels[0]].view(3, 4), full[numels[0]:numels[0] + numels[1]]
    
    # --- 2) one FSDP step with SGD (Algorithm 2), data split over F ranks ---
    X, Y = torch.randn(8, 4), torch.randn(8, 3)
    lr, grads = 0.1, []
    for r in range(F):
        full = all_gather(shards).requires_grad_(True)       # unshard
        w, b = views(full)
        x, y = X.chunk(F)[r], Y.chunk(F)[r]
        loss = ((x @ w.T + b - y) ** 2).mean() / F           # /F: average over ranks
        loss.backward()                                      # gradient of the whole flat tensor
        grads.append(full.grad)                              # (then the full weights are freed)
    g_shards = reduce_scatter(grads)                         # each rank: summed gradient of its own chunk
    shards = [s - lr * g for s, g in zip(shards, g_shards)]  # local optimizer step on the shard
    
    # --- 3) reference: plain single-device SGD on the whole batch ---
    ref = torch.nn.Linear(4, 3); ref.load_state_dict(unit.state_dict())
    ((ref(X) - Y) ** 2).mean().backward()
    with torch.no_grad():
        for p in ref.parameters(): p -= lr * p.grad
    w, b = views(all_gather(shards))
    print("max diff:", max((w - ref.weight).abs().max().item(), (b - ref.bias).abs().max().item()))  # about 3e-8
    print("padding stays zero:", all_gather(shards)[-pad:].abs().max().item())                     # 0.0

    Now the real thing. This script runs PyTorch’s FSDP1 on 4 CPU processes with the gloo backend, so it works on a laptop. It wraps each block as a unit, trains 3 AdamW steps with each rank on a quarter of the batch, and compares the gathered weights with single-process AdamW on the full batch. We ran it with PyTorch 2.8: it prints 4 units, local shard sizes [53, 540, 540, 540] (the root has \(144 + 68 = 212\) parameters, each block 2,160), matching keys, and a maximum difference of about \(10^{-7}\).

    import os, functools, torch, torch.nn as nn
    import torch.distributed as dist, torch.multiprocessing as mp
    from torch.distributed.fsdp import (FullyShardedDataParallel as FSDP, ShardingStrategy,
        StateDictType, FullStateDictConfig)
    from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy
    
    class Block(nn.Module):                       # one "transformer block" (an MLP block here)
        def __init__(s, d):
            super().__init__(); s.norm = nn.LayerNorm(d); s.up = nn.Linear(d, 4 * d); s.down = nn.Linear(4 * d, d)
        def forward(s, x): return x + s.down(torch.relu(s.up(s.norm(x))))
    
    def make_model():
        torch.manual_seed(0)                      # same init on every rank and for the reference
        return nn.Sequential(nn.Linear(8, 16), Block(16), Block(16), Block(16), nn.Linear(16, 4))
    
    def worker(rank, W):
        os.environ.update(MASTER_ADDR="127.0.0.1", MASTER_PORT="29541")
        dist.init_process_group("gloo", rank=rank, world_size=W)     # CPU backend; use "nccl" on GPUs
        policy = functools.partial(transformer_auto_wrap_policy, transformer_layer_cls={Block})
        model = FSDP(make_model(), auto_wrap_policy=policy, device_id=torch.device("cpu"),
                     sharding_strategy=ShardingStrategy.FULL_SHARD)
        units = [m for m in model.modules() if isinstance(m, FSDP)]
        if rank == 0:
            print("FSDP units:", len(units))
            print("local shard numels:", [u._flat_param.numel() for u in units])  # private attribute, for teaching only
        opt = torch.optim.AdamW(model.parameters(), lr=1e-2)        # created AFTER wrapping: sees only shards
        ref = make_model(); ropt = torch.optim.AdamW(ref.parameters(), lr=1e-2)
        torch.manual_seed(1); X, Y = torch.randn(16, 8), torch.randn(16, 4)
        for step in range(3):
            x, y = X.chunk(W)[rank], Y.chunk(W)[rank]                 # each rank: its own quarter
            loss = nn.functional.mse_loss(model(x), y)
            loss.backward(); opt.step(); opt.zero_grad()              # gradients are averaged over ranks
            rl = nn.functional.mse_loss(ref(X), Y); rl.backward(); ropt.step(); ropt.zero_grad()
        cfg = FullStateDictConfig(rank0_only=True)
        with FSDP.state_dict_type(model, StateDictType.FULL_STATE_DICT, cfg):
            sd = model.state_dict()                                   # gathers full, original-shaped weights on rank 0
        if rank == 0:
            diff = max((sd[k] - v).abs().max().item() for k, v in ref.state_dict().items())
            print("keys match:", sorted(sd) == sorted(ref.state_dict()), " max diff vs 1-GPU AdamW:", f"{diff:.1e}")
        dist.destroy_process_group()
    
    if __name__ == "__main__":
        mp.spawn(worker, args=(4,), nprocs=4)     # 4 ranks = 4 "GPUs"

    Change FULL_SHARD to SHARD_GRAD_OP and the result is the same. With NO_SHARD the local sizes become [212, 2160, 2160, 2160]: nothing is sharded. With MixedPrecision(param_dtype=torch.bfloat16, reduce_dtype=torch.float32) (and the loss computed on model(x).float()), the difference grows to about \(2 \times 10^{-2}\) after 3 steps: that is BF16 rounding in forward and backward, not a bug. On a GPU cluster, a typical FSDP1 setup for a Llama-style model looks like this (launched with torchrun --nproc_per_node=8; not run here, it needs GPUs):

    from torch.distributed.fsdp import (FullyShardedDataParallel as FSDP, ShardingStrategy,
        MixedPrecision, BackwardPrefetch, CPUOffload)
    from transformers.models.llama.modeling_llama import LlamaDecoderLayer
    
    torch.cuda.set_device(local_rank)
    model = FSDP(model,
        auto_wrap_policy=functools.partial(transformer_auto_wrap_policy,
                                           transformer_layer_cls={LlamaDecoderLayer}),
        sharding_strategy=ShardingStrategy.FULL_SHARD,          # or HYBRID_SHARD across nodes
        mixed_precision=MixedPrecision(param_dtype=torch.bfloat16,
                                       reduce_dtype=torch.float32, buffer_dtype=torch.bfloat16),
        backward_prefetch=BackwardPrefetch.BACKWARD_PRE,        # the default
        limit_all_gathers=True,                                 # the rate limiter (default)
        cpu_offload=CPUOffload(offload_params=False),           # True only if nothing else fits
        device_id=torch.cuda.current_device(),                  # shard on GPU, fast init
        sync_module_states=True,                                # broadcast rank 0's weights (e.g. loaded checkpoint)
        use_orig_params=True)                                   # needed for torch.compile and per-parameter groups
    optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)  # after wrapping!
    ...
    model.clip_grad_norm_(1.0)                                  # FSDP's own: computes the norm over all shards

    State dicts, gradient accumulation and other details

    • Saving. StateDictType.FULL_STATE_DICT gathers normal, original-shaped weights (use rank0_only=True and offload_to_cpu=True on GPUs to avoid running out of memory). SHARDED_STATE_DICT saves each rank’s piece with the original names, and can be loaded with a different number of GPUs. LOCAL_STATE_DICT saves the raw flat shards and only works with the same setup. [Certain] Big runs use sharded checkpoints (session 51).
    • Gradient accumulation (session 39). Without anything special, FSDP still reduce-scatters after every micro-batch and adds the sharded gradients. Inside model.no_sync() it skips the communication but keeps unsharded gradients, which costs much more memory. [Certain]
    • Initialization of huge models. Build the model on the meta device (no memory), and let FSDP materialize and shard it one unit at a time with param_init_fn, or load weights on rank 0 only and use sync_module_states=True. [Certain]
    • Activation checkpointing combines well with FSDP: apply it to the same blocks you wrap. FSDP shards the model states; checkpointing cuts the activations. The paper’s 175B run used both. [Certain]

    Where this is used

    SettingTypical FSDP1 setupWhy
    Pretraining 1B–70B models on a clusterFULL_SHARD or HYBRID_SHARD, per-block wrap, BF16 params, BACKWARD_PRE, activation checkpointingModel states no longer limit the model size; hybrid keeps slow cross-node traffic small
    Full fine-tuning of a 7B–13B model on one 8-GPU nodeFULL_SHARD over 8 GPUs, often through Hugging Face Accelerate or Trainer [Likely]108 GB of states for 7B becomes 13.5 GB per GPU
    Fine-tuning a big model on very few GPUsFULL_SHARD + CPUOffload (slow) or QLoRA insteadTrades speed for memory when nothing else fits
    Vision-language models (vision encoder + LLM)Wrap the vision blocks and the LLM blocks as separate units; often freeze the vision tower with use_orig_params=True [Likely]Units of very different sizes; frozen and trainable tensors must not share one FlatParameter without use_orig_params
    Video encoders and video-language models (video classification, action recognition, Video-RAG retrieval encoders)FSDP for the model states, plus activation checkpointing, because long frame sequences make activations huge [Likely]FSDP alone does not shrink activations; the two tools solve different halves of the memory bill
    Recommendation models (DHEN in the paper)Dense part with FSDP, huge sparse embedding tables with a different shardingThe paper trained 550M dense parameters with FSDP next to 768B sparse ones [Certain]

    Common mistakes

    • Creating the optimizer before wrapping. The optimizer then holds the original, full parameters, which FSDP has replaced. Always wrap first, then build the optimizer from model.parameters().
    • No auto-wrap policy. The whole model becomes one unit, so the forward pass gathers the full model: the peak memory is the full model again (Figure 4, “No wrapping”).
    • Using torch.nn.utils.clip_grad_norm_ on sharded gradients. Each rank would compute the norm of its own shard only. Use model.clip_grad_norm_(), which combines the norms of all ranks. [Certain]
    • Calling a submodule’s forward directly (for example model.lm_head(h) outside the wrapped forward) when it belongs to an FSDP unit whose weights are not gathered at that moment. You get an error about a size-0 or 1-D tensor. Only run code through the wrapped module’s forward, or use summon_full_params.
    • Mixing frozen and trainable parameters in one unit without use_orig_params=True. A FlatParameter has one requires_grad, so this fails. Common in LoRA and VLM fine-tuning (session 47 makes it easy).
    • Saving with FULL_STATE_DICT on every rank. Each rank gathers the full model in GPU memory and may run out. Use rank0_only=True, offload_to_cpu=True, or a sharded state dict.
    • Using no_sync() for accumulation to “save communication” and then running out of memory: it keeps full, unsharded gradients for every unit.
    • Using the plain GradScaler with FP16. Use ShardedGradScaler, or switch to BF16.

    Interview questions

    What does FSDP do in the forward and backward pass of one unit?basic
    Forward: all-gather the unit’s sharded FlatParameter so every rank has the full weights, run the unit’s forward, then free the full weights and keep only the local shard. Backward: all-gather the weights again, compute the unit’s full gradients from the local micro-batch, reduce-scatter the gradients so each rank keeps the averaged gradient of its own shard, then free the full weights and gradients. The optimizer updates only the local shard. The root unit is kept gathered from the start of the forward to the end of the backward.
    What is a FlatParameter, and why does FSDP1 use it?core
    For each FSDP unit, FSDP flattens all its parameters into 1-D, concatenates them into one vector, pads it with zeros to a multiple of the sharding factor \(F\) (at most \(F - 1\) zeros), and splits it into \(F\) equal shards. Reasons: one big collective per unit instead of one per tensor (less launch overhead, better bandwidth use; the paper saw all-gather time rise quickly below about 33M elements per call), equal sizes on all ranks (what NCCL’s fastest all-gather and reduce-scatter want), and a layout that collectives can read and write without extra copies. The original parameters become views into it.
    How much does FSDP communicate compared to DDP?core
    Per GPU and step, with ring algorithms: DDP does one all-reduce of the gradients, \(2\Psi(F-1)/F\) elements. FULL_SHARD does two all-gathers (forward and backward) and one reduce-scatter, \(3\Psi(F-1)/F\): 1.5× DDP. The root unit skips its second all-gather, so it is a bit less. SHARD_GRAD_OP does one all-gather and one reduce-scatter, the same volume as DDP. With BF16 communication each element is 2 bytes.
    FULL_SHARD vs SHARD_GRAD_OP: what is the trade-off?core
    FULL_SHARD frees the gathered weights after each unit’s forward and gathers them again for the backward: lowest memory, 1.5× DDP communication. SHARD_GRAD_OP keeps the gathered weights from the forward until the end of the backward, so it saves the second all-gather (same volume as DDP) but holds the full low-precision weights during the step, \(2\Psi\) bytes in BF16 (13.5 GB for a 7B model). Use SHARD_GRAD_OP when the model fits that way and communication is the bottleneck.
    When would you use HYBRID_SHARD? What exactly does it communicate?advanced
    On multi-node clusters where the network between nodes is much slower than NVLink inside a node, for models that fit when sharded over one node. It does FULL_SHARD inside each node (all-gathers and reduce-scatters over NVLink) and keeps one copy per node. After the in-node reduce-scatter, GPUs holding the same shard in different nodes all-reduce it. Only that all-reduce crosses nodes, and it moves only \(1/G\) of the gradients: \(2(\Psi/G)(H-1)/H\) per GPU. The sum is exact because addition can be regrouped (formula 46.7). The cost is memory: \(16\Psi/G\) per GPU instead of \(16\Psi/W\).
    Why wrap each transformer block as its own FSDP unit?core
    Peak parameter memory is about \(\Psi/F + \max_i \psi_i\): the shards plus the largest unit that is gathered at once. With one unit for the whole model, the second term is the whole model, so FSDP would gather everything. One unit per block keeps it at one or two blocks, and blocks are natural points for prefetching and activation checkpointing. Smaller units (attention and MLP apart) lower the peak a little more, but double the number of collectives, which hurts throughput. Watch the root: embedding plus output head can be larger than a block.
    Explain BACKWARD_PRE vs BACKWARD_POST.advanced
    All FSDP collectives use one NCCL stream, so their issue order matters. Without prefetch, after a block’s gradients FSDP issues its reduce-scatter, then the next block’s all-gather, and the next compute waits for both. BACKWARD_PRE issues the next all-gather before the current gradient computation: maximum overlap, but at the peak it holds current weights, next weights and current gradients. BACKWARD_POST issues it after the current gradient computation but before the reduce-scatter: the reduce-scatter overlaps the next compute; less overlap, less memory. FSDP finds “next” by reversing the recorded forward order each step. The paper measured about 18% speedup from backward prefetching on a 175B model.
    What is the rate limiter (limit_all_gathers) and why does it exist?advanced
    The CPU can issue all-gathers far ahead of the GPU. Their output buffers are allocated on the communication stream, and the caching allocator cannot reuse those blocks for compute-stream allocations until they are free, so memory is over-reserved and the allocator must do slow cudaFree/cudaMalloc retries. The rate limiter blocks the CPU so at most two units are gathered at a time (current and prefetched), the minimum for overlap. It gave up to 5× speedup on T5-11B in the paper, but about 5% slowdown on DeepViT, so check num_alloc_retries in torch.cuda.memory_stats() before tuning it.
    How does FSDP’s MixedPrecision save memory and bandwidth compared to autocast?core
    The sharded master weights stay FP32. Before each all-gather, FSDP casts the local shard to param_dtype (for example BF16), so the all-gather moves half the bytes and the gathered full unit costs 2 instead of 4 bytes per element; the FP32 full unit never exists. Peak parameter memory goes from \(K_\text{full}\Psi/F + K_\text{full}\max\psi_i\) to \(K_\text{full}\Psi/F + K_\text{low}\max\psi_i\). Autocast instead keeps FP32 weights and casts per operation. reduce_dtype sets the dtype of the gradient reduction (often FP32 for accuracy); buffer_dtype sets the buffers’ dtype. With FP16 you need ShardedGradScaler.
    Why is FSDP1 awkward for LoRA, frozen layers and torch.compile, and what fixes it?advanced
    A FlatParameter merges many tensors into one 1-D vector with one dtype and one requires_grad, and shards cut across tensor borders. So frozen and trainable tensors cannot share a unit, per-parameter optimizer groups do not map to shards, and the state dict needs special handling. use_orig_params=True exposes the original parameters as views (some may be empty on some ranks), which helps with freezing, parameter groups and torch.compile, but adds complexity. FSDP2 shards each parameter separately as a DTensor on dim 0, which solves these problems cleanly.
    A 7B model on 8 × 80 GB GPUs runs out of memory with FSDP. What do you check?core
    Model states should be only about 13.5 GB per GPU (\(16 \times 6.74\text{B}/8\)), so look elsewhere. (1) Is there an auto-wrap policy? Without one the whole model is gathered. (2) Is the root unit too big (embedding and head)? Wrap them separately. (3) Activations: enable activation checkpointing per block, reduce micro-batch or sequence length. (4) Was the optimizer created before wrapping? (5) Is no_sync() keeping unsharded gradients? (6) Saving a FULL_STATE_DICT without rank0_only and offload_to_cpu. (7) Fragmentation: check reserved versus allocated memory and num_alloc_retries.

    Exercises

    1. A unit has parameters of 1,000, 300 and 7 elements, and \(F = 8\). How much padding does FSDP add, and how large is each shard?
    2. A 13B model is trained with FULL_SHARD on 32 GPUs. How many GB of model states (16 bytes per parameter) does each GPU hold? How many GB does each GPU send per step in BF16, by formula (46.4)?
    3. Same model and GPUs, but SHARD_GRAD_OP. What are the memory (states plus the kept BF16 weights) and the communication now?
    4. In Figure 3, why are there 7 all-gathers and not 8? What would you get with 10 non-root blocks?
    5. With 128 GPUs in nodes of 8, and a 7B (6.74B) model, compute the cross-node bytes per GPU per step in BF16 for HYBRID_SHARD by formula (46.8).
    6. In Figure 5, with transfer time 0.6 and BACKWARD_PRE, the exposed time is 1.8. Name the three transfers that cannot be hidden.
    Answerscheck
    1. \(\psi = 1{,}307\); the next multiple of 8 is 1,312, so 5 zeros of padding and shards of 164 elements. 2. \(16 \times 13 \times 10^9 / 32 = 6.5\) GB. Communication: \(3 \times 13 \times 10^9 \times 31/32 \times 2\) bytes \(= 75.6\) GB per step. 3. Memory: \(6.5 + 2 \times 13 = 32.5\) GB. Communication: \(2 \times 13 \times 10^9 \times 31/32 \times 2 = 50.4\) GB. 4. The root unit is gathered once at the start of the forward and kept until the end of the backward, so it does not need a second all-gather. With 10 blocks: \(1 + 2 \times 10 = 21\) all-gathers and 11 reduce-scatters. 5. \(H = 16\): \(2 \times (6.738 \times 10^9 / 8) \times 15/16 \times 2\) bytes \(= 3.16\) GB. 6. The all-gather of the first block in the forward pass, the all-gather of the last block at the start of the backward pass (nothing is computing at that moment in this toy model), and the reduce-scatter of the first block at the very end, which the optimizer step must wait for.

    Key takeaways

    • FSDP is PyTorch’s ZeRO-3: each of \(F\) GPUs keeps \(1/F\) of the weights, gradients and optimizer states (46.1). A 7B model drops from 108 GB to 13.5 GB of states per GPU on 8 GPUs.
    • FSDP1 groups parameters into units and stores each unit as a FlatParameter: flatten, concatenate, pad to a multiple of \(F\), split into equal shards (46.2).
    • Forward: all-gather, compute, free. Backward: all-gather, gradients, reduce-scatter, then a local step on the shard (46.3). That is 1.5× the communication of DDP (46.4); the root unit is gathered once.
    • Peak memory is the shards plus the largest gathered unit (46.5): wrap per transformer block, and watch the root.
    • BACKWARD_PRE prefetching and the rate limiter hide most communication; the first gather and the last reduce-scatter always stay exposed (46.6).
    • FULL_SHARD, SHARD_GRAD_OP, NO_SHARD, HYBRID_SHARD and _HYBRID_SHARD_ZERO2 trade memory against communication; hybrid keeps cross-node traffic to \(1/G\) of the gradients (46.7, 46.8).
    • MixedPrecision gathers in BF16 from FP32 shards (46.9); CPU offload frees GPU memory at a large speed cost.

    Key formulas

    \[ M_{\text{FSDP}} \approx \frac{16\Psi}{F} + K_{\text{low}} \max_i \psi_i, \quad \tilde\psi = F\left\lceil \frac{\psi}{F} \right\rceil, \quad V_{\text{FULL\_SHARD}} = 3\Psi\frac{F-1}{F}, \quad V^{\text{HYBRID}}_{\text{cross}} = 2\frac{\Psi}{G}\frac{H-1}{H} \]

    References for this session

    1. Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
    2. PyTorch (docs). Getting Started with Fully Sharded Data Parallel (FSDP2)
    3. Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
    4. PyTorch (docs). Getting Started with Fully Sharded Data Parallel (FSDP2)
    5. PyTorch (docs). FullyShardedDataParallel (FSDP1) API reference: ShardingStrategy, BackwardPrefetch, MixedPrecision, CPUOffload
    6. Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
    7. Li et al. (2020). PyTorch Distributed: Experiences on Accelerating Data Parallel Training
    8. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
    9. Micikevicius et al. (2017). Mixed Precision Training
    10. PyTorch (docs). Advanced Model Training with Fully Sharded Data Parallel (FSDP1 tutorial)