FSDP2: per-parameter sharding with DTensor
FSDP2 keeps the FSDP idea (shard everything, gather one block at a time) but drops the FlatParameter. Each parameter becomes a DTensor that is cut along its first dimension, so every rank holds a few clean rows of every tensor. This session shows the fully_shard API, why per-parameter sharding makes state dicts, freezing and LoRA simple, the knobs that set memory and speed (MixedPrecisionPolicy, reshard_after_forward, prefetching), and how to move code from FSDP1 to FSDP2 without the usual bugs.
What you will learn
- What a DTensor is: a local tensor plus a device mesh plus placements such as Shard(0), Replicate() and Partial()
- How fully_shard cuts each parameter on dim 0, how uneven sizes are padded, and how the copy-in, all-gather, copy-out path works
- Why per-parameter sharding gives clean state dicts, easy freezing, LoRA in one group and float8 or NF4 all-gathers
- How MixedPrecisionPolicy, reshard_after_forward (True, False or an integer) and implicit or explicit prefetching change memory and communication, with numbers for Llama 2 7B
- How FSDP2 composes with HSDP (2-D mesh), tensor parallelism and torch.compile
- How to migrate an FSDP1 script to FSDP2, argument by argument, and the bugs people hit most often
Why this matters
In session 46 every FSDP unit became one FlatParameter: all tensors of the unit were flattened, glued into one long vector, padded, and cut into equal pieces. That made the collectives fast, but it also made a mess. A rank held “half of a weight matrix and a piece of a bias”. Freezing one tensor, giving one tensor its own learning rate, saving a clean checkpoint, or gathering one tensor in float8 all became special cases.
FSDP2 keeps the FSDP algorithm (shard the weights, gradients and optimizer states; all-gather one block before it runs; reduce-scatter its gradients after) and changes only the data structure. Each parameter stays a separate tensor. It becomes a DTensor (a “distributed tensor”) that is cut along its first dimension, dim 0. Rank \(r\) holds a few whole rows of every tensor. The entry point is one function, fully_shard. [Certain]
This is now the main FSDP in PyTorch. The official FSDP tutorial teaches FSDP2 and says that FSDP1 is deprecated. [Certain] torchtitan, PyTorch’s reference code for large-scale LLM pretraining, uses FSDP2 as its default data parallelism. [Certain] In the torchtitan paper, FSDP2 used on average 7% less GPU memory than FSDP1 on Llama 2 7B and was on average 1.5% faster, that is, about the same speed. [Certain] So the change is not about speed. It is about simpler code, deterministic memory and composition with other parallelisms (next session: session 48).
DTensor in one page
A DTensor is a normal PyTorch tensor plus two pieces of information that say how it is spread over GPUs: [Certain]
- A device mesh: the list of ranks (GPUs) that share the tensor, arranged as a 1-D or 2-D grid. For plain FSDP over 8 GPUs it is the 1-D mesh
[0, 1, …, 7]. - One placement per mesh dimension.
Shard(0)means “this mesh dimension cuts the tensor along dim 0”.Replicate()means “every rank on this mesh dimension has a full copy”.Partial()means “every rank has a partial value, and the true value is the sum over ranks” (gradients before the reduction look like this).
Each rank stores only its local tensor, which you get with p.to_local(). But p.shape reports the global shape, and math on DTensors gives the same result as on the full tensor: PyTorch adds the needed communication for you. p.full_tensor() all-gathers the full tensor; distribute_tensor(t, mesh, [Shard(0)]) cuts a full tensor into a DTensor. [Certain]
FSDP2 uses Shard(0) with chunks made exactly like torch.chunk. For a parameter with \(n\) rows (its dim-0 size) on \(F\) ranks, every chunk has \(c\) rows, and the last ranks get what is left:
- Read the symbols
- \(n\)
- the size of dim 0 of the parameter (the number of rows of a weight matrix, the length of a bias)
- \(F\)
- sharding factor: the number of ranks that share one copy of the model
- \(c\)
- rows per chunk, rounded up
- \(n_r\)
- the rows that rank \(r\) really holds; it can be smaller than \(c\), and even 0, for the last ranks
For real LLM weights, \(n\) is usually a multiple of \(F\), so every rank holds the same number of rows. Llama 2 7B on 64 GPUs: the MLP up-projection has 11,008 rows, so 172 per rank; the attention projections have 4,096 rows, so 64 per rank; nothing is left over. An odd vocabulary is different: GPT-2’s 50,257-row embedding on 64 ranks gives \(c = 786\), so 63 ranks hold 786 rows and the last one holds 739. To make the all-gather use equal sizes, FSDP2 pads the short chunks with zero rows up to \(c\) rows, here 47 rows. We checked these shapes on PyTorch 2.8: a 3-row weight on 4 ranks is stored as shapes 1, 1, 1 and 0 rows, and the all-gather input of rank 3 is padded to 1 row. [Certain]
All the parameters of one fully_shard call form a parameter group that is all-gathered with one collective. Per rank, the send buffer of the group and the total padding are
- Read the symbols
- \(i\)
- runs over the parameters of the group
- \(c_i\)
- rows per chunk of parameter \(i\), from (47.1)
- \(m_i\)
- elements per row of parameter \(i\) (for a weight of shape \(n \times d\), \(m = d\); for a bias, \(m = 1\))
- \(S\)
- the number of elements each rank sends in the group’s all-gather
Compare with FSDP1, where padding is at most \(F - 1\) elements per unit. FSDP2 can pad more, but only on tiny or oddly shaped tensors; for a transformer block on a power-of-two number of GPUs it is usually zero. Figure 1 puts the two layouts side by side.
- for each parameter \(i\) of the group (\(n_i\) rows, \(m_i\) elements per row):
- \(c_i = \lceil n_i / F \rceil\) // rows per chunk, (47.1)
- for \(r = 0, \dots, F-1\): rank \(r\) keeps rows \([r c_i,\ \min\{(r+1)c_i, n_i\})\); the all-gather pads it to \(c_i\) rows
- \(S \mathrel{+}= c_i\, m_i\) // this rank’s send buffer, (47.2)
- padding \(= F S - \sum_i n_i m_i\)
How to read Figure 1
Start with Linear(4, 3) and \(F = 4\), rank 1. In FSDP1 the rank holds flat elements 4 to 7: the second row of the weight. In FSDP2 it holds row 1 of the weight and element 1 of the bias: a clean \(1 \times 4\) tensor and a clean 1-element tensor. Now pick rank 3: in FSDP2 it holds nothing at all (shapes \(0 \times 4\) and 0), because 3 rows cannot fill 4 ranks. The readout shows the price: FSDP2 sends 5 elements per rank (20 in total, 5 of them padding) where FSDP1 sends 4 (1 padding). Switch to the two-layer unit with \(F = 3\): FSDP1 pads 1 element, FSDP2 pads 13. This overhead disappears when dim 0 is divisible by \(F\), as in real LLM layers. Press Reset and then Step through Linear(4, 3) with \(F = 4\): the weight gets \(c = \lceil 3/4 \rceil = 1\), ranks 0, 1 and 2 each take one row, rank 3 takes the range \([3, 3)\), which is empty, and gets one dashed padding row; the bias repeats the same pattern, and \(S\) grows from 4 to 5.
fully_shard(module, mesh) does at setup time- input: a module whose submodules may already be sharded; a device mesh with \(F\) ranks along the shard dimension; this rank’s index \(r\)
- collect the module’s parameters that are not yet managed by an inner
fully_shardcall // these form one parameter group - for each parameter \(p\) with dim-0 size \(n\): keep rows \([rc, \min\{(r+1)c, n\})\) with \(c = \lceil n/F \rceil\) // formula (47.1)
- replace \(p\) by a DTensor with placement
Shard(0)and that local tensor; the parameter name does not change - register a pre-forward hook (all-gather the group) and a post-forward hook (free it, if
reshard_after_forward) - register backward hooks: all-gather before the group’s backward, reduce-scatter its gradients after
- change the class of the module in place, for example
BlockbecomesFSDPBlock, a subclass of bothBlockandFSDPModule// no wrapper object
Two details from the API reference are worth knowing. First, fully_shard does not wrap the module. It changes the module’s class in place, so model.layers[0] is still the same object, the parameter names (FQNs) stay the same, and the state_dict() keys stay the same. [Certain] Second, you call it bottom-up: first on each transformer block, then on the root model. The root call takes all parameters that are left over (embedding, final norm, output head). [Certain]
One group, one collective: copy-in, all-gather, copy-out
If every parameter is its own DTensor, does FSDP2 run one all-gather per tensor? No. That would bring back the problem FlatParameter solved: many small collectives with launch overhead. FSDP2 still runs one all-gather per parameter group. It does it with copies around the collective: [Certain]
- Copy-in. Each rank copies its local shards of all parameters of the group into one flat send buffer, one after another, with zero padding up to \(c_i\) rows each (formula 47.2).
- All-gather. One collective. The output is rank 0’s buffer, then rank 1’s buffer, and so on. Inside it, the pieces of different parameters are interleaved: (p1 of rank 0, p2 of rank 0, …, p1 of rank 1, p2 of rank 1, …).
- Copy-out. For each parameter, take its piece from every rank’s part of the output, drop the padding, and write it into one contiguous full tensor with the original shape.
The backward pass mirrors this. Each rank copies its full gradients into a reduce-scatter input with the same rank-major, padded layout, runs one reduce-scatter, and gets the summed gradient of its own rows of every parameter. These become the .grad of the sharded DTensor parameters. The FSDP2 design document says that the only performance cost of this design is the extra copies before and after the collectives, and that custom copy kernels make them cheap. In internal benchmarks, copy-in/copy-out used the bandwidth better than NCCL’s alternative of grouping many small collectives together. [Certain] Step through it in Figure 2.
- rank \(r\) holds rows \([rc, \min\{(r+1)c, 3\})\) of \(W\) and of \(b\), \(c = \lceil 3/F \rceil\)
- copy-in: send \(=\) [\(W\) rows, zero rows up to \(c\) | \(b\) rows, zeros up to \(c\)], \(S = 2c + c\)
- all-gather: out \(=\) [send\(_0\) | send\(_1\) | …], the same \(F S\) numbers on every rank
- copy-out: cut \(W\) and \(b\) out of every part, drop the padding, rebuild \(W\) (\(3 \times 2\)) and \(b\) (3)
- forward on own micro-batch: \(y_r = W x_r + b\), \(\ell_r = \tfrac12 \lVert y_r - t_r \rVert^2\)
- backward: \(e_r = y_r - t_r\), \(\nabla W_r = e_r x_r^\top\), \(\nabla b_r = e_r\) // full, different per rank
- RS copy-in: the gradients in the same rank-major padded layout, \(F S\) numbers
- reduce-scatter: \(\bar g = \frac1F \sum_r g_r\); rank \(r\) keeps part \(r\) (\(S\) numbers)
- optimizer: shard \(\leftarrow\) shard \(- \eta\,\bar g_\text{shard}\), \(\eta = 0.1\); free the full \(W\), \(b\) and gradients
How to read Figure 2
Each cell is one number. Start with \(F = 2\): \(c = 2\), so rank 0 holds rows 0–1 and rank 1 holds row 2 only. In the copy-in step, rank 1 has one row of \(W\) and one element of \(b\), so it adds 2 + 1 = 3 zeros to reach the same 6-number buffer as rank 0. After the all-gather, both ranks hold the same 12 numbers, in which \(W\) and \(b\) are interleaved; copy-out takes cells 0–3 and 6–7 for \(W\) and cells 4–5 and 10 for \(b\). In the forward, rank 0 gets \(y_0 = (-1.40, 1.80, 3.80)\) and rank 1 gets \(y_1 = (2.10, 2.55, -2.70)\): the full gradients differ between ranks. The reduce-scatter averages them; for \(W_{00}\) it is \((-2.40 + 4.20)/2 = 0.90\), so the optimizer sets \(W_{00} = 0.50 - 0.1 \times 0.90 = 0.41\). Now pick \(F = 4\): \(c = 1\) and rank 3 holds 0 rows; its 3-number buffer is pure padding, and after the reduce-scatter it keeps nothing.
reshard_after_forward=True, on rank \(r\)- pre-forward hook: cast local shards to
param_dtype; copy-in; all-gather; copy-out into full tensors; swap the full tensors in as the module’s parameters - run the module’s forward; activations are saved by autograd as usual
- post-forward hook: free the full tensors (their storage is resized to 0) and swap the sharded DTensors back in // the root group keeps them by default
- pre-backward hook: all-gather the group again, same as step 1
- autograd computes the full gradients of the group from this rank’s micro-batch
- post-backward: copy the gradients into the reduce-scatter layout; reduce-scatter in
reduce_dtype, averaged over ranks; add the result to the sharded.grad; free the full parameters and gradients - optimizer step: a normal
torch.optimoptimizer updates the DTensor parameters with their DTensor gradients; the optimizer states are DTensors too, so they are sharded
The communication volume is the same as in FSDP1: two all-gathers and one reduce-scatter per group, \(3\Psi(F-1)/F\) elements per GPU per step, or 1.5 times DDP. In our CPU run of the script below, a model with a root group and 3 block groups made 7 all-gathers and 4 reduce-scatters per step, exactly like Figure 2 of session 46: the root group is gathered once.
Memory without surprises
FSDP1 and DeepSpeed ZeRO use record_stream to tell the CUDA caching allocator that a buffer is used by the communication stream. The problem is that the memory is then freed only when the GPU has finished with it, which can be much later than when the CPU says “free”. FSDP1 needed the rate limiter limit_all_gathers=True, which blocks the CPU, to keep this under control. FSDP2 does not use record_stream. It synchronizes the streams with CUDA events and frees buffers at known points, so its memory use is deterministic and it never blocks the CPU thread. [Certain] That is the main source of the 7% lower memory measured in torchtitan [Likely].
Why per-parameter sharding is simpler
Clean state dicts
Because the training representation is already “one tensor per name”, model.state_dict() returns the original keys with DTensor values, each holding its rank’s rows. No communication is needed. [Certain] Distributed Checkpoint (DCP) can save these shards and load them later on a different number of GPUs, because each DTensor says exactly which rows it holds (session 51). For a normal single-file checkpoint you call full_tensor() on each value on every rank, or get_model_state_dict(model, options=StateDictOptions(full_state_dict=True, cpu_offload=True)). In FSDP1, both kinds of checkpoint needed the FlatParameter to be taken apart first.
Freezing, LoRA and per-parameter settings
A FlatParameter has one requires_grad and one dtype. In FSDP2 every parameter keeps its own. So you can freeze one tensor in the middle of a block, put frozen and trainable tensors in the same group, use different optimizer settings (for example no weight decay on norms) by name, and the optimizer sees normal per-parameter tensors. [Certain] The design document lists “mixing frozen and non-frozen parameters in one communication group without extra memory” as one of the goals. [Certain] We tested it: one FSDP2 group with a frozen BF16 base weight and FP32 LoRA matrices trained without errors on PyTorch 2.8, and a frozen tensor inside a block stayed bit-for-bit unchanged (script below). Only the trainable parameters of a group must share one dtype, because their gradients go through one reduce-scatter. [Certain]
LoRA (session 57) with rank \(r = 16\) on all seven linear layers of each block adds \(r(d_\text{in} + d_\text{out})\) parameters per layer. Per block: attention \(4 \times 16 \times (4{,}096 + 4{,}096) = 524{,}288\); MLP \(3 \times 16 \times (4{,}096 + 11{,}008) = 724{,}992\). Over 32 blocks: \(32 \times 1{,}249{,}280 = 39{,}976{,}960\), about 0.59% of the 6.738B base parameters.
With FSDP2 the frozen base weights stay sharded in BF16 with no gradients and no Adam states: \(2 \times 6.738 \times 10^9 / 8 = 1.68\) GB per GPU. The LoRA matrices live in the same groups, in FP32 with gradients and Adam states: \(16 \times 39{,}976{,}960 / 8 = 80\) MB per GPU. Full fine-tuning would need \(16 \times 6.738 \times 10^9 / 8 = 13.48\) GB of model states per GPU. The all-gathers still move the frozen weights every step, because the forward pass needs them.
Custom all-gather: float8 and NF4
Because each parameter is its own tensor, FSDP2 lets a tensor subclass decide what is sent in the all-gather. torchao’s float8 training uses this to all-gather weights in float8 (1 byte instead of 2), and QLoRA-style training uses it to gather NF4 weights (session 58). [Certain] With a FlatParameter, one tensor of the unit could not easily use a different format from the others.
Meta-device initialization
A 70B model in FP32 needs about 276 GB just for its weights, so you cannot build it on one GPU first. With FSDP2 you build the model on the meta device (shapes only, no memory), call fully_shard on it, and then call model.to_empty(device="cuda") and your own initialization (for example reset_parameters()) on the sharded DTensors. Each GPU only ever allocates its own rows. [Certain] FSDP1’s param_init_fn is gone.
The knobs: precision, resharding and prefetching
MixedPrecisionPolicy
MixedPrecisionPolicy(param_dtype, reduce_dtype, output_dtype, cast_forward_inputs) is passed as mp_policy to each fully_shard call. [Certain]
param_dtype: the dtype of the gathered (unsharded) parameters used in forward and backward, and so the dtype of the all-gather. The sharded parameters keep their original dtype (usually FP32), and the optimizer steps on them.reduce_dtype: the dtype of the gradient reduce-scatter. If it isNoneandparam_dtypeis set, it followsparam_dtype. The tutorial recommends FP32: “reducing gradients in float32 can be critical for numerics”.output_dtype(new in FSDP2): casts the floating-point outputs of the module’s forward, for example back to FP32 at the root.cast_forward_inputs(defaultTrue): casts floating-point inputs of the forward toparam_dtype.
There is no buffer_dtype: FSDP2 does not manage buffers (such as RoPE tables), so you cast them yourself. [Certain] The cast happens once per group, at the module border, like FSDP1’s MixedPrecision and unlike torch.autocast. Groups may use different policies, for example FP32 for a sensitive output head.
reshard_after_forward
This one argument replaces FSDP1’s FULL_SHARD / SHARD_GRAD_OP choice: [Certain]
True: free the gathered parameters after the group’s forward, gather again in the backward. Like FULL_SHARD (ZeRO-3).False: keep them until the group’s backward is done. Saves the backward all-gather. Like SHARD_GRAD_OP.None(default):Truefor every group except the root, which keeps its parameters, because the backward starts right where the forward ended.- An integer \(g\): after the forward, keep a copy sharded over only \(g\) ranks (a divisor of \(F\), not 1 and not \(F\); usually the 8 GPUs of a node). The backward all-gather then runs over \(g\) ranks inside the node, at the cost of holding \(1/g\) of each block instead of \(1/F\).
torchtitan uses True for every block except the last one: FSDP would prefetch the last block for the backward immediately after its forward, so freeing it would be wasted work. [Certain]
With \(L\) blocks of \(B\) parameters, a root group of \(R\) parameters (embedding, final norm, head), \(\Psi = LB + R\), and BF16 gathering (\(K_\text{low} = 2\) bytes), a simple model of the peak parameter-related memory per GPU is
- Read the symbols
- \(16\Psi/F\)
- the sharded FP32 weights, FP32 gradients and Adam \(m\), \(v\) (16 bytes per parameter, split over \(F\) ranks)
- \(2K_\text{low}R\)
- the root’s gathered weights and gradients, which live through the whole backward
- \(3B\)
- with resharding, about three block-sized buffers are alive at the peak: the current block’s weights, the next block’s prefetched weights and the current block’s gradients
- \((L+1)B\)
- without resharding, all \(L\) blocks are still gathered when the backward starts, plus one block of gradients
- \(L/g\)
- with an integer \(g\), each block keeps a \(1/g\) slice until its backward
And the bytes each GPU sends per step, with ring collectives (session 43) and \(K_\text{red}\) bytes per element in the reduce-scatter (4 for FP32, 2 for BF16), are
The three terms are the forward all-gather, the backward all-gather and the gradient reduce-scatter. The integer option does not send fewer bytes; it moves the backward all-gather onto the fast links inside a node. Figure 3 evaluates both formulas.
param_dtype. Left: peak parameter-related memory per GPU for the three reshard_after_forward settings (the selected one is outlined). Right: bytes each GPU sends per step; the light part of the integer setting runs inside the node. Activations, temporary buffers and fragmentation are not included. Dashed line: 80 GB.How to read Figure 3
Defaults: Llama 2 7B on 64 GPUs, FP32 reduction. True needs 3.95 GB and sends 52.55 GB per step. False saves the backward all-gather (39.8 GB) but keeps every block gathered: 16.09 GB. The integer 8 sits in between, 5.57 GB and 51.13 GB, of which 11.33 GB stay inside the node. Switch to Llama 3 8B: its 128,256-token vocabulary makes the root (embedding plus head, 1.05B parameters) almost five times larger than one block (218M), so the root dominates the gathered memory. Switch to Llama 2 70B on 8 GPUs: the sharded states alone are 138 GB, so it does not fit whatever you set; you need more GPUs or tensor parallelism. Set reduce_dtype to BF16 to see the reduce-scatter bytes halve.
Parameters: 32 blocks of \(B = 202{,}383{,}360\); root \(R = 2 \times 131{,}072{,}000 + 4{,}096 = 262{,}148{,}096\); \(\Psi = 6{,}738{,}415{,}616\).
Memory, True: \(16\Psi/64 = 1.68\) GB, root \(4R = 1.05\) GB, blocks \(2 \times 3B = 1.21\) GB, total 3.95 GB. False: the blocks term becomes \(2 \times 33 \times B = 13.36\) GB, total 16.09 GB.
Communication, True with an FP32 reduce-scatter: forward all-gather \(2\Psi \times 63/64 = 13.27\) GB, backward all-gather \(2 \times 32B \times 63/64 = 12.75\) GB, reduce-scatter \(4\Psi \times 63/64 = 26.53\) GB, total 52.55 GB. False drops the 12.75 GB: 39.80 GB. A BF16 reduce-scatter would save another 13.27 GB, at some cost in precision.
Prefetching
FSDP2 always prefetches in the backward pass the way FSDP1’s BACKWARD_PRE did: the all-gather of the next block is issued before the current block’s gradients are computed. BACKWARD_POST does not exist anymore. [Certain] The order is the reverse of the order in which groups finished their forward.
In the forward pass, FSDP2 relies on implicit prefetching: the CPU thread runs ahead of the GPU, and it issues the all-gather for block \(i\) on a separate CUDA stream while the GPU still computes block \(i-1\). The tutorial recommends starting with this. [Certain] It fails when the CPU is the bottleneck (many small kernels, a slow Python loop): then the CPU reaches block \(i\) late, and its all-gather starts late. For that case there is explicit prefetching: [Certain]
block_i.set_modules_to_forward_prefetch([block_i+1, …]): when block \(i\) runs, also issue the all-gathers of the listed blocks.block_i.set_modules_to_backward_prefetch([…]): the same for the backward, to prefetch more than one block.model.unshard()beforemodel(x): start the first all-gather early, for example during data loading, so the first block does not wait.
Prefetching two or more blocks gives more overlap, but more gathered blocks are in memory at the same time. Something must stop the CPU from gathering all blocks at once. FSDP1 and FSDP2 do this in different places: [Certain]
- FSDP1 (
limit_all_gathers=True): after each block’s forward it records a “free” event in a queue. Before every all-gather, if the queue already holds 2 events, the CPU takes the oldest one and callsevent.synchronize(): the CPU thread stops until the GPU has finished that block. The code calls this the rate limiter; the limit 2 is “empirically chosen”. - FSDP2: the all-gather streams wait on a CUDA event recorded after an earlier block’s copy-out. This is a wait between GPU streams, so the CPU thread never stops.
Figure 4 simulates the forward pass with both rules, one event at a time.
- for block \(i = 0, \dots, 5\): the CPU launches block \(i\) during \([t_i,\ t_i + c]\)
- FSDP2issue \(\mathrm{AG}_i\) if not prefetched; its stream waits for the copy-out of block \(i-2\)
- FSDP2explicit: also issue \(\mathrm{AG}_{i+1}, \dots, \mathrm{AG}_{i+k}\); they wait for the copy-out of block \(i-1\)
- FSDP1before each AG: if 2 gathered blocks are not freed yet, the CPU blocks until the oldest is freed
- FSDP1issue \(\mathrm{AG}_i\), and \(\mathrm{AG}_{i+1}\) if
forward_prefetch - \(\mathrm{AG}_j\) starts at \(\max\)(issue time, end of the previous AG, stream event)
- \(\mathrm{F}_i\) starts at \(\max\)(end of \(\mathrm{F}_{i-1}\), end of \(\mathrm{AG}_i\), \(t_i\)) and ends at \(\max(\text{start} + 1,\ t_i + c)\)
- FSDP1post-forward: free the flat parameter and put its free event in the queue
How to read Figure 4
Defaults: the CPU needs 1.2 per block but the GPU only 1.0, so the run is CPU-bound. FSDP2 with implicit prefetching ends at 7.8 against 7.2 with a free network: 0.6 is exposed, because each all-gather is issued only when the CPU reaches its block. FSDP1 without forward_prefetch does the same and also ends at 7.8; its CPU is never blocked, because the CPU is the slow side. Explicit, 1 ahead removes the exposed time (7.2), at the price of 3 gathered blocks alive at once instead of 2; FSDP1’s forward_prefetch=True does the same. Now set the CPU time to 0.3: the CPU runs far ahead. Both runs end at 6.8 (only the first all-gather, 0.8, is exposed), but the FSDP1 CPU now spends 3.3 time units blocked in event.synchronize() (red), while the FSDP2 CPU is free to do other work. Tick “unshard early”: the FSDP2 run drops to 6.1, because the first all-gather now runs before the forward starts; with explicit prefetching as well it reaches 6.0, the free-network time. FSDP1 has no such call and stays at 6.8. Push the all-gather time above 1.2 and no setting helps: the network itself is the bottleneck.
Composing with other parallelisms
HSDP: a 2-D mesh
FSDP1’s HYBRID_SHARD becomes a 2-D device mesh. With \(W = H \times G\) GPUs in \(H\) nodes of \(G\) GPUs, you build init_device_mesh("cuda", (H, G), mesh_dim_names=("replicate", "shard")) and pass it to fully_shard. The parameters get the placements (Replicate(), Shard(0)): replicated across nodes, sharded inside a node. [Certain] The gradient is reduce-scattered inside the node and all-reduced across nodes, exactly as in FSDP1’s hybrid sharding, and each GPU holds \(16\Psi/G\) bytes of model states. reshard_after_forward=False with the same mesh gives the old _HYBRID_SHARD_ZERO2. [Certain]
Tensor parallelism
Tensor parallelism (next session) is also written with DTensor. You build a 2-D mesh ("dp", "tp"), apply tensor parallelism with parallelize_module on the "tp" sub-mesh, and then call fully_shard(…, mesh=mesh["dp"]). A weight then has one placement per mesh dimension, for example sharded on dim 0 by FSDP and on dim 1 by TP. This works because both are DTensor placements; with FSDP1 it needed special code. torchtitan applies tensor parallelism, then activation checkpointing, then torch.compile, then FSDP2, in that order. [Certain]
torch.compile
FSDP2’s hooks are not traced by the compiler (in PyTorch 2.8 they are wrapped in torch._dynamo.disable while the config flag torch._dynamo.config.skip_fsdp_hooks is True, its default). [Certain] The common recipe is regional compilation: compile each transformer block with fullgraph=True, then call fully_shard. Each block becomes one graph without graph breaks, and FSDP2 does its communication between the blocks. torchtitan reports that this works with FSDP2 and TP and improves both throughput and memory. [Certain] Compilation is the topic of session 50.
Using FSDP2 in PyTorch
First the mechanics, by hand, with no distributed library: per-parameter dim-0 shards on 4 simulated ranks, copy-in, all-gather, copy-out, and a reduce-scatter in the same layout. One sharded SGD step must equal one normal SGD step on the full batch.
import torch torch.manual_seed(0) F = 4 # ranks unit = torch.nn.Sequential(torch.nn.Linear(4, 6), torch.nn.Linear(6, 3)) shapes = [p.shape for p in unit.parameters()] # (6,4) (6,) (3,6) (3,) rows = [-(-s[0] // F) for s in shapes] # c_i = ceil(n_i / F), formula (47.1) inner = [s[1:].numel() for s in shapes] # m_i = elements per row seg = [c * k for c, k in zip(rows, inner)] # padded elements per rank, per parameter def shard(t, r, i): # dim-0 chunk; last ranks may get fewer rows return t[r * rows[i]:(r + 1) * rows[i]].clone() local = [[shard(p.detach(), r, i) for i, p in enumerate(unit.parameters())] for r in range(F)] print("rank 3 local shapes:", [tuple(t.shape) for t in local[3]]) # [(0, 4), (0,), (0, 6), (0,)] def copy_in(shards): # one flat send buffer per rank, padded per param buf, o = torch.zeros(sum(seg)), 0 for i, t in enumerate(shards): buf[o:o + t.numel()] = t.reshape(-1); o += seg[i] return buf def all_gather(local): # output: F rows, rank r's buffer in row r return torch.stack([copy_in(local[r]) for r in range(F)]) def copy_out(out): # cut each parameter out, drop padding, contiguous full, o = [], 0 for i, s in enumerate(shapes): full.append(out[:, o:o + seg[i]].reshape(F * rows[i], *s[1:])[:s[0]].contiguous()); o += seg[i] return full def reduce_scatter(grads_per_rank): # same layout, sum over ranks, keep own row lay = lambda g: torch.stack([copy_in([shard(x, r, i) for i, x in enumerate(g)]) for r in range(F)]) summed = sum(lay(g) for g in grads_per_rank) # (F, sum(seg)) out = [] for r in range(F): # padded row r -> rank r's unpadded shards o, mine = 0, [] for i, s in enumerate(shapes): n = local[r][i].shape[0] mine.append(summed[r, o:o + n * inner[i]].reshape(n, *s[1:])); o += seg[i] out.append(mine) return out X, Y = torch.randn(8, 4), torch.randn(8, 3) lr, grads = 0.1, [] for r in range(F): # each rank: its own quarter of the batch w1, b1, w2, b2 = [t.requires_grad_(True) for t in copy_out(all_gather(local))] x, y = X.chunk(F)[r], Y.chunk(F)[r] loss = (((x @ w1.T + b1) @ w2.T + b2 - y) ** 2).mean() / F # /F: average over ranks loss.backward() grads.append([w1.grad, b1.grad, w2.grad, b2.grad]) g_local = reduce_scatter(grads) local = [[p - lr * g for p, g in zip(local[r], g_local[r])] for r in range(F)] # sharded SGD ref = torch.nn.Sequential(torch.nn.Linear(4, 6), torch.nn.Linear(6, 3)); ref.load_state_dict(unit.state_dict()) ((ref(X) - Y) ** 2).mean().backward() with torch.no_grad(): for p in ref.parameters(): p -= lr * p.grad new = copy_out(all_gather(local)) print("max diff:", max((a - b).abs().max().item() for a, b in zip(new, ref.parameters()))) # about 3e-8 print("send buffer per rank:", sum(seg), "elements for", sum(s.numel() for s in shapes), "real ones") # 17 for 51
Rank 3 holds nothing in this tiny unit, and the all-gather moves \(4 \times 17 = 68\) elements for 51 real ones: formula (47.2) in action. Now the real API. This script runs FSDP2 on 4 CPU processes with the gloo backend, so it works on a laptop. It shards each block as one group, freezes one tensor inside a block, trains 3 AdamW steps with gradient clipping and each rank on a quarter of the batch, and compares with single-process AdamW on the full batch. We ran it with PyTorch 2.8. It prints FSDPSequential FSDPBlock, then DTensor (64, 16) local (16, 16) (Shard(dim=0),), matching keys, the same gradient norm 0.8319 as the reference, a maximum difference of about \(2 \times 10^{-7}\), and frozen tensor unchanged: True.
import os, torch, torch.nn as nn, torch.distributed as dist, torch.multiprocessing as mp from torch.distributed.fsdp import fully_shard, FSDPModule from torch.distributed.device_mesh import init_device_mesh class Block(nn.Module): # one "transformer block" (an MLP block here) def __init__(s, d): super().__init__(); s.norm = nn.LayerNorm(d); s.up = nn.Linear(d, 4 * d); s.down = nn.Linear(4 * d, d) def forward(s, x): return x + s.down(torch.relu(s.up(s.norm(x)))) def make_model(): torch.manual_seed(0) # same init on every rank and for the reference return nn.Sequential(nn.Linear(8, 16), Block(16), Block(16), Block(16), nn.Linear(16, 4)) def worker(rank, W): os.environ.update(MASTER_ADDR="127.0.0.1", MASTER_PORT="29601") dist.init_process_group("gloo", rank=rank, world_size=W) # CPU backend; "nccl" on GPUs mesh = init_device_mesh("cpu", (W,)) # 1-D mesh: plain FSDP model = make_model() for block in model[1:4]: fully_shard(block, mesh=mesh) # one parameter group per block fully_shard(model, mesh=mesh) # root group: first and last Linear for m in model.modules(): if isinstance(m, FSDPModule): m.set_force_sum_reduction_for_comms(True) # gloo only: see "Common mistakes" model[1].up.weight.requires_grad_(False) # freeze ONE tensor inside a group w = model[1].up.weight if rank == 0: print(type(model).__name__, type(model[1]).__name__) print(type(w).__name__, tuple(w.shape), "local", tuple(w.to_local().shape), w.placements) opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-2) # AFTER fully_shard ref = make_model(); ref[1].up.weight.requires_grad_(False) ropt = torch.optim.AdamW([p for p in ref.parameters() if p.requires_grad], lr=1e-2) torch.manual_seed(1); X, Y = torch.randn(16, 8), torch.randn(16, 4) for step in range(3): x, y = X.chunk(W)[rank], Y.chunk(W)[rank] # each rank: its own quarter nn.functional.mse_loss(model(x), y).backward() # gradients are averaged over ranks gn = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0).full_tensor() # on EVERY rank opt.step(); opt.zero_grad() nn.functional.mse_loss(ref(X), Y).backward() rgn = torch.nn.utils.clip_grad_norm_(ref.parameters(), 1.0); ropt.step(); ropt.zero_grad() sd = model.state_dict() # DTensors, original keys, no communication full = {k: v.full_tensor() for k, v in sd.items()} # all-gather: every rank must call it if rank == 0: rsd = ref.state_dict() print("keys match:", list(sd) == list(rsd), " last grad norm:", f"{gn.item():.4f} vs {rgn.item():.4f}") print("max diff vs 1-process AdamW:", f"{max((full[k] - v).abs().max().item() for k, v in rsd.items()):.1e}") print("frozen tensor unchanged:", torch.equal(full["1.up.weight"], make_model().state_dict()["1.up.weight"])) dist.destroy_process_group() if __name__ == "__main__": mp.spawn(worker, args=(4,), nprocs=4) # 4 ranks = 4 "GPUs"
Things to try. Pass mp_policy=MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32) to every fully_shard call (and use model(x).float() in the loss): the difference grows to about \(2 \times 10^{-2}\) after 3 steps, which is BF16 rounding. Pass reshard_after_forward=False to the blocks: the result is identical, and the all-gathers per step drop from 7 to 4 (we counted them by wrapping dist.all_gather_into_tensor). Note that torch.nn.utils.clip_grad_norm_ works unchanged: the gradients are DTensors, so the norm is computed over all shards (it comes back as a DTensor with a Partial placement, and full_tensor() finishes the sum). On a GPU cluster, a typical FSDP2 setup for a Llama-style model looks like this (launched with torchrun; not run here, it needs GPUs):
from torch.distributed.fsdp import fully_shard, MixedPrecisionPolicy, CPUOffloadPolicy from torch.distributed.device_mesh import init_device_mesh mesh = init_device_mesh("cuda", (world_size,)) # or (nodes, 8) with names for HSDP mp_policy = MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32) with torch.device("meta"): model = Transformer(config) # shapes only, no memory for i, block in enumerate(model.layers): block.compile(fullgraph=True) # regional compile, before sharding fully_shard(block, mesh=mesh, mp_policy=mp_policy, reshard_after_forward=(i != len(model.layers) - 1)) # keep the last block (torchtitan) fully_shard(model, mesh=mesh, mp_policy=mp_policy) # root: embedding, norm, head model.to_empty(device="cuda") # allocate only this rank's rows model.init_weights() # your own init, on DTensors (also buffers!) optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1) # after fully_shard for step, (x, y) in enumerate(loader): loss = loss_fn(model(x), y); loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # the normal function works optimizer.step(); optimizer.zero_grad()
Moving from FSDP1 to FSDP2
The migration is mostly mechanical. Instead of an auto_wrap_policy, you write the loop over blocks yourself and call fully_shard on each, then on the root. The rest maps argument by argument (from the FSDP tutorial): [Certain]
| FSDP1 | FSDP2 |
|---|---|
FSDP(model, auto_wrap_policy=…) | fully_shard(block) for each block, then fully_shard(model) |
ShardingStrategy.FULL_SHARD | reshard_after_forward=True |
ShardingStrategy.SHARD_GRAD_OP | reshard_after_forward=False |
HYBRID_SHARD / _HYBRID_SHARD_ZERO2 | True / False with a 2-D device mesh |
MixedPrecision(param_dtype, reduce_dtype, buffer_dtype) | MixedPrecisionPolicy(param_dtype, reduce_dtype, output_dtype, cast_forward_inputs); no buffer dtype |
CPUOffload(offload_params=True) | offload_policy=CPUOffloadPolicy() (pinned memory by default) |
backward_prefetch=BACKWARD_PRE / BACKWARD_POST | always on / not supported |
forward_prefetch=True | set_modules_to_forward_prefetch, or unshard() by hand |
limit_all_gathers=True | not needed: no CPU synchronization |
use_orig_params=True | always: there is no FlatParameter |
sync_module_states=True | set_model_state_dict(…, StateDictOptions(full_state_dict=True, broadcast_from_rank0=True)) |
param_init_fn | meta device, to_empty(), then your own init |
device_id | taken from the device mesh |
with model.no_sync(): | model.set_requires_gradient_sync(False) |
model.clip_grad_norm_(1.0) | torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) |
FSDP.state_dict_type(…) contexts | model.state_dict() (sharded DTensors) or get_model_state_dict / DCP |
Gradient accumulation (session 39) deserves a note. Calling model.set_requires_gradient_sync(False) before all micro-batches but the last skips the reduce-scatters, but then each rank keeps unsharded gradients (in reduce_dtype) until the sync, which costs memory. [Certain] Leaving sync on reduce-scatters every micro-batch and keeps only sharded gradients. set_reshard_after_backward(False) can additionally keep parameters gathered between micro-batches. Which is better depends on whether memory or the network is your limit.
Where this is used
| Setting | Typical FSDP2 setup | Why FSDP2 helps |
|---|---|---|
| LLM pretraining with torchtitan (Llama 3.1 8B to 405B in the paper) | FSDP2 per block, BF16 params with FP32 reduce, regional torch.compile, plus TP and pipeline parallelism at large scale | One DTensor representation for all parallelisms. The paper reports a 65.08% speedup with 1-D parallelism on 128 GPUs for Llama 3.1 8B over optimized baselines [Certain]; this comes from the whole stack (compile, float8 and more), not from FSDP2 alone [Likely] |
| Multi-node training of 7B–70B models | HSDP: 2-D mesh (replicate across nodes, shard inside) | Only a \(1/G\) slice of the gradients crosses nodes |
| LoRA and QLoRA fine-tuning | Frozen base and trainable adapters in the same groups; NF4 all-gather for QLoRA [Likely] | Per-parameter requires_grad and dtype; no extra gradient memory for frozen tensors |
| Vision-language models (frozen or partly frozen vision tower + LLM) | One group per vision block and per LLM block; vision tower frozen, projector trainable [Likely] | Freezing does not depend on how groups are cut |
| Video encoders and video-language models (action recognition, video classification, Video-RAG retrieval encoders) | FSDP2 for model states, activation checkpointing for the long frame sequences [Likely] | FSDP2 shards the model states; activations of many frames need checkpointing on top |
| Float8 training | torchao float8 linears with float8 all-gather [Certain] | The all-gather of those weights moves 1 byte per element instead of 2 |
Common mistakes
- Creating the optimizer before
fully_shard. The optimizer then holds the old plain parameters, which FSDP2 has replaced by DTensors. Build it after sharding. - Calling
fully_shardonly on the root. Then the whole model is one group, and the forward gathers everything. Call it on each block first, then on the root (bottom-up). - Calling a collective on some ranks only.
full_tensor(),get_model_state_dict(full_state_dict=True)and the grad-norm reduction are collectives. Writingif rank == 0: print(gn.full_tensor())hangs the job: rank 0 waits for ranks that never join. We hit exactly this while preparing this page. Call it on all ranks, then print on one. - Mixing DTensors and plain tensors. An in-place update such as
p.add_(t)with a plain tensortraises an error about mixedtorch.TensorandDTensor. Wraptwithdistribute_tensor, or work onp.to_local()with the matching rows. - Running a method other than
forward. FSDP2 gathers parameters in forward hooks. If you callmodel.generate()ormodel.lm_head(h)directly, the parameters may still be sharded. Useregister_fsdp_forward_method(model, "generate"), orunshard()by hand. [Certain] - Meta-device init without full re-initialization.
to_empty()leaves memory uninitialized, including buffers such as RoPE tables, which FSDP2 does not manage. Re-initialize every parameter and buffer. - Saving with
torch.save(model.state_dict())and loading into a plain model. You saved per-rank DTensor shards. Use DCP for sharded checkpoints, orget_model_state_dict(…, full_state_dict=True, cpu_offload=True)for one full file. - Gradients that are 4 times too large on CPU. In our PyTorch 2.8 runs with the
gloobackend, FSDP2 asked for an averaged reduce-scatter and got a sum: on 4 ranks the gradient norm was exactly 4 times the reference.set_force_sum_reduction_for_comms(True)makes FSDP2 sum and divide by itself, which fixed it. With NCCL on GPUs the average is native. Always compare a few steps with a 1-GPU reference before a long run. [Certain for our setup]
Interview questions
What is the main difference between FSDP1 and FSDP2?basic
Shard(0), so a rank holds whole rows of each tensor. The algorithm (all-gather before forward and backward, reduce-scatter of gradients, sharded optimizer) and the communication volume are the same. The API changes too: fully_shard changes the module’s class in place instead of wrapping it, so names and state-dict keys do not change.What is a DTensor?basic
Shard(d) (cut along dim d), Replicate() (full copy) or Partial() (local values sum to the true value). Its shape is the global shape; operations on it give the same result as on the full tensor, with communication inserted as needed. to_local() returns the local piece, full_tensor() all-gathers, distribute_tensor shards a full tensor. FSDP2 and tensor parallelism are both written on top of it.If each parameter is sharded separately, does FSDP2 run one all-gather per parameter?core
fully_shard call). Each rank copies its shards of all parameters of the group into one flat, padded send buffer (copy-in); one all-gather produces all ranks’ buffers one after another, with parameters interleaved; then each parameter is copied out into its own contiguous full tensor. Gradients go the other way into one reduce-scatter. The cost is the extra copies, which custom kernels make cheap; the design document reports that this used bandwidth better than grouping many NCCL calls.How are uneven sizes handled when dim 0 is not divisible by the number of ranks?core
torch.chunk: each chunk has \(c = \lceil n/F \rceil\) rows, so the last ranks hold fewer rows, possibly zero. For the all-gather every rank pads its chunk with zero rows to \(c\), so all send buffers have the same size. Example: 50,257 embedding rows on 64 ranks: 63 ranks hold 786 rows, the last holds 739, and 47 padding rows are sent. For typical LLM layers on power-of-two GPU counts the padding is zero.Why is LoRA or partial freezing easier with FSDP2?core
requires_grad and one dtype, so frozen and trainable tensors could not share a unit without use_orig_params=True, and even then gradient memory and optimizer mapping were awkward. In FSDP2 each parameter keeps its own requires_grad, and frozen parameters get no gradient; only the trainable parameters of a group must share a dtype because they share one reduce-scatter. For Llama 2 7B with rank-16 LoRA on all linear layers on 8 GPUs: 1.68 GB of sharded frozen BF16 weights plus 80 MB of LoRA states per GPU.Explain reshard_after_forward, including the integer option.core
True frees gathered parameters after the group’s forward and gathers them again in backward (ZeRO-3, FULL_SHARD). False keeps them until the backward (SHARD_GRAD_OP), saving one all-gather per group but holding all gathered blocks at the start of backward. The default None is True except for the root. An integer \(g\) keeps a copy sharded over \(g\) ranks (for example one node), so the backward all-gather runs only inside the node; memory is in between. For Llama 2 7B on 64 GPUs with an FP32 reduce: 3.95 GB and 52.6 GB sent per step for True, 16.1 GB and 39.8 GB for False, 5.6 GB and 51.1 GB for 8.What do param_dtype and reduce_dtype do, and why set reduce_dtype to FP32?core
param_dtype is the dtype of the gathered parameters used in forward and backward, so it also sets the all-gather dtype (BF16 halves the bytes and the gathered memory). The sharded parameters stay FP32 and the optimizer steps on them. reduce_dtype is the dtype of the gradient reduce-scatter; if unset it follows param_dtype. Summing gradients from many ranks in BF16 loses precision, so FP32 is the usual choice, at twice the reduce-scatter bytes. There is no buffer dtype: FSDP2 does not manage buffers.Why does FSDP2 not need limit_all_gathers, and why is its memory “deterministic”?advanced
record_stream so that the caching allocator would not reuse communication buffers too early. Memory marked this way is freed only when the GPU has finished with it, which can be much later than the CPU’s free, so the CPU running ahead caused memory to pile up. FSDP1 fixed this by blocking the CPU (the rate limiter). FSDP2 instead synchronizes streams with CUDA events and frees buffers at fixed points, so memory follows the program order and the CPU is never blocked. torchtitan reports on average 7% lower memory for Llama 2 7B.When would you use explicit forward prefetching?advanced
set_modules_to_forward_prefetch([next blocks]) issues it while the earlier block runs. unshard() before the forward hides the first all-gather. The price: more gathered blocks alive at once. In the toy model of Figure 4, a CPU-bound run goes from 0.6 to 0 exposed time with 1-ahead prefetching, holding 3 instead of 2 blocks.How do you do HSDP and FSDP + TP with FSDP2?advanced
fully_shard. Parameters get placements (Replicate(), Shard(0)); gradients are reduce-scattered inside a node and all-reduced across nodes; memory is \(16\Psi/G\) per GPU. FSDP + TP: build a mesh ("dp", "tp"), apply tensor parallelism with parallelize_module on the TP sub-mesh, then fully_shard(…, mesh=mesh["dp"]). A weight then has one placement per mesh dimension. Because both use DTensor, no special glue code is needed. torchtitan applies TP, activation checkpointing, compile, then FSDP2.How does FSDP2 work with torch.compile?advanced
torch._dynamo.disable by default in PyTorch 2.8). The usual recipe is regional compilation: compile each transformer block with fullgraph=True, then apply fully_shard. Each block compiles to one graph without breaks, and FSDP2’s all-gathers and reduce-scatters happen between blocks. Because the parameters keep their original shapes (no FlatParameter views), the compiled code sees normal tensors.You migrated to FSDP2 and loss is slightly different from your 1-GPU run. How do you debug it?core
set_force_sum_reduction_for_comms(True) fixed it). Check that the optimizer was created after fully_shard, that every rank used the same init, and that the data split is right. Then turn on BF16: differences of about \(10^{-2}\) after a few steps are rounding, not bugs. Finally compare all-gather counts with what you expect from reshard_after_forward.Exercises
- A weight of shape \(11{,}008 \times 4{,}096\) is sharded on 64 ranks. How many rows does each rank hold, and how much padding is sent? Same question for an embedding of \(50{,}257 \times 768\) on 64 ranks.
- For the unit Linear(3, 4) + Linear(4, 2) on \(F = 3\) ranks, compute the FSDP1 shard size and padding, and the FSDP2 send buffer \(S\) and padding by (47.2).
- With formulas (47.3) and (47.4), compute the memory and the bytes sent per step for Llama 2 7B on 8 GPUs with
reshard_after_forward=Trueand an FP32 reduce-scatter. - Why does this code hang?
gn = clip_grad_norm_(model.parameters(), 1.0), thenif rank == 0: print(gn.full_tensor()). - Translate this FSDP1 call to FSDP2:
FSDP(model, auto_wrap_policy=block_policy, sharding_strategy=HYBRID_SHARD, mixed_precision=MixedPrecision(param_dtype=bf16, reduce_dtype=fp32))on 4 nodes of 8 GPUs. - In Figure 4 with CPU time 1.2 and all-gather time 0.8, why does implicit prefetching expose time even though each all-gather (0.8) is shorter than a block’s compute (1.0)?
Answerscheck
Partial placement; full_tensor() runs an all-reduce. Only rank 0 calls it, so rank 0 waits forever for the others. Call full_tensor() on all ranks, then print on rank 0. 5. mesh = init_device_mesh("cuda", (4, 8), mesh_dim_names=("replicate", "shard")); mp = MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32); fully_shard(block, mesh=mesh, mp_policy=mp) for each block, then fully_shard(model, mesh=mesh, mp_policy=mp); reshard_after_forward stays at its default (True for blocks). 6. The run is CPU-bound: the CPU reaches block \(i\) only at time \(1.2\,i\), so its all-gather starts then and ends at \(1.2\,i + 0.8\), while block \(i-1\) finished at \(1.2\,i + 0.6\). The GPU waits 0.2 per block. Overlap needs the all-gather to be issued early, not only to be short; explicit prefetching issues it one block earlier.Key takeaways
- FSDP2 keeps the FSDP algorithm but replaces the FlatParameter with one DTensor per parameter, sharded on dim 0 (
Shard(0)), chunked liketorch.chunk(47.1). fully_shardis called bottom-up, changes the module class in place, and keeps names and state-dict keys. Optimizers,clip_grad_norm_and state dicts work on DTensors directly.- One parameter group still means one all-gather and one reduce-scatter, through copy-in and copy-out with padding per parameter (47.2).
- Per-parameter sharding gives communication-free sharded state dicts, freezing and LoRA inside a group, float8 and NF4 all-gathers, and meta-device init.
reshard_after_forward(True, False, integer) trades memory against communication (47.3, 47.4);MixedPrecisionPolicysets gather and reduce dtypes; prefetching is implicit by default and explicit when the CPU is slow.- HSDP is a 2-D mesh; TP and
torch.compilecompose because everything is DTensor. FSDP2 avoidsrecord_stream, so memory is deterministic and the rate limiter is gone.
Key formulas
References for this session
- PyTorch (docs). Getting Started with Fully Sharded Data Parallel (FSDP2)
- PyTorch (code). torchtitan: reference large-scale training with FSDP2
- PyTorch (docs). Getting Started with Fully Sharded Data Parallel (FSDP2)
- PyTorch (code). torchtitan: reference large-scale training with FSDP2
- PyTorch (docs). torch.distributed.fsdp.fully_shard (FSDP2) API reference: fully_shard, FSDPModule, MixedPrecisionPolicy, CPUOffloadPolicy
- PyTorch (Andrew Gu et al.) (2023). [RFC] Per-Parameter-Sharding FSDP (pytorch/pytorch issue 114299)
- Liang et al. (2024). TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
- Zhao et al. (2023). PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- Rajbhandari et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs
- PyTorch (docs). torch.distributed.tensor (DTensor): DeviceMesh, Shard, Replicate, Partial