Distributed checkpoints, fault tolerance and MFU
A pretraining run on thousands of GPUs stops every few hours: a GPU dies, a link hangs, a chip computes a wrong number. This session shows how to save and load a model that is split over many GPUs, how often to save, how to restart (also on a different number of GPUs), how to catch silent data corruption, and how to compute and read MFU, the number that says how much of the GPU’s speed becomes useful training.
What you will learn
- What a training checkpoint must contain to resume exactly, and why it is about 12 bytes per parameter (847 GB for a 70B model)
- How sharded checkpoints (PyTorch DCP) save in parallel, commit safely, and reshard at load time onto a different number of GPUs
- How asynchronous checkpoints hide the write behind training, and what they cost in CPU memory
- Why a 16,384-GPU job fails every 2 to 3 hours, with the real Llama 3 and Meta cluster numbers, and how crashes, hangs, stragglers and silent data corruption differ
- Young’s formula for the best checkpoint interval, and why the restart cost often matters more
- How to compute MFU correctly (which N, which peak, attention, recomputation), and how to read it together with ETTR (goodput)
- When communication can hide behind compute: the minimum tokens per GPU for FSDP, b_min = (P u / B)(F − 1)/F
Why this matters
A large pretraining run is not one long program that starts and ends. It is a chain of many short runs, cut by crashes, hangs and maintenance. During 54 days of Llama 3 405B pretraining on up to 16,384 H100 GPUs, Meta saw 466 job interruptions: 47 planned and 419 unexpected. That is one unexpected stop about every 3.1 hours. [Certain] Still, they spent more than 90% of the time on useful training. [Certain]
That 90% does not happen by luck. It needs three things, and they are the topic of this session:
- Distributed checkpoints. A checkpoint is a saved copy of the full training state. With thousands of GPUs, each GPU holds only a piece of the state (session 46, session 47). Saving and loading must also be split over all GPUs, and you must be able to load on a different number of GPUs than you saved on.
- Fault tolerance. Find a failure fast, remove the bad machine, restart, and lose as little work as possible. This includes the hardest failure: silent data corruption, where a chip computes a wrong number and reports no error.
- MFU (model FLOPs utilization). When the job runs, how much of the GPU’s peak speed becomes useful training? You met the formula in session 42. Here you learn to compute it correctly, to read it, and to raise it by overlapping communication with compute.
These three ideas are linked by one number. The useful work you get from a cluster is roughly
- Read the symbols
- \(n_{\text{GPU}}, P\)
- number of GPUs, and the dense peak FLOP/s of one GPU (989 TFLOP/s for an H100 in BF16 [Certain])
- \(t_{\text{wall}}\)
- wall-clock time: the time on the clock on the wall, including all stops
- MFU
- model FLOPs utilization: useful model FLOP/s divided by peak FLOP/s, measured while the job runs
- ETTR
- effective training time ratio: the time spent on useful training divided by the wall-clock time. Also called goodput (good + throughput). Meta’s Llama 3 paper calls it “effective training time”.
Checkpoints and fault tolerance raise ETTR. Overlap raises MFU. A run with 45% MFU and 70% ETTR does less work than a run with 40% MFU and 95% ETTR.
What a checkpoint must contain
A checkpoint is useful only if training can continue from it as if nothing had happened. Saving only the weights is not enough. You need everything that changes during training:
| Part | Why it is needed | Size (mixed-precision AdamW) |
|---|---|---|
| Model weights (FP32 master copy) | the model itself | 4 bytes per parameter |
| Optimizer state (Adam \(m\) and \(v\)) | without it, Adam starts again with zero momentum and a wrong step size; the loss usually jumps | 8 bytes per parameter |
| Step counter and LR scheduler state | the learning rate depends on the step (session 4) | a few bytes |
| Data loader position | without it you train on the same data twice, or skip data | small |
| Random number generator states (per rank) | dropout masks and data shuffling must continue the same sequence | small |
| Loss scaler (FP16 only), config, code version | to rebuild the exact same run | small |
The gradients are not saved: you checkpoint between steps, when the gradients are empty. So a full training checkpoint has about 12 bytes per parameter, not the 16 bytes of training memory from session 38:
- Read the symbols
- \(\Psi\)
- number of parameters
- \(b_{\text{w}}, b_{\text{opt}}\)
- bytes per parameter for the weights (4 in FP32) and for the optimizer state (8 for Adam’s \(m\) and \(v\) in FP32)
- \(W\)
- world size: the number of ranks (GPU processes) that share the state. With full sharding each rank holds and saves \(1/W\).
From the Llama 3 70B config (vocabulary 128,256, width 8,192, 80 layers, MLP width 28,672, 8 key-value heads of size 128), the model has 70,553,706,496 parameters.
Training checkpoint: \(12 \times 70.55 \times 10^9 = 846.6\) GB. A weights-only export in BF16 (2 bytes per parameter) is 141.1 GB: six times smaller, but you cannot resume training from it.
On 1,024 GPUs with full sharding, each rank saves \(846.6 / 1024 = 0.83\) GB. Suppose the shared file system can absorb 200 GB/s in total (an assumption; real systems vary a lot). If every rank writes in parallel, the save takes about \(846.6 / 200 = 4.2\) s. If instead rank 0 gathers everything and writes one file at 2 GB/s (also an assumption), the save takes \(846.6 / 2 = 423\) s, about 7 minutes, and rank 0 needs 847 GB of CPU memory to hold it.
Sharded checkpoints: save in parallel
The simple way to checkpoint is torch.save(model.state_dict()) on rank 0. With a sharded model, this means: gather every shard to rank 0 (huge communication and memory), then write one big file with one process (slow). The worked example showed 7 minutes for 70B. At 405B it is worse, and during that time all other GPUs wait.
A sharded (or distributed) checkpoint does the opposite: every rank writes only the pieces it already owns, at the same time, to its own file. One small extra file, the metadata, records where every piece of every tensor lives. PyTorch’s Distributed Checkpoint (DCP, torch.distributed.checkpoint) works this way. It “produces multiple files per checkpoint, with at least one per rank”, plus a global .metadata file. [Certain] On disk a single-rank DCP checkpoint looks like .metadata and __0_0.distcp (checked with PyTorch 2.8). [Certain]
Two details make it safe and fast:
- Plan first, then write. Before writing, the ranks agree on a save plan: which rank writes which piece. This matters for replicated tensors. With data parallelism, many ranks hold the same full copy; only one of them should write it. DCP removes such duplicates in its planning step. [Certain]
- Commit at the end. A crash during a save must never leave a half-written checkpoint that looks valid. The metadata file is written last, after all ranks report success. A common pattern: write into
step-N.tmp/and rename it tostep-N/only when complete. On restart, a folder without metadata is ignored. [Likely]
- input: the local state dict of rank \(r\) (its shards of weights and optimizer state), step \(N\)
- build a local plan: list every local piece as (tensor name, global offset, length) // no data yet
- gather all local plans to rank 0; rank 0 removes duplicates (replicated tensors are written once) and sends back a final plan
- write my pieces from the final plan to my own file in
step-N.tmp/// all ranks in parallel - barrier: wait until every rank reports success
- rank 0 writes
.metadata(for every tensor: global shape, and which file holds which piece) and renames the folder tostep-N/// the commit - delete old checkpoints, but keep the last \(k\) complete ones
Loading on a different number of GPUs: resharding
You will often load a checkpoint on a different layout than you saved it: a node died and you restart on fewer nodes, you move from 256 to 512 GPUs, you change from FSDP to FSDP plus tensor parallelism, or you load a pretraining checkpoint for fine-tuning on 8 GPUs. DCP handles this “load-time resharding”, which “enables saving in one cluster topology and loading into another”. [Certain]
The trick is that the metadata talks about global positions, not ranks. For a 1-D sharded tensor of \(n\) elements, saved by \(W\) ranks and loaded by \(W'\) ranks:
- Read the symbols
- \(n\)
- number of elements in the (flattened) tensor
- \(W, W'\)
- number of ranks at save time and at load time
- \([\ell_k, h_k)\)
- the global range stored in saved file \(k\) (from the metadata); \([\ell_{r'}, h_{r'})\) is the range new rank \(r'\) needs
- \(\max(\ell_{r'}, \ell_k) \lt \min(h_{r'}, h_k)\)
- the two ranges overlap; the overlap is exactly the part to copy
Loading is in place: you first build the model on the new layout (each rank allocates its own new shards, empty), and DCP fills those tensors from the files. [Certain] So the load needs no extra copy of the model in memory. Try it in Figure 1.
How to read Figure 1
Start with 4 → 3. Each old file holds 6 elements; each new rank needs 8. New rank 1 needs elements 8 to 15, so it reads elements 8 to 11 from file 1 and 12 to 15 from file 2. Every new rank reads 2 files, and no element is read twice. Now pick 4 → 8: each new rank reads half of one file. Pick 4 → 1: the single rank reads all 4 files (this is how you turn a sharded checkpoint into one file). Pick 4 → 4: every rank reads exactly its own file; this is the fast common case of a restart on the same layout. Real DCP does the same per tensor and also for 2-D pieces (for example, column shards from tensor parallelism).
- input: checkpoint folder
step-N/; the model and optimizer already built on the new layout (empty local shards) - read
.metadata: for every tensor, its global shape and the list of saved pieces (file, global offset, length) - for every local shard I own, with global range \([\ell, h)\):
- for every saved piece \([\ell_k, h_k)\) of the same tensor: if they overlap (51.3), read the overlap from file \(k\) and copy it into my shard // in place
- restore step counter, scheduler, data loader position and RNG states // the non-tensor state
- check: every element of every local shard was written exactly once
Asynchronous checkpoints: hide the save behind training
Even a parallel save stops training. A synchronous save copies the state from GPU to CPU memory and writes it to storage while all GPUs wait. An asynchronous save splits the work in two:
- Staging (blocking, short): copy the state from GPU memory into CPU memory. Training must wait for this, because the next optimizer step will change the weights.
- Writing (in the background, long): a background thread writes the CPU copy to storage while the GPUs already train the next steps.
This is overlap again: slow I/O runs while the GPUs compute. PyTorch’s dcp.async_save works this way: it copies the state dict to staging storage (CPU memory by default) and then “calls the save in a separate thread”. [Certain] torchtitan reports that async checkpointing cuts the checkpoint overhead by 5 to 15 times compared with synchronous DCP, for Llama 3.1 8B. [Certain] Its option async_mode takes "disabled", "async" or "async_with_pinned_mem". [Certain] Pinned (page-locked) CPU memory makes the GPU-to-CPU copy faster, but it is a scarce resource and the buffer stays allocated between saves. [Certain]
Two costs: CPU memory (each rank needs a CPU copy of its share; the PyTorch tutorial warns that memory grows by “checkpoint_size_per_rank X number_of_ranks” on a host [Certain]), and a limit on frequency: a new save cannot start before the previous write has finished. Figure 2 shows both.
How to read Figure 2
With the defaults, a synchronous checkpoint stalls the GPUs for 0.42 s + 4.15 s = 4.57 s, which is 18.3% of the training time when you save every 5 steps. Asynchronous: only the 0.42 s copy is a stall, 1.7% overhead, about 11 times less. Now set \(k = 1\) (a save after every step) and lower the write speed to 0.1 GB/s: the 8.3 s write is longer than a 5 s step, so each new save waits for the previous one, and the async advantage shrinks. Then raise the GPU → CPU speed (pinned memory): the stall falls further. Async pays off when the write fits between two saves.
Some teams go further and keep checkpoints in memory. Gemini’s training kept “redundant in-memory copies of the model state”, so that after a hardware failure it could “rapidly recover directly from an intact model replica”; this raised the goodput of the largest job from 85% to 97%. [Certain] The idea: with data parallelism, another replica already holds the same weights; reading them over the network is much faster than reading 800 GB from a file system.
Failures at scale
Why do big jobs fail so often? Because a synchronous training job is a chain: if any one of 16,384 GPUs, or any cable, switch or host between them, stops, all the others soon wait at the next collective. As Meta wrote, “a single GPU failure may require a restart of the entire job”. [Certain] If each component fails rarely and independently, the job’s mean time to failure (MTTF) shrinks in proportion to the number of components:
- Read the symbols
- \(M_{\text{job}}\)
- mean time between failures of the whole job
- \(N_{\text{nodes}}\)
- number of nodes (machines) in the job; 8 GPUs per node, so 16,384 GPUs are 2,048 nodes
- \(r_f\)
- failure rate of one node, per day. Meta measured 6.50 failures per 1,000 node-days on its RSC-1 research cluster. [Certain]
With that rate, Meta’s reliability study projects an MTTF of 1.8 hours for a 16,384-GPU job and 0.23 hours (14 minutes) for a 131,072-GPU job; it observed 7.9 hours for 1,024-GPU jobs. [Certain] The Llama 3 numbers agree in size: 419 unexpected interruptions in 54 days is \(54 \times 24 / 419 = 3.09\) hours between unexpected stops. [Certain] Turned around: one GPU slot (GPU plus its share of host and network) fails on average only once every \(3.09 \times 16{,}384\) hours \(\approx 5.8\) years. Each part is reliable. The job is not.
| Cause (Llama 3 405B, 54 days) | Share of the 419 unexpected interruptions |
|---|---|
| Faulty GPU | 30.1% |
| GPU HBM3 memory | 17.2% |
| Software bug | 12.9% |
| Network switch or cable | 8.4% |
| Unplanned host maintenance | 7.6% |
| GPU SRAM memory; GPU system processor | 4.5%; 4.1% |
| NCCL watchdog timeouts (cause unknown) | 1.7% |
| Silent data corruption | 1.4% (6 events) |
From Table 5 of the Llama 3 paper. [Certain] About 78% of the unexpected interruptions were confirmed or suspected hardware issues, and GPU issues alone were 58.7%. Only three needed significant manual work; automation handled the rest. [Certain] (The paper prints 148 interruptions for “Faulty GPU” next to 30.1%; 148 / 419 is 35.3%, so one of the two numbers has a typo. The category percentages add up to the 58.7% only with 30.1%.)
Failures come in four kinds, from easy to hard:
- Crash. A process dies (a GPU “falls off the bus”, out of memory, a Python exception). The launcher sees it at once.
- Hang. A rank stops making progress, for example a stalled NVLink transfer. Everyone else waits forever in the next collective. PyTorch’s NCCL process group has a watchdog with a timeout: by default 10 minutes for NCCL (30 for other backends), after which the collective is aborted and the process crashes. [Certain] Ten minutes of 16,384 idle GPUs is expensive, so large teams lower it and add heartbeats. [Likely]
- Straggler. One slow GPU (overheating, throttled clock, a bad link) slows down thousands of others, because every collective waits for the slowest rank. Nothing crashes. Llama 3’s team noted that “even a single straggler can slow down thousands of other GPUs”. [Certain]
- Silent data corruption. The hardware returns a wrong number with no error. The job continues, maybe with a damaged model. Its own section is below.
How often to checkpoint
Checkpoint too rarely, and each failure throws away hours of work. Checkpoint too often, and you spend your time saving. Let \(\tau\) be the time between checkpoints. Each checkpoint stalls training for \(\delta\). When a failure comes, you lose the work since the last checkpoint (on average half an interval, \(\tau/2\)) plus the restart time \(R\) (find the failure, get a replacement node, start the processes, load the checkpoint). Failures arrive every \(M\) on average. The fraction of wall time wasted is then, to first order,
The first term falls with \(\tau\), the second grows. Setting the derivative \(-\delta/\tau^2 + 1/(2M)\) to zero gives Young’s formula (1974): [Certain]
- Read the symbols
- \(\tau, \tau^{*}\)
- the checkpoint interval, and its best value
- \(\delta\)
- how long one checkpoint stalls training. For an async save this is only the staging time.
- \(M\)
- mean time between failures of the job, formula (51.4)
- \(R\)
- restart cost: detection, rescheduling, process start-up, checkpoint load, warm-up (for example,
torch.compileagain)
Two lessons hide in (51.6). The waste grows only with the square root of the stall \(\delta\), so making saves 100 times cheaper makes the best waste 10 times smaller. And the restart cost \(R\) does not depend on \(\tau\) at all: no checkpoint interval can fix a slow restart. The formula is a first-order estimate; it is good when \(\tau\) and \(\delta\) are much smaller than \(M\). Daly (2006) gives a more exact version. [Certain]
Take \(M = 3.09\) h \(= 11{,}135\) s (the Llama 3 rate of unexpected stops) and a restart cost \(R = 10\) min \(= 600\) s (an assumption).
Synchronous saves, \(\delta = 60\) s: \(\tau^{*} = \sqrt{2 \times 60 \times 11{,}135} = 1{,}156\) s \(\approx 19\) min. Waste \(= \sqrt{120 / 11{,}135} + 600/11{,}135 = 10.4\% + 5.4\% = 15.8\%\).
Async saves, \(\delta = 5\) s: \(\tau^{*} = \sqrt{2 \times 5 \times 11{,}135} = 334\) s \(\approx 5.6\) min. Waste \(= 3.0\% + 5.4\% = 8.4\%\).
Now the restart cost is the larger part. To beat 90% ETTR, as Llama 3 did, you need cheap saves and fast restarts. That is exactly what Meta says they worked on: “we reduced job startup and checkpointing time”. [Certain] And note what a bad choice costs: saving every 3 hours with \(\delta = 60\) s gives \(w = 54\%\) by (51.5) (the formula is rough there, but the message is clear).
How to read Figure 3
The defaults are a 16,384-GPU job at Meta’s measured failure rate: MTTF 1.80 h. Saving every hour with a 60 s stall wastes 38.6%. Press the button: τ* is 14.7 min and the waste drops to 22.8%. Now set δ to 5 s (async): the optimum moves to 4.2 min and the waste to 13.2%, most of it the 10 min restart. Set R to 2 min and you are near 6%. Slide the GPUs to 131,072: MTTF falls to 13.5 min, and even good settings waste a lot. That is why the biggest jobs need in-memory checkpoints and very fast restarts, or training that continues without the failed part.
Restart and elastic training
Most frameworks handle a failure with stop everything and restart from the last checkpoint. It sounds crude, but it is simple and correct. The work is in making each stage fast:
- Detect. Crashes are seen at once. Hangs are seen by the watchdog timeout. Slow steps and NaN losses need your own monitoring.
- Localize. Which of 16,384 GPUs is the culprit? PyTorch’s NCCL flight recorder keeps the last collectives of every rank in a ring buffer and dumps them on a timeout; the rank that never entered the collective is the suspect. Llama 3 used it heavily. [Certain]
- Replace and restart. Keep a few hot spare nodes ready, so a restart does not wait for a repair. The launcher forms a new group by rendezvous: the surviving and new workers meet at a known address, get new ranks, and build the process groups again.
Elastic training goes one step further: the job may continue with a different number of nodes. PyTorch’s launcher supports it with torchrun --nnodes=MIN:MAX --max-restarts=K. Its behavior is still stop-and-restart: “On failures or membership changes ALL surviving workers are killed immediately”, a new worker group is formed with new RANK and WORLD_SIZE, and your script must reload its last checkpoint. [Certain] That is why resharding matters: the new world size is different.
Changing the number of data-parallel ranks changes the math of training, not just the layout. If each rank keeps its micro-batch, the global batch shrinks, and the learning rate and schedule that were tuned for the old batch are now wrong. The usual fix is to keep the global batch fixed and change gradient accumulation (session 39): with \(B_{\text{global}} = W \cdot b \cdot A\), losing ranks means more accumulation steps \(A\) per optimizer step. The run becomes slower but stays the same run. [Likely]
- input: checkpoint folder, interval \(\tau\) (in steps), target global batch \(B\)
- join the rendezvous; get rank \(r\) and world size \(W\); build the model and optimizer sharded over \(W\)
- \(A \leftarrow B / (W \cdot b)\) // keep the global batch fixed when \(W\) changes
- if a complete checkpoint exists: load the newest one with resharding (Algorithm 2); else initialize from a seed
- loop: wait for the previous async write if a new save is due; run one optimizer step; log loss, gradient norm, step time
- if the loss is NaN or a health check fails: stop and let the launcher restart from the last good checkpoint
- if \(\text{step} \bmod \tau = 0\): async save (Algorithm 1), staging only blocks
- on any crash or timeout, the launcher kills all workers, replaces the bad node, and starts again at line 2
Stop-and-restart is not the only design. Newer systems try to keep the healthy part of the job running: for example, when one data-parallel replica fails, the others continue for a few steps with a smaller batch while it rejoins. PyTorch’s torchft project explores this per-step fault tolerance. [Likely] It is still less common than plain restarts in large pretraining. [Speculation]
Silent data corruption
Silent data corruption (SDC) means the hardware gives a wrong result and raises no error: no crash, no ECC alarm, no exception. A defective core may compute a wrong multiply once in a billion operations, often only at a certain temperature or voltage. In a training run, a wrong number in one GPU’s gradient gets all-reduced into everyone’s weights. The job continues, but the model may be damaged, or a loss spike appears hours later with no visible cause.
How common is it? Meta’s fleet study “Silent Data Corruptions at Scale” found hundreds of defective CPUs among hundreds of thousands of machines. [Certain] For Gemini, Google wrote that “we can expect SDC events to impact training every week or two”. [Certain] In Llama 3’s 54 days, 6 interruptions were attributed to SDC. [Certain] Rare per chip, but certain for a large job.
Detection tools, from cheap to expensive:
- Watch the training signals. Sudden spikes in the loss or the gradient norm, or NaN and Inf values. Cheap, but it cannot tell a hardware error from a normal loss spike (session 36).
- Compare replicas. With data parallelism, all replicas should hold bit-identical weights after each step: they apply the same averaged gradient to the same weights. A checksum of the parameters, compared across replicas from time to time, catches a replica that drifted. [Likely]
- Deterministic replay. If the training step is deterministic, you can run the same step again (on the same or other hardware) and compare the result bit by bit. A mismatch points to a faulty device. Gemini’s team detected and removed faulty hardware with techniques “that exploit deterministic replay”, plus “proactive SDC scanners on idle machines”. [Certain]
- Scan machines. Run known computations with known answers on idle or suspect nodes, and remove nodes that fail.
A checkpoint policy helps too: keep several older checkpoints, not only the latest. If corruption is found late, you need a checkpoint from before it happened.
MFU: how to compute it correctly
Recall from session 42: MFU is the model’s useful FLOP/s divided by the peak. For one GPU, in a data-parallel job,
- Read the symbols
- \(F_{\text{token}}\)
- useful training FLOPs per token: forward plus backward, from the model shape, not from a hardware counter
- \(N_{\text{ne}}\)
- the parameters that do matrix multiplications: all parameters minus the input embedding table (a lookup does no FLOPs). The output head stays in.
- \(12\,L\,d\,T\)
- attention scores and the weighted sum of values, forward and backward, for \(L\) layers of width \(d\) and sequence length \(T\)
- \(P\)
- the dense peak of one GPU for the data type you train in (989 TFLOP/s BF16 for H100 SXM)
The details are conventions, and they change the answer, so say which ones you use. torchtitan, for example, excludes the embedding table from \(N\), counts attention as \(12 L d T\) without halving it for the causal mask, and does not count the extra matmul that FlashAttention recomputes in the backward pass. [Certain] Recomputed FLOPs (activation checkpointing) count in HFU (hardware FLOPs utilization), never in MFU.
You train Llama 3 8B on H100s with \(T = 8{,}192\). Each GPU processes 2 sequences per step (16,384 tokens), and one step takes 2.0 s: 8,192 tokens/s per GPU.
Only \(6N\) with all 8,030,261,248 parameters: \(6N = 4.82 \times 10^{10}\), so \(8{,}192 \times 4.82 \times 10^{10} / 989 \times 10^{12} = 39.9\%\).
torchtitan convention: \(N_{\text{ne}} = 8{,}030{,}261{,}248 - 128{,}256 \times 4{,}096 = 7{,}504{,}924{,}672\). Attention: \(12 \times 32 \times 4{,}096 \times 8{,}192 = 1.29 \times 10^{10}\). \(F_{\text{token}} = 4.50 \times 10^{10} + 1.29 \times 10^{10} = 5.79 \times 10^{10}\), and MFU \(= 48.0\%\).
\(6N\) with all parameters plus attention: 50.6%. Same job, same speed, 40% to 51%. Before you compare your MFU with a paper, check the formula. torchtitan reports 33% to 42% MFU for Llama 3.1 8B on 8 or 128 H100s without Float8. [Certain]
How to read an MFU number
- It is a ratio, so check both parts. Use the dense peak, never the “with sparsity” number, and the peak of the data type you really use. With FP8 matmuls the right peak is unclear (some layers run in BF16), and torchtitan does not report MFU for Float8 runs because “the definition of MFU under such scenario is not well-defined”. [Certain] Compare tokens per second instead.
- MFU is measured while running. It says nothing about the hours lost to failures. Combine it with ETTR (51.1): \(40\%\ \mathrm{MFU} \times 90\%\ \mathrm{ETTR} = 36\%\) of the paid compute becomes training.
- Big runs are in the 35–50% range, not 80%. PaLM 540B reached 46.2%, Llama 3 405B 38% to 43% (session 42). [Certain] If you see 70% for a dense model, suspect the formula; if you see 15%, look for exposed communication, small micro-batches, or a slow data loader.
- MFU is per configuration. Short sequences, small models and small micro-batches give lower MFU because matmuls are smaller and fixed costs weigh more. Report it with model, sequence length, batch per GPU and number of GPUs.
- input: model shape \((N_{\text{ne}}, L, d)\), sequence length \(T\), peak \(P\), GPUs \(n\)
- \(F_{\text{token}} \leftarrow 6 N_{\text{ne}} + 12 L d T\) // state the convention
- skip the first steps (compile, warm-up); then time \(K\) steps with
torch.cuda.synchronize()before reading the clock - tokens/s per GPU \(\leftarrow\) (tokens in \(K\) global batches, padding excluded) / (time \(\times\, n\))
- MFU \(\leftarrow\) tokens/s per GPU \(\times F_{\text{token}} / P\)
- HFU \(\leftarrow\) the same with the FLOPs actually run (add recomputed forward passes)
- log both every few hundred steps; a drop in MFU over time means stragglers, thermal throttling or a slower data loader
Overlapping communication with compute
Where does MFU go? A big part is communication that the GPU waits for. Session 43 and session 46 showed the mechanism: run collectives on a second CUDA stream, and start them early enough. Here is the question that decides whether it works: is each transfer shorter than the compute it hides behind?
Take FSDP with \(F\) GPUs. Before block \(i\) runs, its weights (\(\psi\) parameters, 2 bytes each in BF16) are all-gathered. While that happens, block \(i-1\) computes on the \(b\) tokens that this GPU holds. The all-gather takes \(2\psi\,(F-1)/(F B)\) seconds at bus bandwidth \(B\); the forward matmuls take \(2\psi b/(P u)\) seconds at compute efficiency \(u\). The transfer is hidden when
- Read the symbols
- \(b\)
- tokens per GPU per micro-step (micro-batch size × sequence length)
- \(B\)
- bus bandwidth per GPU for this collective (for example, about 50 GB/s for one 400 Gb/s network card per GPU)
- \(u\)
- the fraction of peak that the matmuls reach while they run
- \(\psi\)
- parameters of one block. It cancels out: the condition does not depend on model size.
The backward pass gives the same condition: twice the compute (\(4\psi b\)) and twice the transfers (all-gather and reduce-scatter). With \(P = 989\) TFLOP/s, \(u = 0.6\), \(B = 50\) GB/s and \(F = 64\), you need \(b \ge 11{,}700\) tokens per GPU. Below that, FSDP is communication-bound: more GPUs with smaller batches each will not train faster. This is the same kind of “compute per byte” argument as the roofline in session 42, applied to the network; the scaling book How to Scale Your Model uses it for every parallelism. [Likely] Figure 5 runs the full schedule.
How to read Figure 5
Defaults: \(b = 16{,}384\), 50 GB/s, \(u = 0.6\). With overlap the step takes 1.625 s for 1.599 s of compute: MFU 59.1%, close to the 60% ceiling set by \(u\). Uncheck overlap: 0.82 s of communication is exposed and MFU falls to 39.6%. Turn overlap back on and slide \(b\) down to 4,096: now each all-gather (8.6 ms) is longer than a block’s forward (4.2 ms), and MFU falls to 29%. Change the bandwidth to 450 GB/s (NVLink-class) and the curve stays near 58% down to about 1,000 tokens. The knee sits a little to the left of the \(b_{\min}\) line: (51.9) counts only the weight matmuls, but attention adds compute (it needs no gathered weights) that also hides transfers. So (51.9) is a safe, slightly pessimistic bound.
Overlap appears in every part of a training system:
| What is hidden | Behind what | Mechanism |
|---|---|---|
| DDP gradient all-reduce | the rest of the backward pass | gradient buckets that start as soon as they are full (session 44) |
| FSDP all-gather / reduce-scatter | the previous / next block’s compute | prefetching, formula (51.9) |
| Tensor-parallel all-gather and reduce-scatter | pieces of the same matmul | split the matmul into chunks; torchtitan’s async TP raised Llama 3.1 70B on 256 H100s from 897 to 1,010 tokens/s per GPU (+12.6%), on top of compile and Float8 [Certain] |
| Pipeline sends between stages | other micro-batches | 1F1B and interleaved schedules (session 49) |
| Checkpoint writes | the next training steps | async save (Figure 2) |
| Data loading and host-to-device copies | the current step | loader workers, prefetching, pinned memory, non-blocking copies |
Overlap is not free. Communication kernels use some SMs and memory bandwidth, so the compute next to them runs a bit slower; and prefetching holds more data in memory at once. [Likely] The only way to see if overlap really happens is a profiler trace (session 50, session 52).
The code: sharded save, resharded load, exact resume
This program runs on a CPU in under a second and needs only PyTorch. Part 1 and 2 are a tiny version of Algorithms 1 and 2: four “ranks” each save their slice of a flat parameter vector, plus a metadata file and an atomic rename; then three new “ranks” load it by reading only the overlapping pieces. Part 3 shows why the optimizer and RNG states belong in a checkpoint: a crash-and-resume run matches an uninterrupted run bit for bit, and a weights-only resume does not.
import json, os, shutil, torch, torch.nn as nn torch.manual_seed(0) ROOT = "ckpt_demo"; shutil.rmtree(ROOT, ignore_errors=True) def make_model(): torch.manual_seed(0) return nn.Sequential(nn.Linear(8, 16), nn.GELU(), nn.Linear(16, 4)) def flat(model): # one long vector, like a FlatParameter return torch.cat([p.detach().reshape(-1) for p in model.parameters()]) def shard_range(n, world, rank): # equal slices, last one may be shorter size = -(-n // world) # ceil(n / world) lo = min(rank * size, n); hi = min(lo + size, n) return lo, hi # ---- 1. save: each of W "ranks" writes only its own slice ---- def save_sharded(vec, world, step): d = f"{ROOT}/step-{step}.tmp"; os.makedirs(d) files = [] for r in range(world): # in a real job these run in parallel lo, hi = shard_range(vec.numel(), world, r) torch.save(vec[lo:hi].clone(), f"{d}/rank{r}.pt") files.append({"file": f"rank{r}.pt", "lo": lo, "hi": hi}) json.dump({"numel": vec.numel(), "files": files}, open(f"{d}/.metadata", "w")) os.rename(d, f"{ROOT}/step-{step}") # commit: the folder appears only when complete return f"{ROOT}/step-{step}" # ---- 2. load on a DIFFERENT world size: read only overlapping pieces ---- def load_resharded(path, world, rank): meta = json.load(open(f"{path}/.metadata")) lo, hi = shard_range(meta["numel"], world, rank) out, reads = torch.empty(hi - lo), [] for f in meta["files"]: a, b = max(lo, f["lo"]), min(hi, f["hi"]) if a < b: # this saved file overlaps my new slice piece = torch.load(f"{path}/{f['file']}") out[a - lo:b - lo] = piece[a - f["lo"]:b - f["lo"]] reads.append(f["file"]) return out, reads model = make_model(); vec = flat(model) path = save_sharded(vec, world=4, step=100) parts = [] for r in range(3): piece, reads = load_resharded(path, world=3, rank=r) print(f"new rank {r} of 3 reads {reads}") parts.append(piece) print("numel", vec.numel(), "resharded 4 -> 3 equal:", torch.equal(torch.cat(parts), vec)) # ---- 3. exact resume: model + optimizer + RNG + step must all be saved ---- def batch(gen): x = torch.randn(32, 8, generator=gen); return x, x[:, :4].sin() def train(model, opt, gen, steps): for _ in range(steps): x, y = batch(gen) loss = nn.functional.mse_loss(model(x), y) opt.zero_grad(); loss.backward(); opt.step() return loss.item() def fresh(): m = make_model() o = torch.optim.AdamW(m.parameters(), lr=1e-2, weight_decay=0.1) g = torch.Generator().manual_seed(1) return m, o, g m, o, g = fresh(); train(m, o, g, 6); straight = flat(m) m, o, g = fresh(); train(m, o, g, 3) torch.save({"model": m.state_dict(), "optim": o.state_dict(), "rng": g.get_state(), "step": 3}, f"{ROOT}/full.pt") del m, o, g # the job "crashes" m, o, g = fresh() ck = torch.load(f"{ROOT}/full.pt") m.load_state_dict(ck["model"]); o.load_state_dict(ck["optim"]); g.set_state(ck["rng"]) train(m, o, g, 6 - ck["step"]) print("resume is bit-exact:", torch.equal(flat(m), straight)) m, o, g = fresh(); train(m, o, g, 3) m2 = make_model(); m2.load_state_dict(m.state_dict()) # forgot optimizer + RNG o2 = torch.optim.AdamW(m2.parameters(), lr=1e-2, weight_decay=0.1) g2 = torch.Generator().manual_seed(1) train(m2, o2, g2, 3) print("weights-only resume is exact:", torch.equal(flat(m2), straight), " max diff %.4f" % (flat(m2) - straight).abs().max().item()) # new rank 0 of 3 reads ['rank0.pt', 'rank1.pt'] # new rank 1 of 3 reads ['rank1.pt', 'rank2.pt'] # new rank 2 of 3 reads ['rank2.pt', 'rank3.pt'] # numel 212 resharded 4 -> 3 equal: True # resume is bit-exact: True # weights-only resume is exact: False max diff 0.0342
With a real sharded model, you do not write this yourself: you use DCP. It also works in a single process (no process group), which is handy for tests and for converting checkpoints. This was run with PyTorch 2.8:
import os, shutil, torch, torch.nn as nn import torch.distributed.checkpoint as dcp torch.manual_seed(0) model = nn.Sequential(nn.Linear(8, 16), nn.GELU(), nn.Linear(16, 4)) opt = torch.optim.AdamW(model.parameters(), lr=1e-2) model(torch.randn(4, 8)).sum().backward(); opt.step() # create Adam states shutil.rmtree("dcp_ckpt", ignore_errors=True) state = {"model": model.state_dict(), "optim": opt.state_dict()} dcp.save(state, checkpoint_id="dcp_ckpt/step-1") # no process group: one rank print(sorted(os.listdir("dcp_ckpt/step-1"))) model2 = nn.Sequential(nn.Linear(8, 16), nn.GELU(), nn.Linear(16, 4)) target = {"model": model2.state_dict()} # load IN PLACE into these tensors dcp.load(target, checkpoint_id="dcp_ckpt/step-1") print(all(torch.equal(a, b) for a, b in zip(model.state_dict().values(), model2.state_dict().values()))) # ['.metadata', '__0_0.distcp'] # True
In a real FSDP2 job, the state dict holds DTensor shards, and the same two calls save and load in parallel on all ranks. Use get_state_dict / set_state_dict from torch.distributed.checkpoint.state_dict to get model and optimizer state dicts that can be resharded. [Certain] To make one file for inference, convert with python -m torch.distributed.checkpoint.format_utils dcp_to_torch step-N/ model.pt. [Certain]
Where this is used
| Setting | What matters most | Typical choice |
|---|---|---|
| Pretraining an LLM on thousands of GPUs | ETTR and MFU at once | sharded async checkpoints every few minutes to tens of minutes, hot spares, flight recorder, MFU logging [Likely] |
| Fine-tuning on one 8-GPU node | simple, portable checkpoints | DCP during training, one safetensors file at the end for serving |
| Changing the cluster mid-run (more nodes arrive, a rack fails) | loading on a new layout | DCP resharding; keep the global batch fixed with accumulation |
| Vision-language model training (vision encoder + LLM) | parts with different sharding; frozen parts | save only the trainable parts often; the frozen vision tower is saved once |
| Video models (video classification, action recognition, video-text retrieval encoders for Video-RAG) | long sequences of frame tokens; data loading is often the bottleneck | overlap decoding and loading with compute; checkpoint the data position as a global sample index, because video datasets are huge and expensive to replay [Likely] |
| Long RL or post-training runs | several models (policy, reference, reward) in one job | one checkpoint per model plus the shared step and data state |
| Comparing training systems or hardware | a fair speed number | MFU with a stated formula, or tokens/s per GPU for FP8 runs |
Common mistakes
- Saving only the weights. The resume then starts Adam from zero and repeats or skips data. The code above shows the drift. Save optimizer, scheduler, step, data position and RNG states.
- Gathering the full state to rank 0 for every save. It is slow, it can run rank 0 out of CPU memory, and all other GPUs wait. Save sharded; gather only once at the end for export.
- No atomic commit. A crash during a save leaves a broken “latest” checkpoint, and the restart crashes too. Write the metadata last, rename at the end, and keep the last few complete checkpoints.
- Choosing the checkpoint interval by habit (“every 1,000 steps”). Compute it from (51.6) with your real stall time and failure rate, and recompute when the job grows.
- Ignoring the restart cost. If a restart takes 20 minutes (scheduling, compile, loading), no checkpoint interval saves you. Measure \(R\) and work on it.
- Storing per-rank data positions and then changing the world size. Store a global position.
- Reporting MFU without the formula, with the sparse peak, or counting recomputation. The same run can be 40% or 51% (worked example above).
- Assuming overlap happens. If \(b \lt b_{\min}\) (51.9), or a collective is waited on too early, communication is exposed. Check the trace.
Interview questions
Why not just call torch.save on rank 0 for a model sharded over 1,024 GPUs?basic
What must a checkpoint contain so that training resumes exactly?basic
How does load-time resharding work?core
How often should you checkpoint?core
Why does a 16,384-GPU job fail every few hours if each GPU is reliable?basic
What is asynchronous checkpointing, and what does it cost?core
What is silent data corruption, and how do you detect it in training?advanced
A node dies and you restart on fewer GPUs. What changes besides the sharding?advanced
--nnodes=MIN:MAX) kills all workers and restarts them with new ranks, so the script must reload and reshard.How do you compute MFU, and which choices change the number?core
MFU, HFU, tokens per second, ETTR: which one would you report, and when?advanced
When can FSDP hide its communication completely?advanced
A 4,096-GPU job hangs with no error. How do you debug it?core
Exercises
- A 13B model is trained with AdamW in mixed precision. How large is a full training checkpoint, and how large is a BF16 weights-only export? How much does each of 256 ranks write?
- A tensor of 30 elements was saved by 5 ranks and is loaded by 3. Which elements does new rank 1 need, and which files and slices does it read?
- A 4,096-GPU job (8 GPUs per node) runs on a cluster with 6.5 node failures per 1,000 node-days. Compute the job MTTF, Young’s interval for a 30 s stall, and the waste with a 5 min restart cost.
- In exercise 3, by what factor must the stall shrink to halve the \(\sqrt{2\delta/M}\) part of the waste?
- Your Llama 3 8B run (\(T = 8192\)) on H100s reaches 5,645 tokens/s per GPU. Compute MFU with torchtitan’s convention (\(F_{\text{token}} = 5.79 \times 10^{10}\)).
- Using (51.9) with \(P = 989\) TFLOP/s, \(u = 0.5\) and \(F = 64\): what is \(b_{\min}\) at 25 GB/s? If you train with \(T = 4096\), how many sequences per GPU per micro-step do you need?
Answerscheck
Key takeaways
- Useful work = peak × time × MFU × ETTR (51.1). Checkpoints and fast restarts raise ETTR; overlap raises MFU.
- A training checkpoint holds weights, optimizer state, step, scheduler, data position and RNG states: about 12 bytes per parameter (51.2), 847 GB for a 70B model.
- Sharded checkpoints write every rank’s pieces in parallel plus a metadata file written last; the metadata uses global positions, so any layout can load any checkpoint by reading the overlaps (51.3).
- Async saves stall only for the GPU-to-CPU copy; the write runs in the background (5 to 15 times less overhead in torchtitan).
- Job MTTF falls in proportion to the number of nodes (51.4): about 1.8 h at 16,384 GPUs. Young’s interval \(\sqrt{2\delta M}\) (51.6) balances saving against lost work; the restart cost adds \(R/M\) on top.
- Silent data corruption is rare per chip but certain at scale; catch it with replica checksums, deterministic replay and scanners, and keep older checkpoints.
- State your MFU formula (51.7). FSDP can hide its communication only if each GPU has at least \(b_{\min} = (Pu/B)(F-1)/F\) tokens per micro-step (51.9).
Key formulas
References for this session
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- PyTorch (code). torchtitan: reference large-scale training with FSDP2
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- PyTorch (code). torchtitan: reference large-scale training with FSDP2
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Liang et al. (2024). TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
- PyTorch (docs). Distributed Checkpoint (torch.distributed.checkpoint): save, load, async_save, get_state_dict, format_utils
- PyTorch (docs). Asynchronous Saving with Distributed Checkpoint (DCP) (recipe)
- Kokolis et al. (2024). Revisiting Reliability in Large-Scale Machine Learning Research Clusters
- Gemini Team, Google (2023). Gemini: A Family of Highly Capable Multimodal Models
- Dixit et al. (2021). Silent Data Corruptions at Scale
- Young (1974). A first order approximation to the optimum checkpoint interval
- Daly (2006). A higher order estimate of the optimum checkpoint interval for restart dumps
- Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
- PyTorch (docs). torchrun (Elastic Launch): elastic nnodes MIN:MAX, max-restarts, rendezvous