LLM & VLM courseBlog
51Track 3: Pretraining and scale · session 21 of 22

Distributed checkpoints, fault tolerance and MFU

A pretraining run on thousands of GPUs stops every few hours: a GPU dies, a link hangs, a chip computes a wrong number. This session shows how to save and load a model that is split over many GPUs, how often to save, how to restart (also on a different number of GPUs), how to catch silent data corruption, and how to compute and read MFU, the number that says how much of the GPU’s speed becomes useful training.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 42, 43 and 46 (GPU architecture, collective communication, FSDP); session 38 (memory math) helps

What you will learn

  • What a training checkpoint must contain to resume exactly, and why it is about 12 bytes per parameter (847 GB for a 70B model)
  • How sharded checkpoints (PyTorch DCP) save in parallel, commit safely, and reshard at load time onto a different number of GPUs
  • How asynchronous checkpoints hide the write behind training, and what they cost in CPU memory
  • Why a 16,384-GPU job fails every 2 to 3 hours, with the real Llama 3 and Meta cluster numbers, and how crashes, hangs, stragglers and silent data corruption differ
  • Young’s formula for the best checkpoint interval, and why the restart cost often matters more
  • How to compute MFU correctly (which N, which peak, attention, recomputation), and how to read it together with ETTR (goodput)
  • When communication can hide behind compute: the minimum tokens per GPU for FSDP, b_min = (P u / B)(F − 1)/F

Why this matters

A large pretraining run is not one long program that starts and ends. It is a chain of many short runs, cut by crashes, hangs and maintenance. During 54 days of Llama 3 405B pretraining on up to 16,384 H100 GPUs, Meta saw 466 job interruptions: 47 planned and 419 unexpected. That is one unexpected stop about every 3.1 hours. [Certain] Still, they spent more than 90% of the time on useful training. [Certain]

That 90% does not happen by luck. It needs three things, and they are the topic of this session:

  • Distributed checkpoints. A checkpoint is a saved copy of the full training state. With thousands of GPUs, each GPU holds only a piece of the state (session 46, session 47). Saving and loading must also be split over all GPUs, and you must be able to load on a different number of GPUs than you saved on.
  • Fault tolerance. Find a failure fast, remove the bad machine, restart, and lose as little work as possible. This includes the hardest failure: silent data corruption, where a chip computes a wrong number and reports no error.
  • MFU (model FLOPs utilization). When the job runs, how much of the GPU’s peak speed becomes useful training? You met the formula in session 42. Here you learn to compute it correctly, to read it, and to raise it by overlapping communication with compute.

These three ideas are linked by one number. The useful work you get from a cluster is roughly

\[ \text{useful FLOPs} = \underbrace{n_{\text{GPU}}\, P\, t_{\text{wall}}}_{\text{what you pay for}} \times \underbrace{\mathrm{MFU}}_{\text{speed while running}} \times \underbrace{\mathrm{ETTR}}_{\text{fraction of time running}} \tag{51.1} \]
Read the symbols
\(n_{\text{GPU}}, P\)
number of GPUs, and the dense peak FLOP/s of one GPU (989 TFLOP/s for an H100 in BF16 [Certain])
\(t_{\text{wall}}\)
wall-clock time: the time on the clock on the wall, including all stops
MFU
model FLOPs utilization: useful model FLOP/s divided by peak FLOP/s, measured while the job runs
ETTR
effective training time ratio: the time spent on useful training divided by the wall-clock time. Also called goodput (good + throughput). Meta’s Llama 3 paper calls it “effective training time”.

Checkpoints and fault tolerance raise ETTR. Overlap raises MFU. A run with 45% MFU and 70% ETTR does less work than a run with 40% MFU and 95% ETTR.

What a checkpoint must contain

A checkpoint is useful only if training can continue from it as if nothing had happened. Saving only the weights is not enough. You need everything that changes during training:

PartWhy it is neededSize (mixed-precision AdamW)
Model weights (FP32 master copy)the model itself4 bytes per parameter
Optimizer state (Adam \(m\) and \(v\))without it, Adam starts again with zero momentum and a wrong step size; the loss usually jumps8 bytes per parameter
Step counter and LR scheduler statethe learning rate depends on the step (session 4)a few bytes
Data loader positionwithout it you train on the same data twice, or skip datasmall
Random number generator states (per rank)dropout masks and data shuffling must continue the same sequencesmall
Loss scaler (FP16 only), config, code versionto rebuild the exact same runsmall

The gradients are not saved: you checkpoint between steps, when the gradients are empty. So a full training checkpoint has about 12 bytes per parameter, not the 16 bytes of training memory from session 38:

\[ S_{\text{ckpt}} \approx \Psi\,(b_{\text{w}} + b_{\text{opt}}) = 12\,\Psi \ \text{bytes}, \qquad S_{\text{rank}} = \frac{S_{\text{ckpt}}}{W} \tag{51.2} \]
Read the symbols
\(\Psi\)
number of parameters
\(b_{\text{w}}, b_{\text{opt}}\)
bytes per parameter for the weights (4 in FP32) and for the optimizer state (8 for Adam’s \(m\) and \(v\) in FP32)
\(W\)
world size: the number of ranks (GPU processes) that share the state. With full sharding each rank holds and saves \(1/W\).
Worked example: how big is a Llama 3 70B checkpoint?

From the Llama 3 70B config (vocabulary 128,256, width 8,192, 80 layers, MLP width 28,672, 8 key-value heads of size 128), the model has 70,553,706,496 parameters.

Training checkpoint: \(12 \times 70.55 \times 10^9 = 846.6\) GB. A weights-only export in BF16 (2 bytes per parameter) is 141.1 GB: six times smaller, but you cannot resume training from it.

On 1,024 GPUs with full sharding, each rank saves \(846.6 / 1024 = 0.83\) GB. Suppose the shared file system can absorb 200 GB/s in total (an assumption; real systems vary a lot). If every rank writes in parallel, the save takes about \(846.6 / 200 = 4.2\) s. If instead rank 0 gathers everything and writes one file at 2 GB/s (also an assumption), the save takes \(846.6 / 2 = 423\) s, about 7 minutes, and rank 0 needs 847 GB of CPU memory to hold it.

Sharded checkpoints: save in parallel

The simple way to checkpoint is torch.save(model.state_dict()) on rank 0. With a sharded model, this means: gather every shard to rank 0 (huge communication and memory), then write one big file with one process (slow). The worked example showed 7 minutes for 70B. At 405B it is worse, and during that time all other GPUs wait.

A sharded (or distributed) checkpoint does the opposite: every rank writes only the pieces it already owns, at the same time, to its own file. One small extra file, the metadata, records where every piece of every tensor lives. PyTorch’s Distributed Checkpoint (DCP, torch.distributed.checkpoint) works this way. It “produces multiple files per checkpoint, with at least one per rank”, plus a global .metadata file. [Certain] On disk a single-rank DCP checkpoint looks like .metadata and __0_0.distcp (checked with PyTorch 2.8). [Certain]

Two details make it safe and fast:

  • Plan first, then write. Before writing, the ranks agree on a save plan: which rank writes which piece. This matters for replicated tensors. With data parallelism, many ranks hold the same full copy; only one of them should write it. DCP removes such duplicates in its planning step. [Certain]
  • Commit at the end. A crash during a save must never leave a half-written checkpoint that looks valid. The metadata file is written last, after all ranks report success. A common pattern: write into step-N.tmp/ and rename it to step-N/ only when complete. On restart, a folder without metadata is ignored. [Likely]
Algorithm 1: sharded save of one checkpoint (seen from rank \(r\) of \(W\))
  1. input: the local state dict of rank \(r\) (its shards of weights and optimizer state), step \(N\)
  2. build a local plan: list every local piece as (tensor name, global offset, length) // no data yet
  3. gather all local plans to rank 0; rank 0 removes duplicates (replicated tensors are written once) and sends back a final plan
  4. write my pieces from the final plan to my own file in step-N.tmp/ // all ranks in parallel
  5. barrier: wait until every rank reports success
  6. rank 0 writes .metadata (for every tensor: global shape, and which file holds which piece) and renames the folder to step-N/ // the commit
  7. delete old checkpoints, but keep the last \(k\) complete ones

Loading on a different number of GPUs: resharding

You will often load a checkpoint on a different layout than you saved it: a node died and you restart on fewer nodes, you move from 256 to 512 GPUs, you change from FSDP to FSDP plus tensor parallelism, or you load a pretraining checkpoint for fine-tuning on 8 GPUs. DCP handles this “load-time resharding”, which “enables saving in one cluster topology and loading into another”. [Certain]

The trick is that the metadata talks about global positions, not ranks. For a 1-D sharded tensor of \(n\) elements, saved by \(W\) ranks and loaded by \(W'\) ranks:

\[ \text{new rank } r' \text{ needs } \Big[\, r'\tfrac{n}{W'},\ (r'+1)\tfrac{n}{W'} \Big), \qquad \text{it reads file } k \iff \max(\ell_{r'}, \ell_k) \lt \min(h_{r'}, h_k) \tag{51.3} \]
Read the symbols
\(n\)
number of elements in the (flattened) tensor
\(W, W'\)
number of ranks at save time and at load time
\([\ell_k, h_k)\)
the global range stored in saved file \(k\) (from the metadata); \([\ell_{r'}, h_{r'})\) is the range new rank \(r'\) needs
\(\max(\ell_{r'}, \ell_k) \lt \min(h_{r'}, h_k)\)
the two ranges overlap; the overlap is exactly the part to copy

Loading is in place: you first build the model on the new layout (each rank allocates its own new shards, empty), and DCP fills those tensors from the files. [Certain] So the load needs no extra copy of the model in memory. Try it in Figure 1.

Saved on \(W =\)
Loaded on \(W' =\)
Figure 1. Load-time resharding of one tensor with 24 elements. Top: the files written by the \(W\) saving ranks, each holding one contiguous range. Bottom: the \(W'\) new ranks and the range each one needs. Click (or Tab to and press Enter on) a new rank to see which files it reads and which slices it copies, by formula (51.3).
How to read Figure 1

Start with 4 → 3. Each old file holds 6 elements; each new rank needs 8. New rank 1 needs elements 8 to 15, so it reads elements 8 to 11 from file 1 and 12 to 15 from file 2. Every new rank reads 2 files, and no element is read twice. Now pick 4 → 8: each new rank reads half of one file. Pick 4 → 1: the single rank reads all 4 files (this is how you turn a sharded checkpoint into one file). Pick 4 → 4: every rank reads exactly its own file; this is the fast common case of a restart on the same layout. Real DCP does the same per tensor and also for 2-D pieces (for example, column shards from tensor parallelism).

Algorithm 2: resharded load (seen from new rank \(r'\) of \(W'\))
  1. input: checkpoint folder step-N/; the model and optimizer already built on the new layout (empty local shards)
  2. read .metadata: for every tensor, its global shape and the list of saved pieces (file, global offset, length)
  3. for every local shard I own, with global range \([\ell, h)\):
  4.   for every saved piece \([\ell_k, h_k)\) of the same tensor: if they overlap (51.3), read the overlap from file \(k\) and copy it into my shard // in place
  5. restore step counter, scheduler, data loader position and RNG states // the non-tensor state
  6. check: every element of every local shard was written exactly once

Asynchronous checkpoints: hide the save behind training

Even a parallel save stops training. A synchronous save copies the state from GPU to CPU memory and writes it to storage while all GPUs wait. An asynchronous save splits the work in two:

  1. Staging (blocking, short): copy the state from GPU memory into CPU memory. Training must wait for this, because the next optimizer step will change the weights.
  2. Writing (in the background, long): a background thread writes the CPU copy to storage while the GPUs already train the next steps.

This is overlap again: slow I/O runs while the GPUs compute. PyTorch’s dcp.async_save works this way: it copies the state dict to staging storage (CPU memory by default) and then “calls the save in a separate thread”. [Certain] torchtitan reports that async checkpointing cuts the checkpoint overhead by 5 to 15 times compared with synchronous DCP, for Llama 3.1 8B. [Certain] Its option async_mode takes "disabled", "async" or "async_with_pinned_mem". [Certain] Pinned (page-locked) CPU memory makes the GPU-to-CPU copy faster, but it is a scarce resource and the buffer stays allocated between saves. [Certain]

Two costs: CPU memory (each rank needs a CPU copy of its share; the PyTorch tutorial warns that memory grows by “checkpoint_size_per_rank X number_of_ranks” on a host [Certain]), and a limit on frequency: a new save cannot start before the previous write has finished. Figure 2 shows both.

training step (5 s)GPU stalled by checkpointbackground write
Figure 2. Twenty training steps of 5 s each, with a checkpoint every \(k\) steps. Synchronous: the GPUs stall for the copy and the full write. Asynchronous: the GPUs stall only for the copy to CPU; the write runs in the background, and the next checkpoint must wait if the previous write is not done. The defaults are the 70B example on 1,024 GPUs (0.83 GB per rank); the speeds are assumptions, change them.
How to read Figure 2

With the defaults, a synchronous checkpoint stalls the GPUs for 0.42 s + 4.15 s = 4.57 s, which is 18.3% of the training time when you save every 5 steps. Asynchronous: only the 0.42 s copy is a stall, 1.7% overhead, about 11 times less. Now set \(k = 1\) (a save after every step) and lower the write speed to 0.1 GB/s: the 8.3 s write is longer than a 5 s step, so each new save waits for the previous one, and the async advantage shrinks. Then raise the GPU → CPU speed (pinned memory): the stall falls further. Async pays off when the write fits between two saves.

Some teams go further and keep checkpoints in memory. Gemini’s training kept “redundant in-memory copies of the model state”, so that after a hardware failure it could “rapidly recover directly from an intact model replica”; this raised the goodput of the largest job from 85% to 97%. [Certain] The idea: with data parallelism, another replica already holds the same weights; reading them over the network is much faster than reading 800 GB from a file system.

Failures at scale

Why do big jobs fail so often? Because a synchronous training job is a chain: if any one of 16,384 GPUs, or any cable, switch or host between them, stops, all the others soon wait at the next collective. As Meta wrote, “a single GPU failure may require a restart of the entire job”. [Certain] If each component fails rarely and independently, the job’s mean time to failure (MTTF) shrinks in proportion to the number of components:

\[ M_{\text{job}} = \frac{1}{N_{\text{nodes}}\; r_f}, \qquad \text{for example } \frac{24\ \text{h}}{2048 \times 6.5 / 1000} = 1.80\ \text{h} \tag{51.4} \]
Read the symbols
\(M_{\text{job}}\)
mean time between failures of the whole job
\(N_{\text{nodes}}\)
number of nodes (machines) in the job; 8 GPUs per node, so 16,384 GPUs are 2,048 nodes
\(r_f\)
failure rate of one node, per day. Meta measured 6.50 failures per 1,000 node-days on its RSC-1 research cluster. [Certain]

With that rate, Meta’s reliability study projects an MTTF of 1.8 hours for a 16,384-GPU job and 0.23 hours (14 minutes) for a 131,072-GPU job; it observed 7.9 hours for 1,024-GPU jobs. [Certain] The Llama 3 numbers agree in size: 419 unexpected interruptions in 54 days is \(54 \times 24 / 419 = 3.09\) hours between unexpected stops. [Certain] Turned around: one GPU slot (GPU plus its share of host and network) fails on average only once every \(3.09 \times 16{,}384\) hours \(\approx 5.8\) years. Each part is reliable. The job is not.

Cause (Llama 3 405B, 54 days)Share of the 419 unexpected interruptions
Faulty GPU30.1%
GPU HBM3 memory17.2%
Software bug12.9%
Network switch or cable8.4%
Unplanned host maintenance7.6%
GPU SRAM memory; GPU system processor4.5%; 4.1%
NCCL watchdog timeouts (cause unknown)1.7%
Silent data corruption1.4% (6 events)

From Table 5 of the Llama 3 paper. [Certain] About 78% of the unexpected interruptions were confirmed or suspected hardware issues, and GPU issues alone were 58.7%. Only three needed significant manual work; automation handled the rest. [Certain] (The paper prints 148 interruptions for “Faulty GPU” next to 30.1%; 148 / 419 is 35.3%, so one of the two numbers has a typo. The category percentages add up to the 58.7% only with 30.1%.)

Failures come in four kinds, from easy to hard:

  • Crash. A process dies (a GPU “falls off the bus”, out of memory, a Python exception). The launcher sees it at once.
  • Hang. A rank stops making progress, for example a stalled NVLink transfer. Everyone else waits forever in the next collective. PyTorch’s NCCL process group has a watchdog with a timeout: by default 10 minutes for NCCL (30 for other backends), after which the collective is aborted and the process crashes. [Certain] Ten minutes of 16,384 idle GPUs is expensive, so large teams lower it and add heartbeats. [Likely]
  • Straggler. One slow GPU (overheating, throttled clock, a bad link) slows down thousands of others, because every collective waits for the slowest rank. Nothing crashes. Llama 3’s team noted that “even a single straggler can slow down thousands of other GPUs”. [Certain]
  • Silent data corruption. The hardware returns a wrong number with no error. The job continues, maybe with a damaged model. Its own section is below.

How often to checkpoint

Checkpoint too rarely, and each failure throws away hours of work. Checkpoint too often, and you spend your time saving. Let \(\tau\) be the time between checkpoints. Each checkpoint stalls training for \(\delta\). When a failure comes, you lose the work since the last checkpoint (on average half an interval, \(\tau/2\)) plus the restart time \(R\) (find the failure, get a replacement node, start the processes, load the checkpoint). Failures arrive every \(M\) on average. The fraction of wall time wasted is then, to first order,

\[ w(\tau) = \underbrace{\frac{\delta}{\tau}}_{\text{saving}} + \underbrace{\frac{R + \tau/2}{M}}_{\text{failures}}, \qquad \mathrm{ETTR} \approx 1 - w(\tau) \tag{51.5} \]

The first term falls with \(\tau\), the second grows. Setting the derivative \(-\delta/\tau^2 + 1/(2M)\) to zero gives Young’s formula (1974): [Certain]

\[ \tau^{*} = \sqrt{2\,\delta\,M}, \qquad w(\tau^{*}) = \sqrt{\frac{2\,\delta}{M}} + \frac{R}{M} \tag{51.6} \]
Read the symbols
\(\tau, \tau^{*}\)
the checkpoint interval, and its best value
\(\delta\)
how long one checkpoint stalls training. For an async save this is only the staging time.
\(M\)
mean time between failures of the job, formula (51.4)
\(R\)
restart cost: detection, rescheduling, process start-up, checkpoint load, warm-up (for example, torch.compile again)

Two lessons hide in (51.6). The waste grows only with the square root of the stall \(\delta\), so making saves 100 times cheaper makes the best waste 10 times smaller. And the restart cost \(R\) does not depend on \(\tau\) at all: no checkpoint interval can fix a slow restart. The formula is a first-order estimate; it is good when \(\tau\) and \(\delta\) are much smaller than \(M\). Daly (2006) gives a more exact version. [Certain]

Worked example: the Llama 3 failure rate

Take \(M = 3.09\) h \(= 11{,}135\) s (the Llama 3 rate of unexpected stops) and a restart cost \(R = 10\) min \(= 600\) s (an assumption).

Synchronous saves, \(\delta = 60\) s: \(\tau^{*} = \sqrt{2 \times 60 \times 11{,}135} = 1{,}156\) s \(\approx 19\) min. Waste \(= \sqrt{120 / 11{,}135} + 600/11{,}135 = 10.4\% + 5.4\% = 15.8\%\).

Async saves, \(\delta = 5\) s: \(\tau^{*} = \sqrt{2 \times 5 \times 11{,}135} = 334\) s \(\approx 5.6\) min. Waste \(= 3.0\% + 5.4\% = 8.4\%\).

Now the restart cost is the larger part. To beat 90% ETTR, as Llama 3 did, you need cheap saves and fast restarts. That is exactly what Meta says they worked on: “we reduced job startup and checkpointing time”. [Certain] And note what a bad choice costs: saving every 3 hours with \(\delta = 60\) s gives \(w = 54\%\) by (51.5) (the formula is rough there, but the message is clear).

saving: \(\delta/\tau\)lost work + restart: \((R + \tau/2)/M\)total waste
Figure 3. The checkpoint interval trade-off, formulas (51.4) to (51.6), with 8 GPUs per node. The horizontal axis is the interval \(\tau\) on a log scale (1 minute to 8 hours). The dot is your choice; the ring is Young’s optimum.
How to read Figure 3

The defaults are a 16,384-GPU job at Meta’s measured failure rate: MTTF 1.80 h. Saving every hour with a 60 s stall wastes 38.6%. Press the button: τ* is 14.7 min and the waste drops to 22.8%. Now set δ to 5 s (async): the optimum moves to 4.2 min and the waste to 13.2%, most of it the 10 min restart. Set R to 2 min and you are near 6%. Slide the GPUs to 131,072: MTTF falls to 13.5 min, and even good settings waste a lot. That is why the biggest jobs need in-memory checkpoints and very fast restarts, or training that continues without the failed part.

Restart and elastic training

Most frameworks handle a failure with stop everything and restart from the last checkpoint. It sounds crude, but it is simple and correct. The work is in making each stage fast:

1. Trainsave every τ (async)health metrics, heartbeats 2. Detectcrash, NCCL timeout,NaN loss, slow steps 3. Localizeflight recorder dumps,node health checks 4. Replacecordon bad node,take a hot spare 5. Restartrendezvous, new ranks,process groups, compile 6. Loadlatest complete step-N,reshard if layout changed Lost: τ/2 + Rwork since last saveplus steps 2 to 6
Figure 4. The failure recovery loop. Steps 2 to 6 together are the restart cost \(R\) of formula (51.5). Every minute saved here is worth as much as a minute of saved checkpoint stall per failure.
  • Detect. Crashes are seen at once. Hangs are seen by the watchdog timeout. Slow steps and NaN losses need your own monitoring.
  • Localize. Which of 16,384 GPUs is the culprit? PyTorch’s NCCL flight recorder keeps the last collectives of every rank in a ring buffer and dumps them on a timeout; the rank that never entered the collective is the suspect. Llama 3 used it heavily. [Certain]
  • Replace and restart. Keep a few hot spare nodes ready, so a restart does not wait for a repair. The launcher forms a new group by rendezvous: the surviving and new workers meet at a known address, get new ranks, and build the process groups again.

Elastic training goes one step further: the job may continue with a different number of nodes. PyTorch’s launcher supports it with torchrun --nnodes=MIN:MAX --max-restarts=K. Its behavior is still stop-and-restart: “On failures or membership changes ALL surviving workers are killed immediately”, a new worker group is formed with new RANK and WORLD_SIZE, and your script must reload its last checkpoint. [Certain] That is why resharding matters: the new world size is different.

Changing the number of data-parallel ranks changes the math of training, not just the layout. If each rank keeps its micro-batch, the global batch shrinks, and the learning rate and schedule that were tuned for the old batch are now wrong. The usual fix is to keep the global batch fixed and change gradient accumulation (session 39): with \(B_{\text{global}} = W \cdot b \cdot A\), losing ranks means more accumulation steps \(A\) per optimizer step. The run becomes slower but stays the same run. [Likely]

Algorithm 3: a fault-tolerant training loop
  1. input: checkpoint folder, interval \(\tau\) (in steps), target global batch \(B\)
  2. join the rendezvous; get rank \(r\) and world size \(W\); build the model and optimizer sharded over \(W\)
  3. \(A \leftarrow B / (W \cdot b)\) // keep the global batch fixed when \(W\) changes
  4. if a complete checkpoint exists: load the newest one with resharding (Algorithm 2); else initialize from a seed
  5. loop: wait for the previous async write if a new save is due; run one optimizer step; log loss, gradient norm, step time
  6.   if the loss is NaN or a health check fails: stop and let the launcher restart from the last good checkpoint
  7.   if \(\text{step} \bmod \tau = 0\): async save (Algorithm 1), staging only blocks
  8. on any crash or timeout, the launcher kills all workers, replaces the bad node, and starts again at line 2

Stop-and-restart is not the only design. Newer systems try to keep the healthy part of the job running: for example, when one data-parallel replica fails, the others continue for a few steps with a smaller batch while it rejoins. PyTorch’s torchft project explores this per-step fault tolerance. [Likely] It is still less common than plain restarts in large pretraining. [Speculation]

Silent data corruption

Silent data corruption (SDC) means the hardware gives a wrong result and raises no error: no crash, no ECC alarm, no exception. A defective core may compute a wrong multiply once in a billion operations, often only at a certain temperature or voltage. In a training run, a wrong number in one GPU’s gradient gets all-reduced into everyone’s weights. The job continues, but the model may be damaged, or a loss spike appears hours later with no visible cause.

How common is it? Meta’s fleet study “Silent Data Corruptions at Scale” found hundreds of defective CPUs among hundreds of thousands of machines. [Certain] For Gemini, Google wrote that “we can expect SDC events to impact training every week or two”. [Certain] In Llama 3’s 54 days, 6 interruptions were attributed to SDC. [Certain] Rare per chip, but certain for a large job.

Detection tools, from cheap to expensive:

  • Watch the training signals. Sudden spikes in the loss or the gradient norm, or NaN and Inf values. Cheap, but it cannot tell a hardware error from a normal loss spike (session 36).
  • Compare replicas. With data parallelism, all replicas should hold bit-identical weights after each step: they apply the same averaged gradient to the same weights. A checksum of the parameters, compared across replicas from time to time, catches a replica that drifted. [Likely]
  • Deterministic replay. If the training step is deterministic, you can run the same step again (on the same or other hardware) and compare the result bit by bit. A mismatch points to a faulty device. Gemini’s team detected and removed faulty hardware with techniques “that exploit deterministic replay”, plus “proactive SDC scanners on idle machines”. [Certain]
  • Scan machines. Run known computations with known answers on idle or suspect nodes, and remove nodes that fail.

A checkpoint policy helps too: keep several older checkpoints, not only the latest. If corruption is found late, you need a checkpoint from before it happened.

MFU: how to compute it correctly

Recall from session 42: MFU is the model’s useful FLOP/s divided by the peak. For one GPU, in a data-parallel job,

\[ \mathrm{MFU} = \frac{\text{tokens/s per GPU} \times F_{\text{token}}}{P}, \qquad F_{\text{token}} = 6\,N_{\text{ne}} + 12\,L\,d\,T \tag{51.7} \]
Read the symbols
\(F_{\text{token}}\)
useful training FLOPs per token: forward plus backward, from the model shape, not from a hardware counter
\(N_{\text{ne}}\)
the parameters that do matrix multiplications: all parameters minus the input embedding table (a lookup does no FLOPs). The output head stays in.
\(12\,L\,d\,T\)
attention scores and the weighted sum of values, forward and backward, for \(L\) layers of width \(d\) and sequence length \(T\)
\(P\)
the dense peak of one GPU for the data type you train in (989 TFLOP/s BF16 for H100 SXM)

The details are conventions, and they change the answer, so say which ones you use. torchtitan, for example, excludes the embedding table from \(N\), counts attention as \(12 L d T\) without halving it for the causal mask, and does not count the extra matmul that FlashAttention recomputes in the backward pass. [Certain] Recomputed FLOPs (activation checkpointing) count in HFU (hardware FLOPs utilization), never in MFU.

Worked example: one measurement, three MFU numbers

You train Llama 3 8B on H100s with \(T = 8{,}192\). Each GPU processes 2 sequences per step (16,384 tokens), and one step takes 2.0 s: 8,192 tokens/s per GPU.

Only \(6N\) with all 8,030,261,248 parameters: \(6N = 4.82 \times 10^{10}\), so \(8{,}192 \times 4.82 \times 10^{10} / 989 \times 10^{12} = 39.9\%\).

torchtitan convention: \(N_{\text{ne}} = 8{,}030{,}261{,}248 - 128{,}256 \times 4{,}096 = 7{,}504{,}924{,}672\). Attention: \(12 \times 32 \times 4{,}096 \times 8{,}192 = 1.29 \times 10^{10}\). \(F_{\text{token}} = 4.50 \times 10^{10} + 1.29 \times 10^{10} = 5.79 \times 10^{10}\), and MFU \(= 48.0\%\).

\(6N\) with all parameters plus attention: 50.6%. Same job, same speed, 40% to 51%. Before you compare your MFU with a paper, check the formula. torchtitan reports 33% to 42% MFU for Llama 3.1 8B on 8 or 128 H100s without Float8. [Certain]

How to read an MFU number

  • It is a ratio, so check both parts. Use the dense peak, never the “with sparsity” number, and the peak of the data type you really use. With FP8 matmuls the right peak is unclear (some layers run in BF16), and torchtitan does not report MFU for Float8 runs because “the definition of MFU under such scenario is not well-defined”. [Certain] Compare tokens per second instead.
  • MFU is measured while running. It says nothing about the hours lost to failures. Combine it with ETTR (51.1): \(40\%\ \mathrm{MFU} \times 90\%\ \mathrm{ETTR} = 36\%\) of the paid compute becomes training.
  • Big runs are in the 35–50% range, not 80%. PaLM 540B reached 46.2%, Llama 3 405B 38% to 43% (session 42). [Certain] If you see 70% for a dense model, suspect the formula; if you see 15%, look for exposed communication, small micro-batches, or a slow data loader.
  • MFU is per configuration. Short sequences, small models and small micro-batches give lower MFU because matmuls are smaller and fixed costs weigh more. Report it with model, sequence length, batch per GPU and number of GPUs.
Algorithm 4: measure MFU (and HFU) in a real job
  1. input: model shape \((N_{\text{ne}}, L, d)\), sequence length \(T\), peak \(P\), GPUs \(n\)
  2. \(F_{\text{token}} \leftarrow 6 N_{\text{ne}} + 12 L d T\) // state the convention
  3. skip the first steps (compile, warm-up); then time \(K\) steps with torch.cuda.synchronize() before reading the clock
  4. tokens/s per GPU \(\leftarrow\) (tokens in \(K\) global batches, padding excluded) / (time \(\times\, n\))
  5. MFU \(\leftarrow\) tokens/s per GPU \(\times F_{\text{token}} / P\)
  6. HFU \(\leftarrow\) the same with the FLOPs actually run (add recomputed forward passes)
  7. log both every few hundred steps; a drop in MFU over time means stragglers, thermal throttling or a slower data loader

Overlapping communication with compute

Where does MFU go? A big part is communication that the GPU waits for. Session 43 and session 46 showed the mechanism: run collectives on a second CUDA stream, and start them early enough. Here is the question that decides whether it works: is each transfer shorter than the compute it hides behind?

\[ T_{\text{step}} = T_{\text{comp}} + T_{\text{exposed}}, \qquad T_{\text{exposed}} \approx \sum_i \max\big(0,\ t^{\text{comm}}_i - t^{\text{comp}}_i\big) + t_{\text{first}} + t_{\text{last}} \tag{51.8} \]

Take FSDP with \(F\) GPUs. Before block \(i\) runs, its weights (\(\psi\) parameters, 2 bytes each in BF16) are all-gathered. While that happens, block \(i-1\) computes on the \(b\) tokens that this GPU holds. The all-gather takes \(2\psi\,(F-1)/(F B)\) seconds at bus bandwidth \(B\); the forward matmuls take \(2\psi b/(P u)\) seconds at compute efficiency \(u\). The transfer is hidden when

\[ \frac{2\psi}{B}\cdot\frac{F-1}{F} \le \frac{2\psi\, b}{P\,u} \quad\Longleftrightarrow\quad b \ \ge\ b_{\min} = \frac{P\,u}{B}\cdot\frac{F-1}{F} \tag{51.9} \]
Read the symbols
\(b\)
tokens per GPU per micro-step (micro-batch size × sequence length)
\(B\)
bus bandwidth per GPU for this collective (for example, about 50 GB/s for one 400 Gb/s network card per GPU)
\(u\)
the fraction of peak that the matmuls reach while they run
\(\psi\)
parameters of one block. It cancels out: the condition does not depend on model size.

The backward pass gives the same condition: twice the compute (\(4\psi b\)) and twice the transfers (all-gather and reduce-scatter). With \(P = 989\) TFLOP/s, \(u = 0.6\), \(B = 50\) GB/s and \(F = 64\), you need \(b \ge 11{,}700\) tokens per GPU. Below that, FSDP is communication-bound: more GPUs with smaller batches each will not train faster. This is the same kind of “compute per byte” argument as the roofline in session 42, applied to the network; the scaling book How to Scale Your Model uses it for every parallelism. [Likely] Figure 5 runs the full schedule.

MFU with overlapMFU without overlap\(b_{\min}\), formula (51.9)
Figure 5. A schedule simulation of one FSDP training step for Llama 3 8B (32 blocks of 218.1M parameters, \(T = 8192\)) on 64 H100s. Forward: all-gather each block before it runs; backward: all-gather and reduce-scatter per block, all on one communication stream. The curves show MFU (torchtitan convention) against \(b\); the dot is the current setting. Embedding and head communication, the optimizer step and data loading are ignored.
How to read Figure 5

Defaults: \(b = 16{,}384\), 50 GB/s, \(u = 0.6\). With overlap the step takes 1.625 s for 1.599 s of compute: MFU 59.1%, close to the 60% ceiling set by \(u\). Uncheck overlap: 0.82 s of communication is exposed and MFU falls to 39.6%. Turn overlap back on and slide \(b\) down to 4,096: now each all-gather (8.6 ms) is longer than a block’s forward (4.2 ms), and MFU falls to 29%. Change the bandwidth to 450 GB/s (NVLink-class) and the curve stays near 58% down to about 1,000 tokens. The knee sits a little to the left of the \(b_{\min}\) line: (51.9) counts only the weight matmuls, but attention adds compute (it needs no gathered weights) that also hides transfers. So (51.9) is a safe, slightly pessimistic bound.

Overlap appears in every part of a training system:

What is hiddenBehind whatMechanism
DDP gradient all-reducethe rest of the backward passgradient buckets that start as soon as they are full (session 44)
FSDP all-gather / reduce-scatterthe previous / next block’s computeprefetching, formula (51.9)
Tensor-parallel all-gather and reduce-scatterpieces of the same matmulsplit the matmul into chunks; torchtitan’s async TP raised Llama 3.1 70B on 256 H100s from 897 to 1,010 tokens/s per GPU (+12.6%), on top of compile and Float8 [Certain]
Pipeline sends between stagesother micro-batches1F1B and interleaved schedules (session 49)
Checkpoint writesthe next training stepsasync save (Figure 2)
Data loading and host-to-device copiesthe current steploader workers, prefetching, pinned memory, non-blocking copies

Overlap is not free. Communication kernels use some SMs and memory bandwidth, so the compute next to them runs a bit slower; and prefetching holds more data in memory at once. [Likely] The only way to see if overlap really happens is a profiler trace (session 50, session 52).

The code: sharded save, resharded load, exact resume

This program runs on a CPU in under a second and needs only PyTorch. Part 1 and 2 are a tiny version of Algorithms 1 and 2: four “ranks” each save their slice of a flat parameter vector, plus a metadata file and an atomic rename; then three new “ranks” load it by reading only the overlapping pieces. Part 3 shows why the optimizer and RNG states belong in a checkpoint: a crash-and-resume run matches an uninterrupted run bit for bit, and a weights-only resume does not.

import json, os, shutil, torch, torch.nn as nn

torch.manual_seed(0)
ROOT = "ckpt_demo"; shutil.rmtree(ROOT, ignore_errors=True)

def make_model():
    torch.manual_seed(0)
    return nn.Sequential(nn.Linear(8, 16), nn.GELU(), nn.Linear(16, 4))

def flat(model):                      # one long vector, like a FlatParameter
    return torch.cat([p.detach().reshape(-1) for p in model.parameters()])

def shard_range(n, world, rank):      # equal slices, last one may be shorter
    size = -(-n // world)             # ceil(n / world)
    lo = min(rank * size, n); hi = min(lo + size, n)
    return lo, hi

# ---- 1. save: each of W "ranks" writes only its own slice ----
def save_sharded(vec, world, step):
    d = f"{ROOT}/step-{step}.tmp"; os.makedirs(d)
    files = []
    for r in range(world):            # in a real job these run in parallel
        lo, hi = shard_range(vec.numel(), world, r)
        torch.save(vec[lo:hi].clone(), f"{d}/rank{r}.pt")
        files.append({"file": f"rank{r}.pt", "lo": lo, "hi": hi})
    json.dump({"numel": vec.numel(), "files": files}, open(f"{d}/.metadata", "w"))
    os.rename(d, f"{ROOT}/step-{step}")   # commit: the folder appears only when complete
    return f"{ROOT}/step-{step}"

# ---- 2. load on a DIFFERENT world size: read only overlapping pieces ----
def load_resharded(path, world, rank):
    meta = json.load(open(f"{path}/.metadata"))
    lo, hi = shard_range(meta["numel"], world, rank)
    out, reads = torch.empty(hi - lo), []
    for f in meta["files"]:
        a, b = max(lo, f["lo"]), min(hi, f["hi"])
        if a < b:                     # this saved file overlaps my new slice
            piece = torch.load(f"{path}/{f['file']}")
            out[a - lo:b - lo] = piece[a - f["lo"]:b - f["lo"]]
            reads.append(f["file"])
    return out, reads

model = make_model(); vec = flat(model)
path = save_sharded(vec, world=4, step=100)
parts = []
for r in range(3):
    piece, reads = load_resharded(path, world=3, rank=r)
    print(f"new rank {r} of 3 reads {reads}")
    parts.append(piece)
print("numel", vec.numel(), "resharded 4 -> 3 equal:", torch.equal(torch.cat(parts), vec))

# ---- 3. exact resume: model + optimizer + RNG + step must all be saved ----
def batch(gen):
    x = torch.randn(32, 8, generator=gen); return x, x[:, :4].sin()

def train(model, opt, gen, steps):
    for _ in range(steps):
        x, y = batch(gen)
        loss = nn.functional.mse_loss(model(x), y)
        opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

def fresh():
    m = make_model()
    o = torch.optim.AdamW(m.parameters(), lr=1e-2, weight_decay=0.1)
    g = torch.Generator().manual_seed(1)
    return m, o, g

m, o, g = fresh(); train(m, o, g, 6); straight = flat(m)

m, o, g = fresh(); train(m, o, g, 3)
torch.save({"model": m.state_dict(), "optim": o.state_dict(),
            "rng": g.get_state(), "step": 3}, f"{ROOT}/full.pt")
del m, o, g                           # the job "crashes"
m, o, g = fresh()
ck = torch.load(f"{ROOT}/full.pt")
m.load_state_dict(ck["model"]); o.load_state_dict(ck["optim"]); g.set_state(ck["rng"])
train(m, o, g, 6 - ck["step"])
print("resume is bit-exact:", torch.equal(flat(m), straight))

m, o, g = fresh(); train(m, o, g, 3)
m2 = make_model(); m2.load_state_dict(m.state_dict())           # forgot optimizer + RNG
o2 = torch.optim.AdamW(m2.parameters(), lr=1e-2, weight_decay=0.1)
g2 = torch.Generator().manual_seed(1)
train(m2, o2, g2, 3)
print("weights-only resume is exact:", torch.equal(flat(m2), straight),
      " max diff %.4f" % (flat(m2) - straight).abs().max().item())

# new rank 0 of 3 reads ['rank0.pt', 'rank1.pt']
# new rank 1 of 3 reads ['rank1.pt', 'rank2.pt']
# new rank 2 of 3 reads ['rank2.pt', 'rank3.pt']
# numel 212 resharded 4 -> 3 equal: True
# resume is bit-exact: True
# weights-only resume is exact: False  max diff 0.0342

With a real sharded model, you do not write this yourself: you use DCP. It also works in a single process (no process group), which is handy for tests and for converting checkpoints. This was run with PyTorch 2.8:

import os, shutil, torch, torch.nn as nn
import torch.distributed.checkpoint as dcp

torch.manual_seed(0)
model = nn.Sequential(nn.Linear(8, 16), nn.GELU(), nn.Linear(16, 4))
opt = torch.optim.AdamW(model.parameters(), lr=1e-2)
model(torch.randn(4, 8)).sum().backward(); opt.step()        # create Adam states

shutil.rmtree("dcp_ckpt", ignore_errors=True)
state = {"model": model.state_dict(), "optim": opt.state_dict()}
dcp.save(state, checkpoint_id="dcp_ckpt/step-1")               # no process group: one rank
print(sorted(os.listdir("dcp_ckpt/step-1")))

model2 = nn.Sequential(nn.Linear(8, 16), nn.GELU(), nn.Linear(16, 4))
target = {"model": model2.state_dict()}                        # load IN PLACE into these tensors
dcp.load(target, checkpoint_id="dcp_ckpt/step-1")
print(all(torch.equal(a, b) for a, b in zip(model.state_dict().values(), model2.state_dict().values())))

# ['.metadata', '__0_0.distcp']
# True

In a real FSDP2 job, the state dict holds DTensor shards, and the same two calls save and load in parallel on all ranks. Use get_state_dict / set_state_dict from torch.distributed.checkpoint.state_dict to get model and optimizer state dicts that can be resharded. [Certain] To make one file for inference, convert with python -m torch.distributed.checkpoint.format_utils dcp_to_torch step-N/ model.pt. [Certain]

Where this is used

SettingWhat matters mostTypical choice
Pretraining an LLM on thousands of GPUsETTR and MFU at oncesharded async checkpoints every few minutes to tens of minutes, hot spares, flight recorder, MFU logging [Likely]
Fine-tuning on one 8-GPU nodesimple, portable checkpointsDCP during training, one safetensors file at the end for serving
Changing the cluster mid-run (more nodes arrive, a rack fails)loading on a new layoutDCP resharding; keep the global batch fixed with accumulation
Vision-language model training (vision encoder + LLM)parts with different sharding; frozen partssave only the trainable parts often; the frozen vision tower is saved once
Video models (video classification, action recognition, video-text retrieval encoders for Video-RAG)long sequences of frame tokens; data loading is often the bottleneckoverlap decoding and loading with compute; checkpoint the data position as a global sample index, because video datasets are huge and expensive to replay [Likely]
Long RL or post-training runsseveral models (policy, reference, reward) in one jobone checkpoint per model plus the shared step and data state
Comparing training systems or hardwarea fair speed numberMFU with a stated formula, or tokens/s per GPU for FP8 runs

Common mistakes

  • Saving only the weights. The resume then starts Adam from zero and repeats or skips data. The code above shows the drift. Save optimizer, scheduler, step, data position and RNG states.
  • Gathering the full state to rank 0 for every save. It is slow, it can run rank 0 out of CPU memory, and all other GPUs wait. Save sharded; gather only once at the end for export.
  • No atomic commit. A crash during a save leaves a broken “latest” checkpoint, and the restart crashes too. Write the metadata last, rename at the end, and keep the last few complete checkpoints.
  • Choosing the checkpoint interval by habit (“every 1,000 steps”). Compute it from (51.6) with your real stall time and failure rate, and recompute when the job grows.
  • Ignoring the restart cost. If a restart takes 20 minutes (scheduling, compile, loading), no checkpoint interval saves you. Measure \(R\) and work on it.
  • Storing per-rank data positions and then changing the world size. Store a global position.
  • Reporting MFU without the formula, with the sparse peak, or counting recomputation. The same run can be 40% or 51% (worked example above).
  • Assuming overlap happens. If \(b \lt b_{\min}\) (51.9), or a collective is waited on too early, communication is exposed. Check the trace.

Interview questions

Why not just call torch.save on rank 0 for a model sharded over 1,024 GPUs?basic
Rank 0 would have to gather all shards first: for a 70B model with Adam that is about 847 GB (12 bytes per parameter) through one process and one network link, needing that much CPU memory, and then write it alone. That takes minutes while 1,023 GPUs wait. A sharded checkpoint lets every rank write its own 0.83 GB in parallel, plus a small metadata file, so the time is set by the total storage bandwidth (seconds), not by one writer.
What must a checkpoint contain so that training resumes exactly?basic
The FP32 master weights, the optimizer state (Adam’s \(m\) and \(v\)), the step counter and LR scheduler state, the data loader position, the RNG states of every rank, the loss scaler for FP16, and the config. Not the gradients: you save between steps. Without the optimizer state Adam restarts with zero moments and a wrong bias correction, and the loss usually jumps. Without the data position you repeat or skip data. Without RNG states dropout and shuffling differ. A good test: resume from a checkpoint and compare with an uninterrupted run; with deterministic kernels they should match bit for bit.
How does load-time resharding work?core
The checkpoint metadata describes every saved piece by its global position in the tensor (offsets and lengths), not by rank. At load time, you first build the model on the new layout; each new rank knows the global range of each of its shards. For every saved piece that overlaps that range, it reads exactly the overlap and copies it into its shard in place (formula 51.3). Example: 24 elements saved on 4 ranks (6 each), loaded on 3 (8 each): new rank 1 needs 8 to 15, so it reads 8 to 11 from file 1 and 12 to 15 from file 2. DCP does this per tensor, also for 2-D pieces from tensor parallelism. Non-tensor state like the data position must be stored in a layout-independent way.
How often should you checkpoint?core
Balance saving cost against lost work. The waste fraction is \(\delta/\tau + (R + \tau/2)/M\), with stall \(\delta\), interval \(\tau\), restart cost \(R\) and job MTTF \(M\). The minimum is at Young’s \(\tau^{*} = \sqrt{2\delta M}\), with waste \(\sqrt{2\delta/M} + R/M\). Example: \(M = 3.09\) h (Llama 3), \(\delta = 60\) s gives \(\tau^{*} \approx 19\) min and about 10.4% waste from saving and lost work, plus \(R/M\). Since \(M\) shrinks as the job grows, recompute the interval when you scale up, and make saves cheaper (async) because the waste grows with \(\sqrt{\delta}\).
Why does a 16,384-GPU job fail every few hours if each GPU is reliable?basic
Synchronous training needs every component at every step, so one failure anywhere stops the job. With independent failures, the job’s failure rate is the sum of the parts’ rates: \(M_{\text{job}} = 1/(N_{\text{nodes}} r_f)\). Meta measured 6.5 failures per 1,000 node-days; 2,048 nodes give 13.3 failures per day, an MTTF of 1.8 h. Llama 3 saw an unexpected interruption every 3.1 h on average. That corresponds to each GPU slot failing only about once every 5.8 years.
What is asynchronous checkpointing, and what does it cost?core
The save is split into a short blocking stage (copy the state from GPU to CPU memory, because the next optimizer step will change it) and a long background write from CPU memory to storage while training continues. The stall drops from copy + write to copy only; torchtitan reports 5 to 15 times less checkpoint overhead for Llama 3.1 8B. Costs: CPU memory for a full copy of each rank’s share (pinned memory is faster but scarce), and a new save cannot start before the previous write finishes, which limits how often you can save.
What is silent data corruption, and how do you detect it in training?advanced
Hardware that returns a wrong result without any error signal, for example a defective core that miscomputes rarely. In data-parallel training the wrong value is all-reduced into all replicas. Detection: monitor loss and gradient-norm spikes and NaNs (cheap but ambiguous); compare checksums of the weights across data-parallel replicas, which should be bit-identical; deterministic replay of a step to compare results bit for bit, as Gemini did; and proactive test workloads on idle nodes. Keep several old checkpoints so you can roll back to before the corruption. Gemini expected an SDC event every week or two; Llama 3 had 6 SDC interruptions in 54 days.
A node dies and you restart on fewer GPUs. What changes besides the sharding?advanced
The data-parallel size \(W\) changes, so if each rank keeps its micro-batch, the global batch changes, and the tuned learning rate and schedule no longer fit. Keep the global batch fixed by raising gradient accumulation: \(A = B/(W b)\). The step becomes slower but the optimization is the same. Also: the data position must be global so the new ranks can split the remaining data; RNG states per rank no longer map one to one, so exact bitwise reproducibility is usually lost; and the tensor-parallel or pipeline sizes may need to stay the same if the model does not fit otherwise. torchrun’s elastic mode (--nnodes=MIN:MAX) kills all workers and restarts them with new ranks, so the script must reload and reshard.
How do you compute MFU, and which choices change the number?core
MFU = tokens/s per GPU × useful FLOPs per token ÷ dense peak FLOP/s of the GPU. Useful FLOPs per token are usually \(6N + 12LdT\). Choices: whether \(N\) includes the embedding table (torchtitan excludes it), whether attention is counted and whether it is halved for the causal mask, which peak (dense, not sparse; BF16 vs FP8), and that recomputation is excluded (that is HFU). For Llama 3 8B at 8,192 tokens/s per H100, the same run is 39.9% with \(6N\) only, 48.0% with torchtitan’s convention and 50.6% with \(6N\) plus attention. Always state the formula.
MFU, HFU, tokens per second, ETTR: which one would you report, and when?advanced
MFU measures how well the running job uses the GPU for useful model math; it is fair across systems if the formula is stated. HFU counts all FLOPs run, including recomputation, so it measures the kernels, not the training efficiency. Tokens per second is the raw speed; use it when the peak is unclear (FP8, mixed precision per layer) or when comparing the same model and hardware. ETTR (goodput) measures the fraction of wall time spent on useful training, including failures and restarts. The total useful work is peak × time × MFU × ETTR, so a cluster report should give MFU and ETTR together.
When can FSDP hide its communication completely?advanced
When each block’s all-gather takes no longer than the compute it overlaps. All-gather time is \(2\psi (F-1)/(F B)\) (BF16), the forward of one block is \(2\psi b/(P u)\), so the condition is \(b \ge (P u/B)(F-1)/F\), independent of model size. The backward gives the same condition (twice the compute, twice the transfers). With an H100 at \(u = 0.6\) and 50 GB/s per GPU you need about 11,700 tokens per GPU per micro-step; with NVLink-class 450 GB/s about 1,300. Below it, adding GPUs with smaller per-GPU batches makes the job communication-bound; use hybrid sharding, larger micro-batches, or other parallelism.
A 4,096-GPU job hangs with no error. How do you debug it?core
A hang usually means one rank never entered a collective that all others are waiting in. The NCCL watchdog will abort after its timeout (10 min by default in PyTorch). Before or at the timeout, dump the NCCL flight recorder of every rank: it lists the last collectives with sequence numbers, so the rank that is behind or stuck stands out. Then look at that rank: Python stack (py-spy), GPU health (XID errors, ECC, NVLink status), its node’s network. Common causes are a failed GPU or link, a rank that took a different code path (a data-dependent branch that skips a collective), or a deadlock in the data loader. Remove the bad node, restart from the last checkpoint, and if it is a code path bug, fix it so all ranks call the same collectives in the same order.

Exercises

  1. A 13B model is trained with AdamW in mixed precision. How large is a full training checkpoint, and how large is a BF16 weights-only export? How much does each of 256 ranks write?
  2. A tensor of 30 elements was saved by 5 ranks and is loaded by 3. Which elements does new rank 1 need, and which files and slices does it read?
  3. A 4,096-GPU job (8 GPUs per node) runs on a cluster with 6.5 node failures per 1,000 node-days. Compute the job MTTF, Young’s interval for a 30 s stall, and the waste with a 5 min restart cost.
  4. In exercise 3, by what factor must the stall shrink to halve the \(\sqrt{2\delta/M}\) part of the waste?
  5. Your Llama 3 8B run (\(T = 8192\)) on H100s reaches 5,645 tokens/s per GPU. Compute MFU with torchtitan’s convention (\(F_{\text{token}} = 5.79 \times 10^{10}\)).
  6. Using (51.9) with \(P = 989\) TFLOP/s, \(u = 0.5\) and \(F = 64\): what is \(b_{\min}\) at 25 GB/s? If you train with \(T = 4096\), how many sequences per GPU per micro-step do you need?
Answerscheck
1. \(12 \times 13 \times 10^9 = 156\) GB for training, \(2 \times 13 \times 10^9 = 26\) GB for the BF16 export. Each of 256 ranks writes \(156/256 = 0.61\) GB. 2. Old files hold 6 elements each: [0,6), [6,12), [12,18), [18,24), [24,30). New rank 1 needs [10,20). It reads elements 10 to 11 from file 1, 12 to 17 from file 2 (all of it) and 18 to 19 from file 3. 3. 512 nodes × 6.5/1000 = 3.33 failures per day, so \(M = 7.21\) h \(= 25{,}962\) s. \(\tau^{*} = \sqrt{2 \times 30 \times 25{,}962} = 1{,}248\) s \(\approx 20.8\) min. Waste \(= \sqrt{60/25{,}962} + 300/25{,}962 = 4.8\% + 1.2\% = 6.0\%\). 4. The term grows with \(\sqrt{\delta}\), so the stall must shrink 4 times (30 s to 7.5 s). 5. \(5{,}645 \times 5.79 \times 10^{10} / 989 \times 10^{12} = 33.1\%\) (torchtitan’s FSDP-only baseline on 128 GPUs in its Table 2 is 5,645 tokens/s, and its paper gives 33% to 42% MFU, so this fits if their \(T\) is 8,192 [Likely]). 6. \(b_{\min} = 989 \times 10^{12} \times 0.5 / 25 \times 10^9 \times 63/64 = 19{,}471\) tokens, so 5 sequences of 4,096 tokens (20,480 tokens) per GPU per micro-step.

Key takeaways

  • Useful work = peak × time × MFU × ETTR (51.1). Checkpoints and fast restarts raise ETTR; overlap raises MFU.
  • A training checkpoint holds weights, optimizer state, step, scheduler, data position and RNG states: about 12 bytes per parameter (51.2), 847 GB for a 70B model.
  • Sharded checkpoints write every rank’s pieces in parallel plus a metadata file written last; the metadata uses global positions, so any layout can load any checkpoint by reading the overlaps (51.3).
  • Async saves stall only for the GPU-to-CPU copy; the write runs in the background (5 to 15 times less overhead in torchtitan).
  • Job MTTF falls in proportion to the number of nodes (51.4): about 1.8 h at 16,384 GPUs. Young’s interval \(\sqrt{2\delta M}\) (51.6) balances saving against lost work; the restart cost adds \(R/M\) on top.
  • Silent data corruption is rare per chip but certain at scale; catch it with replica checksums, deterministic replay and scanners, and keep older checkpoints.
  • State your MFU formula (51.7). FSDP can hide its communication only if each GPU has at least \(b_{\min} = (Pu/B)(F-1)/F\) tokens per micro-step (51.9).

Key formulas

\[ S_{\text{ckpt}} \approx 12\Psi, \quad M_{\text{job}} = \frac{1}{N_{\text{nodes}} r_f}, \quad \tau^{*} = \sqrt{2\delta M} \]
\[ \mathrm{ETTR} \approx 1 - \frac{\delta}{\tau} - \frac{R + \tau/2}{M}, \quad \mathrm{MFU} = \frac{\text{tok/s/GPU} \cdot (6N_{\text{ne}} + 12LdT)}{P}, \quad b_{\min} = \frac{Pu}{B}\cdot\frac{F-1}{F} \]

References for this session

  1. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  2. PyTorch (code). torchtitan: reference large-scale training with FSDP2
  3. Austin et al., Google DeepMind (2025). How to Scale Your Model
  4. Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
  5. PyTorch (code). torchtitan: reference large-scale training with FSDP2
  6. Austin et al., Google DeepMind (2025). How to Scale Your Model
  7. Llama Team, Meta (2024). The Llama 3 Herd of Models
  8. Liang et al. (2024). TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
  9. PyTorch (docs). Distributed Checkpoint (torch.distributed.checkpoint): save, load, async_save, get_state_dict, format_utils
  10. PyTorch (docs). Asynchronous Saving with Distributed Checkpoint (DCP) (recipe)
  11. Kokolis et al. (2024). Revisiting Reliability in Large-Scale Machine Learning Research Clusters
  12. Gemini Team, Google (2023). Gemini: A Family of Highly Capable Multimodal Models
  13. Dixit et al. (2021). Silent Data Corruptions at Scale
  14. Young (1974). A first order approximation to the optimum checkpoint interval
  15. Daly (2006). A higher order estimate of the optimum checkpoint interval for restart dumps
  16. Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
  17. PyTorch (docs). torchrun (Elastic Launch): elastic nnodes MIN:MAX, max-restarts, rendezvous