Training recipe and stability
A large training run is a long chain of small risks. This session explains the recipe that makes runs work: how to initialize the weights so signals keep their size, how to shape the learning rate over time, what causes loss spikes and how to stop them, and how μP lets you tune settings on a small model and reuse them on a big one.
What you will learn
- Why initialization variance must depend on width, and why the popular std 0.02 is only right for one width
- How scaled residual initialization keeps the residual stream from growing with depth
- Warmup, cosine, warmup-stable-decay and multi-step schedules, and why decaying the learning rate lowers the loss
- The main causes of loss spikes: Adam after a quiet period, growing attention logits, drifting output logits, bad batches
- The fixes and when each one helps: gradient clipping, lower β₂ and learning rate, z-loss, QK-norm, restart and skip
- μP: why the best learning rate moves with width in standard practice, the μP scaling rules, and the μTransfer workflow
- The real recipes of Llama 3, OLMo 2, PaLM and DeepSeek LLM, with their numbers
Why this matters
A pretraining run of a large model costs millions of dollars and runs for weeks or months. Most of the time, it does not fail because the architecture is wrong. It fails because the loss suddenly jumps up (a loss spike), turns into NaN, or slowly stops improving. The training recipe is the set of choices that prevents this: how the weights start, how the learning rate changes over time, and what guards are in place when something goes wrong.
In session 4 you met AdamW, warmup, cosine decay and gradient clipping. In session 22 you saw Pre-LN, QK-norm and z-loss as architecture choices. In session 35 you learned how to choose model size and token count. This session puts these pieces together into one recipe and asks three practical questions:
- How should the weights start? If the initial weights are too large or too small, signals explode or vanish before training even begins.
- How should the learning rate change over time? The schedule decides both the speed of progress and the final loss.
- How do we keep a long run stable, and how do we pick the learning rate for a model we can afford to train only once? Here we meet loss spikes and their fixes, and μP, a way to tune on a small model and copy the settings to a big one.
The problem is real even for the best teams. For PaLM 540B, Google reports that the loss spiked “roughly 20 times during training, despite the fact that gradient clipping was enabled”, and the spikes did not happen in the smaller models. [Certain] Their fix was simple and expensive: restart from a checkpoint about 100 steps before the spike and skip about 200 to 500 data batches. [Certain] Two years later, the Llama 3 team wrote that their 405B recipe “observed few loss spikes and did not require interventions”. [Certain] The difference between these two stories is mostly the recipe.
The recipe at a glance
Before the details, here are real recipes from published papers. Every number in this table is taken from the paper named in the first column. [Certain]
| Model | Optimizer and learning rate | Schedule | Stability choices |
|---|---|---|---|
| Llama 2 7B | AdamW, \(\beta = (0.9, 0.95)\), \(\varepsilon = 10^{-5}\), weight decay 0.1, peak \(3\times10^{-4}\) | 2,000 warmup steps, cosine to 10% of peak | Clipping at 1.0, Pre-LN with RMSNorm |
| Llama 3 405B | AdamW, peak \(8\times10^{-5}\) | 8,000 warmup steps, cosine to \(8\times10^{-7}\) over 1,200,000 steps | Batch grows from 4M to 8M to 16M tokens during training |
| OLMo 2 7B | AdamW, peak \(3\times10^{-4}\), \(\varepsilon = 10^{-8}\) | 2,000 warmup steps, cosine, then a linear decay to 0 at the end | Init std 0.02 everywhere, QK-norm, norm after each sub-layer, z-loss, no weight decay on embeddings |
| PaLM 540B | Adafactor (no factorization), \(10^{-2}\) for 10,000 steps | Then decay as \(1/\sqrt{k}\) at step \(k\) | \(\beta_2 = 1 - k^{-0.8}\), clipping at 1.0, z-loss \(10^{-4}\), restart-and-skip for spikes |
| DeepSeek LLM | AdamW | 2,000 warmup steps, then a step down to 31.6% of peak after 80% of tokens and to 10% after 90% | Multi-step schedule so the first phase can be reused |
You can already see a pattern. Almost everyone uses AdamW with \(\beta_2\) around 0.95, a short warmup, a decay at the end, clipping at 1.0, and some guard against growing logits. The rest of the session explains why each piece is there.
Initialization: keep signals the right size
Training starts from random weights. The question is: random how large? Think of one linear layer \(\mathbf{y} = W\mathbf{x}\) with input size \(n\) (the fan-in). Each output is a sum of \(n\) products \(W_{ij} x_j\). If the weights are independent with mean 0 and variance \(\sigma^2\), the variances add up:
- Read the symbols
- \(n\)
- fan-in: the number of inputs of the layer (the width \(d\) for most transformer matrices)
- \(\sigma\)
- the standard deviation used to draw each weight at the start
- \(\operatorname{Var}(x_j)\)
- the variance of one input coordinate
So each layer multiplies the variance by \(n\sigma^2\), and the standard deviation (std) by \(\sigma\sqrt{n}\). A ReLU sets half of the values to zero and halves the variance, so for a ReLU layer the std factor is \(\sigma\sqrt{n/2}\). After \(L\) layers without residual connections or normalization:
A factor even a little above 1 explodes; a little below 1 vanishes. The classic fix is to choose \(\sigma\) so that the factor is exactly 1: \(\sigma = 1/\sqrt{n}\) for linear or tanh layers (LeCun or Xavier style) and \(\sigma = \sqrt{2/n}\) for ReLU (He initialization). Both are “fan-in” rules: the std shrinks as the layer gets wider.
How to read Figure 1
Start with the fixed std 0.02 and width 512. Each layer multiplies the std by \(0.02\sqrt{512} = 0.45\), so the signal vanishes: after 32 layers it is about \(10^{-11}\). Now choose \(1/\sqrt{n}\): the line is flat at 1 for every width. Switch the activation to ReLU: with \(1/\sqrt{n}\) the signal now shrinks by \(1/\sqrt 2\) per layer, and \(\sqrt{2/n}\) (He) fixes it. With tanh the signal cannot explode, because tanh is bounded, but it can still shrink. The dashed line is formula (36.2); the dots are the real simulation, and they agree closely.
Why the popular std 0.02 depends on width
GPT-2, Megatron-LM, nanoGPT and OLMo 2 draw most weights from a normal distribution with std 0.02. [Certain for GPT-2 code and OLMo 2] This is a fixed number, not a fan-in rule. Formula (36.1) tells us when it is “right”: \(n \sigma^2 = 1\) means \(n = 1/0.02^2 = 2{,}500\). For a width of 768 (GPT-2 small) the variance factor per layer is 0.31; for 8,192 it is 3.28.
Variance factor per layer \(n\sigma^2\) with \(\sigma = 0.02\): width 768 gives \(768 \times 0.0004 = 0.307\); width 4,096 gives 1.638; width 8,192 gives 3.277. The std factors are \(\sqrt{\cdot}\): 0.554, 1.280 and 1.810. In a plain stack of 32 such layers, the std would be multiplied by \(0.554^{32} \approx 6 \times 10^{-9}\) at width 768 and by \(1.81^{32} \approx 1.8 \times 10^{8}\) at width 8,192. The fan-in rule \(1/\sqrt{n}\) would use 0.036 and 0.011 instead.
So why does std 0.02 work in practice? Because a transformer is not a plain stack. Every sub-layer reads a normalized copy of the residual stream (Pre-LN), so the input to each layer has size about 1 no matter what came before; and the residual connection keeps the signal alive even if a sub-layer’s output is small. The normalization layers hide most of the width effect at initialization. But they do not hide it during training: the updates still depend on width, and that is the problem μP solves at the end of this session.
Scaled residual initialization
The residual stream is a sum. Each of the \(L\) blocks adds two edits (attention and MLP), so after \(L\) blocks there are \(2L\) edits. If the edits are roughly independent at initialization, their variances add:
- Read the symbols
- \(\mathbf{h}_0, \mathbf{h}_L\)
- the residual stream after the embedding, and after the last block
- \(\mathbf{f}_l\)
- the edit that sub-layer \(l\) adds to the stream
- \(\sigma_{\text{out}}\)
- the init std of the output projection of each sub-layer (attention output and MLP down-projection)
If every edit has the same variance \(v\), the stream variance grows to \(\operatorname{Var}(\mathbf{h}_0) + 2Lv\): it grows linearly with depth. GPT-2 fixes this by scaling the weights of the residual layers at initialization by \(1/\sqrt{N}\), where \(N\) is the number of residual layers. [Certain] Since variance scales with the square of std, each edit then has variance proportional to \(1/(2L)\), and their sum stays the same for any depth. nanoGPT uses \(0.02/\sqrt{2L}\) for these projections: 0.0041 for 12 layers and 0.0025 for 32 layers. [Likely] OLMo 2 found that plain std 0.02 for all weights gave fewer gradient spikes than its earlier scaled scheme. [Certain] So this is a choice to test, not a law.
Two more details matter. Embeddings feed the stream directly, so their size sets \(\operatorname{Var}(\mathbf{h}_0)\); PaLM used std 1 for the input embeddings and scaled the output logits by \(1/\sqrt{d}\), because it ties the input and output embeddings. [Certain] And zero-initialized output projections (used in some recipes) make every block start as the identity function; the model then grows its edits during training. [Likely]
Learning-rate schedules, revisited
You know the shapes from session 4: linear warmup (formula 4.11), then cosine decay (formula 4.12). Here we add two newer shapes and, more importantly, explain why decay lowers the loss.
Warmup-stable-decay and multi-step
Cosine needs the total number of steps \(T\) in advance. If you later want to train longer, the learning rate has already gone down, and you cannot simply continue. Warmup-stable-decay (WSD), introduced with MiniCPM, keeps the learning rate at its peak for most of training, then decays it over a short final phase. [Certain] With warmup length \(W\) and decay start \(S\):
The decay here is linear to 0; other decay shapes (exponential, square root, cosine) are also used. [Likely] The big advantage: you can save a checkpoint anywhere in the stable phase, and later either continue training from it or run a short decay to get a finished model. One long run can then produce finished models for several token budgets, which is exactly what you want for the scaling-law fits in session 35. DeepSeek LLM used a related multi-step schedule: after 2,000 warmup steps, the learning rate drops to 31.6% of the peak after 80% of the tokens, and to 10% after 90%. [Certain] They report that the final performance “is essentially consistent with that of a cosine scheduler”, and that the first phase can be reused when they train longer. [Certain] (31.6% is about \(1/\sqrt{10}\).)
Why decaying the learning rate lowers the loss
Stochastic gradients are noisy. Even at the bottom of the valley, each step jumps by a random amount, so the weights bounce around the minimum instead of sitting on it. The size of this bouncing depends on the learning rate. For one direction with curvature \(\lambda\) and gradient noise with std \(\sigma\), plain SGD settles into a cloud whose average loss is exactly
This is the noise floor. It is proportional to \(\eta\): halve the learning rate, and the floor halves. A high learning rate makes fast progress along the flat directions but sits on a high floor; a low one sits on a low floor but makes slow progress. A schedule gets the best of both: high early (progress), low at the end (floor). This also explains the strange shape of WSD loss curves: the loss is worse than cosine during the stable phase, then drops sharply during the decay. The drop is the floor going down. Figure 2 shows this on a toy problem.
How to read Figure 2
With the defaults (noise 0.1, decay over the last 20%), the constant schedule (grey) ends at 0.119: it stops improving once it reaches its noise floor. Cosine (blue) ends at 0.088. WSD (orange) ends at 0.057. Just before its decay starts (step 2,400), WSD is at 0.135 while cosine is already at 0.095: WSD looks worse for most of the run, then drops quickly during the decay. WSD wins here because it spends more steps at the peak, which helps the flat directions (small \(\lambda\)) make progress. Now raise the noise to 0.2: the noise floor matters more, and cosine (0.157) now beats WSD (0.198 with 20% decay). Lower the noise to 0.05 and even the constant schedule beats cosine. The best schedule is a balance between progress and noise floor (formula 36.5). This is a toy: real LLM losses are not quadratic. In real runs, WSD with a decay phase of roughly 10 to 20% is reported to match cosine. [Likely]
How long should warmup be?
Warmup protects the first steps, when Adam’s second-moment estimate is based on very few gradients and the weights are far from any good region. Real runs use a small fraction of the total: Llama 2 used 2,000 of about 500,000 steps (0.4%), and Llama 3 405B used 8,000 of 1,200,000 (0.67%). [Certain] Longer warmup is a common first fix for early instability, and Wortsman et al. found that warmup makes small models less sensitive to the choice of learning rate. [Likely]
Loss spikes: what they are and where they come from
A loss spike is a sudden jump of the training loss, often by a large amount, within a few steps. Sometimes the loss comes back down by itself in a few hundred steps; sometimes it never fully recovers; sometimes it becomes NaN. Spikes become more common in larger models, at higher learning rates, and late in training. The main known causes are these:
- Adam after a quiet period. If the gradients of some weights are tiny for a long time, Adam’s second moment \(\mathbf{v}\) becomes tiny too. When a large gradient then arrives, the update \(\hat m / \sqrt{\hat v}\) is suddenly large. Molybog et al. argue that, in large runs, Adam can enter a state where the update has a large norm and is almost uncorrelated with the descent direction, which leads to divergence. [Certain]
- Growing attention logits. As \(W_Q\) and \(W_K\) grow during training, the attention scores grow with the square of their size; softmax becomes almost one-hot, its gradient vanishes, and training becomes unstable. This was the problem in ViT-22B, fixed with QK-norm. [Certain]
- Drifting output logits. Cross-entropy does not care if all output logits shift by the same amount, so they can drift to very large values, where low-precision numbers lose accuracy. [Likely] z-loss fixes this.
- Data. Some batches are unusual (long runs of repeated tokens, for example). OLMo 2 removed documents with 32 or more repeated n-grams and masked such sequences in the loss, because they caused gradient spikes. [Certain] But PaLM found that the same batches did not cause a spike when trained from an earlier checkpoint: spikes come from a bad batch and a sensitive model state together. [Certain]
Cause 1 in numbers: Adam after a quiet period
Suppose a weight sees a gradient of 0.01 for thousands of steps. Then \(m \approx 0.01\) and \(v \approx 0.01^2\), and each update is about \(\eta\) (Adam normalizes). Now a burst arrives: the gradient jumps to \(g_1 \gg 0.01\). On the first step of the burst, \(m \approx (1-\beta_1) g_1\) and \(v \approx (1-\beta_2) g_1^2\), so
With \(\beta_1 = 0.9\): the limit is 3.16 for \(\beta_2 = 0.999\), 1.00 for \(\beta_2 = 0.99\), and 0.45 for \(\beta_2 = 0.95\). If the burst lasts several steps, \(m\) catches up with the new gradient faster than \(v\) does (because \(\beta_1\) forgets faster than \(\beta_2\)), and the steps get even bigger. This is one clear reason why large runs use \(\beta_2 = 0.95\) instead of the default 0.999. Note what formula (36.6) does not contain: the size of the gradient. Adam divides by the gradient’s own size, so global-norm clipping, which only rescales the gradient, helps less than you might expect.
How to read Figure 3
With the defaults (\(\beta_2 = 0.999\), burst \(\times 100\) for 10 steps), the largest step is about 6.5 times a normal step, and the weight moves more than twice as far as it normally would over the window. Switch to \(\beta_2 = 0.95\): the largest step is about 1.03, because \(\mathbf{v}\) adapts within about 20 steps. Now go back to 0.999 and change the burst size: \(\times 10\) gives a peak of 4.9, \(\times 100\) and \(\times 1000\) both give 6.5. Above a certain size, the size of the gradient no longer matters, as formula (36.6) says. Clipping at \(10\times\) the normal size only turns the burst into a \(\times 10\) burst: the peak is still 4.9. Only clipping very close to the normal size (\(2\times\)) brings it down to 1.6, and in a real model you cannot clip that tightly without slowing all of training. The lesson: for Adam, lower \(\beta_2\) and a lower learning rate are stronger fixes than clipping.
Two related details. First, the Adam \(\varepsilon\) matters when gradients are tiny: if \(\sqrt{\hat v}\) falls near \(\varepsilon\), the update shrinks. Wortsman et al. report that a too-large \(\varepsilon\) can make updates too small in large models, and OLMo 2 lowered \(\varepsilon\) from \(10^{-5}\) to \(10^{-8}\). [Certain for OLMo 2] Second, PaLM chose \(\beta_2 = 1 - k^{-0.8}\) (growing toward 1 with the step \(k\)) because “rare embedding tokens can have poorly estimated second moments over shorter windows”. [Certain] Rare tokens are exactly the “quiet” weights of formula (36.6).
Causes 2 and 3: logits that grow
Output logits. The cross-entropy loss only depends on differences between logits, so the model can add the same number to all logits without changing the loss. Nothing stops this drift. But numbers stored in BF16 have only 8 significant bits, so large values are coarse: between 256 and 512, BF16 can only represent every second integer, and between 512 and 1,024 only every fourth. Large logits lose the small differences that carry the prediction. z-loss adds a small penalty that keeps the log of the softmax normalizer near 0:
- Read the symbols
- \(z_j\)
- the output logit of vocabulary entry \(j\)
- \(Z\)
- the softmax normalizer; \(\log Z\) is the log-sum-exp of the logits
- \(\alpha\)
- a small weight; PaLM used \(10^{-4}\) [Certain]
Its gradient is easy to read. Since \(\partial \log Z / \partial z_i = p_i\) (the softmax probability):
When \(\log Z \gt 0\), every logit is pushed down, the likely ones most; when \(\log Z \lt 0\), they are pushed up. The cross-entropy gradient \(p_i - y_i\) sums to zero over the vocabulary, so it can never move the average logit; the z-loss gradient sums to \(2\alpha \log Z\), so it moves exactly the part that cross-entropy ignores.
How to read Figure 4
At shift 0, \(\log Z = 3.217\), the cross-entropy is 0.2171 and the z-loss is about 0.001. Move the shift to 50: cross-entropy is still 0.2171, but the z-loss is 0.283, so its gradient (formula 36.8) pulls the logits back. Move to 1,000: BF16 can only store multiples of 4 there, so the logits become 1004, 1000, 1000, 1000, 1000, and the computed cross-entropy is 0.0707, which is wrong. The model’s prediction has been damaged by rounding alone. In real code, the loss is computed in FP32, but logits and activations may pass through BF16 before that; session 37 covers the number formats in detail.
Attention logits. The score between query \(\mathbf{q}_i\) and key \(\mathbf{k}_j\) is \(\mathbf{q}_i \cdot \mathbf{k}_j / \sqrt{d_h}\). If \(W_Q\) and \(W_K\) both grow by a factor \(g\), the scores grow by \(g^2\) (session 22). QK-norm applies RMSNorm to each head’s query and key before the dot product:
Now the scores depend only on the directions of \(\mathbf{q}\) and \(\mathbf{k}\) and on the learned gain vectors \(\mathbf{g}_q, \mathbf{g}_k\), which grow slowly. The growth of \(W_Q, W_K\) no longer reaches the softmax. QK-norm was key for ViT-22B [Certain] and is used in OLMo 2 [Certain] and Qwen3 [Certain]. Wortsman et al. showed that attention-logit growth and output-logit drift both appear in small models trained at high learning rates, and that QK-layernorm and z-loss fix them there too, so you can study these problems cheaply. [Certain]
The fixes, and when each one helps
| Fix | What it does | Helps against | Cost |
|---|---|---|---|
| Global-norm gradient clipping (1.0) | Caps the length of the gradient (formula 4.13) | Rare huge gradients; mainly protects \(\mathbf{m}\) and SGD-like updates | Almost free; weak against Adam’s \(\hat m / \sqrt{\hat v}\) (Figure 3) |
| Lower \(\beta_2\) (0.95) | \(\mathbf{v}\) follows gradient changes within about 20 steps | Adam after a quiet period (formula 36.6) | Noisier step sizes |
| Lower peak learning rate, longer warmup | Smaller steps everywhere | Almost every cause | Slower training; the safest first fix |
| z-loss (\(\alpha \approx 10^{-4}\)) | Keeps \(\log Z\) near 0 (formula 36.7) | Output logit drift | Negligible |
| QK-norm | Normalizes \(\mathbf{q}, \mathbf{k}\) per head (formula 36.9) | Attention logit growth, entropy collapse | A small extra norm per layer; part of the architecture, so decide before training |
| No weight decay on embeddings, careful \(\varepsilon\) | Keeps embedding norms healthy; avoids tiny or frozen updates | Slow drifts that end in spikes [Likely] | Free |
| Restart and skip | Go back about 100 steps before the spike, skip the next few hundred batches | A spike that happened anyway (PaLM) | Lost compute; needs frequent checkpoints and a deterministic data loader |
- watch every step: loss, pre-clip gradient norm, max attention logit per layer, mean \(\log Z\), update-to-weight ratio per layer
- if the loss jumps and does not return within a few hundred steps, or becomes NaN:
- find the first step where the gradient norm or a logit statistic started to rise // often well before the loss
- reload the checkpoint from before that point (PaLM: about 100 steps before the spike)
- skip the batches around the spike (PaLM: about 200 to 500) // needs a loader where \(\text{batch} = f(\text{step})\)
- if it spikes again: lower the learning rate (for example by 10 to 30%) or lower \(\beta_2\) from here on
- if attention logits grow: QK-norm (next run) or soft-capping; if \(\log Z\) drifts: add z-loss
- record every intervention in the run log, so the result can be reproduced
μP: tune small, train big
The biggest model in a project is usually trained once. You cannot sweep its learning rate. The usual hope is: sweep on a small model and reuse the best value. In the standard parameterization (SP, what PyTorch does by default) this does not work, because the best learning rate moves as the model gets wider. Figure 1 of the μP paper shows this for transformers trained with Adam: in standard practice the optimum shifts with width, while in μP it stays in place. [Certain] You can see the same effect in the small experiment at the end of the code section: in SP the best Adam learning rate falls from \(2^{-7}\) at width 64 to \(2^{-12}\) at width 1,024.
Why the best learning rate moves
Take one hidden layer \(\mathbf{h} = W\mathbf{x}\) with width \(n\), entries of \(\mathbf{x}\) of size about 1, and \(W\) initialized with variance \(1/n\), so the entries of \(\mathbf{h}\) also have size about 1. Look at the very first Adam step. With bias correction, the first update of every weight is exactly \(-\eta \operatorname{sign}(g)\). The gradient is \(\partial L / \partial W = \delta\, \mathbf{x}^\top\), where the column vector \(\delta = \partial L/\partial \mathbf{h}\) comes from above (from formula 4.4), so \(\operatorname{sign}(g_{ij}) = \operatorname{sign}(\delta_i)\operatorname{sign}(x_j)\). For the same input, the change of \(\mathbf{h}\) is
(For a standard normal \(x_j\), the average of \(|x_j|\) is \(\sqrt{2/\pi} = 0.798\).) Every one of the \(n\) terms pushes \(h_i\) in the same direction, because the update is built from \(\mathbf{x}\) itself. So the change grows linearly with width, while \(\mathbf{h}\) itself stays the same size. A learning rate that changes \(\mathbf{h}\) by 20% at width 256 changes it by 1,300% at width 16,384. To keep the relative change the same, the learning rate of hidden matrices must scale like \(1/n\). That is the core of μP for Adam.
How to read Figure 5
The grey line rises with slope 1 on the log-log plot: doubling the width doubles the relative change. At width 256 with \(\eta \approx 10^{-3}\), the change is about 0.2; at width 16,384, it is about 12, which means the layer’s output is destroyed in one step. The orange line (μP) is flat: the change is the same at every width. Now change the base \(\eta\): both lines move up or down together, but only the orange one keeps its shape. This is why the best \(\eta\) found at width 256 is still the best at width 16,384 in μP.
The μP rules
The μP paper (Tensor Programs V) gives the full rules for every type of parameter. Here is its Table 3 for Adam, with fan-in \(n\). [Certain] Parameters are grouped by how many of their dimensions grow with width: input weights (embeddings, the first layer; only the output side grows), hidden weights (both sides grow), and output weights (the unembedding; only the input side grows).
| Input weights and all biases | Hidden weights | Output weights | |
|---|---|---|---|
| Init variance, μP | \(1/n\) | \(1/n\) | \(1/n^2\) (SP: \(1/n\)) |
| Adam learning rate, μP | \(\eta\) (no scaling) | \(\eta/n\) (SP: \(\eta\)) | \(\eta/n\) (SP: \(\eta\)) |
| SGD learning rate, μP | \(\eta \cdot n_{\text{out}}\) (SP: \(\eta\)) | \(\eta\) | \(\eta/n\) (SP: \(\eta\)) |
For transformers there is one more change: the attention score uses \(1/d_h\) instead of \(1/\sqrt{d_h}\). [Certain] In practice, the rules are written relative to a base width \(n_0\) (the proxy model), so at the base width μP and SP agree in learning rates:
- Read the symbols
- \(n_0\)
- the width of the small proxy model on which you tuned (the base width)
- \(\eta_{\text{base}}\)
- the best learning rate found by the sweep on the proxy
- \(\sigma^2_{\text{out}}\)
- the init variance of the output (unembedding) layer; the paper also gives an equivalent version with a \(1/n\) multiplier on the output logits (its Table 8), which works with tied embeddings
- write the target model in μP (formula 36.11 and the table), with a base width \(n_0\)
- build a proxy: the same model with width \(n_0\) (often a few hundred), same depth or shallower, trained on fewer tokens
- for each candidate setting (learning rate, init scale, output multiplier, warmup, …): train the proxy and record the loss
- pick the best setting; check that it is stable at 2 or 3 widths between \(n_0\) and the target // the optimum should not move
- train the target with the same settings; μP rescales the hidden and output learning rates by \(n_0/n\) automatically
What transfers and what does not. The paper lists learning rate, initialization scale, and parameter multipliers as transferable across width; across depth, batch size, training time and sequence length, transfer was checked only empirically on transformers. Regularization settings such as dropout and weight decay do not transfer. [Certain] The headline result: by transferring from a 40M-parameter proxy, they beat the published numbers of the 6.7B GPT-3 model, with a tuning cost of only 7% of the total pretraining cost. [Certain] Cerebras-GPT and several later model families report using μP; many large labs do not say whether they use it. [Likely]
The recipe in code
The first program builds a small GPT-style model with the stable recipe: std 0.02 init with scaled residual projections (formula 36.3), QK-norm (36.9), a loss with z-loss (36.7), AdamW with \(\beta_2 = 0.95\), no decay on 1-D parameters and embeddings, and clipping. It then reproduces the Adam burst of Figure 3 with torch.optim.Adam. It needs PyTorch 2.4 or newer (for nn.RMSNorm).
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) Initialization: std 0.02 everywhere, residual output projections scaled by 1/sqrt(2L) class Block(nn.Module): def __init__(self, d, n_head, qk_norm=True): super().__init__() self.h, self.dh = n_head, d // n_head self.n1, self.n2 = nn.RMSNorm(d), nn.RMSNorm(d) self.qkv, self.proj = nn.Linear(d, 3 * d, bias=False), nn.Linear(d, d, bias=False) self.up, self.down = nn.Linear(d, 4 * d, bias=False), nn.Linear(4 * d, d, bias=False) self.qn = nn.RMSNorm(self.dh) if qk_norm else nn.Identity() # QK-norm, formula (36.9) self.kn = nn.RMSNorm(self.dh) if qk_norm else nn.Identity() def forward(self, x): B, T, d = x.shape q, k, v = self.qkv(self.n1(x)).view(B, T, 3, self.h, self.dh).unbind(2) q, k = self.qn(q), self.kn(k) # normalize each head's q and k a = F.scaled_dot_product_attention(q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), is_causal=True) x = x + self.proj(a.transpose(1, 2).reshape(B, T, d)) return x + self.down(F.gelu(self.up(self.n2(x)))) class TinyLM(nn.Module): def __init__(self, vocab=1000, d=128, n_head=4, L=4): super().__init__() self.emb = nn.Embedding(vocab, d) self.blocks = nn.ModuleList([Block(d, n_head) for _ in range(L)]) self.norm, self.head = nn.RMSNorm(d), nn.Linear(d, vocab, bias=False) for name, p in self.named_parameters(): # formulas (36.1) and (36.3) if p.ndim == 2: std = 0.02 / math.sqrt(2 * L) if name.endswith(("proj.weight", "down.weight")) else 0.02 nn.init.normal_(p, mean=0.0, std=std) def forward(self, idx): x = self.emb(idx) for b in self.blocks: x = b(x) return self.head(self.norm(x)) # 2) Loss with z-loss, formula (36.7); computed in FP32 def lm_loss(logits, targets, z_alpha=1e-4): logits = logits.float() logz = torch.logsumexp(logits, dim=-1) # log Z for every position ce = (logz - logits.gather(-1, targets[..., None]).squeeze(-1)).mean() return ce + z_alpha * (logz ** 2).mean(), ce, logz.mean() model = TinyLM() idx = torch.randint(0, 1000, (8, 33)) logits = model(idx[:, :-1]) loss, ce, logz = lm_loss(logits, idx[:, 1:]) print(f"CE {ce.item():.3f} (ln 1000 = {math.log(1000):.3f}), mean log Z {logz.item():.3f}") print("check: CE equals F.cross_entropy:", torch.allclose(ce, F.cross_entropy(logits.reshape(-1, 1000), idx[:, 1:].reshape(-1)))) # 3) One step of the stable recipe: AdamW (beta2 0.95), no decay on 1-D params and embeddings, clipping at 1.0 decay = [p for n, p in model.named_parameters() if p.ndim == 2 and "emb" not in n] nodecay = [p for n, p in model.named_parameters() if p.ndim < 2 or "emb" in n] opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.1}, {"params": nodecay, "weight_decay": 0.0}], lr=3e-4, betas=(0.9, 0.95), eps=1e-8) opt.zero_grad(set_to_none=True); loss.backward() gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) opt.step() print(f"grad norm before clipping {gnorm.item():.3f}") # 4) The Adam burst toy of Figure 3: 5000 quiet steps (g = 0.01), then 10 steps with g = 1 for b2 in (0.95, 0.999): w = torch.zeros(1, requires_grad=True) o = torch.optim.Adam([w], lr=1.0, betas=(0.9, b2), eps=1e-8) sizes = [] for t in range(5050): w.grad = torch.tensor([1.0 if 5000 <= t < 5010 else 0.01]) before = w.item(); o.step(); sizes.append(abs(w.item() - before)) print(f"beta2 = {b2}: largest update during the burst = {max(sizes[5000:]):.2f} x lr")
Output with PyTorch 2.8 on CPU:
CE 6.935 (ln 1000 = 6.908), mean log Z 6.929 check: CE equals F.cross_entropy: True grad norm before clipping 2.983 beta2 = 0.95: largest update during the burst = 1.03 x lr beta2 = 0.999: largest update during the burst = 6.51 x lr
The initial cross-entropy is close to \(\ln 1000 = 6.908\), which is what a well-initialized model should give: with small weights, the output is almost uniform over the 1,000 tokens. But \(\log Z \approx 6.93\), far from 0, so the z-loss starts working from step one. The last two lines match Figure 3 (1.03 and 6.51).
The second program is a small μP experiment: a 3-layer MLP trained with Adam on a fixed random regression task, at widths 64, 256 and 1,024, for 10 learning rates. In μP mode, the output layer is initialized with variance \(1/n^2\) and the hidden and output learning rates are divided by \(n / 64\) (base width 64).
import math, torch torch.manual_seed(0) D_IN, BASE = 32, 64 # proxy (base) width = 64 teacher = torch.nn.Sequential(torch.nn.Linear(D_IN, 256), torch.nn.Tanh(), torch.nn.Linear(256, 1)) X = torch.randn(4096, D_IN) with torch.no_grad(): Y = teacher(X) * 3 def make(width, mup): torch.manual_seed(1) l1, l2, l3 = torch.nn.Linear(D_IN, width), torch.nn.Linear(width, width), torch.nn.Linear(width, 1) for l in (l1, l2, l3): # init variance 1/fan_in for input and hidden torch.nn.init.normal_(l.weight, std=l.in_features ** -0.5); torch.nn.init.zeros_(l.bias) if mup: torch.nn.init.normal_(l3.weight, std=1.0 / l3.in_features) # output: variance 1/fan_in^2 return l1, l2, l3 def train(width, lr, mup, steps=300): l1, l2, l3 = make(width, mup) m = width / BASE groups = [{"params": l1.parameters(), "lr": lr}, {"params": [l2.weight], "lr": lr / m if mup else lr}, # hidden: lr / width {"params": [l2.bias], "lr": lr}, {"params": [l3.weight], "lr": lr / m if mup else lr}, # output: lr / width {"params": [l3.bias], "lr": lr}] opt = torch.optim.Adam(groups) g = torch.Generator().manual_seed(2) for t in range(steps): idx = torch.randint(0, 4096, (128,), generator=g) h = torch.relu(l2(torch.relu(l1(X[idx])))) loss = ((l3(h) - Y[idx]) ** 2).mean() if not torch.isfinite(loss): return float("nan") opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): return ((l3(torch.relu(l2(torch.relu(l1(X))))) - Y) ** 2).mean().item() lrs = [2.0 ** k for k in range(-13, -3)] for mup in (False, True): print("muP" if mup else "SP ") for width in (64, 256, 1024): res = [train(width, lr, mup) for lr in lrs] best = min(range(len(lrs)), key=lambda i: res[i] if res[i] == res[i] else 1e9) print(f" width {width:5d}: best log2(lr) = {int(math.log2(lrs[best]))}, loss {res[best]:.4f}")
Output (about one minute on a laptop CPU):
SP width 64: best log2(lr) = -7, loss 0.0090 width 256: best log2(lr) = -9, loss 0.0089 width 1024: best log2(lr) = -12, loss 0.0047 muP width 64: best log2(lr) = -5, loss 0.0090 width 256: best log2(lr) = -7, loss 0.0047 width 1024: best log2(lr) = -7, loss 0.0046
In SP, the best learning rate falls by a factor of \(2^5 = 32\) when the width grows by 16, close to the \(1/n\) of formula (36.10). In μP it stays at \(2^{-7}\) for widths 256 and 1,024. At width 64 the μP curve is almost flat between \(2^{-7}\) (loss 0.010) and \(2^{-5}\) (0.009), so the argmin is noisy there. A small toy, one seed: treat it as an illustration, not a benchmark. Adding width 4,096 (which takes about ten times longer) moved the SP optimum to \(2^{-14}\), the edge of the grid, while μP stayed at \(2^{-7}\).
Where this is used
| Setting | Recipe parts that matter most | Why |
|---|---|---|
| LLM pretraining at scale | AdamW with \(\beta_2 = 0.95\), short warmup, cosine or WSD, clipping 1.0, QK-norm, z-loss, frequent checkpoints | One run, months long; a spike costs days of compute |
| Planning a model family (scaling-law sweeps) | WSD or multi-step schedules; μP for learning-rate transfer | One stable-phase run gives many finished models; tuned settings carry over across sizes |
| Continued pretraining (new language, code, long context) | Re-warmup from a low learning rate, then decay again; a WSD stable-phase checkpoint is the best start [Likely] | A cosine-finished model has a tiny learning rate; jumping back to the peak causes a spike |
| Vision transformers and VLM encoders | QK-norm (ViT-22B), careful warmup, low learning rate for the pretrained encoder | Attention-logit growth was first a large-ViT problem |
| Video models (action recognition, video retrieval, key-moment detection) | Very long token sequences per clip; QK-norm and z-loss as for LLMs; often a frozen or low-learning-rate image encoder [Likely] | Many tokens per sample means large attention logits and noisy, memory-limited small batches |
| Mixture-of-Experts models | Router z-loss on the router logits, plus the same recipe [Likely] | Router logits drift like output logits; see session 29 |
| Fine-tuning and RL | Much lower peak learning rate, short warmup, clipping | The model is already good; big steps cause forgetting or collapse |
Common mistakes
- Copying “std 0.02” without checking the width. It equals the fan-in rule only near width 2,500. It works in Pre-LN transformers because the norms hide it, but it is not a universal constant. If you remove norms or change the architecture, recheck formula (36.1).
- Forgetting the residual scaling when adding depth. Without it, the stream variance grows with \(2L\) (formula 36.3), and deeper models are harder to train.
- Tuning the learning rate on a small SP model and using it on a big one. In SP the optimum moves down with width (Figure 5 and the code output); the big model will be unstable. Either use μP or fit how the best learning rate scales with size.
- Relying on clipping alone. Clipping at 1.0 did not prevent PaLM’s spikes. For Adam, the update size is set by \(\hat m/\sqrt{\hat v}\), which clipping barely changes (Figure 3).
- Resuming a finished cosine run at the peak learning rate. The sudden jump causes a spike. Re-warm up, or keep a stable-phase (WSD) checkpoint for later continuation.
- No deterministic data order. Restart-and-skip needs to know exactly which batches came before the spike. Make the batch a function of the step number, as PaLM did.
- Looking only at the loss. The gradient norm, the max attention logit and the mean \(\log Z\) usually start to rise well before the loss spikes. Log them from step one.
- Half-implementing μP. One layer left in SP (often the output layer or the attention scaling) silently breaks transfer. Run a coordinate check before trusting the transfer.
Interview questions
Why does initialization variance need to depend on width?basic
Why do GPT-style models scale the output projection of each block by \(1/\sqrt{2L}\)?core
Cosine vs warmup-stable-decay: which and why?core
Why does lowering the learning rate at the end reduce the loss so much?core
Your loss spiked at step 80,000 of a large run. What do you do?core
Why does gradient clipping not prevent all loss spikes with Adam?advanced
What is z-loss, and what exactly does its gradient do?core
How does QK-norm stop attention entropy collapse?core
What is μP, and why does the best learning rate move with width without it?advanced
Which hyperparameters transfer with μP, and which do not?advanced
Why do large runs use \(\beta_2 = 0.95\) rather than 0.999, and what did PaLM do instead?advanced
Exercises
- A plain linear stack of width 2,048 uses std 0.02. By what factor does the std change per layer, and after 24 layers?
- A GPT-style model has 24 blocks. With the GPT-2/nanoGPT rule, what init std do the attention output and MLP down-projections get?
- A WSD run has 100,000 steps, peak \(3 \times 10^{-4}\), and decays linearly to 0 over the last 20%. What is the learning rate at step 90,000?
- Using formula (36.6), what is the largest first-step burst update (in units of \(\eta\)) for \(\beta_1 = 0.9\), \(\beta_2 = 0.98\)?
- During training, the mean \(\log Z\) is 20. With \(\alpha = 10^{-4}\), what is the z-loss, and what is the sum of its gradient over all logits?
- On a μP proxy of width 256, the best hidden learning rate is \(2^{-7}\). What hidden learning rate and output-layer init std does the width-4,096 target use?
- A logit of about 200 is stored in BF16. What is the spacing between representable numbers there? (BF16 has 8 significant bits.)
Answerscheck
Key takeaways
- Initialization variance must scale like \(1/n\) to keep signals the same size (36.1, 36.2). Std 0.02 equals that only near width 2,500; Pre-LN hides the difference at init. Scaled residual init (36.3) keeps the stream from growing with depth.
- Warmup protects the first steps; decay lowers the noise floor, which is proportional to \(\eta\) (36.5). WSD (36.4) and multi-step schedules match cosine and let you reuse or extend runs.
- Loss spikes come from fragile model states plus unusual batches: Adam after a quiet period (36.6), growing attention logits, drifting output logits.
- Fixes: \(\beta_2 = 0.95\) and a lower learning rate (strong), z-loss (36.7, 36.8) for output logits, QK-norm (36.9) for attention logits, clipping (weak against Adam), and restart-and-skip as a last resort.
- In standard parameterization, Adam updates change hidden layers by about \(0.8\,\eta n\) (36.10), so the best learning rate moves with width. μP (36.11) rescales learning rates and inits per layer type so the best settings found on a small proxy transfer to the big model.
Key formulas
References for this session
- Yang et al. (2022). Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Stanford (2025). CS336: Language Modeling from Scratch (course)
- Yang et al. (2022). Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
- Hugging Face (Nanotron team) (2025). The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- Stanford (2025). CS336: Language Modeling from Scratch (course)
- Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Team OLMo (Walsh et al.) (2024). 2 OLMo 2 Furious
- Wortsman et al. (2023). Small-scale proxies for large-scale Transformer training instabilities
- Molybog et al. (2023). A Theory on Adam Instability in Large-Scale Machine Learning
- Dehghani et al. (2023). Scaling Vision Transformers to 22 Billion Parameters
- Henry, Dachapally, Pawar, Chen (2020). Query-Key Normalization for Transformers
- Radford et al. (2019). Language Models are Unsupervised Multitask Learners
- Hu et al. (2024). MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- DeepSeek-AI (2024). DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Xiong et al. (2020). On Layer Normalization in the Transformer Architecture