Scaling laws
If you train a language model with more parameters, more data or more compute, its loss falls in a smooth and predictable way: a power law. This session shows the two famous laws (Kaplan 2020 and Chinchilla 2022), why they disagree about how many tokens each parameter needs (about 20), why real models are trained on far more tokens than that, and how to use a scaling law to plan a training run before you spend the money.
What you will learn
- What a power law is, and why it is a straight line on log-log axes
- The Kaplan et al. laws for parameters, data and compute, and their advice: grow the model faster than the data
- The Chinchilla parametric law L = E + A/N^α + B/D^β, isoFLOP curves, and the closed-form compute-optimal model size
- Why Kaplan and Chinchilla disagreed, and where the rule of about 20 tokens per parameter comes from
- Inference-optimal training: why Llama 3 8B saw about 1,900 tokens per parameter
- How to plan a run: GPUs and days to FLOPs, FLOPs to model size and token count, and the cost of leaving the optimum
- How to fit a scaling law yourself, and the pitfalls that make fits wrong
Why this matters
Training a large language model costs millions of dollars, and you usually get only one try. Before you start, you must answer two questions: how big should the model be, and on how many tokens should it train? Scaling laws answer both, using only small, cheap experiments.
A scaling law is a simple formula that predicts the final loss of a model from its size \(N\) (number of parameters), the amount of training data \(D\) (number of tokens) and the training compute \(C\) (number of floating-point operations, FLOPs). The surprising fact behind this session is that these formulas work. The loss of a language model falls smoothly and predictably, over many orders of magnitude, as a power law. So you can train twenty small models for a few hours each, fit a curve, and predict the loss of a model one thousand times larger.
The history is also a lesson. In 2020, Kaplan et al. concluded that models should grow much faster than their data, and the field built GPT-3 (175B parameters, 300B tokens). In 2022, the Chinchilla paper showed that, for a fixed compute budget, parameters and tokens should grow at the same rate, about 20 tokens per parameter. A 70B model trained on 1.4T tokens beat the 280B Gopher model that used about the same compute. [Certain] Then practice moved again: Llama 3 8B was trained on more than 15T tokens, about 75 times the Chinchilla-optimal amount for its size (about 200B tokens), because a small model is cheaper to serve. [Certain]
In session 25 you learned \(C \approx 6ND\). In session 34 you saw where the tokens come from and how many good ones exist. This session joins the two: given \(C\), how should you split it between \(N\) and \(D\)?
What a scaling law is: power laws on log-log axes
Train many models of different sizes, each on plenty of data, and plot the final test loss against the number of parameters. On normal axes you see a curve that falls fast and then flattens. On log-log axes (log of loss against log of size) the same points lie on a straight line. A straight line on log-log axes is the signature of a power law:
- Read the symbols
- \(L\)
- the test loss: average cross-entropy per token, in nats (natural-log units), as in session 6
- \(X\)
- the resource you scale: parameters \(N\), tokens \(D\) or compute \(C\)
- \(X_c\)
- a fitted constant that sets the height of the line; it depends on the data and the tokenizer
- \(\alpha_X\)
- the fitted exponent: how fast the loss falls; it is the slope of the line on log-log axes
Take the log of both sides and the straight line appears:
The key property of a power law is that every multiplication of the resource gives the same fraction of improvement. If \(\alpha = 0.076\), doubling the model always multiplies the loss by \(2^{-0.076} = 0.949\): a 5.1% drop, whether you go from 1M to 2M parameters or from 100B to 200B. This is also the bad news. Each new 5% costs twice as much as the last one.
Real curves cannot fall forever. Text has some randomness that no model can predict (the next word after “I had a” is truly uncertain). So a better form adds a floor, the irreducible loss \(E\): \(L(X) = E + A / X^{\alpha}\). Far above the floor this looks like a straight line on log-log axes. Close to the floor it bends and flattens. Figure 1 shows both forms.
How to read Figure 1
With log-log axes and the pure power law, the line is straight and the drop per doubling (in the readout) is the same 5.1% everywhere. Switch to the floor version: at small \(N\) it is almost straight, but above about \(10^{11}\) parameters it bends toward 1.69, and the gain per doubling shrinks. Switch to linear axes to see why people use log axes: almost all the action is squeezed into the left edge. The “local slope” in the readout is \(d \log L / d \log N\); for a pure power law it is always \(-\alpha\).
With Kaplan’s exponent \(\alpha_N = 0.076\), we want \((N_1/N_2)^{0.076} = 0.9\). So \(N_2/N_1 = 0.9^{-1/0.076} = 4.0\). A model 4 times larger gives a 10% lower loss (if data and compute do not limit it). A model 10 times larger multiplies the loss by \(10^{-0.076} = 0.839\), a 16% drop. Small exponents mean large models: this is why the field went from millions to billions of parameters.
Kaplan et al. (2020): the first laws
Kaplan and colleagues at OpenAI trained decoder-only transformers from 768 to 1.5 billion non-embedding parameters, on 22 million to 23 billion tokens of WebText2. [Certain] They found three power laws, each valid when the other two resources do not limit the model:
| Law | Exponent | Constant | Meaning of 2× more |
|---|---|---|---|
| \(L(N)\), parameters, trained to convergence | \(\alpha_N = 0.076\) | \(N_c = 8.8\times10^{13}\) | loss × 0.949 |
| \(L(D)\), tokens, large model with early stopping | \(\alpha_D = 0.095\) | \(D_c = 5.4\times10^{13}\) | loss × 0.936 |
| \(L(C_{\min})\), compute, best model size for that compute | \(\alpha_C^{\min} = 0.050\) | \(C_c = 3.1\times10^{8}\) PF-days | loss × 0.966 |
A PF-day is \(10^{15}\) FLOP/s for one day: \(8.64\times10^{19}\) FLOPs. They also combined parameters and data into one formula, which tells you how much a model overfits when its data is limited:
Two more results matter. First, the shape of the model (depth versus width, number of heads) changes the loss very little when \(N\) is fixed. [Certain] So \(N\) is the number that counts. Second, the compute of training is, to a good approximation,
(2 FLOPs per parameter per token forward, 4 backward; see session 25). With a fixed budget \(C\), a larger \(N\) means a smaller \(D\). So the real question is: which split gives the lowest loss?
Kaplan’s answer: spend compute mostly on size
Kaplan et al. found that the compute-efficient model size grows as \(N_{\text{opt}} \propto C^{0.73}\) and the data only as \(D_{\text{opt}} \propto C^{0.27}\). [Certain] Ten times more compute should buy a model \(10^{0.73} = 5.4\) times larger but only \(10^{0.27} = 1.9\) times more tokens. Their summary: large models are more sample-efficient, and compute-efficient training “stops far short of convergence”. GPT-3 followed this logic: 175B parameters on 300B tokens, only 1.7 tokens per parameter. [Certain]
Chinchilla (2022): parameters and tokens grow together
Hoffmann et al. at DeepMind trained over 400 models, from 70M to over 16B parameters, on 5B to 500B tokens. [Certain] They used three different methods to find the best split of compute, and all three gave nearly the same answer: \(N_{\text{opt}} \propto C^{a}\) and \(D_{\text{opt}} \propto C^{b}\) with \(a \approx b \approx 0.5\).
| Method | How it works | \(a\) (params) | \(b\) (tokens) |
|---|---|---|---|
| 1. Training-curve envelope | For each FLOP count, take the lowest loss reached by any run | 0.50 | 0.50 |
| 2. IsoFLOP profiles | Fix the FLOPs, vary \(N\), find the valley | 0.49 | 0.51 |
| 3. Parametric fit | Fit formula (35.5) to all final losses | 0.46 | 0.54 |
| Kaplan et al. (2020) | for comparison | 0.73 | 0.27 |
“Exponent 0.5 for both” means: when compute grows 4 times, make the model 2 times larger and train it on 2 times more tokens. The ratio \(D/N\) stays constant. In their Table 3 (Approach 1) this ratio is about 20 at every scale: 400M parameters with 8.0B tokens, 1B with 20.2B, 10B with 205.1B, 67B with 1.5T. [Certain] That is where the famous “20 tokens per parameter” comes from.
The parametric law
Approach 3 writes the loss as a sum of three terms:
- Read the symbols
- \(E\)
- the irreducible loss: the entropy of the text itself, what a perfect model would still pay
- \(A / N^{\alpha}\)
- the cost of a finite model: even with infinite data, \(N\) parameters cannot represent the perfect predictor
- \(B / D^{\beta}\)
- the cost of finite data and finite training: the model saw only \(D\) tokens, once
- Fitted values
- \(E = 1.69\), \(A = 406.4\), \(B = 410.7\), \(\alpha = 0.34\), \(\beta = 0.28\) (Hoffmann et al., MassiveText data) [Certain]
They fit these five numbers by minimizing a Huber loss (a squared loss that becomes linear for large errors, so outliers matter less) between the predicted and the observed log loss, with \(\delta = 10^{-3}\) and the L-BFGS optimizer. [Certain] You will do the same in the code below.
Gopher: \(N = 280\)B, \(D = 300\)B, so \(C = 6 \cdot 280\times10^9 \cdot 300\times10^9 = 5.04\times10^{23}\) FLOPs. Chinchilla: \(N = 70\)B, \(D = 1.4\)T, so \(C = 5.88\times10^{23}\) FLOPs: about the same budget.
Gopher’s terms: \(406.4 / (2.8\times10^{11})^{0.34} = 0.052\) and \(410.7 / (3\times10^{11})^{0.28} = 0.251\). Total \(1.69 + 0.052 + 0.251 = 1.993\).
Chinchilla’s terms: \(406.4 / (7\times10^{10})^{0.34} = 0.083\) and \(410.7 / (1.4\times10^{12})^{0.28} = 0.163\). Total \(1.69 + 0.083 + 0.163 = 1.937\).
Gopher has a tiny model term and a big data term: it is under-trained. Chinchilla balances the two and gets a lower loss for the same compute, with a model 4 times smaller (so 4 times cheaper to run). In real tests, Chinchilla reached 67.5% on MMLU, more than 7 points above Gopher. [Certain]
IsoFLOP curves
Approach 2 is the easiest one to understand and the one most labs use today. Fix the compute \(C\). For each model size \(N\), the token count is forced: \(D = C/(6N)\). Too small a model has plenty of data but too little capacity; too big a model has capacity but sees too few tokens. In between there is a valley. Its bottom is the compute-optimal \(N\) for that budget. Repeat for several budgets and fit a line through the valley bottoms on log-log axes.
Put \(D = C/(6N)\) into (35.5). Along one isoFLOP curve the loss depends on \(N\) only:
The model term falls as \(N\) grows and the data term rises. The slope of the valley, per factor \(e\) in model size, is
- Read the symbols
- \(L_C(N)\)
- the final loss of a model with \(N\) parameters that uses the whole budget \(C\), so it sees \(D = C/(6N)\) tokens
- \(\alpha A/N^{\alpha}\)
- how much loss you gain from a slightly larger model
- \(\beta B/D^{\beta}\)
- how much loss you lose because that larger model sees fewer tokens
- slope \(\lt 0\)
- a larger model is still better; slope \(\gt 0\): you have passed the valley bottom; slope \(= 0\): the optimum
Figure 2 runs this search step by step. It evaluates 13 model sizes, four per factor of 10, keeps the best one, and then compares it with the exact answer of formula (35.6) below.
How to read Figure 2
Press Play. The orange dot walks from small to large models. In the lower panel, the purple model term falls and the teal data term rises; the loss above is their sum plus \(E\). The live panel shows every number put into the formulas. Watch the sign of the slope (35.5b): it is negative while a bigger model still helps and turns positive after the valley bottom. At \(10^{23}\) FLOPs with the refit, the best of the 13 sizes is 28.9B parameters (loss 2.0321) and the exact optimum is 29.4B on 566B tokens, 19.2 tokens per parameter. On the last step, move the budget. The valley moves right (bigger models) and down (lower loss) as compute grows. With the Besiroglu refit, the optimal tokens per parameter stays close to 20 at every budget (about 26 at \(10^{18}\), 18 at \(10^{24}\)). Switch to the published Hoffmann constants: the ratio drifts upward with compute (about 93 at the Gopher budget), because the rounded constants give \(a = 0.45\). This is the inconsistency the replication found. Notice also that the valley is flat: neighbours of the best grid size, 1.8 times smaller or larger, are only 0.004 to 0.005 nats worse. Being 2 times off in model size costs only a little loss.
The closed-form optimum
We want the lowest \(L(N, D)\) under the constraint \(6ND = C\). Put \(D = C/(6N)\) into (35.5), take the derivative with respect to \(N\), and set it to zero. At the optimum the two shrinking terms are balanced: \(\alpha A / N^{\alpha} = \beta B / D^{\beta}\). Solving gives
So the exponents are \(a = \beta/(\alpha+\beta)\) and \(b = \alpha/(\alpha+\beta)\). They add up to 1, as they must, because \(N \cdot D = C/6\). If the model term falls faster (larger \(\alpha\)), you need fewer parameters and more data, and the other way round. When \(\alpha \approx \beta\), both exponents are about 0.5, and the tokens-per-parameter ratio \(D/N\) is almost constant. With a constant ratio of 20, the whole plan fits in one line:
Rule of 20: \(N = \sqrt{10^{24}/120} = 9.13\times10^{10}\), about 91B parameters, and \(D = 1.83\times10^{12}\), about 1.8T tokens. The exact optimum of the Besiroglu refit (formula 35.6) is 95.9B parameters on 1.74T tokens, 18 tokens per parameter. The rule and the formula agree within 5%.
Figure 3 puts both laws and real models on one chart.
How to read Figure 3
Click each model. GPT-3 and Gopher sit near the gray Kaplan line: about 1 to 2 tokens per parameter. Chinchilla sits on the blue line by design. The Llama models sit far below it: Llama 3 8B used about 1,900 tokens per parameter, about 90 times the Chinchilla ratio. The readout gives the compute-optimal size for the same compute, so you can see how far each model is from the optimum. Llama 3 405B is close to compute-optimal: Meta’s own scaling law gave 402B for its budget, and it is within 1.4 times of the rule-of-20 size.
Why Kaplan and Chinchilla disagreed
The two papers disagree about \(a\): 0.73 or 0.50. At the Gopher budget, Kaplan’s slope suggests a model of about 485B parameters; Chinchilla’s suggests about 68B. That is a 7 times difference, and it decides whether you build Gopher or Chinchilla. Several reasons are known:
- The learning-rate schedule. Kaplan’s runs used one cosine schedule length and read the loss of shorter trainings from intermediate checkpoints. A checkpoint in the middle of a cosine schedule has not yet had its learning rate decayed, so its loss is too high. The Chinchilla authors found that if the cosine cycle is more than 25% longer than the run, the result is noticeably worse, and that intermediate losses therefore make small-data training look bad. [Certain] This pushes the fit toward “bigger model, fewer tokens”.
- Small models and non-embedding parameters. Most of Kaplan’s runs were small (many under 100M parameters), and he counted only non-embedding parameters. At small sizes, the embeddings are a large fraction of the model, and the curve is not straight yet. Pearce and Song showed that redoing the Chinchilla analysis with Kaplan’s counting and scale gives exponents close to Kaplan’s. [Certain]
- Other details. Porian et al. reproduced Kaplan’s law and found three more causes: the compute of the last (output) layer was ignored, warmup was too long for small models, and the optimizer was not tuned for each scale. After fixing them, their result matches Chinchilla. [Certain]
And Chinchilla itself had a flaw. Besiroglu et al. rebuilt the data from the paper’s plots and found that the published Approach 3 fit does not match Approaches 1 and 2, and that its confidence intervals were implausibly narrow. Their refit gives about 20 tokens per parameter, in agreement with the other two approaches. [Certain] That refit is what Figures 2, 4, 5 and 6 use.
Compute-optimal is not the goal: inference-optimal training
Chinchilla answers one question: what is the lowest loss for a fixed training budget? But a deployed model also costs money every time it runs. Generating one token costs about \(2N\) FLOPs (session 25). If the model will serve \(D_{\text{inf}}\) tokens over its life, the total compute is
Now fix a target quality instead of a budget. For each model size \(N\), formula (35.5) tells you how many tokens you need to reach a target loss \(L^*\):
A smaller model needs more training tokens to reach the same loss, but every served token is cheaper. If \(D_{\text{inf}}\) is large, the best \(N\) moves below the Chinchilla size. If \(N\) is too small, the model can never reach \(L^*\), even with infinite data (the condition in 35.9). Sardana et al. built exactly this analysis and concluded that a model expecting large demand (around a billion requests) should be smaller and trained longer than Chinchilla-optimal. They trained 47 models and saw quality still improving at ratios up to 10,000 tokens per parameter. [Certain]
How to read Figure 4
With inference ignored, the best size is the Chinchilla size (about 20 tokens per parameter). Turn inference on and slide the served tokens up. At \(10^{13}\) served tokens for a \(10^{23}\)-FLOP quality target, the best model is about 11B instead of 29B, trained on about 214 tokens per parameter, and the total compute falls by about 43%. At \(10^{14}\) the ratio passes 1,000. The training cost goes up (blue curve to the left of the gray line), but the total goes down. That is the Llama strategy.
How much does over-training cost?
Very little, because the isoFLOP valley is flat. At a fixed budget of \(1.05\times10^{24}\) FLOPs, the Besiroglu refit predicts a loss of 1.9585 at 20 tokens per parameter, 1.9651 at 100, 1.9715 at 200 and 1.9957 at 1,000. Training a 5 times smaller model (100 instead of 20 tokens per parameter) costs only 0.007 nats of loss for the same compute, and the model is 2.2 times cheaper to run. This is a prediction of the fit, far outside the ratios it was fit on; Sardana et al. warn that fits made on normal ratios overestimate the value of extra tokens at extreme ratios. [Certain]
Real practice: Llama 2 7B saw 2T tokens (286 per parameter). Llama 3 8B saw over 15T tokens (about 1,900 per parameter). Meta wrote that the Chinchilla-optimal amount of data for an 8B model is about 200B tokens, and that both the 8B and 70B models “continued to improve log-linearly” up to 15T tokens. [Certain]
What scaling laws do not tell you
- Constants do not transfer. \(E, A, B\) depend on the data mixture, the tokenizer and the architecture. A loss of 1.9 on MassiveText and 1.9 on your data are not the same thing. Fit your own law on your own data.
- Loss is not accuracy. The laws predict cross-entropy, not benchmark scores. Downstream accuracy often looks like a sudden jump. Llama 3 used two steps: fit the model’s log-likelihood of the correct answer against FLOPs, then fit a sigmoid from that log-likelihood to accuracy. They report that this predicted the final ARC Challenge score of the 405B model from runs up to \(10^{22}\) FLOPs, a little below the real score. [Certain]
- Extrapolation is risky. Fits are made over 2 to 4 orders of magnitude and used 2 to 4 more beyond. Small errors in the exponents grow fast: Kaplan noted his own laws contradict each other near \(10^{12}\) parameters. [Certain] The code below shows how 1% noise in 45 runs moves the predicted ratio from 18 to 27.
- Hyperparameters matter. A law fit with badly tuned learning rates is a law of bad tuning. Use a schedule that matches each run’s length, or WSD (warmup-stable-decay) with a decay at each measured point (Hägele et al.), and transfer learning rates across widths with μP (session 36). [Likely]
- The architecture changes the law. Mixture-of-experts models need their own laws, with the number of experts as a new variable (Clark et al.). [Certain] Vision transformers (Zhai et al.) and CLIP (Cherti et al.) also follow power laws, with different exponents. [Certain]
Using scaling laws to plan a training run
Here is the full planning chain. It starts from hardware, not from a model size. The budget in FLOPs is
- Read the symbols
- \(n_{\text{GPU}}\)
- number of GPUs
- \(P_{\text{peak}}\)
- peak dense BF16 FLOP/s of one GPU: about 989 TFLOP/s for an H100 SXM, 312 for an A100 (session 42) [Certain]
- \(\text{MFU}\)
- model FLOPs utilization: the fraction of peak you actually reach; 35% to 50% is typical for large dense training [Likely]
- \(t\)
- wall-clock training time in seconds
- input: 4 to 6 small budgets \(C_1 \lt C_2 \lt \dots\), spaced by about 3× (for example \(10^{18}\) to \(10^{20}\) FLOPs)
- for each budget \(C_j\):
- choose 7 to 9 model sizes \(N\) spread over about 10× around the expected optimum
- train each one on \(D = C_j / (6N)\) tokens, with the learning-rate schedule matched to that length
- fit a parabola to the final loss against \(\log N\); its minimum is \(N_{\text{opt}}(C_j)\)
- fit a straight line \(\log N_{\text{opt}} = a \log C + \text{const}\) through the minima // the slope is a, near 0.5
- return the line; check it on one larger budget before trusting it
Step 5 of Algorithm 1 fits a parabola in \(x = \ln N\) to the measured losses of one budget. Its lowest point gives the optimal size:
The parabola needs \(c_2 \gt 0\) (a valley, not a hill). It is only an approximation: the true curve (35.5a) is not symmetric in \(\ln N\), so even with perfect measurements the parabola misses the true minimum by a few percent. Figure 5 runs the whole of Algorithm 1 on the same grid as the PyTorch code below: 5 budgets, 9 sizes each, 45 runs.
How to read Figure 5
Each budget takes two steps: first its 9 runs appear, then the parabola is fitted and its minimum is read with (35.11). The live panel shows the fitted coefficients and the error against the true optimum. Start with “No noise”: the parabola minima are already off (−2.1% at \(10^{18}\), −8.3% at \(10^{20}\), where the grid is lopsided around the valley), the fitted slope is 0.503 instead of 0.513, and the prediction at \(10^{24}\) is 82.9B parameters with 24.2 tokens per parameter, while the truth is 95.9B with 18.1. Then add noise. With 1% noise and seed 1, the slope drops to 0.459 and the prediction becomes 48.9B parameters with 69.6 tokens per parameter: half the true size, from noise you could not see in any single run. Press “New noise seed” a few times: the slope and the prediction move from seed to seed. Extrapolating four orders of magnitude amplifies small errors in the slope.
- input: GPUs, days, expected MFU, expected serving volume, available unique tokens \(D_{\text{data}}\)
- \(C \leftarrow n_{\text{GPU}} \cdot P_{\text{peak}} \cdot \text{MFU} \cdot t\) // formula (35.10)
- \(N_{\text{opt}}, D_{\text{opt}} \leftarrow\) your fitted law at \(C\) // formula (35.6), or roughly (35.7)
- if the model will serve many tokens: choose a ratio \(r = D/N\) above 20 (formula 35.8, Figure 4); set \(N = \sqrt{C/(6r)}\), \(D = rN\)
- if \(D \gt\) about \(4 D_{\text{data}}\): the plan needs too many epochs; use a bigger model and fewer tokens, or get more data
- check memory: the model, its optimizer states and activations must fit the cluster (session 38)
- predict the final loss with (35.5); during training, compare the real loss curve against it
How to read Figure 6
The defaults (1,024 H100s, 30 days, 40% MFU) give about \(1.05\times10^{24}\) FLOPs, so a compute-optimal plan of about 94B parameters on 1.9T tokens. Raise the tokens per parameter to see the inference-friendly plans: the model shrinks with the square root of the ratio, and the loss penalty stays small until the ratio is in the hundreds. The “tokens/s needed” box tells you the throughput your cluster must sustain, which is a good sanity check against your measured speed.
\(C = 1024 \times 989\times10^{12} \times 0.40 \times 30 \times 86{,}400 = 1.05\times10^{24}\) FLOPs. Rule of 20: \(N = \sqrt{1.05\times10^{24}/120} = 93.5\)B parameters, \(D = 1.87\)T tokens. The cluster must process \(1.87\times10^{12} / (30 \times 86{,}400) = 722{,}000\) tokens per second for the whole month.
If the model will serve a lot of traffic, choose 200 tokens per parameter instead: \(N = \sqrt{1.05\times10^{24}/1200} = 29.6\)B on 5.9T tokens. The refit predicts a loss of 1.9715 instead of 1.9585, and the model is 3.2 times cheaper to serve. But it needs 5.9T good tokens, so check the data first.
Fitting a scaling law in PyTorch
The code creates 45 fake runs (5 budgets × 9 sizes) from a known law plus 1% noise, then (a) finds each isoFLOP valley with a parabola (Algorithm 1), and (b) fits all five constants of (35.5) with a Huber loss on log loss, as in the Chinchilla paper. The log-sum-exp form keeps \(E\), \(A\), \(B\) positive.
import math, torch torch.manual_seed(0) # 1) a "true" law (the Besiroglu et al. refit) to create fake runs E0, A0, B0, a0, b0 = 1.8172, 482.01, 2085.43, 0.3478, 0.3658 def true_loss(N, D): return E0 + A0 / N**a0 + B0 / D**b0 # 2) an isoFLOP grid: 5 budgets x 9 model sizes, D = C / (6N), 1% noise runs = [] for C in [1e18, 3e18, 1e19, 3e19, 1e20]: for N in torch.logspace(7, 9.5, 9).tolist(): D = C / (6 * N) runs.append((N, D, true_loss(N, D) * math.exp(0.01 * torch.randn(1).item()))) N = torch.tensor([r[0] for r in runs], dtype=torch.float64) D = torch.tensor([r[1] for r in runs], dtype=torch.float64) L = torch.tensor([r[2] for r in runs], dtype=torch.float64) # 3) Algorithm 1 (isoFLOP): fit a parabola in log N per budget, take its minimum for C in [1e18, 3e18, 1e19, 3e19, 1e20]: m = (6 * N * D - C).abs() / C < 1e-6 x = N[m].log(); X = torch.stack([x**2, x, torch.ones_like(x)], 1) c2, c1, _ = torch.linalg.lstsq(X, L[m].unsqueeze(1)).solution.squeeze(1) Nopt = math.exp(-c1 / (2 * c2)) print(f"C={C:.0e} N_opt={Nopt:.3g} D_opt/N_opt={C / (6 * Nopt) / Nopt:.1f}") # 4) Approach 3: fit L = E + A/N^a + B/D^b with a Huber loss on log L (formula 35.5) # parameters: e = log E, la = log A, lb = log B, a, b p = torch.tensor([0.5, 6.0, 7.0, 0.3, 0.3], dtype=torch.float64, requires_grad=True) def pred_log(p): e, la, lb, a, b = p terms = torch.stack([e.expand_as(N), la - a * N.log(), lb - b * D.log()]) return torch.logsumexp(terms, 0) # log(E + A/N^a + B/D^b) huber = torch.nn.HuberLoss(delta=1e-3, reduction="sum") opt = torch.optim.LBFGS([p], lr=1.0, max_iter=500, line_search_fn="strong_wolfe") def closure(): opt.zero_grad(); loss = huber(pred_log(p), L.log()); loss.backward(); return loss opt.step(closure) e, la, lb, a, b = p.detach().tolist() E_, A_, B_ = math.exp(e), math.exp(la), math.exp(lb) print(f"fit: E={E_:.3f} A={A_:.1f} B={B_:.1f} alpha={a:.3f} beta={b:.3f}") # 5) use the fit: compute-optimal N and D for a big budget (formula 35.6) C = 1e24 G = (a * A_ / (b * B_)) ** (1 / (a + b)) N_opt = G * (C / 6) ** (b / (a + b)); D_opt = C / 6 / N_opt print(f"C=1e24: N_opt={N_opt:.3g} D_opt={D_opt:.3g} tokens/param={D_opt / N_opt:.1f}")
Output (PyTorch 2.x, CPU):
C=1e+18 N_opt=8.27e+07 D_opt/N_opt=24.4 C=3e+18 N_opt=1.36e+08 D_opt/N_opt=27.1 C=1e+19 N_opt=2.65e+08 D_opt/N_opt=23.8 C=3e+19 N_opt=4.63e+08 D_opt/N_opt=23.4 C=1e+20 N_opt=8.65e+08 D_opt/N_opt=22.3 fit: E=1.721 A=400.2 B=1097.9 alpha=0.335 beta=0.331 C=1e24: N_opt=7.83e+10 D_opt=2.13e+12 tokens/param=27.2
Read it carefully. The isoFLOP minima are close to the truth (the true optima are \(8.05\times10^7\) at \(10^{18}\) and \(8.53\times10^8\) at \(10^{20}\)). The parametric fit looks reasonable, but its constants are off (\(B\) is half the true value), and its extrapolation to \(10^{24}\) says 27 tokens per parameter, while the true law says 18. Only 1% noise did that. Five constants are hard to pin down from a small grid, because \(E\), \(B\) and \(\beta\) can trade off against each other. This is why Chinchilla’s Approach 3 disagreed with Approaches 1 and 2, and why you should always cross-check methods.
Where this is used
| Setting | How scaling laws are used | Example |
|---|---|---|
| Choosing a flagship model size | isoFLOP experiments up to about \(10^{22}\) FLOPs, extrapolated to the real budget | Llama 3: law predicted 402B parameters on 16.55T tokens for \(3.8\times10^{25}\) FLOPs; Meta trained 405B [Certain] |
| Small models for cheap serving | inference-aware cost (35.8): train far past 20 tokens per parameter | Llama 3 8B on over 15T tokens [Certain] |
| Predicting benchmark scores | loss → log-likelihood of correct answer → accuracy (sigmoid) | Llama 3 predicted ARC Challenge for 405B [Certain] |
| Data-limited domains (code, a language with little text) | data-constrained laws: how many epochs, when to add other data | Muennighoff et al.: up to about 4 epochs nearly free [Certain] |
| Mixture of experts | laws in parameters and number of experts | Clark et al. [Certain]; see session 29 |
| Vision backbones and CLIP-style encoders | power laws in model size, data and compute for image classification and zero-shot retrieval | ViT (Zhai et al.), OpenCLIP (Cherti et al.) [Certain]; see session 166 |
| Video retrieval and action recognition encoders | the same planning logic: decide the encoder size versus the number of video-text pairs before a big pretraining run | [Speculation] public video scaling studies are fewer; fit your own small grid |
| Comparing architectures or data recipes | compare whole curves (loss versus compute), not one point; a better recipe shifts the line down or makes it steeper | data-filtering ablations in session 34 |
Common mistakes
- Reading losses from intermediate checkpoints of one long cosine schedule. Their learning rate has not decayed, so they overestimate the loss of a short run. This is the main bug behind Kaplan’s too-large models. Match the schedule to each run, or use WSD with a cooldown at each point.
- Treating “20 tokens per parameter” as a law of nature. It is the compute-optimal ratio for one data mixture and one family of models. Your data, tokenizer and architecture can move it. And compute-optimal is often not what you want (Figure 4).
- Comparing losses across tokenizers or datasets. Loss per token depends on what a token is. Use bits per byte (or per character) when tokenizers differ (session 40).
- Mixing parameter definitions. Kaplan counts non-embedding parameters; Chinchilla counts all. For small models the difference is large and changes the fitted exponent.
- Fitting on too narrow a range. A fit over one order of magnitude cannot pin down the exponents; extrapolating it 1,000 times is a guess.
- Trusting one fitting method. As the code shows, a five-constant fit can look fine and still extrapolate badly. Cross-check isoFLOP minima and the parametric fit.
- Forgetting the data supply. An over-trained plan that needs 20T tokens is useless if you have 3T unique good tokens.
- Ignoring MFU in planning. Peak FLOP/s is not real FLOP/s. A plan made at 100% MFU is 2 to 3 times too optimistic.
Interview questions
What is a scaling law, and why is it useful?basic
Why do people plot scaling results on log-log axes?basic
What does “Chinchilla-optimal” mean? Give the numbers.core
Why did Kaplan et al. recommend much larger models than Chinchilla?core
Derive the compute-optimal \(N\) and \(D\) from \(L = E + A/N^\alpha + B/D^\beta\) and \(C = 6ND\).advanced
You have \(10^{24}\) FLOPs. What model do you train?core
Why was Llama 3 8B trained on 15T tokens, about 75× its Chinchilla-optimal data?core
What is the irreducible loss \(E\), and why does it matter for extrapolation?core
How would you fit your own scaling law, and what can go wrong?advanced
Do scaling laws predict downstream accuracy and “emergent” abilities?advanced
Your unique data is limited to 2T tokens but your budget suggests 6T. What do you do?advanced
Exercises
- With Kaplan’s \(\alpha_N = 0.076\), how much larger must a model be to cut the loss by 10%? By how much does a 10× larger model cut it?
- Your budget is \(6\times10^{23}\) FLOPs. Use the rule of 20 to find \(N\) and \(D\). Which famous model is this?
- Llama 3 8B trained on 15T tokens. What is its training compute? What would the Chinchilla-optimal model size and token count be for that compute?
- A 7B model trained on 2T tokens serves \(10^{12}\) tokens per day for one year. Compare training and inference FLOPs.
- Plan a run with 256 H100s for 14 days at 40% MFU, compute-optimal. Give \(C\), \(N\) and \(D\).
- If \(a = b = 0.5\) and the compute grows 10 times, how much larger should the model be, and how many more tokens?
- In formula (35.6), what happens to the optimal tokens per parameter as \(C\) grows if \(\alpha \gt \beta\)? Use the published Hoffmann constants as an example.
Answerscheck
Key takeaways
- Language-model loss falls as a power law (35.1) in parameters, data and compute: a straight line on log-log axes (35.2), with an irreducible floor \(E\) near the end.
- Kaplan et al. (35.3) found small exponents (0.076, 0.095, 0.050) and advised \(N \propto C^{0.73}\): big models, few tokens (GPT-3: 1.7 tokens per parameter).
- Chinchilla’s law \(E + A/N^\alpha + B/D^\beta\) (35.5), with \(C = 6ND\) (35.4), gives the closed-form optimum (35.6): \(N\) and \(D\) both grow as \(C^{0.5}\), about 20 tokens per parameter, so \(N \approx \sqrt{C/120}\) (35.7).
- Kaplan was biased by an unmatched learning-rate schedule, small models and non-embedding counting; Chinchilla’s own parametric fit was later corrected (Besiroglu et al.).
- Including inference (35.8, 35.9) makes smaller, longer-trained models optimal; the flat isoFLOP valley makes over-training cheap in loss, as long as you have the data.
- To plan: GPUs × peak × MFU × time (35.10) → \(C\) → your own fitted law → \(N\), \(D\) → check data, memory, and serving cost.
Key formulas
References for this session
- Kaplan et al. (2020). Scaling Laws for Neural Language Models
- Hoffmann et al. (2022). Training Compute-Optimal Large Language Models
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Kaplan et al. (2020). Scaling Laws for Neural Language Models
- Hoffmann et al. (2022). Training Compute-Optimal Large Language Models
- Austin et al., Google DeepMind (2025). How to Scale Your Model
- Besiroglu, Erdil, Barnett, You (2024). Chinchilla Scaling: A replication attempt
- Porian, Wortsman, Jitsev, Schmidt, Carmon (2024). Resolving Discrepancies in Compute-Optimal Scaling of Language Models
- Pearce, Song (2024). Reconciling Kaplan and Chinchilla Scaling Laws
- Sardana, Portes, Doubov, Frankle (2023). Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Meta AI (2024). Introducing Meta Llama 3: The most capable openly available LLM to date
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Brown et al. (2020). Language Models are Few-Shot Learners
- Rae et al. (2021). Scaling Language Models: Methods, Analysis & Insights from Training Gopher
- Muennighoff et al. (2023). Scaling Data-Constrained Language Models
- Hägele et al. (2024). Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
- Clark et al. (2022). Unified Scaling Laws for Routed Language Models
- Zhai, Kolesnikov, Houlsby, Beyer (2022). Scaling Vision Transformers
- Cherti et al. (2023). Reproducible scaling laws for contrastive language-image learning