LLM & VLM courseBlog
4Track 1: Foundations · session 4 of 15

Optimization: backpropagation, SGD, Adam and AdamW, learning-rate schedules

Training means changing billions of numbers so the loss goes down. This session shows how the gradient of every weight is found (backpropagation), how we use it to take a step (SGD, momentum, Adam, AdamW), and how the step size changes during training (warmup, cosine, warmup-stable-decay). These choices decide whether a large training run works or blows up.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 1 to 3

What you will learn

  • What a gradient is, and how backpropagation computes all gradients with one backward pass
  • Why the backward pass costs about twice the forward pass, and why it needs memory for activations
  • Gradient descent, its stability limit, and why badly shaped losses are slow
  • SGD with mini-batches, momentum, Adam with bias correction, and why AdamW is different from Adam with L2
  • Optimizer memory: 16 bytes per parameter, and what that means for an 8B model
  • Learning-rate schedules (warmup, cosine, linear, warmup-stable-decay) and gradient clipping, with the real Llama 2 recipe

Why this matters

A model starts with random weights and a loss of about 11.8 nats. Training is the process that brings the loss down, one small step at a time, for hundreds of thousands of steps. The optimizer decides each step.

Two questions must be answered at every step: which direction should each of the billions of weights move (the gradient, computed by backpropagation), and how far (the learning rate, the optimizer’s rules, and the schedule). Get the first wrong and nothing learns. Get the second wrong and the loss explodes into NaN after a week of GPU time.

This session gives you the full toolkit used to train real LLMs. Llama 2, for example, was trained with AdamW (\(\beta_1 = 0.9\), \(\beta_2 = 0.95\), \(\varepsilon = 10^{-5}\)), weight decay 0.1, gradient clipping at 1.0, 2,000 warmup steps, and a cosine schedule down to 10% of the peak learning rate (\(3 \times 10^{-4}\) for the 7B model). [Certain] By the end of this session you will know why each of those numbers is there.

Gradients: which way is downhill?

For a function of one number, the derivative \(f'(w)\) is the slope: how much \(f\) changes when \(w\) moves a tiny bit. For a function of many numbers \(\theta = (\theta_1, \dots, \theta_N)\), each partial derivative \(\partial \mathcal{L} / \partial \theta_i\) is the slope along one weight, with all others held still. The list of all of them is the gradient:

\[ \nabla_\theta \mathcal{L} = \left( \frac{\partial \mathcal{L}}{\partial \theta_1},\ \frac{\partial \mathcal{L}}{\partial \theta_2},\ \dots,\ \frac{\partial \mathcal{L}}{\partial \theta_N} \right) \tag{4.1} \]

The gradient points in the direction where the loss grows fastest. So the steepest way down is \(-\nabla_\theta \mathcal{L}\). For Llama 3 8B, this vector has about 8 billion numbers, one per weight.

The chain rule

A network is a chain of simple functions. If \(L\) depends on \(a\), and \(a\) depends on \(z\), then the effect of \(z\) on \(L\) is the product of the two local effects:

\[ \frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z} \tag{4.2} \]

“If moving \(z\) by 1 moves \(a\) by 0.2, and moving \(a\) by 1 moves \(L\) by −0.8, then moving \(z\) by 1 moves \(L\) by −0.16.” Backpropagation is nothing more than this rule, applied from the loss back to every weight.

Backpropagation

We draw the computation as a graph: each node is one simple operation. Then we do two passes:

  1. Forward pass: compute every node’s value from the inputs to the loss, and keep the values we will need later.
  2. Backward pass: start from \(\partial L / \partial L = 1\) and walk back. At each node, multiply the incoming gradient by the node’s local derivative (formula 4.2) and send the result to its inputs. If a value feeds several nodes, add up the gradients that come back from each of them.

Let us do it by hand on the smallest possible network: one input \(x\), one weight \(w\), one bias \(b\), a sigmoid, and a squared error to the target \(y\):

\[ z = w x + b, \qquad a = \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad L = (a - y)^2 \tag{4.3} \]

With \(x = 2\), \(w = 0.5\), \(b = -0.5\) and \(y = 1\), step through Figure 1. It runs the real forward pass, backward pass and update in your browser, line by line, for several training iterations. The box on the right shows each formula with the current numbers put in.

forward valuegradient \(\partial L/\partial(\text{node})\), flowing backupdated weight
  1. for iteration \(= 1, \dots, K\):
  2. \(z \leftarrow w x + b\) // forward (4.3)
  3. \(a \leftarrow \sigma(z)\)
  4. \(L \leftarrow (a - y)^2\) // keep \(z, a\) in memory
  5. \(\partial L/\partial a \leftarrow 1 \cdot 2(a - y)\) // backward starts at 1
  6. \(\partial L/\partial z \leftarrow \partial L/\partial a \cdot a(1 - a)\) // chain rule (4.2)
  7. \(\partial L/\partial w \leftarrow \partial L/\partial z \cdot x\), \(\;\partial L/\partial b \leftarrow \partial L/\partial z \cdot 1\)
  8. \(w \leftarrow w - \eta\, \partial L/\partial w\), \(\;b \leftarrow b - \eta\, \partial L/\partial b\) // (4.5)
Figure 1. Backpropagation and gradient descent, computed live. Forward steps fill in the blue values from the inputs to the loss; a blue pulse carries each value along its edge. Backward steps fill in the orange gradients from the loss back to the inputs; an orange pulse carries each gradient back, and the new gradient is the incoming one times a local derivative. The update step turns \(w\) and \(b\) green. Then the next iteration starts.
How to read Figure 1

Press Play with the default values. In iteration 1 the forward pass gives \(z = 0.5\), \(a = 0.622\) and \(L = 0.1425\). The backward pass gives \(\partial L/\partial a = -0.755\), then \(\partial L/\partial z = -0.177\), then \(\partial L/\partial w = -0.355\) and \(\partial L/\partial b = -0.177\). Both gradients are negative, so the update makes \(w\) and \(b\) larger. The loss falls from 0.1425 to 0.0399 (iteration 2) and 0.0236 (iteration 3). The steps get smaller because \(a(1 - a)\) shrinks as \(a\) moves toward 1: the sigmoid becomes flat. Notice two things. First, the backward pass needs the forward values: \(\sigma'(z) = a(1-a)\) uses \(a\), and \(\partial z/\partial w = x\) uses \(x\). That is why training must store activations, and why memory is the big problem in training (Track 3). Second, the gradients for \(w\) and \(b\) both reuse \(\partial L/\partial z\). Backprop computes each shared piece once; that reuse is what makes it fast. Try \(y = 0\) (exercise 1). Or set the start \(w = -2\): then \(z = -4.5\) and \(a = 0.011\), the sigmoid is almost flat there (\(a(1 - a) = 0.011\)), and in 6 iterations the loss only falls from 0.9781 to 0.9572. This is a vanishing gradient: the chain rule multiplies by a tiny local derivative.

The numbers from Figure 1

Forward: \(z = 0.5 \cdot 2 - 0.5 = 0.5\), \(a = \sigma(0.5) = 0.622\), \(L = (0.622 - 1)^2 = 0.143\).

Backward: \(\partial L/\partial a = 2(a - y) = -0.755\). Then \(\partial a/\partial z = a(1 - a) = 0.235\), so \(\partial L/\partial z = -0.755 \times 0.235 = -0.177\). Finally \(\partial L/\partial w = \partial L/\partial z \cdot x = -0.355\) and \(\partial L/\partial b = -0.177\).

One step with learning rate 1: \(w \leftarrow 0.5 + 0.355 = 0.855\), \(b \leftarrow -0.5 + 0.177 = -0.323\). The new loss is \(0.040\), down from \(0.143\).

Backprop through a linear layer

The same rule, written for matrices, gives the two formulas you need for every linear layer \(Y = XW\) with \(X \in \R^{B \times n}\) and \(W \in \R^{n \times m}\). Let \(G = \partial L / \partial Y\) (same shape as \(Y\)) be the gradient that arrives from above:

\[ \frac{\partial L}{\partial W} = X^\top G \in \R^{n \times m}, \qquad \frac{\partial L}{\partial X} = G\, W^\top \in \R^{B \times n} \tag{4.4} \]

Check the shapes: \((n \times B)(B \times m) = n \times m\), the shape of \(W\). Good. Each formula is one matrix multiplication with the same cost as the forward \(XW\). So the backward pass costs about twice the forward pass: one multiplication for the weight gradient, one for the input gradient. Forward \(2N\) plus backward \(4N\) gives the \(6N\) FLOPs per token from Session 1.

Gradient descent, and why it can be slow or unstable

The simplest optimizer takes a step against the gradient, scaled by the learning rate \(\eta\):

\[ \theta_{t+1} = \theta_t - \eta \, \nabla_\theta \mathcal{L}(\theta_t) \tag{4.5} \]

Too small an \(\eta\), and training crawls. Too large, and each step jumps over the valley and lands higher than before. For a bowl-shaped loss we can say exactly where “too large” starts. Near a minimum, the loss looks like \(\tfrac12 \sum_i \lambda_i u_i^2\), where the \(\lambda_i\) are the curvatures along its main directions (the eigenvalues of the Hessian). Gradient descent is stable only if

\[ \eta \lt \frac{2}{\lambda_{\max}} \tag{4.6} \]

But progress along the flattest direction is proportional to \(\eta \lambda_{\min}\). So when \(\lambda_{\max} / \lambda_{\min}\) (the condition number) is large, the steep direction forces a small \(\eta\), and the flat direction then takes forever. The loss looks like a long, narrow ravine. Deep networks have such ravines everywhere, and this is the main reason we need better optimizers than plain gradient descent. Figure 2 shows a ravine with \(\lambda = 1\) and \(\lambda = 50\).

gradient descentmomentum (\(\beta = 0.9\))Adam (\(\beta_1 = 0.9,\ \beta_2 = 0.999\))
Figure 2. Three optimizers on the loss \(f(x, y) = \tfrac12 (x^2 + 50 y^2)\), starting from the same point. The minimum is at the center. Gradient descent and momentum share one learning rate. Adam has its own, because Adam’s learning rate is a step size in weight units, not a multiplier on the gradient.
How to read Figure 2

Plain gradient descent zig-zags across the narrow direction and creeps along the long one. Push its \(\eta\) above 0.04 and it diverges, exactly as formula (4.6) says (\(\lambda_{\max} = 50\)): it can never go fast along the flat direction. Momentum cancels the zig-zag and builds speed along the ravine; it stays stable up to about \(\eta = 0.076\). Adam scales each coordinate separately, so the steep and flat directions move at similar speed. Turn on noisy gradients: every method wanders, which is what small batches do in real training.

Stochastic gradient descent and mini-batches

The true loss is an average over the whole dataset (trillions of tokens). We cannot compute its gradient for every step. Instead we take a random mini-batch of \(B\) examples and use their average gradient as an estimate:

\[ \mathbf{g}_B = \frac{1}{B} \sum_{i=1}^{B} \nabla_\theta \ell_i(\theta), \qquad \E[\mathbf{g}_B] = \nabla_\theta \mathcal{L}, \qquad \operatorname{Var}(\mathbf{g}_B) = \frac{\sigma^2}{B} \tag{4.7} \]

The estimate is right on average (unbiased) but noisy, and the noise shrinks as \(1/B\). This is stochastic gradient descent (SGD). Bigger batches give cleaner gradients but cost more per step; beyond a certain “critical batch size”, doubling the batch no longer halves the number of steps. A common rule for SGD is the linear scaling rule: multiply the batch by \(k\), multiply the learning rate by \(k\) (with warmup). For Adam, the best learning rate grows more slowly with batch size. [Likely] LLMs count batches in tokens: Llama 2 used about 4 million tokens per step.

Momentum

Momentum keeps a running average of past gradients and steps along it:

\[ \mathbf{v}_{t} = \beta\, \mathbf{v}_{t-1} + \mathbf{g}_t, \qquad \theta_{t+1} = \theta_t - \eta\, \mathbf{v}_t \tag{4.8} \]

Components that keep the same sign add up; components that flip sign (the zig-zag) cancel. With \(\beta = 0.9\), a steady gradient eventually produces steps \(1/(1 - \beta) = 10\) times larger, so the effective learning rate along a consistent direction is \(10\eta\).

Adam

Adam keeps two running averages for every weight: the mean of the gradient \(\mathbf{m}\), and the mean of the squared gradient \(\mathbf{v}\). Then it divides one by the square root of the other:

\[ \begin{aligned} \mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1 - \beta_1)\, \mathbf{g}_t \\ \mathbf{v}_t &= \beta_2 \mathbf{v}_{t-1} + (1 - \beta_2)\, \mathbf{g}_t^2 \\ \hat{\mathbf{m}}_t &= \frac{\mathbf{m}_t}{1 - \beta_1^t}, \qquad \hat{\mathbf{v}}_t = \frac{\mathbf{v}_t}{1 - \beta_2^t} \\ \theta_{t+1} &= \theta_t - \eta\, \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t} + \varepsilon} \end{aligned} \tag{4.9} \]
Read the symbols
\(\mathbf{g}_t^2\)
the gradient squared, each component separately
\(\beta_1, \beta_2\)
how long each average remembers: 0.9 is about the last 10 steps; 0.95 about 20; 0.999 about 1,000
\(\hat{\mathbf{m}}_t, \hat{\mathbf{v}}_t\)
bias-corrected averages (explained below)
\(\varepsilon\)
a tiny number (\(10^{-8}\) or \(10^{-5}\)) that prevents division by zero

What does this achieve? The ratio \(\hat m / \sqrt{\hat v}\) is roughly “average gradient divided by its typical size”. If a weight’s gradient is consistently positive, the ratio is near 1 and the weight moves about \(\eta\) per step. If the gradient is mostly noise that flips sign, \(\hat m\) is near 0 and the weight barely moves. And a weight with tiny gradients moves as fast as one with huge gradients. Each weight gets its own step size. This fixes the ravine problem of Figure 2.

Why bias correction?

Both averages start at 0. After the first step, \(m_1 = (1 - \beta_1) g_1 = 0.1\, g_1\): ten times too small. In general, \(\E[m_t] = (1 - \beta_1^t)\, \E[g]\), so dividing by \(1 - \beta_1^t\) removes the bias exactly. The same holds for \(v\).

Worked example: the very first Adam step

With bias correction, \(\hat m_1 = g_1\) and \(\hat v_1 = g_1^2\), so the first update is \(\eta\, g_1 / |g_1| = \pm\eta\), whatever the gradient’s size. Without correction and with \(\beta_1 = 0.9\), \(\beta_2 = 0.999\): the update would be \(\eta \cdot 0.1 g_1 / \sqrt{0.001\, g_1^2} = 3.16\,\eta\). The first steps would be more than 3 times too large, at the moment the model is most fragile. With \(\beta_2 = 0.95\) the factor would be 0.45 instead. Explore it in Figure 3.

gradient \(g\) (dots)raw average \(m\)corrected average \(\hat m\)update size \(\div\,\eta\)
Figure 3. Adam on a single weight whose gradient is about 1 plus noise. The orange line is the size of each update in units of \(\eta\). Ideally it should be close to 1 from the first step.

AdamW: weight decay done right

Weight decay shrinks every weight a little each step, which keeps weights small and acts as regularization. With plain SGD, adding an L2 penalty \(\tfrac{\lambda}{2}\lVert\theta\rVert^2\) to the loss gives exactly the same thing. With Adam it does not. In “Adam with L2”, the penalty’s gradient \(\lambda\theta\) is added to \(\mathbf{g}\) before the division by \(\sqrt{\hat{\mathbf{v}}}\). Weights with large gradients then get almost no decay, and weights with small gradients get a lot. AdamW (Loshchilov and Hutter) decouples the decay from the adaptive part:

\[ \theta_{t+1} = \theta_t - \eta \left( \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t} + \varepsilon} + \lambda\, \theta_t \right) \tag{4.10} \]

Here \(\mathbf{m}\) and \(\mathbf{v}\) are computed from the loss gradient only, and every weight shrinks by the same fraction \(\eta\lambda\) per step. With \(\eta = 3 \times 10^{-4}\) and \(\lambda = 0.1\), that is \(0.003\%\) per step. AdamW is the standard optimizer for transformers. Weight decay is usually not applied to biases and normalization weights, and often not to embeddings. [Likely]

Watch every optimizer line run

Figure 4 runs SGD, momentum, Adam, AdamW and “Adam with L2” side by side on one 2D loss, \(f(x, y) = \tfrac12\big((x - 2.5)^2 + 20\,(y - 0.5)^2\big)\), all from the start point \((-4.5, -1.5)\). The minimum is at \((2.5, 0.5)\), away from the origin, so you can see weight decay pull toward 0. Pick one optimizer to inspect: its algorithm box highlights the line being run, and the table shows its real \(\mathbf{g}\), \(\mathbf{m}\), \(\mathbf{v}\), \(\hat{\mathbf{m}}\), \(\hat{\mathbf{v}}\) for both coordinates at every step.

Inspect
SGDmomentumAdamAdamWAdam + L2\(-\mathbf{g}\) direction
  1. \(\theta \leftarrow \theta_0\)
  2. for \(t = 1, 2, \dots\):
  3. \(\mathbf{g} \leftarrow \nabla f(\theta)\)
  4. \(\theta \leftarrow \theta - \eta\, \mathbf{g}\) // (4.5)
Figure 4. Five optimizers stepping on \(f(x, y) = \tfrac12\big((x - 2.5)^2 + 20\,(y - 0.5)^2\big)\), computed live with \(\varepsilon = 10^{-8}\). The black ring is the start, the black dot the minimum. For the inspected optimizer, the red arrow shows the direction \(-\mathbf{g}\), and the dashed arrow shows the step it is about to take. For AdamW the step is split into its two parts: the adaptive part (orange) and the decay part \(-\eta\lambda\theta\) (green), which always points at the origin.
How to read Figure 4

Step through the first iteration of AdamW with the defaults. The gradient at the start is \(\mathbf{g} = (-7, -40)\): the \(y\) direction is 20 times steeper. Then \(\mathbf{m} = 0.1\,\mathbf{g} = (-0.7, -4)\) and \(\mathbf{v} = 0.001\,\mathbf{g}^2 = (0.049, 1.6)\). Bias correction divides by \(1 - 0.9 = 0.1\) and \(1 - 0.999 = 0.001\), so \(\hat{\mathbf{m}} = \mathbf{g}\) and \(\hat{\mathbf{v}} = \mathbf{g}^2\), and the adaptive part is exactly \((-1, -1)\): both coordinates move by \(\eta = 0.1\), although their gradients differ by a factor of 20. That is Adam’s per-weight step size. The decay part is \(\eta\lambda\theta = 0.1 \times 0.2 \times (-4.5, -1.5) = (-0.09, -0.03)\), so AdamW lands at \((-4.31, -1.37)\), while Adam lands at \((-4.4, -1.4)\). Now press Play. After 120 steps, SGD (\(\eta = 0.03\)) is at \((2.319, 0.500)\): the steep \(y\) direction was solved early, the flat \(x\) direction is still closing in. Momentum is at \((2.497, 0.503)\) and Adam at \((2.335, 0.496)\). AdamW stops short at \((1.60, 0.46)\): the decay keeps pulling it toward 0. Adam + L2 is at \((2.00, 0.49)\); it is heading to the minimum of \(f + \tfrac{\lambda}{2}\lVert\theta\rVert^2\), which is \((2.083, 0.495)\). In Adam + L2 the pull \(\lambda\theta\) is part of \(\mathbf{g}\), so it is divided by \(\sqrt{\hat{\mathbf{v}}}\) like the rest; the live box shows how much smaller that makes it in the steep \(y\) direction. Turn off bias correction: the first Adam step becomes \((1 - \beta_1)/\sqrt{1 - \beta_2} = 3.16\) times larger in each coordinate, so Adam jumps from \((-4.5, -1.5)\) to \((-4.18, -1.18)\). Finally, push \(\eta\) for SGD to 0.1, which is \(2/\lambda_{\max} = 2/20\): the \(y\) coordinate of SGD now bounces between \(-1.5\) and \(2.5\) forever. That is the stability limit (4.6) in action.

The memory cost of Adam

Adam stores \(\mathbf{m}\) and \(\mathbf{v}\) for every weight, usually in FP32. With mixed-precision training, the full bill per parameter is:

WhatBytes per parameter
BF16 weights (used in forward and backward)2
BF16 gradients2
FP32 master copy of the weights4
Adam \(\mathbf{m}\) (FP32)4
Adam \(\mathbf{v}\) (FP32)4
Total, before activations16

For an 8B model: \(8 \times 10^9 \times 16 = 128\) GB, before a single activation is stored. That is why full fine-tuning of an 8B model does not fit on one 80 GB GPU, and why sharding (FSDP, ZeRO), 8-bit optimizers, Adafactor and LoRA exist (session 38, session 57). Newer optimizers such as Lion (one state instead of two) and Muon (for the matrices of hidden layers) are being used too; Adam(W) is still the default. [Likely]

Learning-rate schedules

Almost no LLM is trained with a constant learning rate. The learning rate follows a schedule \(\eta(t)\) with two phases.

Warmup

For the first \(W\) steps, the learning rate rises linearly from near 0 to its peak:

\[ \eta(t) = \eta_{\max} \cdot \frac{t}{W}, \qquad t \lt W \tag{4.11} \]

Why? At the start, the weights are random, the gradients are large and change fast, and Adam’s \(\mathbf{v}\) has seen only a few gradients, so its scaling is unreliable. Big steps at that moment can push the model into a bad region it never leaves, or cause a loss spike. Warmup lets the statistics settle first. Llama 2 used 2,000 warmup steps out of about 500,000 (2 trillion tokens at 4 million tokens per step).

Decay

After warmup, the learning rate goes down. Early on, big steps make fast progress; at the end, small steps let the model settle into the bottom of the valley instead of bouncing around it. The most common shape is cosine decay to a minimum \(\eta_{\min}\):

\[ \eta(t) = \eta_{\min} + \tfrac12 (\eta_{\max} - \eta_{\min}) \left( 1 + \cos\!\left( \pi \, \frac{t - W}{T - W} \right) \right), \qquad W \le t \le T \tag{4.12} \]

At the start of decay the cosine is 1 and \(\eta = \eta_{\max}\); at the end it is \(-1\) and \(\eta = \eta_{\min}\). Halfway, \(\eta\) is the average of the two: with \(\eta_{\max} = 3\times10^{-4}\) and \(\eta_{\min} = 3\times10^{-5}\), that is \(1.65\times10^{-4}\).

The weakness of cosine: you must choose the total length \(T\) in advance. Warmup-stable-decay (WSD) keeps the learning rate constant for most of training, then decays it quickly (for example linearly) over the last 10–20% of steps. You can continue training from any point in the stable phase and only run a short decay when you need a finished model. Several recent LLMs use it. [Likely]

Figure 5 runs the schedule function step by step, the way a training loop calls it once per optimizer step. The highlighted line is the branch that runs at the current step, with the numbers put in.

  1. if \(t \lt W\): \(\eta \leftarrow \eta_{\max}\,(t + 1)/W\) // warmup (4.11), \(t\) counts from 0
  2. else: \(p \leftarrow (t - W)/(T - W)\) // progress of the decayelse if \(t \lt T - D\): \(\eta \leftarrow \eta_{\max}\) // stable phaseelse: \(\eta \leftarrow \eta_{\max}\)
  3. \(\eta \leftarrow \eta_{\min} + \tfrac12(\eta_{\max} - \eta_{\min})(1 + \cos \pi p)\) // (4.12)\(\eta \leftarrow \eta_{\max} - (\eta_{\max} - \eta_{\min})\,p\)else: \(\eta \leftarrow \eta_{\max} - (\eta_{\max} - \eta_{\min})(t - T + D)/D\) // decay(no decay)
Figure 5. Learning-rate schedules over \(T = 10{,}000\) steps with peak \(\eta_{\max} = 3 \times 10^{-4}\). Play moves the current step forward; the part of the curve already used is drawn bold, and the shaded area is the sum of all learning rates so far (how far the optimizer has been allowed to travel).
How to read Figure 5

With the defaults (cosine, warmup 2% = 200 steps, minimum 10%), step 0 gives \(\eta = 3\times10^{-4} \times 1/200 = 1.5\times10^{-6}\). At step 199 warmup ends at the peak. At step 5,100 the decay is exactly half done (\(p = 0.5\)), and \(\eta = 1.65\times10^{-4}\), the average of the peak and the minimum. Switch to warmup-stable-decay: the learning rate stays at the peak until step 8,500 and then falls linearly to \(3\times10^{-5}\) at the end. The sum of all learning rates is larger for WSD (2.77 against 1.65 for cosine and for linear decay, summed over all steps), because it spends longer at the peak.

Gradient clipping

Sometimes one batch produces a huge gradient (an odd example, or a sharp cliff in the loss). One such step can undo days of training. Global-norm clipping rescales the whole gradient when its length is above a threshold \(c\):

\[ \mathbf{g} \leftarrow \mathbf{g} \cdot \min\!\left( 1,\ \frac{c}{\lVert \mathbf{g} \rVert_2} \right), \qquad \lVert \mathbf{g} \rVert_2 = \sqrt{\textstyle\sum_i g_i^2} \ \text{ over all weights} \tag{4.13} \]

The direction is kept; only the length is capped. Example: \(\mathbf{g} = (3, 4)\) has norm 5; with \(c = 1\) it becomes \((0.6, 0.8)\). LLMs typically use \(c = 1.0\). Watching the gradient norm over training is one of the best early warnings of instability: rising norms often come before a loss spike (session 36).

Putting it together: one optimizer step

Algorithm 1: one training step with AdamW, clipping and a schedule
  1. input: weights \(\theta\), states \(\mathbf{m}, \mathbf{v}\), step \(t\), batch, hyper-parameters \(\beta_1, \beta_2, \varepsilon, \lambda, c\)
  2. \(L \leftarrow\) forward pass on the batch, storing activations
  3. \(\mathbf{g} \leftarrow \nabla_\theta L\) // backward pass, formula (4.4) per layer
  4. \(\mathbf{g} \leftarrow \mathbf{g} \cdot \min(1, c / \lVert \mathbf{g} \rVert)\) // clip, formula (4.13)
  5. \(\eta \leftarrow \text{schedule}(t)\) // warmup then cosine, formulas (4.11), (4.12)
  6. \(t \leftarrow t + 1\)
  7. \(\mathbf{m} \leftarrow \beta_1 \mathbf{m} + (1 - \beta_1)\mathbf{g}\), \(\;\mathbf{v} \leftarrow \beta_2 \mathbf{v} + (1 - \beta_2)\mathbf{g}^2\)
  8. \(\hat{\mathbf{m}} \leftarrow \mathbf{m}/(1 - \beta_1^t)\), \(\;\hat{\mathbf{v}} \leftarrow \mathbf{v}/(1 - \beta_2^t)\) // bias correction
  9. \(\theta \leftarrow \theta - \eta\,\big(\hat{\mathbf{m}}/(\sqrt{\hat{\mathbf{v}}} + \varepsilon) + \lambda\,\theta \cdot \text{decay\_mask}\big)\) // formula (4.10); no decay on norms and biases
  10. set all gradients to zero for the next step

The PyTorch version, with a check that a hand-written AdamW matches the built-in one:

import math, torch
torch.manual_seed(0)

model = torch.nn.Sequential(torch.nn.Linear(16, 32), torch.nn.LayerNorm(32), torch.nn.Linear(32, 4))

# 1) no weight decay on biases and normalization weights (all 1-D parameters)
decay   = [p for n, p in model.named_parameters() if p.ndim >= 2]
nodecay = [p for n, p in model.named_parameters() if p.ndim <  2]
opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.1},
                         {"params": nodecay, "weight_decay": 0.0}],
                        lr=3e-4, betas=(0.9, 0.95), eps=1e-8)

# 2) warmup + cosine schedule, formulas (4.11) and (4.12), as a multiplier of the peak lr
W, T, min_ratio = 100, 1000, 0.1
def lr_mult(step):
    if step < W:
        return (step + 1) / W
    p = (step - W) / max(1, T - W)
    return min_ratio + 0.5 * (1 - min_ratio) * (1 + math.cos(math.pi * min(1.0, p)))
sched = torch.optim.lr_scheduler.LambdaLR(opt, lr_mult)

# 3) one training step: forward, backward, clip, step, schedule
x, y = torch.randn(64, 16), torch.randint(0, 4, (64,))
loss = torch.nn.functional.cross_entropy(model(x), y)
opt.zero_grad(set_to_none=True)
loss.backward()
gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)   # formula (4.13); returns the norm before clipping
opt.step()
sched.step()                                    # once per optimizer step, not per epoch
print(loss.item(), gnorm.item(), sched.get_last_lr())

# 4) a hand-written AdamW update, formula (4.10), matches torch.optim.AdamW
w = torch.randn(5, requires_grad=True); w2 = w.detach().clone().requires_grad_(True)
ref = torch.optim.AdamW([w2], lr=1e-2, betas=(0.9, 0.95), eps=1e-8, weight_decay=0.1)
m = torch.zeros(5); v = torch.zeros(5)
for t in range(1, 6):
    g = torch.sin(w.detach() * t)                  # any gradient sequence
    w2.grad = g.clone(); ref.step()
    with torch.no_grad():
        w.mul_(1 - 1e-2 * 0.1)                     # decoupled decay (PyTorch applies it first)
        m = 0.9 * m + 0.1 * g; v = 0.95 * v + 0.05 * g * g
        mh, vh = m / (1 - 0.9 ** t), v / (1 - 0.95 ** t)
        w -= 1e-2 * mh / (vh.sqrt() + 1e-8)
print((w - w2).abs().max().item())                 # about 1e-8: the same update

Where this is used

SettingTypical choicesWhy
LLM pretraining (Llama 2 7B)AdamW, \(\beta = (0.9, 0.95)\), \(\varepsilon = 10^{-5}\), wd 0.1, clip 1.0, warmup 2,000 steps, cosine to 10%, peak \(3\times10^{-4}\)\(\beta_2 = 0.95\) reacts faster to sudden gradient changes than 0.999, which helps stability at scale [Likely]
Supervised fine-tuning of an LLMAdamW, peak lr around 10–100× lower than pretraining, short warmup, cosine or linear decay, 1–3 epochs [Likely]The model is already good; big steps cause forgetting (session 54)
LoRA fine-tuningAdamW on the small adapter matrices only, often lr around \(10^{-4}\) to \(2\times10^{-4}\) [Likely]Few trainable weights, so optimizer memory is tiny
Heads on a frozen vision backbone (action recognition, shot classification)AdamW on the heads only; if you unfreeze the backbone, a much smaller lr for it than for the headsProtects good pretrained features while the new head learns fast
RL fine-tuning (PPO, GRPO)AdamW with a very small lr (around \(10^{-6}\)), clipping, KL penalty [Likely]The policy must change slowly or it collapses (session 64)

Common mistakes

  • Forgetting to zero the gradients. PyTorch adds new gradients to old ones. Without zero_grad() every step uses the sum of all past gradients. (Gradient accumulation does this on purpose, then divides the loss by the number of micro-batches.)
  • Calling the scheduler once per epoch when it was designed for steps, or the other way round. Plot your schedule before training.
  • Weight decay on norms and biases. Decaying LayerNorm or RMSNorm gains toward 0 hurts. Use parameter groups as in the code.
  • Adam + L2 when you meant AdamW. torch.optim.Adam(weight_decay=...) is the L2 version. Use AdamW.
  • Clipping before unscaling in FP16 training. With a gradient scaler, call scaler.unscale_(opt) before clip_grad_norm_, or you clip the scaled (huge) gradients.
  • Too small \(\varepsilon\) in low precision. In FP16, \(10^{-8}\) rounds to 0 and \(\sqrt{\hat v} + \varepsilon\) can be 0. Keep optimizer math in FP32 (the standard), or use a larger \(\varepsilon\).
  • No warmup with a large peak learning rate. Classic cause of an early loss spike or NaN.
  • Changing the batch size without changing the learning rate. They are linked (formula 4.7); re-tune or scale.

Interview questions

Explain backpropagation in two sentences. Why is it efficient?basic
Backprop applies the chain rule from the loss back through the computation graph, multiplying each incoming gradient by a local derivative and summing over paths. It is efficient because it computes the gradient for all weights in one backward pass that costs about twice the forward pass, reusing shared intermediate gradients instead of differentiating each weight separately.
Why does the backward pass cost about 2× the forward pass?core
For each linear layer \(Y = XW\), the backward pass needs two matrix multiplications of the same size as the forward one: \(\partial L/\partial W = X^\top G\) and \(\partial L/\partial X = GW^\top\). So forward \(\approx 2N\) FLOPs per token and backward \(\approx 4N\), giving the \(6N\) per token training rule. (The first layer does not need the input gradient, a small saving.)
What is the difference between Adam and AdamW?core
Adam with weight decay adds \(\lambda\theta\) to the gradient, so the decay is divided by \(\sqrt{\hat v}\) and becomes uneven: weights with large gradient variance are barely decayed. AdamW applies \(\theta \leftarrow \theta - \eta\lambda\theta\) directly, decoupled from the adaptive scaling. This gives true weight decay and makes \(\lambda\) and \(\eta\) easier to tune separately. AdamW is the standard for transformers.
Why does Adam need bias correction?core
\(\mathbf{m}\) and \(\mathbf{v}\) start at zero, so early averages are shrunk: \(\E[m_t] = (1-\beta_1^t)\E[g]\). Dividing by \(1-\beta_1^t\) and \(1-\beta_2^t\) removes the bias. Without it, with \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), the first update is about \(3.16\eta\) instead of \(\eta\), exactly when the model is most fragile.
How much memory does AdamW training of an 8B model need, before activations?core
With mixed precision: 2 bytes (BF16 weights) + 2 (gradients) + 4 (FP32 master weights) + 4 + 4 (Adam m and v) = 16 bytes per parameter, so 128 GB. Plus activations, which depend on batch size and sequence length. Ways to reduce: shard across GPUs (ZeRO or FSDP), 8-bit optimizer states, Adafactor (factored second moment), or LoRA (train few parameters).
Why do we use learning-rate warmup?core
At initialization gradients are large and change quickly, and Adam’s second-moment estimate is based on very few steps, so adaptive step sizes are unreliable. Large early steps can cause loss spikes or move the model into a poor region. Ramping the learning rate up over the first steps lets the statistics stabilize; it also matters more with large batches and with Post-LN transformers.
What is the stability limit of gradient descent on a quadratic, and what does it tell you about deep networks?advanced
On \(\tfrac12\sum\lambda_i u_i^2\), gradient descent is stable only for \(\eta \lt 2/\lambda_{\max}\), while convergence along the flattest direction scales with \(\eta\lambda_{\min}\). A large condition number therefore forces slow training. Deep nets have very uneven curvature, which motivates momentum and per-parameter adaptive methods. Empirically, training often runs near this “edge of stability”, where the largest curvature hovers around \(2/\eta\). [Likely]
Cosine vs warmup-stable-decay schedules?core
Cosine decays smoothly from the peak to a minimum over a fixed total length, so you must know the total steps in advance; stopping early leaves the model at a high learning rate. WSD holds the peak learning rate for most of training and decays quickly at the end, so you can keep training from any stable-phase checkpoint and produce a finished model with a short decay. Both work well; WSD is more flexible for continued pretraining.
What does gradient clipping do, and how would you use it to detect problems?basic
Global-norm clipping rescales the full gradient to length \(c\) (often 1.0) whenever its norm is larger, keeping the direction. It prevents one bad batch from causing a huge update. Log the pre-clip norm: a steady rise or sudden jumps usually come before loss spikes, and frequent clipping means the learning rate may be too high.
Batch size doubles. What do you do with the learning rate?advanced
Gradient noise variance halves (formula 4.7). For SGD, the linear scaling rule suggests doubling the learning rate, with warmup. For Adam, the best learning rate typically grows more slowly (often closer to \(\sqrt{2}\)), and beyond the critical batch size, larger batches stop reducing the number of steps needed. In practice, re-tune on a small sweep, and keep tokens-per-step in mind for LLMs.
Why is \(\beta_2 = 0.95\) common for LLMs instead of the default 0.999?advanced
\(\beta_2 = 0.999\) averages squared gradients over about 1,000 steps. If gradients suddenly grow, \(\hat v\) lags behind, the denominator is too small, and updates become too large, which can trigger loss spikes. \(\beta_2 = 0.95\) (about 20 steps) adapts faster. Large-scale recipes such as Llama 2 use 0.95. [Likely]

Exercises

  1. In Figure 1, change the target to \(y = 0\) (by hand). Compute \(\partial L/\partial w\) and \(\partial L/\partial b\). Which way do \(w\) and \(b\) move?
  2. For \(f(x, y) = \tfrac12(x^2 + 50y^2)\), what is the largest stable learning rate for gradient descent? Check it in Figure 2.
  3. Write the first Adam update for a weight with \(g_1 = 0.002\), \(\eta = 10^{-3}\), with bias correction.
  4. Compute the learning rate at step 7,500 of a 10,000-step cosine schedule with no warmup, \(\eta_{\max} = 3\times10^{-4}\) and \(\eta_{\min} = 0\).
  5. How much memory do Adam states (\(\mathbf{m}\) and \(\mathbf{v}\) only) need for a 70B model in FP32? In 8-bit?
  6. Clip \(\mathbf{g} = (6, 8, 0)\) to norm 2.
Answerscheck
1. \(\partial L/\partial a = 2(0.622 - 0) = 1.245\); \(\partial L/\partial z = 1.245 \times 0.235 = 0.293\); \(\partial L/\partial w = 0.585\), \(\partial L/\partial b = 0.293\). Both are positive, so both decrease, which lowers \(a\) toward 0. 2. \(\eta \lt 2/50 = 0.04\). 3. \(\hat m = g\), \(\hat v = g^2\), so the update is \(10^{-3} \times 0.002/0.002 = 10^{-3}\) (minus a tiny \(\varepsilon\) effect): the size of the gradient does not matter. 4. Progress \(p = 0.75\): \(\eta = 0.5 \times 3\times10^{-4} \times (1 + \cos 0.75\pi) = 1.5\times10^{-4} \times (1 - 0.707) = 4.39\times10^{-5}\). 5. \(70\times10^9 \times 8\) bytes = 560 GB; in 8-bit, \(70\times10^9 \times 2\) = 140 GB (plus small block scales). 6. Norm 10, factor 0.2: \((1.2, 1.6, 0)\).

Key takeaways

  • The gradient (4.1) gives the downhill direction for every weight; backpropagation computes all of it with the chain rule (4.2) in one backward pass, at about twice the forward cost (4.4).
  • Gradient descent (4.5) is stable only for \(\eta \lt 2/\lambda_{\max}\) (4.6); badly conditioned losses are slow. Mini-batches give unbiased but noisy gradients (4.7).
  • Momentum (4.8) smooths the path; Adam (4.9) gives each weight its own step size, with bias correction for the first steps; AdamW (4.10) applies weight decay correctly.
  • AdamW with mixed precision costs 16 bytes per parameter: 128 GB for an 8B model before activations.
  • Warmup (4.11) then decay (4.12, or WSD), plus global-norm clipping at 1.0 (4.13): the standard recipe, as in Llama 2.

Key formulas

\[ \theta \leftarrow \theta - \eta\left(\frac{\hat{\mathbf{m}}}{\sqrt{\hat{\mathbf{v}}}+\varepsilon} + \lambda\theta\right), \quad \frac{\partial L}{\partial W} = X^\top G, \quad \eta \lt \frac{2}{\lambda_{\max}}, \quad \mathbf{g} \leftarrow \mathbf{g}\min\!\left(1, \frac{c}{\lVert\mathbf{g}\rVert}\right) \]

References for this session

  1. Kingma & Ba (2014). Adam: A Method for Stochastic Optimization
  2. Loshchilov & Hutter (2017). Decoupled Weight Decay Regularization
  3. Karpathy (2023). Neural Networks: Zero to Hero (video course)
  4. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  5. Goyal et al. (2017). Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
  6. Hu et al. (2024). MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
  7. Shazeer & Stern (2018). Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
  8. Dettmers et al. (2021). 8-bit Optimizers via Block-wise Quantization
  9. Chen et al. (2023). Symbolic Discovery of Optimization Algorithms (Lion)
  10. Jordan et al. (2024). Muon: An optimizer for hidden layers in neural networks (blog)