LLM & VLM courseBlog
66Track 4: Post-training and RL · session 14 of 25

PPO

PPO (Proximal Policy Optimization) is the policy-gradient method behind classic RLHF. It reuses each batch of samples for several gradient steps, and a clipped importance ratio stops any one update from moving the policy too far. This session builds the full loss (clipped policy term, value loss, entropy bonus), shows how a language model turns into an RL agent with a per-token KL penalty to a frozen reference, and counts the memory of the four models PPO keeps on the GPUs.

Time: 3 to 4 hours (reading 2 to 2.5 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 3, 38, 63, 64 and 65

What you will learn

  • Why plain policy gradient can use each sample only once, and how the importance ratio between the new and the old policy lets PPO reuse a batch
  • The clipped objective: the six cases of sign and ratio, when the gradient is zero, and why the minimum makes it a pessimistic bound
  • The adaptive KL penalty, the value loss, the entropy bonus, and how they combine into one PPO loss
  • PPO for language models: tokens as actions, a per-token KL penalty to a frozen reference model, the reward model score at the last token, and GAE with gamma = 1
  • The four models in memory (policy, reference, reward model, critic), their byte counts, and the real InstructGPT setup
  • A full PPO-RLHF iteration in pseudo-code and in runnable PyTorch, plus the signals to watch during training

Why this matters

In session 64 we learned to push up the log-probability of actions that got a good reward. In session 65 we learned to measure “good” with an advantage from a learned critic. One problem is left: plain policy gradient is wasteful and fragile. Each batch of samples is used for one gradient step and then thrown away, and one step that is too large can wreck the policy. PPO (Proximal Policy Optimization, Schulman et al., 2017) fixes both with one small idea: a clipped probability ratio that lets us reuse a batch for several steps while keeping the new policy close to the old one.

PPO is the algorithm of classic RLHF (reinforcement learning from human feedback). InstructGPT, the model behind the first ChatGPT-style assistants, was trained with PPO against a reward model, with a clip ratio of 0.2 and a KL penalty to the supervised model. [Certain] Llama 2-Chat used PPO after rounds of rejection-sampling fine-tuning. [Certain] Newer methods such as GRPO (session 70) keep PPO’s clipped objective and change only how the advantage is computed. So if you understand PPO, you understand the core of almost every RL method used on LLMs today.

  • Reuse samples. Generating answers with a large model is the most expensive part of RLHF. An importance ratio lets us train several mini-batch steps on one batch of generated answers.
  • Small, safe steps. The clipped objective removes the incentive to move the probability of any sample by more than a factor \(1 \pm \epsilon\) in one update.
  • Stay close to the start. A KL penalty to a frozen reference model stops the policy from drifting into strange text that fools the reward model.
  • Four models. Policy, reference, reward model and critic all sit in GPU memory at the same time. This is why PPO-RLHF is expensive, and why DPO and GRPO were invented.

From policy gradient to a trust region

Recall the setting from session 63. A policy \(\pi_\theta(a \mid s)\) gives a probability to each action \(a\) in state \(s\). For an LLM, the state is the prompt plus the tokens written so far, and the action is the next token. The policy-gradient estimate from session 64, with an advantage \(\hat A_t\) from session 65, is

\[ \hat g = \hat{\E}_t\!\left[ \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, \hat A_t \right] \]

where \(\hat{\E}_t\) means the average over a batch of collected steps. This formula is only correct when the samples come from the current policy \(\pi_\theta\). After one gradient step, \(\theta\) has changed, the batch now comes from an older policy, and the estimate is biased. So REINFORCE must generate a fresh batch for every step. For an LLM, generating a batch of long answers takes much more time than one backward pass. [Likely]

The importance ratio

The fix is importance sampling: if samples come from one distribution but we want an average under another, we weight each sample by the ratio of the two probabilities. Call the policy that generated the batch \(\pi_{\theta_\text{old}}\) (the “old” policy, frozen while we train on this batch). The probability ratio of one sample is

\[ r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)} = \exp\!\big(\log \pi_\theta(a_t \mid s_t) - \log \pi_{\theta_\text{old}}(a_t \mid s_t)\big) \tag{66.1} \]
Read the symbols
\(\pi_\theta\)
the policy we are training now; its weights change at every mini-batch step
\(\pi_{\theta_\text{old}}\)
the same network at the moment the batch was generated; its log-probabilities are computed once and stored
\(r_t(\theta)\)
how much more (above 1) or less (below 1) likely the action is now than when it was sampled; \(r_t = 1\) before the first update

Weighting the advantage by the ratio gives the surrogate objective that TRPO and PPO start from (CPI stands for “conservative policy iteration”, the older work where it appeared):

\[ L^{\text{CPI}}(\theta) = \hat{\E}_t\!\left[ r_t(\theta)\, \hat A_t \right] \tag{66.2} \]

At \(\theta = \theta_\text{old}\), its gradient is exactly the policy gradient, because \(\nabla_\theta r_t = r_t \nabla_\theta \log \pi_\theta\) and \(r_t = 1\) there. So we can now take several steps on the same batch. The danger is that nothing stops us. Maximizing (66.2) says “if an action had positive advantage, make it infinitely more likely”. The ratio grows, the importance weights get huge, and the estimate becomes meaningless: the batch only tells us about actions near the old policy.

TRPO (Trust Region Policy Optimization, Schulman et al., 2015) solved this with a hard constraint: maximize (66.2) subject to an average KL divergence between old and new policy below a small \(\delta\). That needs second-order math (conjugate gradients and a line search), which is complex and hard to combine with large networks. [Certain] PPO keeps the idea of a trust region, a region around the old policy where the surrogate can be trusted, and gets it with a first-order trick that is a few lines of code.

The clipped objective

PPO replaces (66.2) with

\[ L^{\text{CLIP}}(\theta) = \hat{\E}_t\!\left[ \min\!\Big( r_t(\theta)\, \hat A_t,\ \operatorname{clip}\big(r_t(\theta),\, 1-\epsilon,\, 1+\epsilon\big)\, \hat A_t \Big) \right] \tag{66.3} \]
Read the symbols
\(\epsilon\)
the clip range; 0.2 in the PPO paper’s best setting and in InstructGPT
\(\operatorname{clip}(r, 1-\epsilon, 1+\epsilon)\)
\(r\) forced into the interval \([1-\epsilon, 1+\epsilon]\): values below are raised to \(1-\epsilon\), values above are lowered to \(1+\epsilon\)
\(\min(\cdot,\cdot)\)
take the smaller of the unclipped and the clipped term, for each sample separately
\(\hat A_t\)
the advantage of the sample, from GAE; computed once per batch and kept fixed during the epochs

Read it one sample at a time. The second term has no gradient once \(r\) leaves the interval, because a clipped value is a constant. The minimum then picks which term is active. There are only a few cases:

AdvantageRatioActive termGradient pushesMeaning
\(\hat A \gt 0\)\(r \le 1+\epsilon\)\(r \hat A\)\(r\) upa good action, not yet pushed too far: make it more likely
\(\hat A \gt 0\)\(r \gt 1+\epsilon\)\((1+\epsilon)\hat A\)nothing (0)already made much more likely: stop
\(\hat A \gt 0\)\(r \lt 1-\epsilon\)\(r \hat A\)\(r\) upa good action that became less likely: always allowed to fix
\(\hat A \lt 0\)\(r \ge 1-\epsilon\)\(r \hat A\)\(r\) downa bad action, not yet pushed too far: make it less likely
\(\hat A \lt 0\)\(r \lt 1-\epsilon\)\((1-\epsilon)\hat A\)nothing (0)already made much less likely: stop
\(\hat A \lt 0\)\(r \gt 1+\epsilon\)\(r \hat A\)\(r\) downa bad action that became more likely: always allowed to fix

So the clip only acts in the direction that would improve the objective. In the words of the paper, “we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse”. [Certain] This makes \(L^{\text{CLIP}}\) a pessimistic (lower) bound on \(L^{\text{CPI}}\): it never promises more improvement than the unclipped surrogate, and it stops counting improvement beyond the trust region. Near \(r = 1\) the two are equal, so the first step is a normal policy-gradient step.

The gradient of one term with respect to the ratio is therefore very simple:

\[ \frac{\partial}{\partial r}\min\!\big(r \hat A,\ \operatorname{clip}(r)\, \hat A\big) = \begin{cases} 0 & \text{if } (\hat A \gt 0 \text{ and } r \gt 1+\epsilon) \text{ or } (\hat A \lt 0 \text{ and } r \lt 1-\epsilon) \\ \hat A & \text{otherwise} \end{cases} \]

In code, the share of samples in the first case is logged as the clip fraction. It is the simplest health signal of a PPO run.

Worked example: five samples, \(\epsilon = 0.2\)

(a) \(\hat A = +2\), \(r = 1.3\): unclipped \(1.3 \times 2 = 2.6\), clipped \(1.2 \times 2 = 2.4\). The minimum is 2.4, a constant, so the gradient is 0. The action is already 30% more likely; PPO stops pushing.

(b) \(\hat A = +2\), \(r = 1.1\): both terms are 2.2, gradient \(\hat A = 2\). Inside the trust region, PPO is plain policy gradient.

(c) \(\hat A = +2\), \(r = 0.6\): unclipped 1.2, clipped \(0.8 \times 2 = 1.6\). The minimum is 1.2, the unclipped term, so the gradient is 2: a good action that lost probability is always pushed back.

(d) \(\hat A = -1\), \(r = 0.7\): unclipped \(-0.7\), clipped \(0.8 \times (-1) = -0.8\). The minimum is \(-0.8\), clipped, gradient 0. The bad action is already 30% less likely; stop.

(e) \(\hat A = -1\), \(r = 1.5\): unclipped \(-1.5\), clipped \(-1.2\). The minimum is \(-1.5\), unclipped, gradient \(-1\). The bad action became 50% more likely, and PPO pushes it down with full force. This is the pessimistic side of the min.

unclipped \(r \hat A\)PPO term \(\min(r\hat A, \operatorname{clip}(r)\hat A)\)trust region \([1-\epsilon, 1+\epsilon]\)
Figure 1. One term of formula (66.3) for a single sample, as the ratio \(r\) between new and old probability changes. The dot is the current ratio. Where the blue curve is flat, the gradient is zero and this sample no longer moves the policy.
How to read Figure 1

With \(\hat A = 2\), the blue curve follows the grey line up to \(r = 1 + \epsilon\) and then goes flat: there is no reward for making a good action more than 20% more likely in one update. Move \(r\) below 1: the curve follows the line again, so the gradient is back. Now set \(\hat A = -2\): the flat part moves to the left of \(1 - \epsilon\), and for large \(r\) the curve keeps falling steeply. Change \(\epsilon\) to see the flat part move. Setting \(\hat A = 0\) gives a flat line: samples with zero advantage never move the policy.

One batch, many epochs: what clipping buys

A PPO iteration has two phases. First, generate a batch with the current policy and freeze it as \(\pi_{\theta_\text{old}}\); store the old log-probabilities and compute advantages. Second, run \(K\) epochs (passes) over this batch, split into mini-batches, maximizing (66.3) with Adam. Then the trained policy becomes the new old policy, and the loop repeats. The PPO paper used 10 epochs with mini-batches of 64 on MuJoCo robotics tasks and 3 epochs on Atari. [Certain]

Why not train on one batch forever? Because a batch is a noisy sample. With a few dozen answers, some bad answers get high scores by luck. The importance-weighted estimate (66.2) believes the batch completely, so with many epochs the policy chases the noise: the batch says “big improvement” while the true improvement is small or negative. Clipping limits how far the policy can move on one batch. Figure 2 shows this on a problem small enough to compute exactly.

old policy \(\pi_{\theta_\text{old}}\)new policy after \(K\) epochstrue gain in expected rewardgain estimated from the batch\(\KL(\pi_{\theta_\text{old}} \,\|\, \pi_\theta)\), dashed
Figure 2. A one-step problem (a “bandit”): the policy picks one of six answers A to F with true rewards 1.0, 0.6, 0.3, 0.0, −0.4 and −1.0. One seeded batch of 64 answers is drawn from the old policy, and each observed reward has noise added (standard deviation 1.0). Advantages are the batch rewards minus their mean, divided by their standard deviation. Each epoch is one full-batch gradient-ascent step (step size 1.0) on the logits. Because the problem is tiny, the true expected reward and the exact KL can be computed.
How to read Figure 2

The numbers under the bars are the true reward and, below it, the average noisy reward the batch saw. Answer D is truly worth 0.0, but the batch saw an average of 0.38. Choose “No clipping” and 20 epochs: the policy puts most of its mass on D, the KL grows to about 0.88 nats, and the batch estimate of the gain (orange) climbs to about 0.41 while the true gain (green) is only about 0.05 and falling. Switch to PPO: the policy moves a little, the KL stays near 0.03, and the true gain is higher. Move \(K\) to 1: both methods are identical, because on the first step every ratio is 1.

This toy problem has no function approximation and no changing states, which are the reasons large policy steps are even more harmful in real RL. [Likely] Still, it shows the key point: one batch supports only a small step, and the clip is a cheap way to enforce that.

The alternative: a KL penalty with an adaptive weight

The PPO paper also proposed a penalty instead of a clip:

\[ L^{\text{KLPEN}}(\theta) = \hat{\E}_t\!\left[ r_t(\theta)\, \hat A_t - \beta\, \KL\!\big[\pi_{\theta_\text{old}}(\cdot \mid s_t),\ \pi_\theta(\cdot \mid s_t)\big] \right] \tag{66.4} \]

After each update, measure the average KL \(d\). If \(d \lt d_\text{targ}/1.5\), halve \(\beta\); if \(d \gt 1.5\, d_\text{targ}\), double it. [Certain] On the paper’s MuJoCo benchmark (7 tasks, 3 seeds each, scores normalized so random = 0 and best = 1), no clipping or penalty scored −0.39, clipping with \(\epsilon = 0.1, 0.2, 0.3\) scored 0.76, 0.82 and 0.70, the adaptive KL penalty scored 0.68 to 0.74, and a fixed penalty 0.62 to 0.72. [Certain] So the clip became the default. Keep the two KL terms apart in your head: this one is to the old policy (a trust region per iteration). The RLHF KL below is to the frozen reference model (a leash for the whole run).

The full PPO loss: value loss and entropy bonus

The advantage needs a critic \(V_\phi(s)\) that predicts the return. The critic is trained on the same batch, by regression to the return targets \(\hat R_t = \hat A_t + V_{\phi_\text{old}}(s_t)\) (the GAE return from session 65). Many implementations also clip the value update, in the same spirit as the policy:

\[ L^{\text{VF}}(\phi) = \hat{\E}_t\!\left[ \max\!\Big( \big(V_\phi(s_t) - \hat R_t\big)^2,\ \big(V_{\phi_\text{old}}(s_t) + \operatorname{clip}(V_\phi(s_t) - V_{\phi_\text{old}}(s_t), -\epsilon_v, \epsilon_v) - \hat R_t\big)^2 \Big) \right] \tag{66.5} \]

The paper itself uses the plain squared error \((V_\phi(s_t) - \hat R_t)^2\). [Certain] Value clipping came from later code; studies of PPO implementations found no clear evidence that it helps and some evidence that it can hurt. [Certain] Treat it as an option.

To keep the policy from becoming certain too early (and stopping exploration), we can add an entropy bonus, the entropy of the policy’s distribution at each state:

\[ S[\pi_\theta](s_t) = -\sum_{a} \pi_\theta(a \mid s_t) \log \pi_\theta(a \mid s_t) \tag{66.6} \]

For an LLM the sum runs over the whole vocabulary, which is cheap because the logits are already computed. Together, the paper’s combined objective (to be maximized) is

\[ L^{\text{CLIP+VF+S}}(\theta) = \hat{\E}_t\!\left[ L^{\text{CLIP}}_t(\theta) - c_1\, L^{\text{VF}}_t(\theta) + c_2\, S[\pi_\theta](s_t) \right] \tag{66.7} \]
Read the symbols
\(L^{\text{CLIP}}_t\)
the clipped policy term of sample \(t\), from (66.3)
\(L^{\text{VF}}_t\)
the squared value error of sample \(t\), from (66.5); it enters with a minus sign because we maximize
\(c_1\), \(c_2\)
weights of the value loss and the entropy bonus; the paper’s Atari setting used \(c_1 = 1\), \(c_2 = 0.01\)
\(S\)
the entropy bonus (66.6)

In code we minimize the negative: loss = -L_clip + c1 * L_vf - c2 * entropy. The value weight \(c_1\) matters only when policy and critic share parameters; with separate networks, each has its own optimizer or its own loss scale. In LLM RLHF, the policy and critic are usually separate models, and the entropy bonus is often set to 0, because the pretrained distribution already has plenty of entropy. [Likely]

PPO for language models

How does a chatbot become an RL agent? InstructGPT describes the environment as a bandit: it “presents a random customer prompt and expects a response to the prompt”, then produces a reward from the reward model and ends the episode. [Certain] For PPO we still look inside the response: each generated token is one action, the state is the prompt plus the tokens so far, and an episode has as many steps as the response has tokens.

The quantity we want to maximize is the reward model score, minus a penalty for moving away from the model we started from:

\[ \max_\theta\ \ \E_{x \sim \mathcal{D},\ y \sim \pi_\theta(\cdot \mid x)}\big[ r_\psi(x, y) \big] \;-\; \beta\, \E_{x \sim \mathcal{D}}\Big[ \KL\big(\pi_\theta(\cdot \mid x)\ \|\ \pi_\text{ref}(\cdot \mid x)\big) \Big] \tag{66.8} \]
Read the symbols
\(x\), \(\mathcal{D}\)
a prompt, and the set of training prompts
\(y\)
a full response sampled from the policy
\(r_\psi(x, y)\)
the reward model’s score of the whole response (one number); session 67 shows how it is trained
\(\pi_\text{ref}\)
the reference model: a frozen copy of the SFT model the policy started from
\(\beta\)
the KL coefficient: how short the leash is; InstructGPT used \(\beta = 0.02\)

Why the KL term? The reward model is only a learned approximation of human judgment, and it is reliable only on text similar to what it was trained on. Without the leash, the policy finds text the reward model scores highly but humans do not like: reward hacking. The KL also keeps the language fluent and the general skills of the SFT model. InstructGPT added the KL penalty “to mitigate over-optimization of the reward model”. [Certain] The KL-regularized objective also has a closed-form best policy, \(\pi^*(y \mid x) \propto \pi_\text{ref}(y \mid x)\, e^{r_\psi(x,y)/\beta}\), which is the starting point of DPO (session 68).

The per-token reward

The sequence KL is a sum over tokens, \(\log \frac{\pi_\theta(y \mid x)}{\pi_\text{ref}(y \mid x)} = \sum_t \log \frac{\pi_\theta(y_t \mid x, y_{\lt t})}{\pi_\text{ref}(y_t \mid x, y_{\lt t})}\). Following Ziegler et al. (2019) and Stiennon et al. (2020), the standard implementation puts this sample estimate into the reward, one piece per token, and adds the reward model score at the last token: [Certain]

\[ r_t = -\beta \Big( \log \pi_{\theta_\text{old}}(y_t \mid s_t) - \log \pi_\text{ref}(y_t \mid s_t) \Big) + \mathbb{1}[t = T]\; r_\psi(x, y) \tag{66.9} \]
Read the symbols
\(s_t\)
the state before token \(t\): the prompt \(x\) plus \(y_1, \dots, y_{t-1}\)
\(T\)
the index of the last response token (usually the end-of-turn token)
\(\mathbb{1}[t = T]\)
1 at the last token, 0 elsewhere: the reward model speaks only once, at the end
\(\log \pi_{\theta_\text{old}} - \log \pi_\text{ref}\)
the per-token log-ratio, computed once during the rollout; it is part of the reward, so it gets no gradient

Then GAE runs over the tokens of each response, exactly as in session 65:

\[ \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t), \qquad \hat A_t = \sum_{l=0}^{T-t} (\gamma\lambda)^l\, \delta_{t+l}, \qquad V(s_{T+1}) = 0 \tag{66.10} \]

InstructGPT used no discount, \(\gamma = 1\). [Certain] With \(\gamma = 1\) and \(\lambda = 1\), the advantage of every token is just the total remaining reward minus the critic’s guess. With \(\lambda \lt 1\), the critic’s guesses at later tokens are used more, which lowers variance but adds bias. Because the score arrives only at the end, the critic is what spreads it back to the early tokens.

Worked example: a four-token response

Take \(\beta = 0.05\), log-ratios to the reference \([0.30, -0.10, 0.50, 0.20]\), reward model score 1.2, critic values \(V = [0.6, 0.7, 0.8, 1.0]\), \(\gamma = 1\), \(\lambda = 0.95\).

Per-token rewards (66.9): \(-0.05 \times 0.30 = -0.015\), then \(+0.005\), \(-0.025\), and at the last token \(-0.010 + 1.2 = 1.19\). The total KL penalty is \(-0.05 \times 0.9 = -0.045\).

TD errors (66.10): \(\delta_3 = 1.19 + 0 - 1.0 = 0.19\), \(\delta_2 = -0.025 + 1.0 - 0.8 = 0.175\), \(\delta_1 = 0.005 + 0.8 - 0.7 = 0.105\), \(\delta_0 = -0.015 + 0.7 - 0.6 = 0.085\).

Advantages, from the end: \(\hat A_3 = 0.19\), \(\hat A_2 = 0.175 + 0.95 \times 0.19 = 0.3555\), \(\hat A_1 = 0.105 + 0.95 \times 0.3555 = 0.4427\), \(\hat A_0 = 0.085 + 0.95 \times 0.4427 = 0.5056\). The critic targets are \(\hat A_t + V_t = [1.106, 1.143, 1.156, 1.19]\). Every token gets a positive advantage, the first one the most, because the critic underestimated how good this response would be.

per-token reward \(r_t\)critic value \(V(s_t)\)advantage \(\hat A_t\)
Figure 3. One response of eight tokens (the last is the end-of-turn token). The top row shows the fixed per-token log-ratio \(\log \pi_{\theta_\text{old}} - \log \pi_\text{ref}\); the token “Paris” is much more likely under the policy than under the reference. Rewards follow (66.9), advantages follow (66.10) with \(\gamma = 1\). The critic values are fixed example numbers.
How to read Figure 3

With \(\beta = 0.05\) the KL part of each reward is tiny, and the score at the end dominates. Raise \(\beta\) to 0.5: the token “Paris” now has a clear negative reward, and its advantage drops below its neighbours; this is how the leash pushes back on tokens where the policy has moved far from the reference. Set the score to −1: all advantages turn negative. Set \(\lambda = 0\): each advantage becomes a one-step TD error and depends only on the critic’s next value. Set \(\lambda = 1\): each advantage is the full remaining reward minus \(V(s_t)\).

Measuring the KL

We never compute the exact KL over all possible responses. We estimate it from the sampled tokens. With \(\rho_t = \pi_\text{ref}(y_t \mid s_t) / \pi_\theta(y_t \mid s_t)\) and samples from \(\pi_\theta\), two common per-token estimators of \(\KL(\pi_\theta \,\|\, \pi_\text{ref})\) are

\[ k_1 = -\log \rho_t, \qquad k_3 = (\rho_t - 1) - \log \rho_t \tag{66.11} \]

Both have the right average (they are unbiased). \(k_1\) is the log-ratio used in (66.9); it can be negative on single tokens. \(k_3\) is never negative and usually has lower variance; it is from Schulman’s note on KL approximation, and GRPO uses it as a loss term. [Certain] We checked numerically that both average to the exact KL on a small distribution (0.02582 nats in our example). When people report “KL” in a PPO-RLHF run, it is usually the sum of \(k_1\) over the response, averaged over the batch. [Likely]

Four models in memory

A PPO-RLHF iteration needs four networks:

  1. Policy (also called the actor): the model being trained. It generates the responses and gets gradients. Initialized from the SFT model.
  2. Reference model: a frozen copy of the SFT model. Only forward passes, to get \(\log \pi_\text{ref}\) for (66.9).
  3. Reward model: frozen. One forward pass per response gives \(r_\psi(x, y)\).
  4. Critic (value model): trained. A language model with a scalar head that predicts \(V(s_t)\) at every token. InstructGPT initialized it from the reward model. [Certain]

Two of them are trained, so they need gradients and optimizer states. With mixed-precision AdamW (session 38), a trained model needs about 16 bytes per parameter (2 for bf16 weights, 2 for gradients, 4 for the fp32 master copy, 4 + 4 for Adam’s two moments). A frozen model in bf16 needs 2 bytes per parameter. So

\[ M \approx 16\,(N_\pi + N_V) + 2\,(N_\text{ref} + N_\text{RM}) \ \text{bytes} \;+\; \text{activations} + \text{KV cache} \tag{66.12} \]
Read the symbols
\(N_\pi, N_V\)
parameters of the policy and the critic (trained)
\(N_\text{ref}, N_\text{RM}\)
parameters of the reference and the reward model (frozen); \(N_\text{ref} = N_\pi\)
KV cache
memory for generation; for long responses and big batches it can be tens of GB on its own
Worked example: PPO on a 7B model

All four models 7B: policy \(16 \times 7 = 112\) GB, critic 112 GB, reference 14 GB, reward model 14 GB. Total 252 GB of model state, before activations and the KV cache: at least 4 GPUs of 80 GB just to hold it. Compare SFT of the same model: 112 GB. PPO needs more than twice the memory, and each iteration also runs generation, three extra forward passes and two backward passes.

InstructGPT used a 6B reward model and a 6B critic for all policy sizes, partly because “a 175B RM and value function greatly increase the compute requirements of PPO”. [Certain] With a 175B policy: policy 2,800 GB, reference 350 GB, reward model 12 GB, critic 96 GB, total 3,258 GB by (66.12).

Figure 4. One PPO-RLHF iteration. Solid arrows are data flow during the rollout (generation and scoring), the dashed arrows are the PPO update. Bars show the memory of each model by formula (66.12): 16 bytes per parameter for trained models, 2 bytes for frozen bf16 models. Activations and the KV cache are not included.
How to read Figure 4

Start at 7B with all models the same size: the two trained models take 89% of the memory. Choose “6B (InstructGPT)” with a 175B policy: the policy and its frozen copy now dominate, and the small reward model and critic hardly matter. Choose “No critic”: GRPO drops the critic and estimates the advantage from a group of answers to the same prompt instead, which saves the second trained model. That is one main reason it became popular (session 70).

Ways to fit the four models, used in real frameworks: shard all of them with ZeRO-3 or FSDP (session 45); put the frozen models on separate GPUs or offload them to CPU; use LoRA (session 57) so the reference is just the base weights with the adapter turned off; share one backbone between policy and critic (rare for LLMs, since the two losses interfere [Likely]); and use a fast inference engine such as vLLM for the generation phase, with weights copied from the trainer after each update (session 75). TRL, OpenRLHF and verl all implement versions of these tricks. [Likely]

The algorithm

Algorithm 1: PPO, actor-critic style (Schulman et al., 2017)
  1. input: policy \(\pi_\theta\), critic \(V_\phi\), clip \(\epsilon\), epochs \(K\), mini-batch size \(M\)
  2. for iteration \(= 1, 2, \dots\):
  3.   run \(\pi_\theta\) in the environment to collect \(N T\) steps; \(\theta_\text{old} \leftarrow \theta\) // the batch is now fixed
  4.   store \(\log \pi_{\theta_\text{old}}(a_t \mid s_t)\), \(V_{\phi_\text{old}}(s_t)\), rewards
  5.   compute \(\hat A_t\) with GAE and returns \(\hat R_t = \hat A_t + V_{\phi_\text{old}}(s_t)\) // (66.10)
  6.   for epoch \(= 1 \dots K\): for each shuffled mini-batch of size \(M\):
  7.     \(r_t \leftarrow \exp(\log \pi_\theta - \log \pi_{\theta_\text{old}})\) // (66.1)
  8.     loss \(\leftarrow -L^{\text{CLIP}} + c_1 L^{\text{VF}} - c_2 S\) // (66.3), (66.5)–(66.7)
  9.     Adam step on \(\theta, \phi\) (after gradient clipping)
  10.     (optional) stop the epochs early if the approximate KL to \(\pi_{\theta_\text{old}}\) is too large
Algorithm 2: one PPO-RLHF iteration for a language model
  1. input: prompts batch \(\{x_i\}\); policy \(\pi_\theta\); frozen \(\pi_\text{ref}\), \(r_\psi\); critic \(V_\phi\); \(\beta\), \(\epsilon\), \(\gamma = 1\), \(\lambda\)
  2. \(y_i \sim \pi_\theta(\cdot \mid x_i)\), sampling at temperature 1 // rollout: the slow part
  3. one forward pass each: \(\log \pi_{\theta_\text{old}}(y_{i,t})\), \(\log \pi_\text{ref}(y_{i,t})\), \(V_\phi(s_{i,t})\), and \(r_\psi(x_i, y_i)\)
  4. \(r_{i,t} \leftarrow -\beta(\log \pi_{\theta_\text{old}} - \log \pi_\text{ref})\); add \(r_\psi(x_i, y_i)\) at the last token // (66.9)
  5. GAE backwards over each response; returns \(\hat R = \hat A + V\); whiten \(\hat A\) over the batch // (66.10)
  6. for epoch \(= 1 \dots K\): for each mini-batch: // InstructGPT: \(K = 1\), 8 mini-batches
  7.   recompute \(\log \pi_\theta\) and \(V_\phi\) on the stored \((x, y)\) with teacher forcing
  8.   policy loss \(-L^{\text{CLIP}}\) over response tokens only (mask prompt and padding); value loss (66.5)
  9.   backward, clip gradients, Adam steps for policy and critic
  10. log: mean score, KL to reference, clip fraction, entropy, value error; copy new weights to the generation engine

Watch Algorithm 2 run, line by line

Formulas are easier to trust when you can see them produce numbers. Figure 5 runs Algorithm 2 for real in your browser, on a language model small enough to follow by hand. There are four prompts, “2+3=”, “4+3=”, “6+2=” and “3+3=”. The model answers with two tokens: a number from 0 to 9, then either the closing tag “</a>” or a period. A program checks the answer, as in session 70: the reward is \(R = r_\text{acc} + r_\text{fmt}\), so 2 for the right number with a closed tag, 1 for a wrong number with a closed tag, and 0 when the tag is missing (the parser cannot read the answer). The reference model is a weak “SFT model”: it gives the right number only 18% of the time, likes the number one too high (a typical slip), and closes the tag 40% of the time. The policy starts as a copy of the reference.

The policy’s parameters are its logits (the scores before the softmax): a table of 10 scores per prompt for the number, and 2 scores per prompt for the tag. The critic is a table too: one value for the state before the number and one for each state before the tag. Every line of the algorithm then needs only one more formula, the gradient of the PPO term with respect to a logit:

\[ \frac{\partial\, (\rho\, \hat A)}{\partial z_k} = \hat A\, \rho\, \big( \mathbb{1}[k = y] - \pi_\theta(k \mid s) \big), \qquad g \leftarrow g \cdot \min\!\Big( 1, \frac{c}{\lVert g \rVert} \Big) \tag{66.13} \]
Read the symbols
\(z_k\)
the logit of token \(k\) in state \(s\); \(\pi_\theta(k \mid s)\) is its softmax probability
\(y\)
the token that was sampled; its logit goes up by \(\hat A \rho (1 - \pi)\) and every other logit goes down by \(\hat A \rho\, \pi_k\)
\(\rho\), \(\hat A\)
the ratio (66.1) and the advantage of this token; when the token is clipped in (66.3), its gradient is 0
\(g\), \(c\)
the mini-batch gradient and the gradient-norm limit (here \(c = 0.1\)); almost every PPO code clips the gradient norm like this

We write the ratio as \(\rho\) in the figure, to keep the letter \(r\) for rewards. The update is plain gradient ascent \(\theta \leftarrow \theta + \alpha g\) on the logits; real runs use Adam, which changes the step size but not the direction logic.

Algorithm 2, as the simulation runs it
  1. for iteration \(= 1, 2, \dots\) (at most 200):
  2.   sample 4 answers per prompt, \(y \sim \pi_\theta\); freeze \(\pi_{\theta_\text{old}} \leftarrow \pi_\theta\)
  3.   store \(\log \pi_{\theta_\text{old}}\), \(\log \pi_\text{ref}\), \(V_\text{old}\) for both tokens
  4.   \(R \leftarrow r_\text{acc} + r_\text{fmt}\) // a program, not a model
  5.   \(r_t \leftarrow -\beta(\log \pi_{\theta_\text{old}} - \log \pi_\text{ref})\), plus \(R\) at the last token // (66.9)
  6.   \(\delta_t\), \(\hat A_t\) by GAE, \(\gamma = 1\), \(\lambda = 0.95\); \(\hat R_t \leftarrow \hat A_t + V_\text{old}\) // (66.10)
  7.   for epoch \(= 1 \dots K\): shuffle, cut into mini-batches of \(M\)
  8.     \(\rho \leftarrow \exp(\log \pi_\theta - \log \pi_{\theta_\text{old}})\) // (66.1)
  9.     \(L^\text{CLIP} \leftarrow\) mean of \(\min(\rho \hat A, \operatorname{clip}(\rho, 1 \pm \epsilon)\hat A)\) // (66.3)
  10.     \(L^\text{VF} \leftarrow\) mean of the clipped value loss // (66.5)
  11.     \(\theta \leftarrow \theta + \alpha\, g\) (norm-clipped); \(V \leftarrow V - 2\, \nabla L^\text{VF}\) // (66.13)
  12.   log accuracy, KL to \(\pi_\text{ref}\), clip fraction
accuracy (exact)format rate (dashed) and clip fractionKL to reference; purple ticks: \(\pi_\text{ref}\)right numbertrust region \(1 \pm \epsilon\)
Figure 5. PPO-RLHF (Algorithm 2) running live on the toy model. The highlighted line is the line that just ran; the panel next to it shows that line’s formula with the numbers of the followed answer. Answer cards: left edge green for reward 2, orange for 1, red for 0; blue background marks the current mini-batch; red ratios are clipped tokens. Accuracy and KL are computed exactly from the policy’s probabilities, not estimated from samples. Each seed gives the same run every time.
How to read Figure 5

Press “Step one line” a few times. Line 5 shows that the first answer of seed 1, “2+3= 6</a>”, gets \(r_1 = 0\) and \(r_2 = 1\): in iteration 1 the policy equals the reference, so the KL part is 0, and the closed tag earns 1. The critic still says 0, so line 6 gives \(\hat A_2 = 1\) and \(\hat A_1 = 0.95\): the wrong number 6 is pushed up, because the critic has not yet learned that a closed tag is the normal case. Keep stepping to line 8: in the first mini-batch every ratio is exactly 1; in later mini-batches they spread, and the histogram shows them leaving the shaded band. Then switch to “Fast” and press Play. The format rate (dashed) rises first, then the accuracy. Try “No clip”: the clip fraction drops to 0 and the ratio histogram grows a red bar above 2. Try “Strong KL”: the purple KL curve stops much lower and the accuracy stays well below 1, because the leash to the reference holds the policy back.

Worked example: the first answer of Figure 5 (seed 1, default settings)

The answer is “6</a>” to “2+3=”. Under the reference, \(\pi(6) = 0.302\) and \(\pi(\text{</a>}) = 0.401\), so \(\log \pi_{\theta_\text{old}} = -1.198\) and \(-0.913\). The verifier gives \(R = 0 + 1 = 1\).

Rewards (66.9): \(r_1 = -0.05 \times (-1.198 - (-1.198)) = 0\) and \(r_2 = 0 + 1 = 1\). With the critic at 0: \(\delta_2 = 1 - 0 = 1\), \(\delta_1 = 0 + 0 - 0 = 0\), so \(\hat A_2 = 1\) and \(\hat A_1 = 0 + 0.95 \times 1 = 0.95\).

After iteration 1 (8 mini-batch steps), \(\pi_\theta(6 \mid \text{2+3=})\) has grown from 0.302 to 0.395 and \(\pi_\theta(\text{</a>} \mid \text{2+3=})\) from 0.401 to 0.678. 32% of the token updates in this iteration were clipped. The exact accuracy goes from 0.073 to 0.089, then 0.736 after 10 iterations and 0.986 after 30, with a KL to the reference of 2.517 nats.

The PyTorch code below runs this loop on a toy problem in a few seconds on a CPU. The “language model” writes 6 tokens from a vocabulary of 8, the frozen “reward model” scores the fraction of tokens equal to 3, and all four roles exist: policy, frozen reference, reward model and critic. It is deterministic (fixed seed, CPU).

import copy, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
V, T, B = 8, 6, 64            # vocab size, response length, prompts per batch
GOOD = 3                       # the "reward model" likes token 3

class TinyLM(nn.Module):       # next-token logits from (previous token, position)
    def __init__(s, out):
        super().__init__()
        s.tok, s.pos = nn.Embedding(V + 1, 32), nn.Embedding(T, 32)
        s.mlp = nn.Sequential(nn.Linear(32, 64), nn.Tanh(), nn.Linear(64, out))
    def forward(s, prev):      # prev: (B, T) previous tokens, V = "start"
        return s.mlp(s.tok(prev) + s.pos.weight[None, :prev.size(1)])

def reward_model(y):           # frozen scorer: one number per response
    return (y == GOOD).float().mean(1)

policy = TinyLM(V)
ref = copy.deepcopy(policy).eval()            # frozen reference = start policy
for p in ref.parameters(): p.requires_grad_(False)
critic = TinyLM(1)                             # value head: one number per token
opt = torch.optim.Adam(list(policy.parameters()) + list(critic.parameters()), lr=1e-3)
beta, eps, gamma, lam, c1, c2 = 0.05, 0.2, 1.0, 0.95, 0.5, 0.01

def prev_of(y):                # shift right: the token before each position
    return torch.cat([torch.full((y.size(0), 1), V), y[:, :-1]], 1)

def logp_of(model, y):         # log pi(y_t | y_<t) for every position, plus full log-probs
    lp = F.log_softmax(model(prev_of(y)), -1)
    return lp.gather(-1, y[..., None]).squeeze(-1), lp

for it in range(60):
    # 1) rollout: sample responses token by token from the current policy
    with torch.no_grad():
        y = torch.zeros(B, 0, dtype=torch.long)
        for t in range(T):
            prev = prev_of(torch.cat([y, torch.zeros(B, 1, dtype=torch.long)], 1))
            nxt = torch.distributions.Categorical(logits=policy(prev)[:, t]).sample()
            y = torch.cat([y, nxt[:, None]], 1)
        old_lp, _ = logp_of(policy, y)
        ref_lp, _ = logp_of(ref, y)
        values = critic(prev_of(y)).squeeze(-1)
        score = reward_model(y)
        # 2) per-token reward: KL penalty at every token, score at the last one
        kl_t = old_lp - ref_lp
        rew = -beta * kl_t
        rew[:, -1] += score
        # 3) GAE, backwards over the response
        adv, last = torch.zeros(B, T), torch.zeros(B)
        for t in reversed(range(T)):
            nv = values[:, t + 1] if t + 1 < T else torch.zeros(B)
            delta = rew[:, t] + gamma * nv - values[:, t]
            last = delta + gamma * lam * last
            adv[:, t] = last
        ret = adv + values                       # targets for the critic
        adv = (adv - adv.mean()) / (adv.std() + 1e-8)
    # 4) PPO epochs over minibatches of the same rollout
    clipped = []
    for epoch in range(4):
        for idx in torch.randperm(B).split(16):
            new_lp, lp_all = logp_of(policy, y[idx])
            ratio = torch.exp(new_lp - old_lp[idx])
            a = adv[idx]
            pg = -torch.min(ratio * a, ratio.clamp(1 - eps, 1 + eps) * a).mean()
            v = critic(prev_of(y[idx])).squeeze(-1)
            vf = ((v - ret[idx]) ** 2).mean()
            ent = -(lp_all.exp() * lp_all).sum(-1).mean()
            loss = pg + c1 * vf - c2 * ent
            opt.zero_grad(); loss.backward()
            nn.utils.clip_grad_norm_(list(policy.parameters()) + list(critic.parameters()), 1.0)
            opt.step()
            clipped.append(((ratio - 1).abs() > eps).float().mean().item())
    if it % 10 == 0 or it == 59:
        print(f"iter {it:2d}  score {score.mean():.3f}  KL to ref {kl_t.sum(1).mean():.3f}"
              f"  clip frac {sum(clipped)/len(clipped):.3f}")
# Output (CPU, torch 2.8):
# iter  0  score 0.164  KL to ref 0.000  clip frac 0.136
# iter 10  score 0.828  KL to ref 5.978  clip frac 0.005
# iter 20  score 0.768  KL to ref 5.130  clip frac 0.082
# iter 30  score 0.849  KL to ref 6.492  clip frac 0.047
# iter 40  score 0.776  KL to ref 5.190  clip frac 0.031
# iter 50  score 0.844  KL to ref 6.329  clip frac 0.031
# iter 59  score 0.784  KL to ref 5.394  clip frac 0.029

The score rises from 0.164 to about 0.8 in 10 iterations and then stays there, held by the KL leash. Is 0.8 the right place to stop? For this reward, the best policy of (66.8) has a closed form per token: \(\pi^*(\text{GOOD}) \propto \pi_\text{ref}(\text{GOOD})\, e^{(1/6)/\beta}\), because each good token adds \(1/6\) to the score. If the reference were uniform over the 8 tokens, that gives \(e^{3.33} = 28.0\) and \(\pi^* = 28.0/(28.0 + 7) = 0.80\) for \(\beta = 0.05\). With \(\beta = 0.2\) the same code (we ran it) levels off at a score of about 0.24 to 0.34 and a KL below 0.61, close to the closed form \(\pi^* = 2.30/(2.30 + 7) = 0.25\). So \(\beta\) directly sets how far the policy goes from the reference.

Practical recipe and what to watch

SettingPPO paper (MuJoCo)PPO paper (Atari)InstructGPT (RLHF)
Clip \(\epsilon\)searched; 0.2 best\(0.1 \times \alpha\) (\(\alpha\) annealed \(1 \to 0\))0.2
Epochs per batch \(K\)1031 (batch 512 split into 8 mini-batches of 64)
Discount \(\gamma\), GAE \(\lambda\)0.99, 0.950.99, 0.95\(\gamma = 1\) (no discount)
Learning rate\(3 \times 10^{-4}\)\(2.5 \times 10^{-4} \times \alpha\)constant after 10-iteration warmup; critic \(9 \times 10^{-6}\) (1.3B, 6B policies)
Value / entropy weightsseparate networks, no entropy bonus\(c_1 = 1\), \(c_2 = 0.01\)separate 6B critic, initialized from the reward model
KL to referencenonenone\(\beta = 0.02\), per token; plus pretraining loss with \(\gamma_\text{ptx} = 27.8\) (PPO-ptx)
Scale1M steps per task8 actors \(\times\) 128 steps256k episodes, about 31k unique prompts

All values in this table are from the two papers. [Certain] InstructGPT also kept an exponential moving average of the weights (decay 0.992) and sampled at temperature 1. [Certain] Its “PPO-ptx” variant mixes ordinary next-token loss on pretraining data into the PPO gradient to reduce the drop in public NLP benchmarks (the “alignment tax”). [Certain]

Signals to monitor

  • Reward model score on the rollouts: should rise, then level off. A score that keeps rising fast while human or judge ratings stop improving suggests reward hacking (session 67).
  • KL to the reference: should grow slowly. A sudden jump means the policy found a strange region (repetition, odd formatting).
  • Clip fraction and approximate KL to the old policy: if the clip fraction is very high (say above 0.3 [Likely]), the steps are too large; lower the learning rate or the number of epochs.
  • Entropy of the policy: a fast collapse means the model is becoming deterministic and stops exploring.
  • Value loss and explained variance of the critic: a critic that cannot predict returns gives noisy advantages.
  • Response length: reward models often prefer longer answers, so a steady growth in length is a classic hack. [Likely]

Many small implementation details matter. Two careful studies collected them: one for PPO in general (37 details, such as advantage normalization per mini-batch, global gradient-norm clipping at 0.5 and Adam epsilon \(10^{-5}\)) [Certain], and one for PPO-RLHF on summarization (more than 20 details). [Certain] Examples from RLHF practice: whiten the advantages, normalize or whiten the reward model scores, give a fixed penalty to responses that never produce the end token, and compute the value only on response tokens. [Likely]

Two runs side by side: what the clip and the KL weight do

Figure 6 runs the simulation of Figure 5 twice with the same seed, so both runs see the same random numbers, and changes only one setting. Both runs use \(\alpha = 20\), \(K = 4\) epochs and mini-batches of 8.

The clip. Without the clip (\(\epsilon = 100\), so nothing is ever clipped), a lucky batch can move a prompt all the way in one iteration. For seed 3, the prompt “4+3=” jumps to the answer “3</a>” in the very first iteration (from probability 0.05 to 0.90): wrong, but with a closed tag, so it earns reward 1. Once the policy says “3” with probability 0.9 and more, it almost never samples “7” again, every answer gets the same reward, the critic predicts it, the advantages go to 0, and the prompt is stuck. After 150 iterations run B has accuracy 0.750 (three prompts of four) and run A, with \(\epsilon = 0.2\), has 0.998. In 4 of the first 6 seeds the unclipped run ends stuck at 0.75. This is the RL meaning of “instability” in a run with exact rewards: not a crash, but a policy that commits too early and stops exploring.

The KL weight. With \(\beta = 0\) nothing pulls the policy back, and its KL to the reference climbs to 2.60 nats: it says the right answer with probability near 1, far from the reference’s 0.18. With \(\beta = 0.5\) the run settles near a KL of 1.21 and an accuracy of 0.559 for seed 3: the KL penalty in the reward (66.9) has become as large as the gain from being right. This is formula (66.8) at work: \(\beta\) chooses how far from the reference the best policy is.

Run ARun B (dashed)purple ticks: \(\pi_\text{ref}\)
Figure 6. Two PPO runs of the Figure 5 simulation with the same seed and one different setting, 150 iterations. Run A is solid blue, run B is dashed red. The bottom panels show each run’s policy: bars are \(\pi_\theta\) over the ten numbers, green for the right one, purple ticks for the reference.
How to read Figure 6

Press “Run to the end” with the clip comparison and seed 3: run B’s accuracy curve stops at 0.75, and its policy panel shows the prompt “4+3=” with a tall grey bar on 3. Its clip fraction is almost 0, because with \(\epsilon = 100\) nothing is clipped. Run A’s clip fraction is high in the first iterations (0.30, 0.63, 0.20): that is the clip doing its job, holding back the big early steps. Change the seed: some seeds are lucky and run B ends at 1.0 too, but run A ends above 0.99 for every seed from 1 to 6. Switch to the KL comparison: run B (\(\beta = 0.5\)) keeps the bars close to the purple ticks.

Where this is used

SettingRewardHow PPO is used
Classic RLHF chat assistants (InstructGPT, Llama 2-Chat)A learned reward model of human preferencesThe full four-model loop with a KL leash to the SFT model; see session 67
Summarization from human feedback (Stiennon et al., 2020)Reward model of summary preferencesThe same loop on TL;DR summaries; the setting where PPO-RLHF for LLMs was worked out
Reasoning and math with verifiable rewardsA checker: correct final answer or passing testsPPO with a critic, or GRPO, which keeps the clipped loss but drops the critic (session 70)
Robotics and games (the original PPO paper)Environment rewardMany short episodes, discount 0.99, several epochs per batch
VLM alignment (less hallucination in image answers)Reward model or judge that checks answers against the imageSame loop; the image tokens are part of the prompt and get no loss. [Likely]
Video-LLMs: video QA and key-moment detectionVerifiable rewards such as the overlap between a predicted and a true time span, or exact-match answersPPO or GRPO-style updates on top of a video SFT model (session 205); long visual prompts make rollouts and memory even more expensive. [Likely]
Video captioning and retrieval-friendly descriptionsA judge or a retrieval score of the caption against the clipReward the caption that retrieves the right clip; the KL leash keeps captions fluent. [Speculation]

Common mistakes

  • Gradient through the old log-probabilities. \(\log \pi_{\theta_\text{old}}\) and the advantages must be constants (computed under no_grad and stored). If they carry gradients, the ratio is always 1 and the loss is wrong.
  • Recomputing old log-probabilities with a different setup than generation: different temperature, different padding, or a different numeric path in the inference engine. Then \(r \ne 1\) before the first step. Check that the ratio is 1.000 at the start of each iteration.
  • Mixing up the two KLs. The clip (or adaptive penalty) controls the step from \(\pi_{\theta_\text{old}}\). The \(\beta\) term controls the distance to the frozen \(\pi_\text{ref}\). Using the policy itself as the reference makes the leash vanish.
  • Loss on prompt or padding tokens. Only response tokens are actions. Mask the prompt and padding in the policy loss, the value loss, the KL and the entropy.
  • Wrong sign of the min for negative advantages. Writing min(ratio, clip(ratio)) * adv instead of min(ratio * adv, clip(ratio) * adv) clips in the wrong direction when the advantage is negative.
  • Too many epochs or too large a learning rate. A high clip fraction and a fast-growing KL are the signs. LLM RLHF often uses 1 to 4 epochs. [Likely]
  • An untrained critic. A random critic gives random advantages. Initialize it from the reward model (as InstructGPT did) or from the SFT model, and consider a few critic-only warmup steps. [Likely]
  • Ignoring the end token. Responses cut by the length limit get a reward model score for text that never finished; penalize or drop them.

Interview questions

Why does PPO need the importance ratio at all?basic
The policy-gradient formula is valid only for samples from the current policy. PPO wants to take several gradient steps on one batch, so after the first step the samples come from an older policy. Multiplying the advantage by \(r = \pi_\theta / \pi_{\theta_\text{old}}\) corrects for this (importance sampling): the surrogate \(\E[r \hat A]\) has the policy gradient as its gradient at \(\theta_\text{old}\) and stays a reasonable estimate while the policy stays close. Without it, every step needs a fresh batch, and generation is the expensive part of RLHF.
Write the PPO clipped objective and explain why it takes a minimum.core
\(L^{\text{CLIP}} = \E_t[\min(r_t \hat A_t, \operatorname{clip}(r_t, 1-\epsilon, 1+\epsilon)\hat A_t)]\). The clipped term alone would remove the gradient whenever \(r\) leaves the interval, also when the change made things worse. The minimum keeps the unclipped term whenever it is smaller, so the clip only removes the incentive to move further in the direction that improves the objective. The result is a pessimistic lower bound on the unclipped surrogate: improvement beyond \(1 \pm \epsilon\) is not counted, but damage is always counted and corrected.
When is the gradient of a PPO term zero? What happens if \(\hat A \lt 0\) and \(r = 1.5\)?core
The gradient is zero when \(\hat A \gt 0\) and \(r \gt 1+\epsilon\), or when \(\hat A \lt 0\) and \(r \lt 1-\epsilon\): the sample has already moved far enough in the right direction (and also when \(\hat A = 0\)). With \(\hat A = -1\), \(r = 1.5\), \(\epsilon = 0.2\): unclipped \(-1.5\), clipped \(-1.2\), the min is \(-1.5\), the unclipped term, so the gradient with respect to \(r\) is \(\hat A = -1\). A bad action that became more likely is pushed back down with full strength.
Clipping versus an adaptive KL penalty: what are they and which works better?core
The KL-penalty version maximizes \(\E[r \hat A] - \beta \KL[\pi_{\theta_\text{old}}, \pi_\theta]\) and adapts \(\beta\) after each update: halve it if the measured KL is below \(d_\text{targ}/1.5\), double it if above \(1.5\, d_\text{targ}\). Clipping needs no KL computation and no adaptation. In the PPO paper’s MuJoCo comparison, clipping with \(\epsilon = 0.2\) scored 0.82 (normalized), the best adaptive KL setting 0.74, the best fixed KL 0.72, and no constraint −0.39. In practice, people use clipping and often also monitor the KL, sometimes stopping the epochs early when it grows too large.
What is the difference between the KL to the old policy and the KL to the reference model?core
The KL to \(\pi_{\theta_\text{old}}\) (enforced by the clip) is a per-iteration trust region: it limits how far one update moves, and \(\pi_{\theta_\text{old}}\) changes every iteration. The KL to \(\pi_\text{ref}\) (the \(\beta\) term in RLHF) is part of the task: it limits how far the whole training run can drift from the frozen SFT model, to prevent reward hacking and keep fluency. The first is an optimization tool; the second changes what the optimal policy is (\(\pi^* \propto \pi_\text{ref} e^{r/\beta}\)).
Why put the reference KL into the per-token reward instead of into the loss?advanced
The objective (66.8) is an expected return where the KL is part of the reward: each token pays \(-\beta \log(\pi/\pi_\text{ref})\). Putting it in the reward means the critic and GAE treat it like any other reward, so a token is also charged for the KL cost of the tokens it causes later. It also gives dense feedback at every token, while the reward model only speaks at the end. GRPO instead adds a KL term (the \(k_3\) estimator) directly to the loss, which is simpler without a critic; the two are not identical in gradient, and the choice is one of the known differences between methods.
Which models are in memory during PPO-RLHF, and how much memory for a 7B setup?core
Policy (trained), reference (frozen SFT copy), reward model (frozen) and critic (trained, often initialized from the reward model). With mixed-precision AdamW at 16 bytes per trained parameter and 2 bytes per frozen bf16 parameter, all at 7B: \(112 + 112 + 14 + 14 = 252\) GB before activations and the KV cache for generation, versus 112 GB for SFT. Ways to reduce it: smaller reward model and critic (InstructGPT used 6B for a 175B policy), ZeRO-3/FSDP sharding, offloading the frozen models, LoRA so the reference is the base with adapters off, or dropping the critic as GRPO does.
Why does InstructGPT initialize the critic from the reward model, and what goes wrong with a bad critic?advanced
The critic must predict the expected final score from partial responses. The reward model already understands which responses score well, so it is a much better starting point than a random head. InstructGPT notes that 175B reward models were unstable and less suitable as value-function initializations, and used 6B for both. A bad critic gives noisy or biased advantages: with \(\lambda \lt 1\), GAE trusts the critic’s values, so errors in \(V\) directly become wrong credit for tokens. Symptoms are a high value loss, slow or unstable reward growth, and advantages that are noise.
What do you monitor during a PPO-RLHF run, and what do the signals tell you?core
Mean reward model score (should rise then level off), KL to the reference (should grow slowly; jumps signal strange text), clip fraction and approximate KL to the old policy (too high means steps too large), policy entropy (collapse means no exploration), value loss or explained variance (critic quality), response length (a common reward hack), and periodic evaluations with humans or a separate judge, because the reward model score alone can rise while true quality falls.
Is PPO on-policy or off-policy? Why does RLHF often use only one epoch?advanced
PPO is on-policy in spirit: each batch comes from the current policy and is used only for a short time. Within one iteration it is slightly off-policy, because later mini-batch steps train on samples from \(\pi_{\theta_\text{old}}\); the ratio and the clip handle that small gap. In LLM RLHF the batch is large and the reward is noisy and exploitable, so a large step on one batch is risky; InstructGPT used a single epoch over 8 mini-batches. More epochs save generation compute, but a high clip fraction shows when they stop being useful. Asynchronous systems that generate with slightly stale weights increase the off-policy gap (session 71).
Why is \(\gamma = 1\) normal in RLHF, and how does the end-of-response score reach the first token?core
An episode is one response of finite length, and we care about the whole response equally, so there is no reason to discount; InstructGPT used no discount. The reward model score appears only at the last token (plus small KL rewards everywhere). With GAE, \(\hat A_t\) sums the TD errors after \(t\) with weight \(\lambda^l\); with \(\lambda = 1\) it equals the full remaining reward minus \(V(s_t)\), so the score reaches every token directly. With \(\lambda \lt 1\), the score reaches early tokens through the critic’s values: the critic learns to predict the final score and so passes credit back.

Exercises

  1. With \(\epsilon = 0.2\) and \(\hat A = -3\), compute the PPO term and its gradient with respect to \(r\) for \(r = 0.75\) and for \(r = 1.25\).
  2. In the adaptive KL penalty with \(d_\text{targ} = 0.01\), what happens to \(\beta\) when the measured KL is 0.02, 0.005 and 0.012?
  3. A response has two tokens with log-ratios 0.2 and 0.4 to the reference, \(\beta = 0.1\), reward model score 0.5, critic values \(V = [0.3, 0.4]\), \(\gamma = \lambda = 1\). Compute the per-token rewards and the two advantages.
  4. Compute the model-state memory by (66.12) for a 13B policy with a 13B reward model and a 13B critic. How many 80 GB GPUs is that at least?
  5. InstructGPT trained for 256k episodes with batches of 512 and mini-batches of 64, one epoch. About how many PPO iterations and how many optimizer steps is that?
  6. Explain in one sentence why \(\hat A = 0\) for every sample would give zero policy gradient no matter what the ratios are.
Answerscheck
1. \(r = 0.75\): unclipped \(-2.25\), clipped \(0.8 \times (-3) = -2.4\); the min is \(-2.4\) (clipped), gradient 0. \(r = 1.25\): unclipped \(-3.75\), clipped \(1.2 \times (-3) = -3.6\); the min is \(-3.75\) (unclipped), gradient \(-3\), pushing \(r\) down. 2. The thresholds are \(0.01/1.5 = 0.00667\) and \(0.015\). KL 0.02 is above 0.015, so \(\beta\) doubles; 0.005 is below 0.00667, so \(\beta\) halves; 0.012 is in between, so \(\beta\) stays. 3. Rewards: \(r_1 = -0.1 \times 0.2 = -0.02\), \(r_2 = -0.1 \times 0.4 + 0.5 = 0.46\). \(\delta_2 = 0.46 + 0 - 0.4 = 0.06\), \(\delta_1 = -0.02 + 0.4 - 0.3 = 0.08\). \(\hat A_2 = 0.06\), \(\hat A_1 = 0.08 + 0.06 = 0.14\), which equals the total reward \(0.44\) minus \(V = 0.3\). 4. \(16 \times (13 + 13) + 2 \times (13 + 13) = 416 + 52 = 468\) GB, at least \(\lceil 468/80 \rceil = 6\) GPUs before activations and KV cache. 5. \(256{,}000 / 512 = 500\) iterations; each has \(512/64 = 8\) mini-batch steps, so about 4,000 optimizer steps. 6. Every term is \(\min(r \cdot 0, \operatorname{clip}(r) \cdot 0) = 0\) for any \(r\), so the objective is constant and has no gradient.

Key takeaways

  • The importance ratio \(r = \pi_\theta / \pi_{\theta_\text{old}}\) (66.1) lets PPO reuse one batch of samples for several mini-batch steps.
  • The clipped objective (66.3) is a pessimistic bound: it stops rewarding changes beyond \(1 \pm \epsilon\) in the improving direction, but always corrects changes in the harmful direction. \(\epsilon = 0.2\) is the standard value.
  • The full loss adds a value loss and an entropy bonus (66.5)–(66.7); in LLM RLHF the critic is a separate model and the entropy bonus is often zero.
  • For LLMs, each token is an action. The reward is a per-token KL penalty to a frozen reference plus the reward model score at the last token (66.9); GAE with \(\gamma = 1\) spreads it over the tokens (66.10).
  • Two KLs: the clip limits the step from the old policy; \(\beta\) limits the drift from the reference and sets the optimum \(\pi^* \propto \pi_\text{ref} e^{r/\beta}\).
  • In the live toy run (Figures 5 and 6), removing the clip lets one lucky batch lock a prompt onto a wrong answer that earns the format reward, and the run stays stuck; \(\beta\) sets how far the policy ends from the reference (KL 2.60 nats with \(\beta = 0\), 1.21 with \(\beta = 0.5\), seed 3).
  • PPO-RLHF keeps four models in memory; two are trained, so a 7B setup needs about 252 GB of model state (66.12). This cost is what DPO and GRPO try to remove.

Key formulas

\[ r_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)}, \qquad L^{\text{CLIP}} = \hat{\E}_t\big[\min(r_t \hat A_t,\ \operatorname{clip}(r_t, 1-\epsilon, 1+\epsilon)\hat A_t)\big] \]
\[ \text{loss} = -L^{\text{CLIP}} + c_1 L^{\text{VF}} - c_2 S, \qquad r_t = -\beta \log\frac{\pi_{\theta_\text{old}}(y_t \mid s_t)}{\pi_\text{ref}(y_t \mid s_t)} + \mathbb{1}[t = T]\, r_\psi(x, y) \]
\[ \hat A_t = \sum_{l \ge 0} (\gamma\lambda)^l \delta_{t+l}, \qquad M \approx 16\,(N_\pi + N_V) + 2\,(N_\text{ref} + N_\text{RM}) \ \text{bytes} \]

References for this session

  1. Schulman et al. (2017). Proximal Policy Optimization Algorithms
  2. Ouyang et al. (2022). Training language models to follow instructions with human feedback
  3. Lambert (2025). Reinforcement Learning from Human Feedback (book)
  4. Schulman et al. (2017). Proximal Policy Optimization Algorithms
  5. Ouyang et al. (2022). Training language models to follow instructions with human feedback
  6. Lambert (2025). Reinforcement Learning from Human Feedback (book)
  7. Schulman et al. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation
  8. OpenAI (Achiam) (2018). Spinning Up in Deep RL
  9. Stiennon et al. (2020). Learning to summarize from human feedback
  10. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  11. Schulman (2020). Approximating KL Divergence (blog)
  12. Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
  13. Hugging Face (code). TRL library
  14. Schulman et al. (2015). Trust Region Policy Optimization
  15. Ziegler et al. (2019). Fine-Tuning Language Models from Human Preferences
  16. Huang et al. (2024). The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
  17. Huang et al. (2022). The 37 Implementation Details of Proximal Policy Optimization (ICLR blog track)