PPO
PPO (Proximal Policy Optimization) is the policy-gradient method behind classic RLHF. It reuses each batch of samples for several gradient steps, and a clipped importance ratio stops any one update from moving the policy too far. This session builds the full loss (clipped policy term, value loss, entropy bonus), shows how a language model turns into an RL agent with a per-token KL penalty to a frozen reference, and counts the memory of the four models PPO keeps on the GPUs.
What you will learn
- Why plain policy gradient can use each sample only once, and how the importance ratio between the new and the old policy lets PPO reuse a batch
- The clipped objective: the six cases of sign and ratio, when the gradient is zero, and why the minimum makes it a pessimistic bound
- The adaptive KL penalty, the value loss, the entropy bonus, and how they combine into one PPO loss
- PPO for language models: tokens as actions, a per-token KL penalty to a frozen reference model, the reward model score at the last token, and GAE with gamma = 1
- The four models in memory (policy, reference, reward model, critic), their byte counts, and the real InstructGPT setup
- A full PPO-RLHF iteration in pseudo-code and in runnable PyTorch, plus the signals to watch during training
Why this matters
In session 64 we learned to push up the log-probability of actions that got a good reward. In session 65 we learned to measure “good” with an advantage from a learned critic. One problem is left: plain policy gradient is wasteful and fragile. Each batch of samples is used for one gradient step and then thrown away, and one step that is too large can wreck the policy. PPO (Proximal Policy Optimization, Schulman et al., 2017) fixes both with one small idea: a clipped probability ratio that lets us reuse a batch for several steps while keeping the new policy close to the old one.
PPO is the algorithm of classic RLHF (reinforcement learning from human feedback). InstructGPT, the model behind the first ChatGPT-style assistants, was trained with PPO against a reward model, with a clip ratio of 0.2 and a KL penalty to the supervised model. [Certain] Llama 2-Chat used PPO after rounds of rejection-sampling fine-tuning. [Certain] Newer methods such as GRPO (session 70) keep PPO’s clipped objective and change only how the advantage is computed. So if you understand PPO, you understand the core of almost every RL method used on LLMs today.
- Reuse samples. Generating answers with a large model is the most expensive part of RLHF. An importance ratio lets us train several mini-batch steps on one batch of generated answers.
- Small, safe steps. The clipped objective removes the incentive to move the probability of any sample by more than a factor \(1 \pm \epsilon\) in one update.
- Stay close to the start. A KL penalty to a frozen reference model stops the policy from drifting into strange text that fools the reward model.
- Four models. Policy, reference, reward model and critic all sit in GPU memory at the same time. This is why PPO-RLHF is expensive, and why DPO and GRPO were invented.
From policy gradient to a trust region
Recall the setting from session 63. A policy \(\pi_\theta(a \mid s)\) gives a probability to each action \(a\) in state \(s\). For an LLM, the state is the prompt plus the tokens written so far, and the action is the next token. The policy-gradient estimate from session 64, with an advantage \(\hat A_t\) from session 65, is
where \(\hat{\E}_t\) means the average over a batch of collected steps. This formula is only correct when the samples come from the current policy \(\pi_\theta\). After one gradient step, \(\theta\) has changed, the batch now comes from an older policy, and the estimate is biased. So REINFORCE must generate a fresh batch for every step. For an LLM, generating a batch of long answers takes much more time than one backward pass. [Likely]
The importance ratio
The fix is importance sampling: if samples come from one distribution but we want an average under another, we weight each sample by the ratio of the two probabilities. Call the policy that generated the batch \(\pi_{\theta_\text{old}}\) (the “old” policy, frozen while we train on this batch). The probability ratio of one sample is
- Read the symbols
- \(\pi_\theta\)
- the policy we are training now; its weights change at every mini-batch step
- \(\pi_{\theta_\text{old}}\)
- the same network at the moment the batch was generated; its log-probabilities are computed once and stored
- \(r_t(\theta)\)
- how much more (above 1) or less (below 1) likely the action is now than when it was sampled; \(r_t = 1\) before the first update
Weighting the advantage by the ratio gives the surrogate objective that TRPO and PPO start from (CPI stands for “conservative policy iteration”, the older work where it appeared):
At \(\theta = \theta_\text{old}\), its gradient is exactly the policy gradient, because \(\nabla_\theta r_t = r_t \nabla_\theta \log \pi_\theta\) and \(r_t = 1\) there. So we can now take several steps on the same batch. The danger is that nothing stops us. Maximizing (66.2) says “if an action had positive advantage, make it infinitely more likely”. The ratio grows, the importance weights get huge, and the estimate becomes meaningless: the batch only tells us about actions near the old policy.
TRPO (Trust Region Policy Optimization, Schulman et al., 2015) solved this with a hard constraint: maximize (66.2) subject to an average KL divergence between old and new policy below a small \(\delta\). That needs second-order math (conjugate gradients and a line search), which is complex and hard to combine with large networks. [Certain] PPO keeps the idea of a trust region, a region around the old policy where the surrogate can be trusted, and gets it with a first-order trick that is a few lines of code.
The clipped objective
PPO replaces (66.2) with
- Read the symbols
- \(\epsilon\)
- the clip range; 0.2 in the PPO paper’s best setting and in InstructGPT
- \(\operatorname{clip}(r, 1-\epsilon, 1+\epsilon)\)
- \(r\) forced into the interval \([1-\epsilon, 1+\epsilon]\): values below are raised to \(1-\epsilon\), values above are lowered to \(1+\epsilon\)
- \(\min(\cdot,\cdot)\)
- take the smaller of the unclipped and the clipped term, for each sample separately
- \(\hat A_t\)
- the advantage of the sample, from GAE; computed once per batch and kept fixed during the epochs
Read it one sample at a time. The second term has no gradient once \(r\) leaves the interval, because a clipped value is a constant. The minimum then picks which term is active. There are only a few cases:
| Advantage | Ratio | Active term | Gradient pushes | Meaning |
|---|---|---|---|---|
| \(\hat A \gt 0\) | \(r \le 1+\epsilon\) | \(r \hat A\) | \(r\) up | a good action, not yet pushed too far: make it more likely |
| \(\hat A \gt 0\) | \(r \gt 1+\epsilon\) | \((1+\epsilon)\hat A\) | nothing (0) | already made much more likely: stop |
| \(\hat A \gt 0\) | \(r \lt 1-\epsilon\) | \(r \hat A\) | \(r\) up | a good action that became less likely: always allowed to fix |
| \(\hat A \lt 0\) | \(r \ge 1-\epsilon\) | \(r \hat A\) | \(r\) down | a bad action, not yet pushed too far: make it less likely |
| \(\hat A \lt 0\) | \(r \lt 1-\epsilon\) | \((1-\epsilon)\hat A\) | nothing (0) | already made much less likely: stop |
| \(\hat A \lt 0\) | \(r \gt 1+\epsilon\) | \(r \hat A\) | \(r\) down | a bad action that became more likely: always allowed to fix |
So the clip only acts in the direction that would improve the objective. In the words of the paper, “we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse”. [Certain] This makes \(L^{\text{CLIP}}\) a pessimistic (lower) bound on \(L^{\text{CPI}}\): it never promises more improvement than the unclipped surrogate, and it stops counting improvement beyond the trust region. Near \(r = 1\) the two are equal, so the first step is a normal policy-gradient step.
The gradient of one term with respect to the ratio is therefore very simple:
In code, the share of samples in the first case is logged as the clip fraction. It is the simplest health signal of a PPO run.
(a) \(\hat A = +2\), \(r = 1.3\): unclipped \(1.3 \times 2 = 2.6\), clipped \(1.2 \times 2 = 2.4\). The minimum is 2.4, a constant, so the gradient is 0. The action is already 30% more likely; PPO stops pushing.
(b) \(\hat A = +2\), \(r = 1.1\): both terms are 2.2, gradient \(\hat A = 2\). Inside the trust region, PPO is plain policy gradient.
(c) \(\hat A = +2\), \(r = 0.6\): unclipped 1.2, clipped \(0.8 \times 2 = 1.6\). The minimum is 1.2, the unclipped term, so the gradient is 2: a good action that lost probability is always pushed back.
(d) \(\hat A = -1\), \(r = 0.7\): unclipped \(-0.7\), clipped \(0.8 \times (-1) = -0.8\). The minimum is \(-0.8\), clipped, gradient 0. The bad action is already 30% less likely; stop.
(e) \(\hat A = -1\), \(r = 1.5\): unclipped \(-1.5\), clipped \(-1.2\). The minimum is \(-1.5\), unclipped, gradient \(-1\). The bad action became 50% more likely, and PPO pushes it down with full force. This is the pessimistic side of the min.
How to read Figure 1
With \(\hat A = 2\), the blue curve follows the grey line up to \(r = 1 + \epsilon\) and then goes flat: there is no reward for making a good action more than 20% more likely in one update. Move \(r\) below 1: the curve follows the line again, so the gradient is back. Now set \(\hat A = -2\): the flat part moves to the left of \(1 - \epsilon\), and for large \(r\) the curve keeps falling steeply. Change \(\epsilon\) to see the flat part move. Setting \(\hat A = 0\) gives a flat line: samples with zero advantage never move the policy.
One batch, many epochs: what clipping buys
A PPO iteration has two phases. First, generate a batch with the current policy and freeze it as \(\pi_{\theta_\text{old}}\); store the old log-probabilities and compute advantages. Second, run \(K\) epochs (passes) over this batch, split into mini-batches, maximizing (66.3) with Adam. Then the trained policy becomes the new old policy, and the loop repeats. The PPO paper used 10 epochs with mini-batches of 64 on MuJoCo robotics tasks and 3 epochs on Atari. [Certain]
Why not train on one batch forever? Because a batch is a noisy sample. With a few dozen answers, some bad answers get high scores by luck. The importance-weighted estimate (66.2) believes the batch completely, so with many epochs the policy chases the noise: the batch says “big improvement” while the true improvement is small or negative. Clipping limits how far the policy can move on one batch. Figure 2 shows this on a problem small enough to compute exactly.
How to read Figure 2
The numbers under the bars are the true reward and, below it, the average noisy reward the batch saw. Answer D is truly worth 0.0, but the batch saw an average of 0.38. Choose “No clipping” and 20 epochs: the policy puts most of its mass on D, the KL grows to about 0.88 nats, and the batch estimate of the gain (orange) climbs to about 0.41 while the true gain (green) is only about 0.05 and falling. Switch to PPO: the policy moves a little, the KL stays near 0.03, and the true gain is higher. Move \(K\) to 1: both methods are identical, because on the first step every ratio is 1.
This toy problem has no function approximation and no changing states, which are the reasons large policy steps are even more harmful in real RL. [Likely] Still, it shows the key point: one batch supports only a small step, and the clip is a cheap way to enforce that.
The alternative: a KL penalty with an adaptive weight
The PPO paper also proposed a penalty instead of a clip:
After each update, measure the average KL \(d\). If \(d \lt d_\text{targ}/1.5\), halve \(\beta\); if \(d \gt 1.5\, d_\text{targ}\), double it. [Certain] On the paper’s MuJoCo benchmark (7 tasks, 3 seeds each, scores normalized so random = 0 and best = 1), no clipping or penalty scored −0.39, clipping with \(\epsilon = 0.1, 0.2, 0.3\) scored 0.76, 0.82 and 0.70, the adaptive KL penalty scored 0.68 to 0.74, and a fixed penalty 0.62 to 0.72. [Certain] So the clip became the default. Keep the two KL terms apart in your head: this one is to the old policy (a trust region per iteration). The RLHF KL below is to the frozen reference model (a leash for the whole run).
The full PPO loss: value loss and entropy bonus
The advantage needs a critic \(V_\phi(s)\) that predicts the return. The critic is trained on the same batch, by regression to the return targets \(\hat R_t = \hat A_t + V_{\phi_\text{old}}(s_t)\) (the GAE return from session 65). Many implementations also clip the value update, in the same spirit as the policy:
The paper itself uses the plain squared error \((V_\phi(s_t) - \hat R_t)^2\). [Certain] Value clipping came from later code; studies of PPO implementations found no clear evidence that it helps and some evidence that it can hurt. [Certain] Treat it as an option.
To keep the policy from becoming certain too early (and stopping exploration), we can add an entropy bonus, the entropy of the policy’s distribution at each state:
For an LLM the sum runs over the whole vocabulary, which is cheap because the logits are already computed. Together, the paper’s combined objective (to be maximized) is
- Read the symbols
- \(L^{\text{CLIP}}_t\)
- the clipped policy term of sample \(t\), from (66.3)
- \(L^{\text{VF}}_t\)
- the squared value error of sample \(t\), from (66.5); it enters with a minus sign because we maximize
- \(c_1\), \(c_2\)
- weights of the value loss and the entropy bonus; the paper’s Atari setting used \(c_1 = 1\), \(c_2 = 0.01\)
- \(S\)
- the entropy bonus (66.6)
In code we minimize the negative: loss = -L_clip + c1 * L_vf - c2 * entropy. The value weight \(c_1\) matters only when policy and critic share parameters; with separate networks, each has its own optimizer or its own loss scale. In LLM RLHF, the policy and critic are usually separate models, and the entropy bonus is often set to 0, because the pretrained distribution already has plenty of entropy. [Likely]
PPO for language models
How does a chatbot become an RL agent? InstructGPT describes the environment as a bandit: it “presents a random customer prompt and expects a response to the prompt”, then produces a reward from the reward model and ends the episode. [Certain] For PPO we still look inside the response: each generated token is one action, the state is the prompt plus the tokens so far, and an episode has as many steps as the response has tokens.
The quantity we want to maximize is the reward model score, minus a penalty for moving away from the model we started from:
- Read the symbols
- \(x\), \(\mathcal{D}\)
- a prompt, and the set of training prompts
- \(y\)
- a full response sampled from the policy
- \(r_\psi(x, y)\)
- the reward model’s score of the whole response (one number); session 67 shows how it is trained
- \(\pi_\text{ref}\)
- the reference model: a frozen copy of the SFT model the policy started from
- \(\beta\)
- the KL coefficient: how short the leash is; InstructGPT used \(\beta = 0.02\)
Why the KL term? The reward model is only a learned approximation of human judgment, and it is reliable only on text similar to what it was trained on. Without the leash, the policy finds text the reward model scores highly but humans do not like: reward hacking. The KL also keeps the language fluent and the general skills of the SFT model. InstructGPT added the KL penalty “to mitigate over-optimization of the reward model”. [Certain] The KL-regularized objective also has a closed-form best policy, \(\pi^*(y \mid x) \propto \pi_\text{ref}(y \mid x)\, e^{r_\psi(x,y)/\beta}\), which is the starting point of DPO (session 68).
The per-token reward
The sequence KL is a sum over tokens, \(\log \frac{\pi_\theta(y \mid x)}{\pi_\text{ref}(y \mid x)} = \sum_t \log \frac{\pi_\theta(y_t \mid x, y_{\lt t})}{\pi_\text{ref}(y_t \mid x, y_{\lt t})}\). Following Ziegler et al. (2019) and Stiennon et al. (2020), the standard implementation puts this sample estimate into the reward, one piece per token, and adds the reward model score at the last token: [Certain]
- Read the symbols
- \(s_t\)
- the state before token \(t\): the prompt \(x\) plus \(y_1, \dots, y_{t-1}\)
- \(T\)
- the index of the last response token (usually the end-of-turn token)
- \(\mathbb{1}[t = T]\)
- 1 at the last token, 0 elsewhere: the reward model speaks only once, at the end
- \(\log \pi_{\theta_\text{old}} - \log \pi_\text{ref}\)
- the per-token log-ratio, computed once during the rollout; it is part of the reward, so it gets no gradient
Then GAE runs over the tokens of each response, exactly as in session 65:
InstructGPT used no discount, \(\gamma = 1\). [Certain] With \(\gamma = 1\) and \(\lambda = 1\), the advantage of every token is just the total remaining reward minus the critic’s guess. With \(\lambda \lt 1\), the critic’s guesses at later tokens are used more, which lowers variance but adds bias. Because the score arrives only at the end, the critic is what spreads it back to the early tokens.
Take \(\beta = 0.05\), log-ratios to the reference \([0.30, -0.10, 0.50, 0.20]\), reward model score 1.2, critic values \(V = [0.6, 0.7, 0.8, 1.0]\), \(\gamma = 1\), \(\lambda = 0.95\).
Per-token rewards (66.9): \(-0.05 \times 0.30 = -0.015\), then \(+0.005\), \(-0.025\), and at the last token \(-0.010 + 1.2 = 1.19\). The total KL penalty is \(-0.05 \times 0.9 = -0.045\).
TD errors (66.10): \(\delta_3 = 1.19 + 0 - 1.0 = 0.19\), \(\delta_2 = -0.025 + 1.0 - 0.8 = 0.175\), \(\delta_1 = 0.005 + 0.8 - 0.7 = 0.105\), \(\delta_0 = -0.015 + 0.7 - 0.6 = 0.085\).
Advantages, from the end: \(\hat A_3 = 0.19\), \(\hat A_2 = 0.175 + 0.95 \times 0.19 = 0.3555\), \(\hat A_1 = 0.105 + 0.95 \times 0.3555 = 0.4427\), \(\hat A_0 = 0.085 + 0.95 \times 0.4427 = 0.5056\). The critic targets are \(\hat A_t + V_t = [1.106, 1.143, 1.156, 1.19]\). Every token gets a positive advantage, the first one the most, because the critic underestimated how good this response would be.
How to read Figure 3
With \(\beta = 0.05\) the KL part of each reward is tiny, and the score at the end dominates. Raise \(\beta\) to 0.5: the token “Paris” now has a clear negative reward, and its advantage drops below its neighbours; this is how the leash pushes back on tokens where the policy has moved far from the reference. Set the score to −1: all advantages turn negative. Set \(\lambda = 0\): each advantage becomes a one-step TD error and depends only on the critic’s next value. Set \(\lambda = 1\): each advantage is the full remaining reward minus \(V(s_t)\).
Measuring the KL
We never compute the exact KL over all possible responses. We estimate it from the sampled tokens. With \(\rho_t = \pi_\text{ref}(y_t \mid s_t) / \pi_\theta(y_t \mid s_t)\) and samples from \(\pi_\theta\), two common per-token estimators of \(\KL(\pi_\theta \,\|\, \pi_\text{ref})\) are
Both have the right average (they are unbiased). \(k_1\) is the log-ratio used in (66.9); it can be negative on single tokens. \(k_3\) is never negative and usually has lower variance; it is from Schulman’s note on KL approximation, and GRPO uses it as a loss term. [Certain] We checked numerically that both average to the exact KL on a small distribution (0.02582 nats in our example). When people report “KL” in a PPO-RLHF run, it is usually the sum of \(k_1\) over the response, averaged over the batch. [Likely]
Four models in memory
A PPO-RLHF iteration needs four networks:
- Policy (also called the actor): the model being trained. It generates the responses and gets gradients. Initialized from the SFT model.
- Reference model: a frozen copy of the SFT model. Only forward passes, to get \(\log \pi_\text{ref}\) for (66.9).
- Reward model: frozen. One forward pass per response gives \(r_\psi(x, y)\).
- Critic (value model): trained. A language model with a scalar head that predicts \(V(s_t)\) at every token. InstructGPT initialized it from the reward model. [Certain]
Two of them are trained, so they need gradients and optimizer states. With mixed-precision AdamW (session 38), a trained model needs about 16 bytes per parameter (2 for bf16 weights, 2 for gradients, 4 for the fp32 master copy, 4 + 4 for Adam’s two moments). A frozen model in bf16 needs 2 bytes per parameter. So
- Read the symbols
- \(N_\pi, N_V\)
- parameters of the policy and the critic (trained)
- \(N_\text{ref}, N_\text{RM}\)
- parameters of the reference and the reward model (frozen); \(N_\text{ref} = N_\pi\)
- KV cache
- memory for generation; for long responses and big batches it can be tens of GB on its own
All four models 7B: policy \(16 \times 7 = 112\) GB, critic 112 GB, reference 14 GB, reward model 14 GB. Total 252 GB of model state, before activations and the KV cache: at least 4 GPUs of 80 GB just to hold it. Compare SFT of the same model: 112 GB. PPO needs more than twice the memory, and each iteration also runs generation, three extra forward passes and two backward passes.
InstructGPT used a 6B reward model and a 6B critic for all policy sizes, partly because “a 175B RM and value function greatly increase the compute requirements of PPO”. [Certain] With a 175B policy: policy 2,800 GB, reference 350 GB, reward model 12 GB, critic 96 GB, total 3,258 GB by (66.12).
How to read Figure 4
Start at 7B with all models the same size: the two trained models take 89% of the memory. Choose “6B (InstructGPT)” with a 175B policy: the policy and its frozen copy now dominate, and the small reward model and critic hardly matter. Choose “No critic”: GRPO drops the critic and estimates the advantage from a group of answers to the same prompt instead, which saves the second trained model. That is one main reason it became popular (session 70).
Ways to fit the four models, used in real frameworks: shard all of them with ZeRO-3 or FSDP (session 45); put the frozen models on separate GPUs or offload them to CPU; use LoRA (session 57) so the reference is just the base weights with the adapter turned off; share one backbone between policy and critic (rare for LLMs, since the two losses interfere [Likely]); and use a fast inference engine such as vLLM for the generation phase, with weights copied from the trainer after each update (session 75). TRL, OpenRLHF and verl all implement versions of these tricks. [Likely]
The algorithm
- input: policy \(\pi_\theta\), critic \(V_\phi\), clip \(\epsilon\), epochs \(K\), mini-batch size \(M\)
- for iteration \(= 1, 2, \dots\):
- run \(\pi_\theta\) in the environment to collect \(N T\) steps; \(\theta_\text{old} \leftarrow \theta\) // the batch is now fixed
- store \(\log \pi_{\theta_\text{old}}(a_t \mid s_t)\), \(V_{\phi_\text{old}}(s_t)\), rewards
- compute \(\hat A_t\) with GAE and returns \(\hat R_t = \hat A_t + V_{\phi_\text{old}}(s_t)\) // (66.10)
- for epoch \(= 1 \dots K\): for each shuffled mini-batch of size \(M\):
- \(r_t \leftarrow \exp(\log \pi_\theta - \log \pi_{\theta_\text{old}})\) // (66.1)
- loss \(\leftarrow -L^{\text{CLIP}} + c_1 L^{\text{VF}} - c_2 S\) // (66.3), (66.5)–(66.7)
- Adam step on \(\theta, \phi\) (after gradient clipping)
- (optional) stop the epochs early if the approximate KL to \(\pi_{\theta_\text{old}}\) is too large
- input: prompts batch \(\{x_i\}\); policy \(\pi_\theta\); frozen \(\pi_\text{ref}\), \(r_\psi\); critic \(V_\phi\); \(\beta\), \(\epsilon\), \(\gamma = 1\), \(\lambda\)
- \(y_i \sim \pi_\theta(\cdot \mid x_i)\), sampling at temperature 1 // rollout: the slow part
- one forward pass each: \(\log \pi_{\theta_\text{old}}(y_{i,t})\), \(\log \pi_\text{ref}(y_{i,t})\), \(V_\phi(s_{i,t})\), and \(r_\psi(x_i, y_i)\)
- \(r_{i,t} \leftarrow -\beta(\log \pi_{\theta_\text{old}} - \log \pi_\text{ref})\); add \(r_\psi(x_i, y_i)\) at the last token // (66.9)
- GAE backwards over each response; returns \(\hat R = \hat A + V\); whiten \(\hat A\) over the batch // (66.10)
- for epoch \(= 1 \dots K\): for each mini-batch: // InstructGPT: \(K = 1\), 8 mini-batches
- recompute \(\log \pi_\theta\) and \(V_\phi\) on the stored \((x, y)\) with teacher forcing
- policy loss \(-L^{\text{CLIP}}\) over response tokens only (mask prompt and padding); value loss (66.5)
- backward, clip gradients, Adam steps for policy and critic
- log: mean score, KL to reference, clip fraction, entropy, value error; copy new weights to the generation engine
Watch Algorithm 2 run, line by line
Formulas are easier to trust when you can see them produce numbers. Figure 5 runs Algorithm 2 for real in your browser, on a language model small enough to follow by hand. There are four prompts, “2+3=”, “4+3=”, “6+2=” and “3+3=”. The model answers with two tokens: a number from 0 to 9, then either the closing tag “</a>” or a period. A program checks the answer, as in session 70: the reward is \(R = r_\text{acc} + r_\text{fmt}\), so 2 for the right number with a closed tag, 1 for a wrong number with a closed tag, and 0 when the tag is missing (the parser cannot read the answer). The reference model is a weak “SFT model”: it gives the right number only 18% of the time, likes the number one too high (a typical slip), and closes the tag 40% of the time. The policy starts as a copy of the reference.
The policy’s parameters are its logits (the scores before the softmax): a table of 10 scores per prompt for the number, and 2 scores per prompt for the tag. The critic is a table too: one value for the state before the number and one for each state before the tag. Every line of the algorithm then needs only one more formula, the gradient of the PPO term with respect to a logit:
- Read the symbols
- \(z_k\)
- the logit of token \(k\) in state \(s\); \(\pi_\theta(k \mid s)\) is its softmax probability
- \(y\)
- the token that was sampled; its logit goes up by \(\hat A \rho (1 - \pi)\) and every other logit goes down by \(\hat A \rho\, \pi_k\)
- \(\rho\), \(\hat A\)
- the ratio (66.1) and the advantage of this token; when the token is clipped in (66.3), its gradient is 0
- \(g\), \(c\)
- the mini-batch gradient and the gradient-norm limit (here \(c = 0.1\)); almost every PPO code clips the gradient norm like this
We write the ratio as \(\rho\) in the figure, to keep the letter \(r\) for rewards. The update is plain gradient ascent \(\theta \leftarrow \theta + \alpha g\) on the logits; real runs use Adam, which changes the step size but not the direction logic.
- for iteration \(= 1, 2, \dots\) (at most 200):
- sample 4 answers per prompt, \(y \sim \pi_\theta\); freeze \(\pi_{\theta_\text{old}} \leftarrow \pi_\theta\)
- store \(\log \pi_{\theta_\text{old}}\), \(\log \pi_\text{ref}\), \(V_\text{old}\) for both tokens
- \(R \leftarrow r_\text{acc} + r_\text{fmt}\) // a program, not a model
- \(r_t \leftarrow -\beta(\log \pi_{\theta_\text{old}} - \log \pi_\text{ref})\), plus \(R\) at the last token // (66.9)
- \(\delta_t\), \(\hat A_t\) by GAE, \(\gamma = 1\), \(\lambda = 0.95\); \(\hat R_t \leftarrow \hat A_t + V_\text{old}\) // (66.10)
- for epoch \(= 1 \dots K\): shuffle, cut into mini-batches of \(M\)
- \(\rho \leftarrow \exp(\log \pi_\theta - \log \pi_{\theta_\text{old}})\) // (66.1)
- \(L^\text{CLIP} \leftarrow\) mean of \(\min(\rho \hat A, \operatorname{clip}(\rho, 1 \pm \epsilon)\hat A)\) // (66.3)
- \(L^\text{VF} \leftarrow\) mean of the clipped value loss // (66.5)
- \(\theta \leftarrow \theta + \alpha\, g\) (norm-clipped); \(V \leftarrow V - 2\, \nabla L^\text{VF}\) // (66.13)
- log accuracy, KL to \(\pi_\text{ref}\), clip fraction
How to read Figure 5
Press “Step one line” a few times. Line 5 shows that the first answer of seed 1, “2+3= 6</a>”, gets \(r_1 = 0\) and \(r_2 = 1\): in iteration 1 the policy equals the reference, so the KL part is 0, and the closed tag earns 1. The critic still says 0, so line 6 gives \(\hat A_2 = 1\) and \(\hat A_1 = 0.95\): the wrong number 6 is pushed up, because the critic has not yet learned that a closed tag is the normal case. Keep stepping to line 8: in the first mini-batch every ratio is exactly 1; in later mini-batches they spread, and the histogram shows them leaving the shaded band. Then switch to “Fast” and press Play. The format rate (dashed) rises first, then the accuracy. Try “No clip”: the clip fraction drops to 0 and the ratio histogram grows a red bar above 2. Try “Strong KL”: the purple KL curve stops much lower and the accuracy stays well below 1, because the leash to the reference holds the policy back.
The answer is “6</a>” to “2+3=”. Under the reference, \(\pi(6) = 0.302\) and \(\pi(\text{</a>}) = 0.401\), so \(\log \pi_{\theta_\text{old}} = -1.198\) and \(-0.913\). The verifier gives \(R = 0 + 1 = 1\).
Rewards (66.9): \(r_1 = -0.05 \times (-1.198 - (-1.198)) = 0\) and \(r_2 = 0 + 1 = 1\). With the critic at 0: \(\delta_2 = 1 - 0 = 1\), \(\delta_1 = 0 + 0 - 0 = 0\), so \(\hat A_2 = 1\) and \(\hat A_1 = 0 + 0.95 \times 1 = 0.95\).
After iteration 1 (8 mini-batch steps), \(\pi_\theta(6 \mid \text{2+3=})\) has grown from 0.302 to 0.395 and \(\pi_\theta(\text{</a>} \mid \text{2+3=})\) from 0.401 to 0.678. 32% of the token updates in this iteration were clipped. The exact accuracy goes from 0.073 to 0.089, then 0.736 after 10 iterations and 0.986 after 30, with a KL to the reference of 2.517 nats.
The PyTorch code below runs this loop on a toy problem in a few seconds on a CPU. The “language model” writes 6 tokens from a vocabulary of 8, the frozen “reward model” scores the fraction of tokens equal to 3, and all four roles exist: policy, frozen reference, reward model and critic. It is deterministic (fixed seed, CPU).
import copy, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) V, T, B = 8, 6, 64 # vocab size, response length, prompts per batch GOOD = 3 # the "reward model" likes token 3 class TinyLM(nn.Module): # next-token logits from (previous token, position) def __init__(s, out): super().__init__() s.tok, s.pos = nn.Embedding(V + 1, 32), nn.Embedding(T, 32) s.mlp = nn.Sequential(nn.Linear(32, 64), nn.Tanh(), nn.Linear(64, out)) def forward(s, prev): # prev: (B, T) previous tokens, V = "start" return s.mlp(s.tok(prev) + s.pos.weight[None, :prev.size(1)]) def reward_model(y): # frozen scorer: one number per response return (y == GOOD).float().mean(1) policy = TinyLM(V) ref = copy.deepcopy(policy).eval() # frozen reference = start policy for p in ref.parameters(): p.requires_grad_(False) critic = TinyLM(1) # value head: one number per token opt = torch.optim.Adam(list(policy.parameters()) + list(critic.parameters()), lr=1e-3) beta, eps, gamma, lam, c1, c2 = 0.05, 0.2, 1.0, 0.95, 0.5, 0.01 def prev_of(y): # shift right: the token before each position return torch.cat([torch.full((y.size(0), 1), V), y[:, :-1]], 1) def logp_of(model, y): # log pi(y_t | y_<t) for every position, plus full log-probs lp = F.log_softmax(model(prev_of(y)), -1) return lp.gather(-1, y[..., None]).squeeze(-1), lp for it in range(60): # 1) rollout: sample responses token by token from the current policy with torch.no_grad(): y = torch.zeros(B, 0, dtype=torch.long) for t in range(T): prev = prev_of(torch.cat([y, torch.zeros(B, 1, dtype=torch.long)], 1)) nxt = torch.distributions.Categorical(logits=policy(prev)[:, t]).sample() y = torch.cat([y, nxt[:, None]], 1) old_lp, _ = logp_of(policy, y) ref_lp, _ = logp_of(ref, y) values = critic(prev_of(y)).squeeze(-1) score = reward_model(y) # 2) per-token reward: KL penalty at every token, score at the last one kl_t = old_lp - ref_lp rew = -beta * kl_t rew[:, -1] += score # 3) GAE, backwards over the response adv, last = torch.zeros(B, T), torch.zeros(B) for t in reversed(range(T)): nv = values[:, t + 1] if t + 1 < T else torch.zeros(B) delta = rew[:, t] + gamma * nv - values[:, t] last = delta + gamma * lam * last adv[:, t] = last ret = adv + values # targets for the critic adv = (adv - adv.mean()) / (adv.std() + 1e-8) # 4) PPO epochs over minibatches of the same rollout clipped = [] for epoch in range(4): for idx in torch.randperm(B).split(16): new_lp, lp_all = logp_of(policy, y[idx]) ratio = torch.exp(new_lp - old_lp[idx]) a = adv[idx] pg = -torch.min(ratio * a, ratio.clamp(1 - eps, 1 + eps) * a).mean() v = critic(prev_of(y[idx])).squeeze(-1) vf = ((v - ret[idx]) ** 2).mean() ent = -(lp_all.exp() * lp_all).sum(-1).mean() loss = pg + c1 * vf - c2 * ent opt.zero_grad(); loss.backward() nn.utils.clip_grad_norm_(list(policy.parameters()) + list(critic.parameters()), 1.0) opt.step() clipped.append(((ratio - 1).abs() > eps).float().mean().item()) if it % 10 == 0 or it == 59: print(f"iter {it:2d} score {score.mean():.3f} KL to ref {kl_t.sum(1).mean():.3f}" f" clip frac {sum(clipped)/len(clipped):.3f}") # Output (CPU, torch 2.8): # iter 0 score 0.164 KL to ref 0.000 clip frac 0.136 # iter 10 score 0.828 KL to ref 5.978 clip frac 0.005 # iter 20 score 0.768 KL to ref 5.130 clip frac 0.082 # iter 30 score 0.849 KL to ref 6.492 clip frac 0.047 # iter 40 score 0.776 KL to ref 5.190 clip frac 0.031 # iter 50 score 0.844 KL to ref 6.329 clip frac 0.031 # iter 59 score 0.784 KL to ref 5.394 clip frac 0.029
The score rises from 0.164 to about 0.8 in 10 iterations and then stays there, held by the KL leash. Is 0.8 the right place to stop? For this reward, the best policy of (66.8) has a closed form per token: \(\pi^*(\text{GOOD}) \propto \pi_\text{ref}(\text{GOOD})\, e^{(1/6)/\beta}\), because each good token adds \(1/6\) to the score. If the reference were uniform over the 8 tokens, that gives \(e^{3.33} = 28.0\) and \(\pi^* = 28.0/(28.0 + 7) = 0.80\) for \(\beta = 0.05\). With \(\beta = 0.2\) the same code (we ran it) levels off at a score of about 0.24 to 0.34 and a KL below 0.61, close to the closed form \(\pi^* = 2.30/(2.30 + 7) = 0.25\). So \(\beta\) directly sets how far the policy goes from the reference.
Practical recipe and what to watch
| Setting | PPO paper (MuJoCo) | PPO paper (Atari) | InstructGPT (RLHF) |
|---|---|---|---|
| Clip \(\epsilon\) | searched; 0.2 best | \(0.1 \times \alpha\) (\(\alpha\) annealed \(1 \to 0\)) | 0.2 |
| Epochs per batch \(K\) | 10 | 3 | 1 (batch 512 split into 8 mini-batches of 64) |
| Discount \(\gamma\), GAE \(\lambda\) | 0.99, 0.95 | 0.99, 0.95 | \(\gamma = 1\) (no discount) |
| Learning rate | \(3 \times 10^{-4}\) | \(2.5 \times 10^{-4} \times \alpha\) | constant after 10-iteration warmup; critic \(9 \times 10^{-6}\) (1.3B, 6B policies) |
| Value / entropy weights | separate networks, no entropy bonus | \(c_1 = 1\), \(c_2 = 0.01\) | separate 6B critic, initialized from the reward model |
| KL to reference | none | none | \(\beta = 0.02\), per token; plus pretraining loss with \(\gamma_\text{ptx} = 27.8\) (PPO-ptx) |
| Scale | 1M steps per task | 8 actors \(\times\) 128 steps | 256k episodes, about 31k unique prompts |
All values in this table are from the two papers. [Certain] InstructGPT also kept an exponential moving average of the weights (decay 0.992) and sampled at temperature 1. [Certain] Its “PPO-ptx” variant mixes ordinary next-token loss on pretraining data into the PPO gradient to reduce the drop in public NLP benchmarks (the “alignment tax”). [Certain]
Signals to monitor
- Reward model score on the rollouts: should rise, then level off. A score that keeps rising fast while human or judge ratings stop improving suggests reward hacking (session 67).
- KL to the reference: should grow slowly. A sudden jump means the policy found a strange region (repetition, odd formatting).
- Clip fraction and approximate KL to the old policy: if the clip fraction is very high (say above 0.3 [Likely]), the steps are too large; lower the learning rate or the number of epochs.
- Entropy of the policy: a fast collapse means the model is becoming deterministic and stops exploring.
- Value loss and explained variance of the critic: a critic that cannot predict returns gives noisy advantages.
- Response length: reward models often prefer longer answers, so a steady growth in length is a classic hack. [Likely]
Many small implementation details matter. Two careful studies collected them: one for PPO in general (37 details, such as advantage normalization per mini-batch, global gradient-norm clipping at 0.5 and Adam epsilon \(10^{-5}\)) [Certain], and one for PPO-RLHF on summarization (more than 20 details). [Certain] Examples from RLHF practice: whiten the advantages, normalize or whiten the reward model scores, give a fixed penalty to responses that never produce the end token, and compute the value only on response tokens. [Likely]
Two runs side by side: what the clip and the KL weight do
Figure 6 runs the simulation of Figure 5 twice with the same seed, so both runs see the same random numbers, and changes only one setting. Both runs use \(\alpha = 20\), \(K = 4\) epochs and mini-batches of 8.
The clip. Without the clip (\(\epsilon = 100\), so nothing is ever clipped), a lucky batch can move a prompt all the way in one iteration. For seed 3, the prompt “4+3=” jumps to the answer “3</a>” in the very first iteration (from probability 0.05 to 0.90): wrong, but with a closed tag, so it earns reward 1. Once the policy says “3” with probability 0.9 and more, it almost never samples “7” again, every answer gets the same reward, the critic predicts it, the advantages go to 0, and the prompt is stuck. After 150 iterations run B has accuracy 0.750 (three prompts of four) and run A, with \(\epsilon = 0.2\), has 0.998. In 4 of the first 6 seeds the unclipped run ends stuck at 0.75. This is the RL meaning of “instability” in a run with exact rewards: not a crash, but a policy that commits too early and stops exploring.
The KL weight. With \(\beta = 0\) nothing pulls the policy back, and its KL to the reference climbs to 2.60 nats: it says the right answer with probability near 1, far from the reference’s 0.18. With \(\beta = 0.5\) the run settles near a KL of 1.21 and an accuracy of 0.559 for seed 3: the KL penalty in the reward (66.9) has become as large as the gain from being right. This is formula (66.8) at work: \(\beta\) chooses how far from the reference the best policy is.
How to read Figure 6
Press “Run to the end” with the clip comparison and seed 3: run B’s accuracy curve stops at 0.75, and its policy panel shows the prompt “4+3=” with a tall grey bar on 3. Its clip fraction is almost 0, because with \(\epsilon = 100\) nothing is clipped. Run A’s clip fraction is high in the first iterations (0.30, 0.63, 0.20): that is the clip doing its job, holding back the big early steps. Change the seed: some seeds are lucky and run B ends at 1.0 too, but run A ends above 0.99 for every seed from 1 to 6. Switch to the KL comparison: run B (\(\beta = 0.5\)) keeps the bars close to the purple ticks.
Where this is used
| Setting | Reward | How PPO is used |
|---|---|---|
| Classic RLHF chat assistants (InstructGPT, Llama 2-Chat) | A learned reward model of human preferences | The full four-model loop with a KL leash to the SFT model; see session 67 |
| Summarization from human feedback (Stiennon et al., 2020) | Reward model of summary preferences | The same loop on TL;DR summaries; the setting where PPO-RLHF for LLMs was worked out |
| Reasoning and math with verifiable rewards | A checker: correct final answer or passing tests | PPO with a critic, or GRPO, which keeps the clipped loss but drops the critic (session 70) |
| Robotics and games (the original PPO paper) | Environment reward | Many short episodes, discount 0.99, several epochs per batch |
| VLM alignment (less hallucination in image answers) | Reward model or judge that checks answers against the image | Same loop; the image tokens are part of the prompt and get no loss. [Likely] |
| Video-LLMs: video QA and key-moment detection | Verifiable rewards such as the overlap between a predicted and a true time span, or exact-match answers | PPO or GRPO-style updates on top of a video SFT model (session 205); long visual prompts make rollouts and memory even more expensive. [Likely] |
| Video captioning and retrieval-friendly descriptions | A judge or a retrieval score of the caption against the clip | Reward the caption that retrieves the right clip; the KL leash keeps captions fluent. [Speculation] |
Common mistakes
- Gradient through the old log-probabilities. \(\log \pi_{\theta_\text{old}}\) and the advantages must be constants (computed under
no_gradand stored). If they carry gradients, the ratio is always 1 and the loss is wrong. - Recomputing old log-probabilities with a different setup than generation: different temperature, different padding, or a different numeric path in the inference engine. Then \(r \ne 1\) before the first step. Check that the ratio is 1.000 at the start of each iteration.
- Mixing up the two KLs. The clip (or adaptive penalty) controls the step from \(\pi_{\theta_\text{old}}\). The \(\beta\) term controls the distance to the frozen \(\pi_\text{ref}\). Using the policy itself as the reference makes the leash vanish.
- Loss on prompt or padding tokens. Only response tokens are actions. Mask the prompt and padding in the policy loss, the value loss, the KL and the entropy.
- Wrong sign of the min for negative advantages. Writing
min(ratio, clip(ratio)) * advinstead ofmin(ratio * adv, clip(ratio) * adv)clips in the wrong direction when the advantage is negative. - Too many epochs or too large a learning rate. A high clip fraction and a fast-growing KL are the signs. LLM RLHF often uses 1 to 4 epochs. [Likely]
- An untrained critic. A random critic gives random advantages. Initialize it from the reward model (as InstructGPT did) or from the SFT model, and consider a few critic-only warmup steps. [Likely]
- Ignoring the end token. Responses cut by the length limit get a reward model score for text that never finished; penalize or drop them.
Interview questions
Why does PPO need the importance ratio at all?basic
Write the PPO clipped objective and explain why it takes a minimum.core
When is the gradient of a PPO term zero? What happens if \(\hat A \lt 0\) and \(r = 1.5\)?core
Clipping versus an adaptive KL penalty: what are they and which works better?core
What is the difference between the KL to the old policy and the KL to the reference model?core
Why put the reference KL into the per-token reward instead of into the loss?advanced
Which models are in memory during PPO-RLHF, and how much memory for a 7B setup?core
Why does InstructGPT initialize the critic from the reward model, and what goes wrong with a bad critic?advanced
What do you monitor during a PPO-RLHF run, and what do the signals tell you?core
Is PPO on-policy or off-policy? Why does RLHF often use only one epoch?advanced
Why is \(\gamma = 1\) normal in RLHF, and how does the end-of-response score reach the first token?core
Exercises
- With \(\epsilon = 0.2\) and \(\hat A = -3\), compute the PPO term and its gradient with respect to \(r\) for \(r = 0.75\) and for \(r = 1.25\).
- In the adaptive KL penalty with \(d_\text{targ} = 0.01\), what happens to \(\beta\) when the measured KL is 0.02, 0.005 and 0.012?
- A response has two tokens with log-ratios 0.2 and 0.4 to the reference, \(\beta = 0.1\), reward model score 0.5, critic values \(V = [0.3, 0.4]\), \(\gamma = \lambda = 1\). Compute the per-token rewards and the two advantages.
- Compute the model-state memory by (66.12) for a 13B policy with a 13B reward model and a 13B critic. How many 80 GB GPUs is that at least?
- InstructGPT trained for 256k episodes with batches of 512 and mini-batches of 64, one epoch. About how many PPO iterations and how many optimizer steps is that?
- Explain in one sentence why \(\hat A = 0\) for every sample would give zero policy gradient no matter what the ratios are.
Answerscheck
Key takeaways
- The importance ratio \(r = \pi_\theta / \pi_{\theta_\text{old}}\) (66.1) lets PPO reuse one batch of samples for several mini-batch steps.
- The clipped objective (66.3) is a pessimistic bound: it stops rewarding changes beyond \(1 \pm \epsilon\) in the improving direction, but always corrects changes in the harmful direction. \(\epsilon = 0.2\) is the standard value.
- The full loss adds a value loss and an entropy bonus (66.5)–(66.7); in LLM RLHF the critic is a separate model and the entropy bonus is often zero.
- For LLMs, each token is an action. The reward is a per-token KL penalty to a frozen reference plus the reward model score at the last token (66.9); GAE with \(\gamma = 1\) spreads it over the tokens (66.10).
- Two KLs: the clip limits the step from the old policy; \(\beta\) limits the drift from the reference and sets the optimum \(\pi^* \propto \pi_\text{ref} e^{r/\beta}\).
- In the live toy run (Figures 5 and 6), removing the clip lets one lucky batch lock a prompt onto a wrong answer that earns the format reward, and the run stays stuck; \(\beta\) sets how far the policy ends from the reference (KL 2.60 nats with \(\beta = 0\), 1.21 with \(\beta = 0.5\), seed 3).
- PPO-RLHF keeps four models in memory; two are trained, so a 7B setup needs about 252 GB of model state (66.12). This cost is what DPO and GRPO try to remove.
Key formulas
References for this session
- Schulman et al. (2017). Proximal Policy Optimization Algorithms
- Ouyang et al. (2022). Training language models to follow instructions with human feedback
- Lambert (2025). Reinforcement Learning from Human Feedback (book)
- Schulman et al. (2017). Proximal Policy Optimization Algorithms
- Ouyang et al. (2022). Training language models to follow instructions with human feedback
- Lambert (2025). Reinforcement Learning from Human Feedback (book)
- Schulman et al. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation
- OpenAI (Achiam) (2018). Spinning Up in Deep RL
- Stiennon et al. (2020). Learning to summarize from human feedback
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Schulman (2020). Approximating KL Divergence (blog)
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Hugging Face (code). TRL library
- Schulman et al. (2015). Trust Region Policy Optimization
- Ziegler et al. (2019). Fine-Tuning Language Models from Human Preferences
- Huang et al. (2024). The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
- Huang et al. (2022). The 37 Implementation Details of Proximal Policy Optimization (ICLR blog track)