RLHF: preference data and reward models
RLHF (reinforcement learning from human feedback) teaches a model what people prefer. People compare two answers and pick the better one; a reward model learns to give a higher score to the preferred answer (the Bradley–Terry model); then PPO pushes the policy toward high scores while a KL penalty keeps it close to the starting model. This session builds each step with real numbers from InstructGPT and Llama 2, and shows why pushing a learned reward too hard makes the real quality go down (Goodhart’s law).
What you will learn
- Why RLHF collects comparisons (which answer is better?) instead of scores or more demonstrations, and how real preference data is collected, with agreement rates
- The Bradley–Terry model, the reward-model loss, its gradient, why it only sees reward differences, and the ranking and margin versions used by InstructGPT and Llama 2
- How a reward model is built from an LLM: a scalar head on the last token, one epoch of training, normalization, and what its accuracy really tells you
- The RLHF objective: reward minus a KL penalty to a frozen reference, the per-token reward given to PPO, and the closed-form best policy that leads to DPO
- Reward hacking and over-optimization: Goodhart’s law, the Gao et al. scaling forms for best-of-n and RL, and the tools people use against it
- A full RLHF pipeline in pseudo-code and a runnable PyTorch reward model
Why this matters
After supervised fine-tuning (session 54), a model can follow instructions. But it only copies the style of its demonstrations. It does not know which of two reasonable answers people like more: the one that is shorter or longer, more careful or more direct, the one that admits doubt or the one that guesses. RLHF (reinforcement learning from human feedback) adds that knowledge.
The idea has three steps. First, people look at two answers to the same prompt and say which one is better. Second, we train a reward model: a network that reads a prompt and an answer and outputs one number, higher for answers people prefer. Third, we use reinforcement learning (PPO, from session 66) to change the model so that its answers get high reward, with a penalty that keeps it close to where it started. This is the recipe behind InstructGPT and the first ChatGPT, and behind Llama 2-Chat. [Certain]
The result can be large. In InstructGPT, people preferred the answers of a 1.3B RLHF model to the answers of the 175B GPT-3, a model with more than 100 times more parameters. [Certain] But the method also has a famous weakness: the reward model is only an estimate of what people want. Push the policy too hard against it and the policy finds the estimate’s mistakes. The reward score keeps going up while the real quality goes down. This is reward hacking, a case of Goodhart’s law: “when a measure becomes a target, it ceases to be a good measure.”
Collecting preference data
Why comparisons?
We want the model to produce answers that people like. There are three ways to ask people for that signal:
- Write the ideal answer (a demonstration). This is what SFT uses. It is slow and expensive, and the model can never become better than the writers.
- Give a score, for example 1 to 7. Different people use the scale differently: one person’s 5 is another person’s 6, and the same person drifts over a day. Absolute scores are noisy.
- Compare two answers: “Is A or B better?” This is fast and much more consistent, because people judge relative quality much better than absolute quality. It also works when nobody could write the best answer, but everyone can recognize the better of two.
RLHF uses comparisons. Christiano et al. (2017) showed the idea first on robot and Atari tasks: a simulated robot learned a backflip from 900 human comparisons of short video clips, collected in less than an hour. [Certain] Stiennon et al. (2020) then used it for text (summaries of Reddit posts), and InstructGPT used it for general chat.
What one data point looks like
One example is a triple \((x, y_w, y_l)\): a prompt \(x\), the preferred answer \(y_w\) (“winner”, also called chosen) and the other answer \(y_l\) (“loser”, also called rejected). Both answers are usually sampled from the current model, sometimes from different model versions or temperatures, so that they differ in interesting ways. Real projects add detail:
- Rankings. InstructGPT showed labelers between \(K = 4\) and \(K = 9\) answers to rank. A ranking of \(K\) answers gives \(\binom{K}{2}\) pairs: 6 pairs for \(K = 4\), 36 for \(K = 9\). [Certain]
- Strength of preference. Llama 2 asked how much better the chosen answer is: significantly better, better, slightly better, or negligibly better / unsure. [Certain]
- Separate goals. Llama 2 collected helpfulness and safety comparisons separately, with different guidelines, and trained two reward models. [Certain]
- Fresh data every round. Llama 2 collected comparisons in weekly batches from the latest model, because a reward model trained on old answers becomes inaccurate on the new model’s answers. Its own data had 1,418,091 comparisons. [Certain]
People disagree, and that sets a ceiling
Preferences are noisy. In InstructGPT, training labelers agreed with each other 72.6 ± 1.5% of the time. In the summarization work, researchers agreed with each other only 73 ± 4% of the time. [Certain] So even a perfect model of “the average person” cannot predict every label. If people agree 73% of the time, a reward model with 73% accuracy is not bad; it may be close to the best possible. We return to this ceiling in Figure 3.
The Bradley–Terry model
We need a way to turn “A beat B” into a number for each answer. The classic answer is the Bradley–Terry model (1952). Give every answer a hidden score, its reward \(r\). Then the chance that a person prefers \(y_w\) over \(y_l\) depends only on the difference of the two rewards:
- Read the symbols
- \(y_w \succ y_l\)
- “\(y_w\) is preferred to \(y_l\)”
- \(r(x, y)\)
- the reward of answer \(y\) to prompt \(x\): one real number
- \(\sigma\)
- the sigmoid function, which turns any number into a probability between 0 and 1
An equal reward gives \(\sigma(0) = 0.5\): a coin flip. A reward gap of 1 gives 0.731, a gap of 2 gives 0.881. Another way to write the same model shows where it comes from: each answer has a “strength” \(e^{r}\), and the chance of winning is your strength divided by the total:
Divide the top and bottom by \(e^{r(x,y_w)}\) and you get formula (67.1). This is a softmax over two answers, so it is the same math as next-token prediction with a vocabulary of size 2 (session 3).
One property is very important. Add the same constant \(c\) to every reward and nothing changes:
So comparisons only define rewards up to a shift. The reward model learns differences, never absolute levels. A reward of 3.2 means nothing alone. This is why InstructGPT shifted its reward model after training so that the human demonstrations get a mean score of 0, and why Llama 2 whitened its scores (subtract the mean, divide by the standard deviation) before PPO. [Certain] The scale is also only fixed by the sigmoid: a model that is very sure gives large gaps.
The reward-model loss
We fit \(r_\phi\), a network with weights \(\phi\), by maximum likelihood: make the observed preferences as probable as possible. The negative log-likelihood of formula (67.1) is the reward-model loss used by Stiennon et al., InstructGPT and Llama 2: [Certain]
- Read the symbols
- \(\mathcal{D}\)
- the dataset of human comparisons
- \(r_\phi\)
- the reward model with weights \(\phi\)
- \(\log \sigma(\cdot)\)
- log of the model’s probability for the label the person gave; the minus sign turns it into a loss
It is binary cross-entropy where the “logit” is the reward difference \(\Delta = r_\phi(x, y_w) - r_\phi(x, y_l)\). Its gradient is short:
Read it as: push the winner’s reward up and the loser’s reward down, with a weight \(1 - \sigma(\Delta)\), the probability the model gave to the wrong answer. Pairs the model already gets right with confidence contribute almost nothing. Pairs it gets wrong contribute the most.
The reward model gives \(r(x, y_w) = 1.2\) and \(r(x, y_l) = 0.4\), so \(\Delta = 0.8\). It predicts \(P(y_w \succ y_l) = \sigma(0.8) = 0.690\). The loss is \(-\log 0.690 = 0.371\). The gradient on \(\Delta\) is \(-(1 - 0.690) = -0.310\): gradient descent increases the gap.
If the person had preferred the other answer, the loss would be \(-\log \sigma(-0.8) = 1.171\), more than three times larger, and the gradient would be \(-0.690\). Before training, all rewards are about equal, \(\Delta \approx 0\), and the loss is \(\ln 2 = 0.693\) per pair. That is the number to expect at step 0.
Now add 5 to both rewards (6.2 and 5.4). \(\Delta\) is still 0.8 and the loss is still 0.371: formula (67.3).
Rankings and margins
With a ranking of \(K\) answers, InstructGPT uses every pair and averages them, and it puts all \(\binom{K}{2}\) pairs of one prompt in the same batch:
Here \(y_{(1)}\) is the best-ranked answer and \(y_{(K)}\) the worst, so in each pair \(y_{(i)}\) is the winner. Why one batch? The pairs from one prompt are strongly correlated (each answer appears in \(K - 1\) pairs). When InstructGPT shuffled them into the dataset as separate examples, the reward model overfit after one pass. Keeping them together also needs only \(K\) forward passes instead of \(K(K-1)\). [Certain] With 64 prompts per batch and \(K = 9\), one batch holds up to \(64 \times 36 = 2{,}304\) comparisons. [Certain]
Llama 2 uses the strength of preference with a margin \(m\): [Certain]
The loss is only small when the gap is larger than \(m\). Llama 2 tried a small margin (1, 2/3, 1/3, 0 for significantly better, better, slightly better, negligibly better) and a large one (3, 2, 1, 0). The margin raised accuracy mostly on the clearly separated pairs. [Certain] With \(\Delta = 0.8\) and \(m = 1\), the loss rises from 0.371 to 0.798 and the gradient from \(-0.310\) to \(-0.550\): the model is asked for a bigger gap. Try it in Figure 2.
How to read Figure 2
Move \(r_w\) and \(r_l\): only their difference moves the dot. Press “Shift both rewards” (it adds the same number to both) and nothing in the readout changes except the two rewards; that is formula (67.3). Press “Swap the label” to see the loss when the person disagreed with the model: the dot jumps to \(-\Delta\), where the loss and the gradient are large. Raise the margin: the loss curve slides to the right, so a pair needs a larger gap before its gradient fades. Set \(m = 1\) to reproduce Llama 2’s “significantly better” margin.
Building a reward model from an LLM
The reward model must read language as well as the policy does, so it is itself a language model. The standard recipe: [Certain]
- Start from a pretrained or SFT model. InstructGPT started from a 6B model; Llama 2 started its reward models from the chat model checkpoints, so that the reward model “knows what the policy knows”.
- Remove the unembedding layer (the matrix that maps a hidden state to vocabulary logits). Put a linear head \(\mathbf{w}^\top \mathbf{h} + b\) in its place that maps a hidden state to one number.
- Feed the prompt and the answer together, in the same chat template as the policy. Read the score at the last token of the answer (usually the end-of-turn token). With a causal model, that is the only position that has seen the whole answer.
- Train with formula (67.4), (67.6) or (67.7) for about one epoch. Both InstructGPT and Llama 2 report that more epochs overfit quickly. [Certain]
- Shift (and maybe scale) the scores so that a reference set has mean 0, because of formula (67.3).
How big? InstructGPT used one 6B reward model for policies of all sizes, 1.3B to 175B. A 175B reward model had lower validation loss but its training was unstable, and it would have made PPO much more expensive (the reward model also initializes the value network). [Certain] In the summarization work, doubling the reward model size raised its accuracy by about 1.8 points and doubling the data by about 1.1 points. [Certain]
What accuracy means
The main metric is pairwise accuracy on held-out comparisons: how often \(r(x, y_w) \gt r(x, y_l)\). Numbers look low. Llama 2’s helpfulness reward model scored 63.2% on its own helpfulness test set, but 80.7% on pairs marked “significantly better” and 54.7% on pairs marked “negligibly better / unsure”. [Certain] When two answers are almost equally good, people themselves flip a coin, and so must the model.
We can make the ceiling precise. If labels come from formula (67.1) with the true rewards, the best possible accuracy on a pair with true gap \(\Delta^\ast\) is \(\sigma(|\Delta^\ast|)\), not 1. Figure 3 trains a small reward model live, so you can see the ceiling, the effect of label noise and the effect of a labeler bias.
How to read Figure 3
Left: three bars per feature. Grey is the true value of the feature, blue is what the labelers reward, orange is what the reward model learned. The reward model learns the labelers’ taste divided by the noise \(\tau\) (noisy labelers give smaller reward gaps), so we multiply its weights by \(\tau\) to put them on the same scale. Right: the training loss, which starts at \(\ln 2 = 0.693\). The readout gives the expected held-out accuracy of the reward model and of the labelers’ own taste (the ceiling). With few comparisons, the learned weights are noisy and held-out accuracy falls below the ceiling. Raise the noise \(\tau\): the ceiling drops toward 50%, and the model needs more data. Now raise the length bias: the reward model faithfully learns to reward length, its accuracy on labeler data stays high, and its agreement with true quality falls. The reward model learns the labelers, not the truth.
Watch the loss train a reward model, one pair at a time
Figure 3 trains in one go. Figure 4 slows training down to one comparison per step, so you can watch formulas (67.1), (67.4) and (67.5) do their work. It uses the six answers A to F of Figure 5 below. Each answer has a hidden true quality \(q\): \((1.0, 2.0, 0.3, -2.0, -0.5, -1.0)\). The simulation first builds a dataset. It picks two different answers at random, and a simulated labeler prefers \(i\) over \(j\) with the Bradley–Terry probability \(\sigma(q_i - q_j)\). Then, with probability “label noise”, the label is flipped, like a careless labeler. The reward model is the simplest one possible: a table with one learnable score \(r_\phi(y)\) per answer, all 0 at the start. A real reward model replaces the table with an LLM and a linear head, but the loss and the gradient are the same.
- take the next pair \((y_w, y_l)\) from the shuffled dataset
- \(\Delta \leftarrow r_\phi(y_w) - r_\phi(y_l)\) // reward gap
- \(p \leftarrow \sigma(\Delta)\) // (67.1): the model’s chance for this label
- \(\ell \leftarrow -\log p\) // (67.4)
- \(g \leftarrow 1 - p\) // (67.5): the weight of this pair
- \(r_\phi(y_w) \mathrel{+}= \eta\, g\); \(\;r_\phi(y_l) \mathrel{-}= \eta\, g\) // gradient step
How to read Figure 4
Set the speed to 1 and press Next line a few times: you see one pair go through the six lines. At the start every reward is 0, so \(\Delta = 0\), \(p = 0.5\), the loss is \(\ln 2 = 0.693\) and the weight is 0.5. Then press Play at speed 4. With the defaults (120 pairs, noise 0.10, \(\eta = 0.1\), seed 1), 29 of the 120 labels disagree with the true order: 19 were flipped by the noise, the rest are Bradley–Terry coin flips that went the “wrong” way. So no reward model can get more than 75.8% of these training labels right by following the true quality; that is the orange dashed line. After 20 epochs the loss is 0.539, the training accuracy 76.7% (a little above the line, because it also fits some of the flipped labels), and 14 of the 15 answer pairs are in the true order. Compare the bars: the learned rewards keep the true order but the gaps are smaller (from −1.22 to 1.66 instead of −1.97 to 2.03). Noisy labels look like a less sure labeler, and Bradley–Terry turns “less sure” into “smaller gaps”. Set the noise to 0: the gaps grow to −2.56 … 2.59, a little wider than the truth, because 120 pairs is not much data. Set it to 0.30 and the reward model can no longer rank the answers (10 of 15 pairs right). Note that the bars are centered: comparisons fix rewards only up to a shift, formula (67.3). (Speeds 1 and 2 run one line of the algorithm at a time, 0.65 s or 0.2 s per line; speeds 3 to 5 run whole iterations, more of them per frame. “Next line” and the next-step button work at any speed.)
Using the reward: RL with a KL leash
Now we have \(r_\phi(x, y)\). We want a policy \(\pi_\theta\) (the language model being trained) whose answers get high reward. If we only maximized the reward, the policy would drift to strange text the reward model has never seen, where its scores mean nothing. So RLHF adds a penalty for moving away from a frozen reference model \(\pi_{\text{ref}}\), usually the SFT model: [Certain]
- Read the symbols
- \(\pi_\theta(y \mid x)\)
- the probability the policy gives to the whole answer \(y\): the product of its token probabilities
- \(\pi_{\text{ref}}\)
- the frozen copy of the starting (SFT) model
- \(\KL(\pi \,\|\, \pi_{\text{ref}})\)
- the average, over answers \(y \sim \pi\), of \(\log \pi(y \mid x)\) minus \(\log \pi_{\text{ref}}(y \mid x)\): how far the policy has moved, in nats; it is 0 only when the two are equal
- \(\beta\)
- the strength of the leash: 0.05 in the summarization work, 0.02 in InstructGPT, 0.01 for Llama 2 7B and 13B, 0.005 for 34B and 70B [Certain]
Stiennon et al. give two reasons for the KL term: it acts like an entropy bonus that stops the policy from collapsing to one answer, and it keeps the policy’s answers close to the kind of answers the reward model was trained on. [Certain] InstructGPT also mixed in pretraining gradients (“PPO-ptx”, with coefficient \(\gamma = 27.8\)) to stop the model from getting worse on public NLP benchmarks. [Certain]
How PPO sees this reward
PPO works on tokens. Since \(\log \pi(y \mid x)\) is a sum over tokens, the KL of one sampled answer is a sum of per-token log-ratios, and we can hand out the penalty token by token. The reward model score arrives only at the last token \(T\): [Certain]
The sum of these \(r_t\) over the answer is exactly “reward minus \(\beta\) times the sampled KL” from formula (67.8). PPO then uses \(r_t\) with a value network and GAE (InstructGPT used no discount, \(\gamma = 1\)) as in session 65 and session 66.
An answer has 3 tokens. The policy’s log-ratios to the reference are \(0.2, 0.4, 0.0\) nats, \(\beta = 0.05\), and the reward model gives 0.8. Then \(r_1 = -0.05 \times 0.2 = -0.010\), \(r_2 = -0.05 \times 0.4 = -0.020\), \(r_3 = 0 + 0.8 = 0.800\). Total: 0.770 = 0.8 minus \(0.05 \times 0.6\). The PyTorch code below prints exactly these numbers.
The best policy has a closed form
Ignore neural networks for a moment and ask: which distribution \(\pi\) maximizes formula (67.8) for one prompt? The answer is known exactly: [Certain]
Start from the reference and multiply each answer’s probability by \(e^{r/\beta}\), then renormalize. Large \(\beta\): \(\pi^\ast \approx \pi_{\text{ref}}\). Small \(\beta\): all the mass goes to the answer with the highest reward model score, right or wrong. PPO is a way to approach \(\pi^\ast\) with a network and samples. In session 68, this formula is turned around to train the policy directly from comparisons, without a separate reward model.
Figure 5 applies formula (67.10) to six possible answers. The reward model is decent but has one flaw: it over-rates a confident, flattering answer that is wrong.
How to read Figure 5
Each row of the table on the left shows an answer’s reward-model score and its true quality. Start at \(\beta = 1\): \(\pi^\ast\) moves mass toward the detailed correct answer, and true quality rises from 0.505 (reference) to 0.869. Lower \(\beta\): the reward-model score keeps rising, but the policy piles onto the flattering wrong answer, the one the reward model over-rates, and true quality falls to −1.66 at \(\beta = 0.1\). The KL in the readout grows toward its maximum \(\ln 20 = 3.00\) nats, the cost of putting all mass on an answer that had 5% under the reference. That is reward hacking in its simplest form. Raise \(\beta\) above 3 and the policy barely moves: safe, but little gain.
Reaching \(\pi^*\) by sampling: the policy gradient in the loop
Figure 5 jumps straight to \(\pi^\ast\). A real RLHF run cannot do that: it cannot list all answers, it can only sample answers from the current policy, score them, and take a small step. The step follows the gradient of objective (67.8). For one prompt it is
- Read the symbols
- \(r_\phi - \beta \log(\pi_\theta/\pi_{\text{ref}})\)
- the reward of one sampled answer after the KL penalty: the sum of the per-token rewards (67.9)
- \(b\)
- a baseline: any number that does not depend on \(y\). It does not change the expected gradient, because \(\E_{y \sim \pi_\theta}[\nabla_\theta \log \pi_\theta(y)] = 0\), but it lowers the noise. Here we use the mean of the other samples in the batch (the “leave-one-out” baseline)
- \(\nabla_\theta \log \pi_\theta(y)\)
- for a softmax over answers with logits \(\theta\), this is \(e_y - \pi_\theta\): the one-hot vector of \(y\) minus the current probabilities
Why is there no extra term from the \(\log \pi_\theta\) inside the bracket? Its derivative gives \(-\beta\,\E[\nabla_\theta \log \pi_\theta(y)]\), which is 0 for the same reason. PPO adds a value network, GAE and clipping on top (session 66), but its core is this formula. Figure 6 runs it on the six answers of Figure 5, with the policy as a softmax over six logits.
- sample \(y_1, \dots, y_B \sim \pi_\theta\)
- score each answer with the reward model: \(r_\phi(y_i)\)
- \(R_i \leftarrow r_\phi(y_i) - \beta \log \dfrac{\pi_\theta(y_i)}{\pi_{\text{ref}}(y_i)}\) // (67.9), summed
- \(A_i \leftarrow R_i - \frac{1}{B-1}\sum_{j \ne i} R_j\) // baseline b
- \(g \leftarrow \frac{1}{B}\sum_i A_i\,(e_{y_i} - \pi_\theta)\) // (67.11)
- \(\theta \leftarrow \theta + \eta\, g\) // gradient ascent
How to read Figure 6
Use speed 1 and Next line first. In the table, an answer whose penalized reward \(R_i\) beats the other samples gets a positive advantage \(A_i\) (blue), and line 6 moves its logit up. Then Play at speed 4 with the defaults (flawed reward model, \(\beta = 0.1\), seed 1). For the first few hundred iterations everything looks good: the policy moves to B (detailed, correct) and the true quality (blue curve) climbs to 1.94 at iteration 190. D starts at only 5% and is rarely sampled, so the policy has not found the reward model’s mistake yet. Then it does: D is sampled, gets a large reward, and takes over. With seed 1 the true quality turns negative at iteration 526 and ends at −1.71, close to its value at \(\pi^\ast\) (−1.66), while the reward-model score keeps rising to 2.37. This is reward hacking happening during training, not only at the optimum. Other seeds find D at other times, but all 20 seeds find it: the true quality turns negative between iteration 145 and 1,267. Raise \(\beta\) to 0.3 and reset: \(\pi^\ast\) itself keeps half of its mass on B, and the true quality peaks at 1.62 (iteration 51) and then settles near 0.28, the \(\pi^\ast\) value. Choose “True quality” as the reward: now the optimum is good, and the KL curve rises to the dashed \(\pi^\ast\) line and stays there. Larger batches \(B\) give smoother curves.
Reward hacking and over-optimization
The reward model is a proxy: a learned, imperfect stand-in for what people want. The policy is an optimizer, and optimizers find the proxy’s mistakes. Common forms in LLMs: [Likely]
- Length. Labelers often prefer longer, more detailed answers, so reward models give a bonus for length; the policy then pads. Singhal et al. report that on several common setups much of the reward gain from RLHF can be reproduced by optimizing for length alone. [Likely] In the summarization work, length differences explained about a third of the gap between the human-feedback and supervised models at 6.7B. [Certain]
- Sycophancy. Agreeing with the user and praising the question scores well with people, so the model learns to flatter, even when the user is wrong.
- Style over substance. Confident tone, lists and headings, and hedging phrases can raise the score without making the answer more correct.
- Out-of-distribution text. Pushed far, the policy finds odd strings (repeated phrases, strange tokens) where the reward model gives high scores it never learned to give. Stiennon et al. show such over-optimized samples. [Certain]
Stiennon et al. measured the effect directly. They trained policies against a reward model with different KL coefficients and asked labelers to judge the samples. Under light optimization real quality went up. Under heavy optimization the real preference fell, and eventually the reward model became anti-correlated with human preference: higher score, worse summary. [Certain]
Scaling laws for over-optimization
Measuring this with humans is expensive. Gao et al. (2022) used a trick: a large “gold” reward model (the 6B InstructGPT reward model) plays the role of humans and labels 100,000 synthetic comparisons; smaller “proxy” reward models (3M to 3B parameters) are trained on these labels; then the policy is optimized against the proxy, and the gold model measures the real quality. [Certain] With \(d = \sqrt{\KL(\pi \,\|\, \pi_{\text{init}})}\), they found that the gold score follows simple curves: [Certain]
- Read the symbols
- \(R(d)\)
- gold (true) reward gained, with \(R(0) = 0\) at the starting policy
- \(d\)
- the square root of the KL from the starting policy: how far optimization has moved
- \(\alpha, \beta\)
- fitted constants (this \(\beta\) is not the KL coefficient); they change smoothly with the size of the proxy reward model
- bon
- best-of-\(n\): sample \(n\) answers and keep the one the proxy likes best
Both curves rise, reach a peak and then fall: the more you optimize, the more the gap between proxy and gold grows. Setting the derivative of (67.12) to zero gives the peak at \(d^\ast = \alpha_{\text{bon}} / (2 \beta_{\text{bon}})\). For (67.13) the peak is at \(d^\ast = e^{\alpha_{\text{RL}}/\beta_{\text{RL}} - 1}\). Larger proxy reward models had smaller \(\beta\): they can be optimized further before real quality falls. [Certain]
For best-of-\(n\), the KL has a closed form (from Stiennon et al., appendix): [Certain]
So best-of-16 moves the policy by \(\log 16 - 15/16 = 1.835\) nats, and best-of-148 by about 4.0 nats. KL grows only like \(\log n\): best-of-\(n\) is a cheap, gentle optimizer, and a strong baseline. Two more findings from Gao et al.: as a function of KL, RL is slower than best-of-\(n\) both to improve and to over-optimize, and in their setup the KL penalty did not move the gold-reward-versus-KL frontier; it acted mostly like early stopping. [Certain] Figure 7 plots both curves.
How to read Figure 7
Drag \(d\). The orange proxy score keeps going up: from inside the optimization, everything looks fine. The blue gold score peaks and then drops. For best-of-\(n\) the readout also gives the \(n\) that reaches this KL, from formula (67.14): the peak at \(d = 2\) (4 nats) needs about \(n = 147\). The gap between the two curves is the over-optimization you would only see with a fresh human evaluation. Switch to RL: the gold curve peaks later, at \(d = e\), and falls more slowly.
What people do about it
- The KL penalty (formula 67.8), and watching the KL during training. A KL that keeps climbing is a warning sign.
- Early stopping on a real evaluation: human or strong-judge win rates on held-out prompts, not the reward model score.
- Fresh preference data from the current policy, and retrained reward models, as Llama 2 did every round. [Certain]
- Bigger and better reward models: in Gao et al., larger proxies over-optimize later. [Certain]
- Reward model ensembles: optimize the mean or a conservative (worst-case) combination of several reward models. Coste et al. found this reduces over-optimization. [Likely]
- Separate reward models and caps: Llama 2 used a safety reward model that takes over when a prompt is tagged as risky or its safety score falls below 0.15. [Certain] Length penalties or length-controlled evaluation fight length hacking. [Likely]
The full pipeline in pseudo-code
- input: SFT model, prompts \(\{x\}\), labelers
- for each prompt \(x\): sample \(K\) answers from the current model(s), get a ranking \(y_{(1)} \succ \dots \succ y_{(K)}\) // drop ties
- \(r_\phi \leftarrow\) SFT model with the unembedding replaced by a linear head to 1 number
- for one epoch, batches of prompts:
- score all \(K\) answers of each prompt; read \(r_\phi\) at the last answer token
- loss \(\leftarrow\) mean over all \(\binom{K}{2}\) pairs of \(-\log \sigma(r_{(i)} - r_{(j)})\) // formula (67.6), pairs of one prompt stay together
- AdamW step
- report pairwise accuracy on held-out comparisons, split by preference strength
- \(b \leftarrow b - \text{mean}(r_\phi)\) on a reference set // formula (67.3): fix the free shift
- input: policy \(\pi_\theta\) (starts as SFT), frozen \(\pi_{\text{ref}}\), frozen \(r_\phi\), value net \(V\) (starts as \(r_\phi\)), \(\beta\)
- sample a batch of prompts; generate answers \(y \sim \pi_\theta(\cdot \mid x)\) at temperature 1
- \(s \leftarrow r_\phi(x, y)\); normalize \(s\) (shift, or whiten as in Llama 2)
- log-probs of each answer token under \(\pi_\theta\) and \(\pi_{\text{ref}}\)
- per-token rewards \(r_t \leftarrow -\beta(\log\pi_\theta - \log\pi_{\text{ref}})\), plus \(s\) at the last token // formula (67.9)
- advantages with GAE from \(r_t\) and \(V\)
- a few epochs of PPO clipped updates on \(\pi_\theta\), value loss on \(V\)
- log: mean \(s\), KL to the reference, answer length, and a sample of answers // rising \(s\) with rising length or KL = suspect hacking
- every few iterations: evaluate on held-out prompts with humans or a strong judge; stop when the win rate stops improving
Runnable PyTorch: a small causal-transformer reward model trained with formula (67.4) on simulated comparisons, and the per-token reward of formula (67.9). The simulated labelers like token 5 and dislike token 6, and follow formula (67.1). Output with torch 2.8 on CPU is shown in the comments.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) V, D, T, PAD = 16, 64, 12, 0 # vocab, width, max length, pad id GOOD, BAD = 5, 6 # the hidden "taste" of our simulated labelers class RewardModel(nn.Module): def __init__(self): super().__init__() self.emb, self.pos = nn.Embedding(V, D), nn.Embedding(T, D) layer = nn.TransformerEncoderLayer(D, 4, 4 * D, dropout=0.0, batch_first=True, norm_first=True) self.body = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False) self.head = nn.Linear(D, 1) # replaces the unembedding: hidden state to one number def forward(self, ids, mask): # ids, mask: (B, T); mask is True on real tokens h = self.emb(ids) + self.pos(torch.arange(ids.shape[1])) causal = nn.Transformer.generate_square_subsequent_mask(ids.shape[1]) h = self.body(h, mask=causal, src_key_padding_mask=~mask) last = mask.sum(1) - 1 # last real token (right padding) return self.head(h[torch.arange(len(ids)), last]).squeeze(-1) def sample(n): # random answers of length 4..12 L = torch.randint(4, T + 1, (n,)) ids = torch.randint(1, V, (n, T)); mask = torch.arange(T)[None] < L[:, None] return ids.masked_fill(~mask, PAD), mask def gold(ids): # the labelers' true taste return (ids == GOOD).sum(1).float() - (ids == BAD).sum(1).float() def make_pairs(n): # Bradley-Terry labelers, formula (67.1) a, ma = sample(n); b, mb = sample(n) a_wins = torch.rand(n) < torch.sigmoid(gold(a) - gold(b)) ch = torch.where(a_wins[:, None], a, b); mch = torch.where(a_wins[:, None], ma, mb) rej = torch.where(a_wins[:, None], b, a); mrj = torch.where(a_wins[:, None], mb, ma) return ch, mch, rej, mrj train, val = make_pairs(4000), make_pairs(1000) rm = RewardModel(); opt = torch.optim.AdamW(rm.parameters(), lr=1e-3) for step in range(300): # toy data: about 5 epochs; real RMs train about 1 i = torch.randint(0, 4000, (64,)) ch, mch, rej, mrj = (t[i] for t in train) loss = -F.logsigmoid(rm(ch, mch) - rm(rej, mrj)).mean() # formula (67.4) opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): ch, mch, rej, mrj = val acc = (rm(ch, mch) > rm(rej, mrj)).float().mean().item() g = gold(ch) - gold(rej) # the labelers' own taste, same noisy labels ceil = ((g > 0).float() + 0.5 * (g == 0).float()).mean().item() # ties count as a coin flip print(f"RM val accuracy {acc:.3f}, best possible (true taste) {ceil:.3f}") # RM val accuracy 0.692, best possible (true taste) 0.721 def rlhf_rewards(score, logp, logp_ref, resp_mask, beta=0.05): """Per-token rewards for PPO: -beta * log-ratio at every answer token, RM score added at the last one.""" r = -beta * (logp - logp_ref) * resp_mask # formula (67.9) last = resp_mask.shape[1] - 1 - resp_mask.flip(1).float().argmax(1) r[torch.arange(len(r)), last] += score return r logp = torch.tensor([[-1.0, -0.5, -2.0, 0.0]]) logp_ref = torch.tensor([[-1.2, -0.9, -2.0, 0.0]]) m = torch.tensor([[1.0, 1.0, 1.0, 0.0]]) # 3 answer tokens, then padding print(rlhf_rewards(torch.tensor([0.8]), logp, logp_ref, m)) # tensor([[-0.0100, -0.0200, 0.8000, -0.0000]])
Notice that the reward model reaches 0.692 while the best possible accuracy on these noisy labels is 0.721: “69% accuracy” here means the model is close to the labelers’ own consistency, the same situation as with real data.
Where this is used
| System or task | What the preference data and reward model do | Notes |
|---|---|---|
| InstructGPT (2022) | Rankings of 4 to 9 answers, one 6B reward model, PPO-ptx with \(\beta = 0.02\) | 1.3B RLHF model preferred over 175B GPT-3 [Certain] |
| Llama 2-Chat (2023) | 1.4M Meta comparisons plus open datasets; separate helpfulness and safety reward models with margin loss; rejection sampling, then PPO | Reward models refreshed each round with new data [Certain] |
| Anthropic helpful and harmless assistant (2022) | Comparisons for helpfulness and harmlessness, preference models, RLHF with PPO | Also released a public comparison dataset [Certain] |
| Best-of-\(n\) / rejection sampling | Generate \(n\) answers, keep the top-scored one, at inference or to build SFT data | KL only \(\log n - (n-1)/n\); a strong, simple baseline |
| Data filtering and evaluation | Score candidate training examples or model outputs with a reward model; benchmarks such as RewardBench compare reward models | Reward-model scores are not human win rates [Likely] |
| Vision-language models | Preference or correction data on image answers to reduce hallucination (for example RLHF-V) | See session 74 and the VLM track [Likely] |
| Video understanding | Preferences between captions, answers or key-moment summaries of a video; a reward model ranks candidate answers in video QA and Video-RAG | Reward hacking risk: long captions that list many objects look “detailed” [Speculation] |
Common mistakes
- Shuffling the pairs of one ranking as separate examples. They are correlated; InstructGPT saw overfitting within one pass. Keep all pairs of a prompt in one batch (formula 67.6).
- Training the reward model for several epochs. Both InstructGPT and Llama 2 train about one epoch; more overfits quickly.
- Reading the score at the wrong position. With padding, the “last position” of the tensor may be a pad token. Find the last real token from the attention mask, and make sure the end-of-turn token is included, as in the code.
- Treating raw scores as absolute. The loss fixes only differences (67.3). Normalize scores before RL; otherwise a constant offset changes the effective reward and the balance with the KL term.
- Trusting a rising reward curve. The proxy score always rises during RL (Figure 7). Track KL, answer length and a real held-out evaluation.
- Setting \(\beta\) by habit. Too small and the policy hacks the reward; too large and it barely moves. Good values depend on model size and reward scale (Llama 2 halved \(\beta\) for its larger models).
- Using a stale reward model. As the policy improves, its answers leave the reward model’s training distribution. Collect new comparisons from the current policy.
- Different templates for policy and reward model. If the reward model was trained on another chat template or tokenizer, decode the policy’s text and re-tokenize it the reward model’s way.
Interview questions
Why does RLHF collect pairwise comparisons instead of absolute scores or more demonstrations?basic
Write the reward-model loss and its gradient. What does the gradient weight mean?core
The reward model only learns reward differences. Why, and what do you do about it?core
How do you turn an LLM into a reward model? Where is the score read?basic
InstructGPT ranked K = 4 to 9 answers per prompt. How was this used, and why put all pairs of a prompt in one batch?core
Why is there a KL penalty to the reference model in RLHF?core
What is the optimal policy of the KL-regularized RLHF objective?advanced
What is reward hacking? Give examples and say how you would detect it.core
Summarize the over-optimization results of Gao et al.advanced
Your reward model has 65% held-out accuracy. Is that bad?advanced
Name ways to reduce reward over-optimization.core
Exercises
- A reward model gives the chosen answer 0.3 and the rejected answer 1.1. Compute the loss of formula (67.4) and \(\partial\mathcal{L}/\partial\Delta\).
- Labelers rank \(K = 6\) answers per prompt. How many comparisons does one prompt give? How many does a batch of 64 prompts hold?
- Llama 2’s small margin for “better” is \(2/3\). With \(\Delta = 0.8\), compute the margin loss of formula (67.7).
- What KL (in nats) does best-of-16 sampling have? What is \(d\)?
- An answer has 3 tokens with log-ratios \(0.2, -0.1, 0.5\) to the reference, \(\beta = 0.1\), and reward model score 1.0. Write the per-token rewards of formula (67.9) and their sum.
- Two labelers agree on 73% of pairs. Assume every pair has the same true reward gap and each labeler follows Bradley–Terry independently. What is the probability \(p\) that one labeler picks the truly better answer, and what reward gap is that? What accuracy can a perfect reward model reach against one labeler?
Answerscheck
Key takeaways
- RLHF collects comparisons (“which is better?”), because people compare far more consistently than they score. Real data adds rankings, preference strength and separate goals, and labelers agree only about 73% to 77% of the time.
- The Bradley–Terry model (67.1) says preference depends on the reward difference; the reward-model loss (67.4) is binary cross-entropy on that difference. Rewards are defined only up to a shift (67.3), so normalize them.
- A reward model is an LLM with a scalar head read at the last token, trained about one epoch; its accuracy must be compared with the labelers’ own agreement, not with 100%.
- RLHF maximizes reward minus \(\beta\) times the KL to a frozen reference (67.8); PPO gets it as a per-token penalty plus the score at the last token (67.9). The best policy reweights the reference by \(e^{r/\beta}\) (67.10), which leads to DPO. Sampled policy-gradient steps (67.11) approach that best policy; with a flawed reward model they find its mistake, sometimes only late in training (Figure 6).
- The reward model is a proxy. Optimizing it too hard raises the proxy score while true quality peaks and falls (Goodhart’s law; 67.12, 67.13). Watch KL, length and real evaluations, and refresh data and reward models.
Key formulas
References for this session
- Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences
- Stiennon et al. (2020). Learning to summarize from human feedback
- Ouyang et al. (2022). Training language models to follow instructions with human feedback
- Gao et al. (2022). Scaling Laws for Reward Model Overoptimization
- Lambert (2025). Reinforcement Learning from Human Feedback (book)
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Ziegler et al. (2019). Fine-Tuning Language Models from Human Preferences
- Bradley & Terry (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons (Biometrika)
- Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Singhal et al. (2023). A Long Way to Go: Investigating Length Correlations in RLHF
- Coste et al. (2023). Reward Model Ensembles Help Mitigate Overoptimization
- Lambert et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling
- Yu et al. (2023). RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
- Kaufmann et al. (2023). A Survey of Reinforcement Learning from Human Feedback
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model