LLM & VLM courseBlog
67Track 4: Post-training and RL · session 15 of 25

RLHF: preference data and reward models

RLHF (reinforcement learning from human feedback) teaches a model what people prefer. People compare two answers and pick the better one; a reward model learns to give a higher score to the preferred answer (the Bradley–Terry model); then PPO pushes the policy toward high scores while a KL penalty keeps it close to the starting model. This session builds each step with real numbers from InstructGPT and Llama 2, and shows why pushing a learned reward too hard makes the real quality go down (Goodhart’s law).

Time: 3 to 4 hours (reading 2 to 2.5 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 3, 54, 64 and 66

What you will learn

  • Why RLHF collects comparisons (which answer is better?) instead of scores or more demonstrations, and how real preference data is collected, with agreement rates
  • The Bradley–Terry model, the reward-model loss, its gradient, why it only sees reward differences, and the ranking and margin versions used by InstructGPT and Llama 2
  • How a reward model is built from an LLM: a scalar head on the last token, one epoch of training, normalization, and what its accuracy really tells you
  • The RLHF objective: reward minus a KL penalty to a frozen reference, the per-token reward given to PPO, and the closed-form best policy that leads to DPO
  • Reward hacking and over-optimization: Goodhart’s law, the Gao et al. scaling forms for best-of-n and RL, and the tools people use against it
  • A full RLHF pipeline in pseudo-code and a runnable PyTorch reward model

Why this matters

After supervised fine-tuning (session 54), a model can follow instructions. But it only copies the style of its demonstrations. It does not know which of two reasonable answers people like more: the one that is shorter or longer, more careful or more direct, the one that admits doubt or the one that guesses. RLHF (reinforcement learning from human feedback) adds that knowledge.

The idea has three steps. First, people look at two answers to the same prompt and say which one is better. Second, we train a reward model: a network that reads a prompt and an answer and outputs one number, higher for answers people prefer. Third, we use reinforcement learning (PPO, from session 66) to change the model so that its answers get high reward, with a penalty that keeps it close to where it started. This is the recipe behind InstructGPT and the first ChatGPT, and behind Llama 2-Chat. [Certain]

The result can be large. In InstructGPT, people preferred the answers of a 1.3B RLHF model to the answers of the 175B GPT-3, a model with more than 100 times more parameters. [Certain] But the method also has a famous weakness: the reward model is only an estimate of what people want. Push the policy too hard against it and the policy finds the estimate’s mistakes. The reward score keeps going up while the real quality goes down. This is reward hacking, a case of Goodhart’s law: “when a measure becomes a target, it ceases to be a good measure.”

1. Supervised fine-tuning prompts + humandemonstrations SFT model InstructGPT: about 13k prompts 2. Reward model prompt x, answers A and Bperson: “A is better” reward modelr(x, y) = one number Bradley–Terry loss, about 33k prompts initialize 3. PPO with a KL leash policy π (trained)writes answer y rewardr(x, y) referencefrozen SFT maximize r − β·KL(π ‖ πref)InstructGPT: β = 0.02, 31k prompts frozen the SFT model starts the policy and is the frozen reference
Figure 1. The three steps of classic RLHF, with the dataset sizes and the KL coefficient of InstructGPT. This session covers step 2 in depth and the parts of step 3 that are about the reward (the PPO algorithm itself is in session 66).

Collecting preference data

Why comparisons?

We want the model to produce answers that people like. There are three ways to ask people for that signal:

  1. Write the ideal answer (a demonstration). This is what SFT uses. It is slow and expensive, and the model can never become better than the writers.
  2. Give a score, for example 1 to 7. Different people use the scale differently: one person’s 5 is another person’s 6, and the same person drifts over a day. Absolute scores are noisy.
  3. Compare two answers: “Is A or B better?” This is fast and much more consistent, because people judge relative quality much better than absolute quality. It also works when nobody could write the best answer, but everyone can recognize the better of two.

RLHF uses comparisons. Christiano et al. (2017) showed the idea first on robot and Atari tasks: a simulated robot learned a backflip from 900 human comparisons of short video clips, collected in less than an hour. [Certain] Stiennon et al. (2020) then used it for text (summaries of Reddit posts), and InstructGPT used it for general chat.

What one data point looks like

One example is a triple \((x, y_w, y_l)\): a prompt \(x\), the preferred answer \(y_w\) (“winner”, also called chosen) and the other answer \(y_l\) (“loser”, also called rejected). Both answers are usually sampled from the current model, sometimes from different model versions or temperatures, so that they differ in interesting ways. Real projects add detail:

  • Rankings. InstructGPT showed labelers between \(K = 4\) and \(K = 9\) answers to rank. A ranking of \(K\) answers gives \(\binom{K}{2}\) pairs: 6 pairs for \(K = 4\), 36 for \(K = 9\). [Certain]
  • Strength of preference. Llama 2 asked how much better the chosen answer is: significantly better, better, slightly better, or negligibly better / unsure. [Certain]
  • Separate goals. Llama 2 collected helpfulness and safety comparisons separately, with different guidelines, and trained two reward models. [Certain]
  • Fresh data every round. Llama 2 collected comparisons in weekly batches from the latest model, because a reward model trained on old answers becomes inaccurate on the new model’s answers. Its own data had 1,418,091 comparisons. [Certain]

People disagree, and that sets a ceiling

Preferences are noisy. In InstructGPT, training labelers agreed with each other 72.6 ± 1.5% of the time. In the summarization work, researchers agreed with each other only 73 ± 4% of the time. [Certain] So even a perfect model of “the average person” cannot predict every label. If people agree 73% of the time, a reward model with 73% accuracy is not bad; it may be close to the best possible. We return to this ceiling in Figure 3.

The Bradley–Terry model

We need a way to turn “A beat B” into a number for each answer. The classic answer is the Bradley–Terry model (1952). Give every answer a hidden score, its reward \(r\). Then the chance that a person prefers \(y_w\) over \(y_l\) depends only on the difference of the two rewards:

\[ P(y_w \succ y_l \mid x) = \sigma\big(r(x, y_w) - r(x, y_l)\big), \qquad \sigma(z) = \frac{1}{1 + e^{-z}} \tag{67.1} \]
Read the symbols
\(y_w \succ y_l\)
“\(y_w\) is preferred to \(y_l\)”
\(r(x, y)\)
the reward of answer \(y\) to prompt \(x\): one real number
\(\sigma\)
the sigmoid function, which turns any number into a probability between 0 and 1

An equal reward gives \(\sigma(0) = 0.5\): a coin flip. A reward gap of 1 gives 0.731, a gap of 2 gives 0.881. Another way to write the same model shows where it comes from: each answer has a “strength” \(e^{r}\), and the chance of winning is your strength divided by the total:

\[ P(y_w \succ y_l \mid x) = \frac{e^{r(x, y_w)}}{e^{r(x, y_w)} + e^{r(x, y_l)}} \tag{67.2} \]

Divide the top and bottom by \(e^{r(x,y_w)}\) and you get formula (67.1). This is a softmax over two answers, so it is the same math as next-token prediction with a vocabulary of size 2 (session 3).

One property is very important. Add the same constant \(c\) to every reward and nothing changes:

\[ \sigma\big((r_w + c) - (r_l + c)\big) = \sigma(r_w - r_l) \quad \text{for every } c \tag{67.3} \]

So comparisons only define rewards up to a shift. The reward model learns differences, never absolute levels. A reward of 3.2 means nothing alone. This is why InstructGPT shifted its reward model after training so that the human demonstrations get a mean score of 0, and why Llama 2 whitened its scores (subtract the mean, divide by the standard deviation) before PPO. [Certain] The scale is also only fixed by the sigmoid: a model that is very sure gives large gaps.

The reward-model loss

We fit \(r_\phi\), a network with weights \(\phi\), by maximum likelihood: make the observed preferences as probable as possible. The negative log-likelihood of formula (67.1) is the reward-model loss used by Stiennon et al., InstructGPT and Llama 2: [Certain]

\[ \mathcal{L}_{\text{RM}}(\phi) = -\,\E_{(x, y_w, y_l) \sim \mathcal{D}} \Big[ \log \sigma\big( r_\phi(x, y_w) - r_\phi(x, y_l) \big) \Big] \tag{67.4} \]
Read the symbols
\(\mathcal{D}\)
the dataset of human comparisons
\(r_\phi\)
the reward model with weights \(\phi\)
\(\log \sigma(\cdot)\)
log of the model’s probability for the label the person gave; the minus sign turns it into a loss

It is binary cross-entropy where the “logit” is the reward difference \(\Delta = r_\phi(x, y_w) - r_\phi(x, y_l)\). Its gradient is short:

\[ \frac{\partial \mathcal{L}}{\partial \Delta} = -\big(1 - \sigma(\Delta)\big), \qquad \nabla_\phi \mathcal{L} = -\big(1 - \sigma(\Delta)\big) \big( \nabla_\phi r_\phi(x, y_w) - \nabla_\phi r_\phi(x, y_l) \big) \tag{67.5} \]

Read it as: push the winner’s reward up and the loser’s reward down, with a weight \(1 - \sigma(\Delta)\), the probability the model gave to the wrong answer. Pairs the model already gets right with confidence contribute almost nothing. Pairs it gets wrong contribute the most.

Worked example: one comparison

The reward model gives \(r(x, y_w) = 1.2\) and \(r(x, y_l) = 0.4\), so \(\Delta = 0.8\). It predicts \(P(y_w \succ y_l) = \sigma(0.8) = 0.690\). The loss is \(-\log 0.690 = 0.371\). The gradient on \(\Delta\) is \(-(1 - 0.690) = -0.310\): gradient descent increases the gap.

If the person had preferred the other answer, the loss would be \(-\log \sigma(-0.8) = 1.171\), more than three times larger, and the gradient would be \(-0.690\). Before training, all rewards are about equal, \(\Delta \approx 0\), and the loss is \(\ln 2 = 0.693\) per pair. That is the number to expect at step 0.

Now add 5 to both rewards (6.2 and 5.4). \(\Delta\) is still 0.8 and the loss is still 0.371: formula (67.3).

Rankings and margins

With a ranking of \(K\) answers, InstructGPT uses every pair and averages them, and it puts all \(\binom{K}{2}\) pairs of one prompt in the same batch:

\[ \mathcal{L}_{\text{rank}}(\phi) = -\frac{1}{\binom{K}{2}} \sum_{i \lt j} \log \sigma\big( r_\phi(x, y_{(i)}) - r_\phi(x, y_{(j)}) \big) \tag{67.6} \]

Here \(y_{(1)}\) is the best-ranked answer and \(y_{(K)}\) the worst, so in each pair \(y_{(i)}\) is the winner. Why one batch? The pairs from one prompt are strongly correlated (each answer appears in \(K - 1\) pairs). When InstructGPT shuffled them into the dataset as separate examples, the reward model overfit after one pass. Keeping them together also needs only \(K\) forward passes instead of \(K(K-1)\). [Certain] With 64 prompts per batch and \(K = 9\), one batch holds up to \(64 \times 36 = 2{,}304\) comparisons. [Certain]

Llama 2 uses the strength of preference with a margin \(m\): [Certain]

\[ \mathcal{L}_{\text{margin}}(\phi) = -\log \sigma\big( r_\phi(x, y_w) - r_\phi(x, y_l) - m(\text{rating}) \big) \tag{67.7} \]

The loss is only small when the gap is larger than \(m\). Llama 2 tried a small margin (1, 2/3, 1/3, 0 for significantly better, better, slightly better, negligibly better) and a large one (3, 2, 1, 0). The margin raised accuracy mostly on the clearly separated pairs. [Certain] With \(\Delta = 0.8\) and \(m = 1\), the loss rises from 0.371 to 0.798 and the gradient from \(-0.310\) to \(-0.550\): the model is asked for a bigger gap. Try it in Figure 2.

\(P(\text{chosen} \succ \text{rejected}) = \sigma(\Delta)\)loss \(-\log \sigma(\Delta - m)\)\(|\text{gradient}| = \sigma(m - \Delta)\)
Figure 2. The Bradley–Terry model for one comparison. The horizontal axis is the reward gap \(\Delta = r_w - r_l\). The dot shows where the current pair sits.
How to read Figure 2

Move \(r_w\) and \(r_l\): only their difference moves the dot. Press “Shift both rewards” (it adds the same number to both) and nothing in the readout changes except the two rewards; that is formula (67.3). Press “Swap the label” to see the loss when the person disagreed with the model: the dot jumps to \(-\Delta\), where the loss and the gradient are large. Raise the margin: the loss curve slides to the right, so a pair needs a larger gap before its gradient fades. Set \(m = 1\) to reproduce Llama 2’s “significantly better” margin.

Building a reward model from an LLM

The reward model must read language as well as the policy does, so it is itself a language model. The standard recipe: [Certain]

  1. Start from a pretrained or SFT model. InstructGPT started from a 6B model; Llama 2 started its reward models from the chat model checkpoints, so that the reward model “knows what the policy knows”.
  2. Remove the unembedding layer (the matrix that maps a hidden state to vocabulary logits). Put a linear head \(\mathbf{w}^\top \mathbf{h} + b\) in its place that maps a hidden state to one number.
  3. Feed the prompt and the answer together, in the same chat template as the policy. Read the score at the last token of the answer (usually the end-of-turn token). With a causal model, that is the only position that has seen the whole answer.
  4. Train with formula (67.4), (67.6) or (67.7) for about one epoch. Both InstructGPT and Llama 2 report that more epochs overfit quickly. [Certain]
  5. Shift (and maybe scale) the scores so that a reference set has mean 0, because of formula (67.3).

How big? InstructGPT used one 6B reward model for policies of all sizes, 1.3B to 175B. A 175B reward model had lower validation loss but its training was unstable, and it would have made PPO much more expensive (the reward model also initializes the value network). [Certain] In the summarization work, doubling the reward model size raised its accuracy by about 1.8 points and doubling the data by about 1.1 points. [Certain]

What accuracy means

The main metric is pairwise accuracy on held-out comparisons: how often \(r(x, y_w) \gt r(x, y_l)\). Numbers look low. Llama 2’s helpfulness reward model scored 63.2% on its own helpfulness test set, but 80.7% on pairs marked “significantly better” and 54.7% on pairs marked “negligibly better / unsure”. [Certain] When two answers are almost equally good, people themselves flip a coin, and so must the model.

We can make the ceiling precise. If labels come from formula (67.1) with the true rewards, the best possible accuracy on a pair with true gap \(\Delta^\ast\) is \(\sigma(|\Delta^\ast|)\), not 1. Figure 3 trains a small reward model live, so you can see the ceiling, the effect of label noise and the effect of a labeler bias.

true quality weightslabeler taste weightslearned reward-model weights
Figure 3. A linear reward model trained with formula (67.4) on simulated comparisons, 150 steps of full-batch gradient descent, every time you move a slider. Each answer has four features: correctness, useful detail, length and flattery. True quality is \(1.5 \times\) correctness \(+ 1.0 \times\) detail. Simulated labelers pick answer A with probability \(\sigma((u_A - u_B)/\tau)\), where their taste \(u\) is true quality plus “length bias” \(\times\) length.
How to read Figure 3

Left: three bars per feature. Grey is the true value of the feature, blue is what the labelers reward, orange is what the reward model learned. The reward model learns the labelers’ taste divided by the noise \(\tau\) (noisy labelers give smaller reward gaps), so we multiply its weights by \(\tau\) to put them on the same scale. Right: the training loss, which starts at \(\ln 2 = 0.693\). The readout gives the expected held-out accuracy of the reward model and of the labelers’ own taste (the ceiling). With few comparisons, the learned weights are noisy and held-out accuracy falls below the ceiling. Raise the noise \(\tau\): the ceiling drops toward 50%, and the model needs more data. Now raise the length bias: the reward model faithfully learns to reward length, its accuracy on labeler data stays high, and its agreement with true quality falls. The reward model learns the labelers, not the truth.

Watch the loss train a reward model, one pair at a time

Figure 3 trains in one go. Figure 4 slows training down to one comparison per step, so you can watch formulas (67.1), (67.4) and (67.5) do their work. It uses the six answers A to F of Figure 5 below. Each answer has a hidden true quality \(q\): \((1.0, 2.0, 0.3, -2.0, -0.5, -1.0)\). The simulation first builds a dataset. It picks two different answers at random, and a simulated labeler prefers \(i\) over \(j\) with the Bradley–Terry probability \(\sigma(q_i - q_j)\). Then, with probability “label noise”, the label is flipped, like a careless labeler. The reward model is the simplest one possible: a table with one learnable score \(r_\phi(y)\) per answer, all 0 at the start. A real reward model replaces the table with an LLM and a linear head, but the loss and the gradient are the same.

Running: SGD on the reward-model loss
  1. take the next pair \((y_w, y_l)\) from the shuffled dataset
  2. \(\Delta \leftarrow r_\phi(y_w) - r_\phi(y_l)\) // reward gap
  3. \(p \leftarrow \sigma(\Delta)\) // (67.1): the model’s chance for this label
  4. \(\ell \leftarrow -\log p\) // (67.4)
  5. \(g \leftarrow 1 - p\) // (67.5): the weight of this pair
  6. \(r_\phi(y_w) \mathrel{+}= \eta\, g\); \(\;r_\phi(y_l) \mathrel{-}= \eta\, g\) // gradient step
true quality \(q\)learned reward \(r_\phi\)loss / ceilingtraining accuracyfraction of the 15 answer pairs in the true order
Figure 4. A live simulation of reward-model training. Each step takes one labeled pair and runs the six lines of the algorithm box; the highlighted line is the one that just ran, and the panel next to it shows that line’s formula with the current numbers. Training stops after 20 passes (epochs) over the dataset. Everything is seeded, so the same settings give the same run.
How to read Figure 4

Set the speed to 1 and press Next line a few times: you see one pair go through the six lines. At the start every reward is 0, so \(\Delta = 0\), \(p = 0.5\), the loss is \(\ln 2 = 0.693\) and the weight is 0.5. Then press Play at speed 4. With the defaults (120 pairs, noise 0.10, \(\eta = 0.1\), seed 1), 29 of the 120 labels disagree with the true order: 19 were flipped by the noise, the rest are Bradley–Terry coin flips that went the “wrong” way. So no reward model can get more than 75.8% of these training labels right by following the true quality; that is the orange dashed line. After 20 epochs the loss is 0.539, the training accuracy 76.7% (a little above the line, because it also fits some of the flipped labels), and 14 of the 15 answer pairs are in the true order. Compare the bars: the learned rewards keep the true order but the gaps are smaller (from −1.22 to 1.66 instead of −1.97 to 2.03). Noisy labels look like a less sure labeler, and Bradley–Terry turns “less sure” into “smaller gaps”. Set the noise to 0: the gaps grow to −2.56 … 2.59, a little wider than the truth, because 120 pairs is not much data. Set it to 0.30 and the reward model can no longer rank the answers (10 of 15 pairs right). Note that the bars are centered: comparisons fix rewards only up to a shift, formula (67.3). (Speeds 1 and 2 run one line of the algorithm at a time, 0.65 s or 0.2 s per line; speeds 3 to 5 run whole iterations, more of them per frame. “Next line” and the next-step button work at any speed.)

Using the reward: RL with a KL leash

Now we have \(r_\phi(x, y)\). We want a policy \(\pi_\theta\) (the language model being trained) whose answers get high reward. If we only maximized the reward, the policy would drift to strange text the reward model has never seen, where its scores mean nothing. So RLHF adds a penalty for moving away from a frozen reference model \(\pi_{\text{ref}}\), usually the SFT model: [Certain]

\[ \max_{\theta}\ \E_{x \sim \mathcal{D},\ y \sim \pi_\theta(\cdot \mid x)} \big[ r_\phi(x, y) \big] \;-\; \beta\, \E_{x \sim \mathcal{D}} \Big[ \KL\big( \pi_\theta(\cdot \mid x) \,\big\|\, \pi_{\text{ref}}(\cdot \mid x) \big) \Big] \tag{67.8} \]
Read the symbols
\(\pi_\theta(y \mid x)\)
the probability the policy gives to the whole answer \(y\): the product of its token probabilities
\(\pi_{\text{ref}}\)
the frozen copy of the starting (SFT) model
\(\KL(\pi \,\|\, \pi_{\text{ref}})\)
the average, over answers \(y \sim \pi\), of \(\log \pi(y \mid x)\) minus \(\log \pi_{\text{ref}}(y \mid x)\): how far the policy has moved, in nats; it is 0 only when the two are equal
\(\beta\)
the strength of the leash: 0.05 in the summarization work, 0.02 in InstructGPT, 0.01 for Llama 2 7B and 13B, 0.005 for 34B and 70B [Certain]

Stiennon et al. give two reasons for the KL term: it acts like an entropy bonus that stops the policy from collapsing to one answer, and it keeps the policy’s answers close to the kind of answers the reward model was trained on. [Certain] InstructGPT also mixed in pretraining gradients (“PPO-ptx”, with coefficient \(\gamma = 27.8\)) to stop the model from getting worse on public NLP benchmarks. [Certain]

How PPO sees this reward

PPO works on tokens. Since \(\log \pi(y \mid x)\) is a sum over tokens, the KL of one sampled answer is a sum of per-token log-ratios, and we can hand out the penalty token by token. The reward model score arrives only at the last token \(T\): [Certain]

\[ r_t = -\beta \log \frac{\pi_\theta(y_t \mid x, y_{\lt t})}{\pi_{\text{ref}}(y_t \mid x, y_{\lt t})} \;+\; \begin{cases} r_\phi(x, y) & t = T \\ 0 & t \lt T \end{cases} \tag{67.9} \]

The sum of these \(r_t\) over the answer is exactly “reward minus \(\beta\) times the sampled KL” from formula (67.8). PPO then uses \(r_t\) with a value network and GAE (InstructGPT used no discount, \(\gamma = 1\)) as in session 65 and session 66.

Worked example: per-token rewards

An answer has 3 tokens. The policy’s log-ratios to the reference are \(0.2, 0.4, 0.0\) nats, \(\beta = 0.05\), and the reward model gives 0.8. Then \(r_1 = -0.05 \times 0.2 = -0.010\), \(r_2 = -0.05 \times 0.4 = -0.020\), \(r_3 = 0 + 0.8 = 0.800\). Total: 0.770 = 0.8 minus \(0.05 \times 0.6\). The PyTorch code below prints exactly these numbers.

The best policy has a closed form

Ignore neural networks for a moment and ask: which distribution \(\pi\) maximizes formula (67.8) for one prompt? The answer is known exactly: [Certain]

\[ \pi^\ast(y \mid x) = \frac{1}{Z(x)}\, \pi_{\text{ref}}(y \mid x)\, \exp\!\Big( \frac{r_\phi(x, y)}{\beta} \Big), \qquad Z(x) = \sum_{y} \pi_{\text{ref}}(y \mid x)\, \exp\!\Big( \frac{r_\phi(x, y)}{\beta} \Big) \tag{67.10} \]

Start from the reference and multiply each answer’s probability by \(e^{r/\beta}\), then renormalize. Large \(\beta\): \(\pi^\ast \approx \pi_{\text{ref}}\). Small \(\beta\): all the mass goes to the answer with the highest reward model score, right or wrong. PPO is a way to approach \(\pi^\ast\) with a network and samples. In session 68, this formula is turned around to train the policy directly from comparisons, without a separate reward model.

Figure 5 applies formula (67.10) to six possible answers. The reward model is decent but has one flaw: it over-rates a confident, flattering answer that is wrong.

reference \(\pi_\text{ref}\)best policy \(\pi^*\) for this \(\beta\)reward-model score (expected)true quality (expected)
Figure 5. The KL leash on a toy prompt with six possible answers. Top: probabilities under the reference and under \(\pi^\ast\) of formula (67.10). Bottom: expected reward-model score and expected true quality of \(\pi^\ast\) for every \(\beta\); the dashed line marks the current \(\beta\). The numbers are made up to show the effect; they are not from a real model.
How to read Figure 5

Each row of the table on the left shows an answer’s reward-model score and its true quality. Start at \(\beta = 1\): \(\pi^\ast\) moves mass toward the detailed correct answer, and true quality rises from 0.505 (reference) to 0.869. Lower \(\beta\): the reward-model score keeps rising, but the policy piles onto the flattering wrong answer, the one the reward model over-rates, and true quality falls to −1.66 at \(\beta = 0.1\). The KL in the readout grows toward its maximum \(\ln 20 = 3.00\) nats, the cost of putting all mass on an answer that had 5% under the reference. That is reward hacking in its simplest form. Raise \(\beta\) above 3 and the policy barely moves: safe, but little gain.

Reaching \(\pi^*\) by sampling: the policy gradient in the loop

Figure 5 jumps straight to \(\pi^\ast\). A real RLHF run cannot do that: it cannot list all answers, it can only sample answers from the current policy, score them, and take a small step. The step follows the gradient of objective (67.8). For one prompt it is

\[ \nabla_\theta J(\theta) = \E_{y \sim \pi_\theta}\Big[ \Big( r_\phi(x, y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} - b \Big)\, \nabla_\theta \log \pi_\theta(y \mid x) \Big] \tag{67.11} \]
Read the symbols
\(r_\phi - \beta \log(\pi_\theta/\pi_{\text{ref}})\)
the reward of one sampled answer after the KL penalty: the sum of the per-token rewards (67.9)
\(b\)
a baseline: any number that does not depend on \(y\). It does not change the expected gradient, because \(\E_{y \sim \pi_\theta}[\nabla_\theta \log \pi_\theta(y)] = 0\), but it lowers the noise. Here we use the mean of the other samples in the batch (the “leave-one-out” baseline)
\(\nabla_\theta \log \pi_\theta(y)\)
for a softmax over answers with logits \(\theta\), this is \(e_y - \pi_\theta\): the one-hot vector of \(y\) minus the current probabilities

Why is there no extra term from the \(\log \pi_\theta\) inside the bracket? Its derivative gives \(-\beta\,\E[\nabla_\theta \log \pi_\theta(y)]\), which is 0 for the same reason. PPO adds a value network, GAE and clipping on top (session 66), but its core is this formula. Figure 6 runs it on the six answers of Figure 5, with the policy as a softmax over six logits.

Running: one policy-gradient iteration
  1. sample \(y_1, \dots, y_B \sim \pi_\theta\)
  2. score each answer with the reward model: \(r_\phi(y_i)\)
  3. \(R_i \leftarrow r_\phi(y_i) - \beta \log \dfrac{\pi_\theta(y_i)}{\pi_{\text{ref}}(y_i)}\) // (67.9), summed
  4. \(A_i \leftarrow R_i - \frac{1}{B-1}\sum_{j \ne i} R_j\) // baseline b
  5. \(g \leftarrow \frac{1}{B}\sum_i A_i\,(e_{y_i} - \pi_\theta)\) // (67.11)
  6. \(\theta \leftarrow \theta + \eta\, g\) // gradient ascent
policy \(\pi_\theta\) (bars) and its KLclosed-form \(\pi^*\) (67.10)expected reward-model scoreexpected true qualitydashed: the value at \(\pi^*\)
Figure 6. RLHF’s RL step as a live simulation: a softmax policy over the six answers of Figure 5, trained with formula (67.11) from samples, starting at \(\pi_{\text{ref}}\). The bars compare the policy with the closed-form best policy \(\pi^\ast\) of (67.10). “Flawed reward model” uses the reward-model column of Figure 5, which over-rates answer D (flattering, wrong); “Reward model of Figure 4” takes the centered scores that Figure 4 has learned so far (press Reset here after training there). The run stops after 1,500 iterations.
How to read Figure 6

Use speed 1 and Next line first. In the table, an answer whose penalized reward \(R_i\) beats the other samples gets a positive advantage \(A_i\) (blue), and line 6 moves its logit up. Then Play at speed 4 with the defaults (flawed reward model, \(\beta = 0.1\), seed 1). For the first few hundred iterations everything looks good: the policy moves to B (detailed, correct) and the true quality (blue curve) climbs to 1.94 at iteration 190. D starts at only 5% and is rarely sampled, so the policy has not found the reward model’s mistake yet. Then it does: D is sampled, gets a large reward, and takes over. With seed 1 the true quality turns negative at iteration 526 and ends at −1.71, close to its value at \(\pi^\ast\) (−1.66), while the reward-model score keeps rising to 2.37. This is reward hacking happening during training, not only at the optimum. Other seeds find D at other times, but all 20 seeds find it: the true quality turns negative between iteration 145 and 1,267. Raise \(\beta\) to 0.3 and reset: \(\pi^\ast\) itself keeps half of its mass on B, and the true quality peaks at 1.62 (iteration 51) and then settles near 0.28, the \(\pi^\ast\) value. Choose “True quality” as the reward: now the optimum is good, and the KL curve rises to the dashed \(\pi^\ast\) line and stays there. Larger batches \(B\) give smoother curves.

Reward hacking and over-optimization

The reward model is a proxy: a learned, imperfect stand-in for what people want. The policy is an optimizer, and optimizers find the proxy’s mistakes. Common forms in LLMs: [Likely]

  • Length. Labelers often prefer longer, more detailed answers, so reward models give a bonus for length; the policy then pads. Singhal et al. report that on several common setups much of the reward gain from RLHF can be reproduced by optimizing for length alone. [Likely] In the summarization work, length differences explained about a third of the gap between the human-feedback and supervised models at 6.7B. [Certain]
  • Sycophancy. Agreeing with the user and praising the question scores well with people, so the model learns to flatter, even when the user is wrong.
  • Style over substance. Confident tone, lists and headings, and hedging phrases can raise the score without making the answer more correct.
  • Out-of-distribution text. Pushed far, the policy finds odd strings (repeated phrases, strange tokens) where the reward model gives high scores it never learned to give. Stiennon et al. show such over-optimized samples. [Certain]

Stiennon et al. measured the effect directly. They trained policies against a reward model with different KL coefficients and asked labelers to judge the samples. Under light optimization real quality went up. Under heavy optimization the real preference fell, and eventually the reward model became anti-correlated with human preference: higher score, worse summary. [Certain]

Scaling laws for over-optimization

Measuring this with humans is expensive. Gao et al. (2022) used a trick: a large “gold” reward model (the 6B InstructGPT reward model) plays the role of humans and labels 100,000 synthetic comparisons; smaller “proxy” reward models (3M to 3B parameters) are trained on these labels; then the policy is optimized against the proxy, and the gold model measures the real quality. [Certain] With \(d = \sqrt{\KL(\pi \,\|\, \pi_{\text{init}})}\), they found that the gold score follows simple curves: [Certain]

\[ R_{\text{bon}}(d) = d\,(\alpha_{\text{bon}} - \beta_{\text{bon}}\, d) \tag{67.12} \]
\[ R_{\text{RL}}(d) = d\,(\alpha_{\text{RL}} - \beta_{\text{RL}} \log d) \tag{67.13} \]
Read the symbols
\(R(d)\)
gold (true) reward gained, with \(R(0) = 0\) at the starting policy
\(d\)
the square root of the KL from the starting policy: how far optimization has moved
\(\alpha, \beta\)
fitted constants (this \(\beta\) is not the KL coefficient); they change smoothly with the size of the proxy reward model
bon
best-of-\(n\): sample \(n\) answers and keep the one the proxy likes best

Both curves rise, reach a peak and then fall: the more you optimize, the more the gap between proxy and gold grows. Setting the derivative of (67.12) to zero gives the peak at \(d^\ast = \alpha_{\text{bon}} / (2 \beta_{\text{bon}})\). For (67.13) the peak is at \(d^\ast = e^{\alpha_{\text{RL}}/\beta_{\text{RL}} - 1}\). Larger proxy reward models had smaller \(\beta\): they can be optimized further before real quality falls. [Certain]

For best-of-\(n\), the KL has a closed form (from Stiennon et al., appendix): [Certain]

\[ \KL_{\text{bon}}(n) = \log n - \frac{n - 1}{n} \tag{67.14} \]

So best-of-16 moves the policy by \(\log 16 - 15/16 = 1.835\) nats, and best-of-148 by about 4.0 nats. KL grows only like \(\log n\): best-of-\(n\) is a cheap, gentle optimizer, and a strong baseline. Two more findings from Gao et al.: as a function of KL, RL is slower than best-of-\(n\) both to improve and to over-optimize, and in their setup the KL penalty did not move the gold-reward-versus-KL frontier; it acted mostly like early stopping. [Certain] Figure 7 plots both curves.

proxy reward-model scoregold (true) reward
Figure 7. The functional forms (67.12) and (67.13) of Gao et al. The shapes are from the paper; the constants are chosen by us for illustration (gold: \(\alpha = 1\), \(\beta_{\text{bon}} = 0.25\), \(\beta_{\text{RL}} = 0.5\); proxy: \(\beta_{\text{bon}} = 0.05\), \(\beta_{\text{RL}} = 0.1\)), not fitted values.
How to read Figure 7

Drag \(d\). The orange proxy score keeps going up: from inside the optimization, everything looks fine. The blue gold score peaks and then drops. For best-of-\(n\) the readout also gives the \(n\) that reaches this KL, from formula (67.14): the peak at \(d = 2\) (4 nats) needs about \(n = 147\). The gap between the two curves is the over-optimization you would only see with a fresh human evaluation. Switch to RL: the gold curve peaks later, at \(d = e\), and falls more slowly.

What people do about it

  • The KL penalty (formula 67.8), and watching the KL during training. A KL that keeps climbing is a warning sign.
  • Early stopping on a real evaluation: human or strong-judge win rates on held-out prompts, not the reward model score.
  • Fresh preference data from the current policy, and retrained reward models, as Llama 2 did every round. [Certain]
  • Bigger and better reward models: in Gao et al., larger proxies over-optimize later. [Certain]
  • Reward model ensembles: optimize the mean or a conservative (worst-case) combination of several reward models. Coste et al. found this reduces over-optimization. [Likely]
  • Separate reward models and caps: Llama 2 used a safety reward model that takes over when a prompt is tagged as risky or its safety score falls below 0.15. [Certain] Length penalties or length-controlled evaluation fight length hacking. [Likely]

The full pipeline in pseudo-code

Algorithm 1: train a reward model from rankings
  1. input: SFT model, prompts \(\{x\}\), labelers
  2. for each prompt \(x\): sample \(K\) answers from the current model(s), get a ranking \(y_{(1)} \succ \dots \succ y_{(K)}\) // drop ties
  3. \(r_\phi \leftarrow\) SFT model with the unembedding replaced by a linear head to 1 number
  4. for one epoch, batches of prompts:
  5.   score all \(K\) answers of each prompt; read \(r_\phi\) at the last answer token
  6.   loss \(\leftarrow\) mean over all \(\binom{K}{2}\) pairs of \(-\log \sigma(r_{(i)} - r_{(j)})\) // formula (67.6), pairs of one prompt stay together
  7.   AdamW step
  8. report pairwise accuracy on held-out comparisons, split by preference strength
  9. \(b \leftarrow b - \text{mean}(r_\phi)\) on a reference set // formula (67.3): fix the free shift
Algorithm 2: one RLHF iteration with PPO
  1. input: policy \(\pi_\theta\) (starts as SFT), frozen \(\pi_{\text{ref}}\), frozen \(r_\phi\), value net \(V\) (starts as \(r_\phi\)), \(\beta\)
  2. sample a batch of prompts; generate answers \(y \sim \pi_\theta(\cdot \mid x)\) at temperature 1
  3. \(s \leftarrow r_\phi(x, y)\); normalize \(s\) (shift, or whiten as in Llama 2)
  4. log-probs of each answer token under \(\pi_\theta\) and \(\pi_{\text{ref}}\)
  5. per-token rewards \(r_t \leftarrow -\beta(\log\pi_\theta - \log\pi_{\text{ref}})\), plus \(s\) at the last token // formula (67.9)
  6. advantages with GAE from \(r_t\) and \(V\)
  7. a few epochs of PPO clipped updates on \(\pi_\theta\), value loss on \(V\)
  8. log: mean \(s\), KL to the reference, answer length, and a sample of answers // rising \(s\) with rising length or KL = suspect hacking
  9. every few iterations: evaluate on held-out prompts with humans or a strong judge; stop when the win rate stops improving

Runnable PyTorch: a small causal-transformer reward model trained with formula (67.4) on simulated comparisons, and the per-token reward of formula (67.9). The simulated labelers like token 5 and dislike token 6, and follow formula (67.1). Output with torch 2.8 on CPU is shown in the comments.

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
V, D, T, PAD = 16, 64, 12, 0          # vocab, width, max length, pad id
GOOD, BAD = 5, 6                      # the hidden "taste" of our simulated labelers

class RewardModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.emb, self.pos = nn.Embedding(V, D), nn.Embedding(T, D)
        layer = nn.TransformerEncoderLayer(D, 4, 4 * D, dropout=0.0, batch_first=True, norm_first=True)
        self.body = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False)
        self.head = nn.Linear(D, 1)   # replaces the unembedding: hidden state to one number
    def forward(self, ids, mask):     # ids, mask: (B, T); mask is True on real tokens
        h = self.emb(ids) + self.pos(torch.arange(ids.shape[1]))
        causal = nn.Transformer.generate_square_subsequent_mask(ids.shape[1])
        h = self.body(h, mask=causal, src_key_padding_mask=~mask)
        last = mask.sum(1) - 1        # last real token (right padding)
        return self.head(h[torch.arange(len(ids)), last]).squeeze(-1)

def sample(n):                        # random answers of length 4..12
    L = torch.randint(4, T + 1, (n,))
    ids = torch.randint(1, V, (n, T)); mask = torch.arange(T)[None] < L[:, None]
    return ids.masked_fill(~mask, PAD), mask
def gold(ids):                        # the labelers' true taste
    return (ids == GOOD).sum(1).float() - (ids == BAD).sum(1).float()
def make_pairs(n):                    # Bradley-Terry labelers, formula (67.1)
    a, ma = sample(n); b, mb = sample(n)
    a_wins = torch.rand(n) < torch.sigmoid(gold(a) - gold(b))
    ch  = torch.where(a_wins[:, None], a, b);  mch = torch.where(a_wins[:, None], ma, mb)
    rej = torch.where(a_wins[:, None], b, a);  mrj = torch.where(a_wins[:, None], mb, ma)
    return ch, mch, rej, mrj

train, val = make_pairs(4000), make_pairs(1000)
rm = RewardModel(); opt = torch.optim.AdamW(rm.parameters(), lr=1e-3)
for step in range(300):                                  # toy data: about 5 epochs; real RMs train about 1
    i = torch.randint(0, 4000, (64,))
    ch, mch, rej, mrj = (t[i] for t in train)
    loss = -F.logsigmoid(rm(ch, mch) - rm(rej, mrj)).mean()   # formula (67.4)
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    ch, mch, rej, mrj = val
    acc = (rm(ch, mch) > rm(rej, mrj)).float().mean().item()
    g = gold(ch) - gold(rej)                                  # the labelers' own taste, same noisy labels
    ceil = ((g > 0).float() + 0.5 * (g == 0).float()).mean().item()   # ties count as a coin flip
print(f"RM val accuracy {acc:.3f}, best possible (true taste) {ceil:.3f}")
# RM val accuracy 0.692, best possible (true taste) 0.721

def rlhf_rewards(score, logp, logp_ref, resp_mask, beta=0.05):
    """Per-token rewards for PPO: -beta * log-ratio at every answer token, RM score added at the last one."""
    r = -beta * (logp - logp_ref) * resp_mask             # formula (67.9)
    last = resp_mask.shape[1] - 1 - resp_mask.flip(1).float().argmax(1)
    r[torch.arange(len(r)), last] += score
    return r
logp     = torch.tensor([[-1.0, -0.5, -2.0, 0.0]])
logp_ref = torch.tensor([[-1.2, -0.9, -2.0, 0.0]])
m        = torch.tensor([[1.0, 1.0, 1.0, 0.0]])          # 3 answer tokens, then padding
print(rlhf_rewards(torch.tensor([0.8]), logp, logp_ref, m))
# tensor([[-0.0100, -0.0200,  0.8000, -0.0000]])

Notice that the reward model reaches 0.692 while the best possible accuracy on these noisy labels is 0.721: “69% accuracy” here means the model is close to the labelers’ own consistency, the same situation as with real data.

Where this is used

System or taskWhat the preference data and reward model doNotes
InstructGPT (2022)Rankings of 4 to 9 answers, one 6B reward model, PPO-ptx with \(\beta = 0.02\)1.3B RLHF model preferred over 175B GPT-3 [Certain]
Llama 2-Chat (2023)1.4M Meta comparisons plus open datasets; separate helpfulness and safety reward models with margin loss; rejection sampling, then PPOReward models refreshed each round with new data [Certain]
Anthropic helpful and harmless assistant (2022)Comparisons for helpfulness and harmlessness, preference models, RLHF with PPOAlso released a public comparison dataset [Certain]
Best-of-\(n\) / rejection samplingGenerate \(n\) answers, keep the top-scored one, at inference or to build SFT dataKL only \(\log n - (n-1)/n\); a strong, simple baseline
Data filtering and evaluationScore candidate training examples or model outputs with a reward model; benchmarks such as RewardBench compare reward modelsReward-model scores are not human win rates [Likely]
Vision-language modelsPreference or correction data on image answers to reduce hallucination (for example RLHF-V)See session 74 and the VLM track [Likely]
Video understandingPreferences between captions, answers or key-moment summaries of a video; a reward model ranks candidate answers in video QA and Video-RAGReward hacking risk: long captions that list many objects look “detailed” [Speculation]

Common mistakes

  • Shuffling the pairs of one ranking as separate examples. They are correlated; InstructGPT saw overfitting within one pass. Keep all pairs of a prompt in one batch (formula 67.6).
  • Training the reward model for several epochs. Both InstructGPT and Llama 2 train about one epoch; more overfits quickly.
  • Reading the score at the wrong position. With padding, the “last position” of the tensor may be a pad token. Find the last real token from the attention mask, and make sure the end-of-turn token is included, as in the code.
  • Treating raw scores as absolute. The loss fixes only differences (67.3). Normalize scores before RL; otherwise a constant offset changes the effective reward and the balance with the KL term.
  • Trusting a rising reward curve. The proxy score always rises during RL (Figure 7). Track KL, answer length and a real held-out evaluation.
  • Setting \(\beta\) by habit. Too small and the policy hacks the reward; too large and it barely moves. Good values depend on model size and reward scale (Llama 2 halved \(\beta\) for its larger models).
  • Using a stale reward model. As the policy improves, its answers leave the reward model’s training distribution. Collect new comparisons from the current policy.
  • Different templates for policy and reward model. If the reward model was trained on another chat template or tokenizer, decode the policy’s text and re-tokenize it the reward model’s way.

Interview questions

Why does RLHF collect pairwise comparisons instead of absolute scores or more demonstrations?basic
People are much more consistent at saying which of two answers is better than at giving absolute scores, which drift between and within labelers. Comparisons are also faster than writing demonstrations, and they can teach the model to go beyond the labelers’ own writing ability, because recognizing the better answer is easier than producing the best one. The Bradley–Terry model turns comparisons into a scalar reward.
Write the reward-model loss and its gradient. What does the gradient weight mean?core
\(\mathcal{L} = -\log\sigma(r_\phi(x, y_w) - r_\phi(x, y_l))\), averaged over comparisons. With \(\Delta\) the reward gap, \(\partial\mathcal{L}/\partial\Delta = -(1 - \sigma(\Delta))\). The weight \(1 - \sigma(\Delta)\) is the probability the model gives to the wrong preference: confidently correct pairs get little gradient, mistaken pairs get a lot. The gradient raises the winner’s reward and lowers the loser’s. At initialization, \(\Delta \approx 0\) and the loss is \(\ln 2 \approx 0.693\).
The reward model only learns reward differences. Why, and what do you do about it?core
The Bradley–Terry likelihood depends only on \(r_w - r_l\), so adding a constant to all rewards (for one prompt, or globally) leaves the loss unchanged. The absolute level is arbitrary. Before RL, normalize: InstructGPT shifted the reward so human demonstrations had mean 0; Llama 2 whitened scores. This matters because PPO with a KL penalty and a value network is sensitive to the reward’s offset and scale.
How do you turn an LLM into a reward model? Where is the score read?basic
Take a pretrained or SFT model, remove the unembedding (vocabulary) layer, and add a linear head that maps a hidden state to one number. Feed prompt plus answer in the policy’s chat template and read the head at the last real token of the answer (the end-of-turn token), the only position in a causal model that has seen everything. Train with the Bradley–Terry loss for about one epoch.
InstructGPT ranked K = 4 to 9 answers per prompt. How was this used, and why put all pairs of a prompt in one batch?core
A ranking gives \(\binom{K}{2}\) pairs (6 to 36). The loss averages \(-\log\sigma(r_i - r_j)\) over all pairs. If the pairs are shuffled as independent examples, each answer is reused in \(K-1\) correlated updates and the reward model overfits within one pass. Keeping a prompt’s pairs in one batch element fixes the overfitting and needs only \(K\) forward passes instead of one per pair member. With 64 prompts per batch, that is up to 2,304 comparisons per batch.
Why is there a KL penalty to the reference model in RLHF?core
The reward model is accurate only near the distribution it was trained on: answers like the SFT model’s. Without a penalty the policy drifts to text where the reward model is wrong and exploits it (reward hacking). The KL term keeps the policy close to the reference; it also acts like an entropy bonus that prevents collapse to a single answer. In PPO it is applied per token as \(-\beta\log(\pi_\theta/\pi_{\text{ref}})\), with the reward model score added at the final token. Typical \(\beta\) values are 0.005 to 0.05.
What is the optimal policy of the KL-regularized RLHF objective?advanced
For each prompt, \(\pi^\ast(y\mid x) = \pi_{\text{ref}}(y\mid x)\exp(r(x,y)/\beta)/Z(x)\). It reweights the reference by the exponentiated reward. As \(\beta\to\infty\) it returns the reference; as \(\beta\to 0\) it puts all mass on the argmax of the reward model, including its errors. Solving it for \(r\) gives \(r = \beta\log(\pi^\ast/\pi_{\text{ref}}) + \beta\log Z\); putting this into Bradley–Terry cancels \(Z\) and gives the DPO loss.
What is reward hacking? Give examples and say how you would detect it.core
The policy increases the learned reward without increasing true quality, by exploiting the reward model’s flaws (Goodhart’s law). Examples: longer and padded answers, sycophancy and flattery, confident style over correctness, odd repeated strings off the training distribution. Detect it by watching reward together with KL and answer length, reading samples, and measuring win rates with humans or a strong judge on held-out prompts; a rising reward with flat or falling win rate is the sign.
Summarize the over-optimization results of Gao et al.advanced
A 6B gold reward model labels synthetic comparisons; proxy reward models (3M to 3B) are trained on them; policies optimize the proxy and the gold measures real quality. With \(d = \sqrt{\mathrm{KL}}\), gold reward follows \(d(\alpha - \beta d)\) for best-of-\(n\) and \(d(\alpha - \beta\log d)\) for RL: both rise, peak, then fall while the proxy keeps rising. Larger proxies have smaller \(\beta\) (they over-optimize later). Best-of-\(n\) has KL \(\log n - (n-1)/n\). RL is slower than best-of-\(n\) per unit KL, and in their setup the KL penalty did not improve the gold–KL frontier; it mostly acted like early stopping.
Your reward model has 65% held-out accuracy. Is that bad?advanced
Not necessarily. In the papers above, labelers agree with each other only about 67% to 77% of the time, depending on the task, so the ceiling is far below 100%. Under Bradley–Terry, a pair with true gap \(\Delta\) can be predicted at best with accuracy \(\sigma(|\Delta|)\). Look at accuracy split by preference strength (Llama 2: 80.7% on “significantly better”, 54.7% on “negligibly better”), compare to inter-labeler agreement, and check calibration. Then test what matters: does optimizing against it improve real win rates?
Name ways to reduce reward over-optimization.core
A KL penalty and a KL budget; early stopping using held-out human or judge evaluation; larger reward models and more data; fresh comparisons from the current policy and periodic reward-model retraining; ensembles of reward models with mean or conservative aggregation; separate reward models for separate goals (helpfulness, safety) with clear rules for combining them; length penalties or length-controlled evaluation; and sometimes replacing RL with best-of-\(n\), which is gentler per unit of KL.

Exercises

  1. A reward model gives the chosen answer 0.3 and the rejected answer 1.1. Compute the loss of formula (67.4) and \(\partial\mathcal{L}/\partial\Delta\).
  2. Labelers rank \(K = 6\) answers per prompt. How many comparisons does one prompt give? How many does a batch of 64 prompts hold?
  3. Llama 2’s small margin for “better” is \(2/3\). With \(\Delta = 0.8\), compute the margin loss of formula (67.7).
  4. What KL (in nats) does best-of-16 sampling have? What is \(d\)?
  5. An answer has 3 tokens with log-ratios \(0.2, -0.1, 0.5\) to the reference, \(\beta = 0.1\), and reward model score 1.0. Write the per-token rewards of formula (67.9) and their sum.
  6. Two labelers agree on 73% of pairs. Assume every pair has the same true reward gap and each labeler follows Bradley–Terry independently. What is the probability \(p\) that one labeler picks the truly better answer, and what reward gap is that? What accuracy can a perfect reward model reach against one labeler?
Answerscheck
1. \(\Delta = -0.8\); loss \(= -\log\sigma(-0.8) = 1.171\); \(\partial\mathcal{L}/\partial\Delta = -(1 - 0.310) = -0.690\). The model is wrong, so the gradient is large. 2. \(\binom{6}{2} = 15\) per prompt; \(64 \times 15 = 960\) per batch. 3. \(-\log\sigma(0.8 - 0.667) = -\log\sigma(0.133) = 0.629\). 4. \(\log 16 - 15/16\) \(= 2.773 - 0.938\) \(= 1.835\) nats; \(d = \sqrt{1.835} = 1.355\). 5. \(r_1 = -0.02\), \(r_2 = +0.01\), \(r_3 = -0.05 + 1.0 = 0.95\); sum 0.94 (the score minus \(0.1 \times 0.6\)). Note that a negative log-ratio gives a small positive reward. 6. Agreement \(= p^2 + (1-p)^2 = 0.73\), so \(p = (1 + \sqrt{0.46})/2 = 0.839\), a gap of \(\log(p/(1-p)) = 1.65\). A perfect reward model always picks the better answer, so it matches one labeler 83.9% of the time: higher than the 73% agreement between two labelers, because it has no noise of its own.

Key takeaways

  • RLHF collects comparisons (“which is better?”), because people compare far more consistently than they score. Real data adds rankings, preference strength and separate goals, and labelers agree only about 73% to 77% of the time.
  • The Bradley–Terry model (67.1) says preference depends on the reward difference; the reward-model loss (67.4) is binary cross-entropy on that difference. Rewards are defined only up to a shift (67.3), so normalize them.
  • A reward model is an LLM with a scalar head read at the last token, trained about one epoch; its accuracy must be compared with the labelers’ own agreement, not with 100%.
  • RLHF maximizes reward minus \(\beta\) times the KL to a frozen reference (67.8); PPO gets it as a per-token penalty plus the score at the last token (67.9). The best policy reweights the reference by \(e^{r/\beta}\) (67.10), which leads to DPO. Sampled policy-gradient steps (67.11) approach that best policy; with a flawed reward model they find its mistake, sometimes only late in training (Figure 6).
  • The reward model is a proxy. Optimizing it too hard raises the proxy score while true quality peaks and falls (Goodhart’s law; 67.12, 67.13). Watch KL, length and real evaluations, and refresh data and reward models.

Key formulas

\[ P(y_w \succ y_l) = \sigma(r_w - r_l), \qquad \mathcal{L}_{\text{RM}} = -\log\sigma\big(r_\phi(x,y_w) - r_\phi(x,y_l) - m\big) \]
\[ \max_\theta\ \E[r_\phi(x,y)] - \beta\,\KL(\pi_\theta \,\|\, \pi_{\text{ref}}), \qquad \pi^\ast \propto \pi_{\text{ref}}\, e^{r/\beta} \]
\[ \nabla_\theta J = \E_{y \sim \pi_\theta}\Big[\Big(r_\phi(x,y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} - b\Big) \nabla_\theta \log \pi_\theta(y \mid x)\Big] \]
\[ r_t = -\beta\log\frac{\pi_\theta(y_t \mid \cdot)}{\pi_{\text{ref}}(y_t \mid \cdot)} + [t = T]\, r_\phi(x, y), \qquad \KL_{\text{bon}} = \log n - \frac{n-1}{n} \]

References for this session

  1. Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences
  2. Stiennon et al. (2020). Learning to summarize from human feedback
  3. Ouyang et al. (2022). Training language models to follow instructions with human feedback
  4. Gao et al. (2022). Scaling Laws for Reward Model Overoptimization
  5. Lambert (2025). Reinforcement Learning from Human Feedback (book)
  6. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  7. Ziegler et al. (2019). Fine-Tuning Language Models from Human Preferences
  8. Bradley & Terry (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons (Biometrika)
  9. Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  10. Singhal et al. (2023). A Long Way to Go: Investigating Length Correlations in RLHF
  11. Coste et al. (2023). Reward Model Ensembles Help Mitigate Overoptimization
  12. Lambert et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling
  13. Yu et al. (2023). RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
  14. Kaufmann et al. (2023). A Survey of Reinforcement Learning from Human Feedback
  15. Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model