Reward models, deeper: outcome rewards, process rewards, verifiers
RL can only be as good as its reward. This session compares the four main sources of reward for an LLM answer: outcome reward models that score the whole answer, process reward models that score every step, verifiers that check math answers and run code tests, and LLM judges with rubrics. It ends with calibration and with the ways a policy learns to fool a reward.
What you will learn
- How an outcome reward model is trained from automatic correctness labels, and why it accepts right answers reached by wrong reasoning
- How a process reward model scores each step, how PRM800K was labeled, and how product and minimum turn step scores into one score
- How Math-Shepherd labels steps with Monte Carlo completions, and why these labels are noisy in a predictable way
- Verifiers for math and code, the pass@k estimator, and why a verifier’s false-positive rate caps best-of-N accuracy
- LLM-as-a-judge: position, length and self-preference biases, order swapping, rubric rewards and generative verifiers
- Calibration of reward models (ECE, temperature scaling) and robustness: distribution shift, spurious features, overoptimization and the usual defenses
Why this matters
Every RL method in this track needs one thing: a number that says how good an answer is. In session 67 that number came from a reward model trained on human comparisons. This session looks closer at where the reward comes from, and at what goes wrong when the reward is wrong.
There are four main sources of reward for an LLM answer. An outcome reward model (ORM) reads the whole answer and gives one score. A process reward model (PRM) gives a score to every step of a reasoning chain. A verifier is a program that checks the answer: it compares a math answer with the known one, or runs unit tests on code. An LLM judge is another language model that reads the answer, often with a list of criteria (a rubric), and gives a score. Real systems mix them. DeepSeek-R1 used rule-based rewards (a verifier plus a format check) for math and code, and a learned reward model only for general chat data. [Certain]
The choice matters for two reasons. First, at test time a reward model picks the best of many samples (best-of-N, session 72); a better scorer gives a better pick. Second, during RL the policy will find and use every flaw in the reward. A reward that is 95% right is fine for ranking a few answers, but RL searches millions of answers and finds the 5%. This is reward hacking, and it is the main reason why reasoning models today rely on verifiers wherever they can.
Four ways to score an answer
All four sources answer the same question: given a prompt \(x\) and an answer \(y\), what is the reward \(r(x, y)\)? They differ in what they look at (only the end, or every step), how they are built (trained on labels, or written as code), and how they fail.
Recall the reward model from session 67. It is an LLM with a scalar head on the last token, trained on pairs (chosen answer \(y_w\), rejected answer \(y_l\)) with the Bradley–Terry model:
This is already an outcome reward model: it scores the whole answer. Its score has no fixed meaning; only differences matter. The reward models in this session are different in one important way: most of them predict a probability of being correct. That gives the score a meaning (“this answer is right with probability 0.8”), and it lets us ask if the model is calibrated, which we do at the end.
Outcome reward models
For math and code there is often a known final answer. Then we do not need human comparisons: we can label each sampled solution as correct or wrong automatically, and train a classifier on those labels. This is the outcome-supervised reward model. It was introduced as a “verifier” by Cobbe et al. (2021) together with the GSM8K dataset (8.5K grade-school math problems: 7.5K for training, 1K for testing). [Certain]
The recipe: fine-tune a generator, sample many solutions per training problem (Cobbe et al. used 100), mark each one correct if its final answer matches the reference, and train a model to predict that label. With \(z = 1\) for correct and \(z = 0\) for wrong, the loss is binary cross-entropy:
- Read the symbols
- \(x, y\)
- the problem and one sampled solution
- \(z\)
- the label: 1 if the final answer of \(y\) is right, 0 if not (found by a program, not a person)
- \(r_\phi(x, y)\)
- the reward model’s logit, read at the last token; \(\sigma(r_\phi)\) is its predicted probability that \(y\) is correct
At test time we sample \(N\) solutions and return the one with the highest score. Cobbe et al. found that a 6B model with a verifier did about as well as a fine-tuned model 30 times larger, and that the gain grows with more training data. [Certain] They also found two details that matter in practice. A token-level verifier, which predicts “will this solution end correct?” after every token (a value function, session 65), worked better than one prediction at the end. And with more samples, performance rose up to about 400 solutions per problem and then fell: with enough tries, the generator finds wrong solutions that fool the verifier. [Certain] That is reward hacking at test time, before any RL.
The ORM says \(\sigma(r) = 0.8\) for a solution. If the solution is correct, the loss is \(-\log 0.8 = 0.223\). If it is wrong, the loss is \(-\log 0.2 = 1.609\), about 7 times larger. The gradient pushes the score of wrong solutions down hard.
The weak spot: right answer, wrong reasoning
The label \(z\) only checks the final answer. A solution with a wrong step that still reaches the right number gets \(z = 1\). Such false positives are labeled as good, so the ORM learns to accept them. Lightman et al. (2023) point out that this automatic grading is “not perfectly reliable” for this reason. [Certain] For math tests, a lucky right answer counts the same. But in RL, the policy is rewarded for a flawed method, and for tutoring or proofs the reasoning is the whole point.
Process reward models
A process reward model scores each step. We split a solution into steps \(s_1, \dots, s_K\) (in practice, usually one step per line or paragraph). After the last token of step \(k\), the PRM predicts \(p_k\): the probability that step \(k\) is correct, given the problem and the steps before it.
Lightman et al. (2023), “Let’s Verify Step by Step”, trained a PRM on PRM800K: 800K step labels from people, on 75K solutions to 12K MATH problems. [Certain] Each step was labeled positive (correct and useful), negative (wrong or unreasonable) or neutral (unclear). They labeled each solution only up to its first wrong step: for a wrong solution, this tells exactly where the mistake is, and nothing after it is needed. The PRM is trained like a language model: after each step it predicts one label token, and we maximize the log-likelihood of the right label. With two labels this is cross-entropy over steps, up to the first error:
- Read the symbols
- \(p_k\)
- the PRM’s probability that step \(k\) is correct, read at the end of step \(k\); one forward pass gives all \(p_k\) at once
- \(z_k\)
- the step label: 1 for good (positive or neutral), 0 for the first wrong step
- \(K'\)
- the number of labeled steps: all \(K\) for a correct solution, up to and including the first error otherwise
From step scores to one solution score
To rank whole solutions we need one number per solution. Two common choices:
The product is “the probability that every step is correct” if step errors were independent; Lightman et al. used it. The minimum says “a solution is as good as its weakest step”; Math-Shepherd used it. In Lightman et al. the difference was small: 78.2% with the product and 77.6% with the minimum. [Certain] One trap: the product shrinks with length. Ten steps at 0.95 give \(0.95^{10} = 0.599\), but three steps at 0.95 give \(0.857\). Ranking by product favors short solutions; the minimum does not have this problem. A third choice, the score of the last step only, turns the PRM back into an outcome scorer.
Figure 2 uses the problem “A train goes 120 km in 2 hours, then 80 km in 2 hours. What is its average speed?” (answer: 50 km/h). Solution B gets the right answer by averaging the two speeds. That method is wrong in general (it works here only because the two times are equal), so a good PRM gives that step a low score. The step probabilities are made-up teaching values, not output of a real model.
How to read Figure 2
Compare the three solutions under each aggregation. The outcome check (does the final answer equal 50?) gives A and B the same reward, 1. The PRM product gives A 0.858 and B 0.284, because step 3 of B is flawed. “Last step” gives B 0.90: it looks only at the end, so it misses the flaw, like an ORM. For Solution C the final answer 40 is wrong, and the PRM also says where: step 2 (2 + 2 = 5) has the lowest score. Mark more steps as errors and see that the minimum only reacts to the worst step, while the product reacts to every one.
Solution A, steps 0.97, 0.96, 0.95, 0.97: product \(= 0.858\), minimum \(= 0.95\). Solution B, steps 0.95, 0.95, 0.35, 0.90: product \(= 0.284\), minimum \(= 0.35\). Solution C, steps 0.97, 0.08, 0.60, 0.85: product \(= 0.040\), minimum \(= 0.08\). Both PRM rules rank \(\text{A} \gt \text{B} \gt \text{C}\). The outcome check says \(\text{A} = \text{B} = 1 \gt \text{C} = 0\). Only the PRM separates sound reasoning from a lucky answer.
Does process supervision help?
Two studies compared the two kinds of supervision. On GSM8K, Uesato et al. (2022) found that outcome supervision gives a similar final-answer error rate with less labeling, but process supervision (or a reward model that imitates it) was needed to get the reasoning right: compared with the previous best, final-answer error went from 16.8% to 12.7%, and the reasoning error among solutions with a correct final answer went from 14.0% to 3.4%. [Certain] On the harder MATH dataset, with a large GPT-4-based generator, Lightman et al. found a clear win for process supervision in best-of-N selection:
| Method (best of 1,860 samples, 500 MATH test problems) | % solved |
|---|---|
| Majority vote (most common final answer) | 69.6 |
| Outcome reward model (ORM) | 72.4 |
| Process reward model (PRM) | 78.2 |
Source: Lightman et al. (2023), main comparison of the large-scale reward models. [Certain] Note what this measures: picking one answer from many. It does not show that a PRM is a better reward for RL; that is a different question, discussed below.
Step labels without people: Monte Carlo estimation
Human step labels are slow and costly. Math-Shepherd (Wang et al., 2023) labels steps automatically. The idea: a step is good if the solution can still reach the right answer from there. So, for each prefix \(s_1, \dots, s_k\), let a “completer” model finish the solution \(N\) times, and check each final answer \(a_j\) against the reference \(a^*\):
- Read the symbols
- \(N\)
- the number of completions per step; Math-Shepherd used \(N = 8\)
- \(y_k^{\text{HE}}\)
- the hard estimate: 1 if at least one completion reaches the right answer
- \(y_k^{\text{SE}}\)
- the soft estimate: the fraction of completions that reach it, an estimate of the step’s value \(V(s_{1:k})\)
Math-Shepherd trained its PRM on the hard estimate and used the minimum over steps (73.4). It improved Mistral-7B with step-by-step PPO from 77.9% to 84.1% on GSM8K, and to 89.1% with PRM reranking on top. [Certain] Note what this label really is: not “is this step correct?” but “can a completer still get the right answer from here?”. These two differ, and Figure 3 shows how.
How to read Figure 3
The grey bars are the true chance of finishing correctly, the orange dots are the soft estimates from \(N\) completions, and the colored squares are the hard labels. With \(N = 8\), the wrong step 4 gets the hard label 1 (“good”) with probability \(1 - 0.96^8 = 0.279\): more than one in four bad steps is labeled good. Raise \(N\): the soft estimate gets more accurate (its noise falls like \(1/\sqrt{N}\)), but the hard label gets worse, because more tries means more lucky recoveries. At \(N = 64\) the bad step is labeled good 92.7% of the time.
Two more problems with Monte Carlo labels. A strong completer can recover from a bad step (it fixes the error and moves on), so the step looks good. A weak completer fails even after a good step, so the step looks bad. Zhang et al. (2025, Qwen team) found that Monte Carlo labels gave PRMs with worse performance and generalization than labels from an LLM judge or from people, and that many PRMs give their lowest score mostly to the final step, so they act more like outcome scorers. They proposed keeping only steps where the Monte Carlo label and an LLM judge agree. [Certain]
Using a PRM: reranking, search and dense rewards
A PRM can be used in three ways. Reranking: score \(N\) finished solutions with (73.4) and keep the best, as in the table above. Search: because the PRM scores partial solutions, we can prune bad paths early. Snell et al. (2024) used PRM-guided beam search to spend test-time compute more efficiently than best-of-N on some problems. [Likely] Dense rewards in RL: give a reward after every step, not only at the end, so the policy gets credit for the right steps (Math-Shepherd’s step-by-step PPO).
- input: problem \(x\), policy \(\pi\), PRM, beam width \(B\), expansions per beam \(M\), max steps \(K\)
- beams \(\leftarrow\) \(\{\,\text{empty solution}\,\}\)
- for \(k = 1\) to \(K\):
- candidates \(\leftarrow\) for each beam, sample \(M\) next steps from \(\pi\) // stop a step at a newline
- score each candidate prefix with the PRM: \(p_k\) for its new step, solution score \(\min(p_1, \dots, p_k)\)
- beams \(\leftarrow\) the \(B\) best candidates // bad paths are dropped early
- move finished candidates (with a final answer) to the output list
- return the finished solution with the highest PRM score (or a vote weighted by the scores)
Why is the PRM not used everywhere in RL? DeepSeek-R1 lists three reasons in its “unsuccessful attempts”: it is hard to define what a “step” is in general reasoning; it is hard to know if an intermediate step is correct (models label badly, and people do not scale); and a model-based PRM “inevitably leads to reward hacking”, while retraining it costs compute and makes the pipeline complex. They concluded that a PRM is good for reranking and guided search, but its benefit in large-scale RL was small compared with its cost. [Certain] This is one reason the GRPO pipelines of session 70 use outcome rewards from verifiers.
Which step gets the credit? Outcome versus process rewards in RL
The costs above are real. So what does a dense, per-step reward buy? The answer is credit assignment: telling the policy which step was good or bad. Write a solution as actions \(a_1, \dots, a_K\) (one per step). The policy gradient with a per-step advantage \(A_t\) is
- Read the symbols
- \(R\)
- the outcome reward: 1 if the final answer is right, else 0 (from a verifier or an ORM)
- \(s_t\)
- the process reward of step \(t\): the PRM’s verdict on that step (1 = looks correct, 0 = looks wrong)
- \(b, b_t\)
- baselines: here the mean of the other samples in the batch, for the whole answer or for step \(t\)
With an outcome reward, every step of a solution gets the same advantage: a right answer praises its wrong steps too, and a wrong answer blames its right steps. With a process reward, each step gets its own signal. Figure 4 trains two policies on a toy reasoning chain. Each step is a yes/no action: “correct step” with probability \(p_t = \sigma(\theta_t)\). The final answer is right if every step is right; if some step is wrong, it is still right with probability “lucky guess” (the two-mistakes-that-cancel case of the ORM section). Step 3 is hard (\(p_3 = 0.30\) at the start); the others start at 0.75. In this toy the steps are independent, so a step’s own score is the right credit; in real chains a step also changes what comes after it, and the PRM reward is usually summed over the remaining steps.
- each learner samples 8 chains \(a_{i,1..K}\), step \(t\) correct with \(p_t = \sigma(\theta_t)\)
- outcome \(R_i = 1\) if the final answer is right (all steps right, or a lucky guess)
- the PRM scores every step: \(s_{i,t}\) // flipped with prob. “PRM label noise”
- ORM credit: \(A_{i,t} = R_i - \frac{1}{7}\sum_{j \ne i} R_j\) for every step
- PRM credit: \(A_{i,t} = s_{i,t} - \frac{1}{7}\sum_{j \ne i} s_{j,t}\) // step by step
- \(\theta_t \mathrel{+}= \eta \cdot \frac{1}{8}\sum_i A_{i,t}\,(a_{i,t} - p_t)\) // (73.6)
How to read Figure 4
Use speed 1 and Next line for one iteration. After line 4, look at the ORM grid: a whole row is blue or red. A “lucky” row is blue on its wrong steps too, so the ORM learner is rewarded for a wrong step; a row that failed only at step 3 gets the same negative (or zero) credit on its seven good steps. After line 5, the PRM grid is colored step by step: only the ✗ cells are red. Then Play at speed 4. With the defaults (8 steps, lucky-guess rate 0.20, seed 1) both start with a 23.2% chance of a right answer (only 4.0% from sound reasoning). After 25 iterations the PRM learner is at 74.7% and the ORM learner at 32.6%; after 50 iterations, 87.4% against 55.6%. Both get there in the end (97.8% and 96.3% at 300): the outcome reward is not wrong, only slow and noisy, because a step’s credit is mixed with the luck of all the other steps. Set the lucky-guess rate to 0: the ORM learner gets almost no signal at first, because almost every chain fails (19.6% after 25 iterations). Add PRM label noise 0.10: the PRM learner slows down a little (70.2% after 25 iterations) but still leads. A learned PRM that can be fooled is another matter; that is the reward hacking DeepSeek-R1 warned about. (Speeds 1 and 2 run one line of the algorithm at a time, 0.65 s or 0.2 s per line; speeds 3 to 5 run whole iterations, more of them per frame. “Next line” and the next-step button work at any speed.)
Verifiers for math and code
A verifier here means a program, not a network. For math it extracts the final answer (for example the text inside \boxed{...}) and checks it against the reference, after normalizing the format (0.5, 1/2 and 2/4 are the same). For code it runs the program on hidden unit tests in a sandbox. The reward is exact:
- Read the symbols
- \(\text{extract}(y)\)
- the part of the answer that is checked: the boxed number, or the code block
- \(a^*_x\)
- the reference: the known answer, or the test cases for problem \(x\)
- \(\lambda_{\text{fmt}}\)
- the weight of an optional format reward (for example, thinking inside think tags); DeepSeek-R1-Zero used accuracy rewards and format rewards [Certain]
This is the reward of RL with verifiable rewards (RLVR). It has no learned parameters, so the policy cannot make it give a high score to a wrong answer by finding a weak spot in a network. But a verifier can still be wrong, in two directions. A false negative rejects a correct answer: the policy wrote “\(x = 0.50\)” but the parser expected “1/2”, or the answer was right but not boxed. A false positive accepts a wrong answer: the unit tests are too weak, so a program that returns a constant passes them. With RL, false positives are the dangerous ones, because the policy learns to produce them.
pass@k: how verifiers are used to measure
Code models are measured with pass@k: the chance that at least one of \(k\) samples passes all tests. Sampling exactly \(k\) answers gives a noisy estimate. Chen et al. (2021, Codex) sample \(n \ge k\) answers, count the \(c\) correct ones, and use the unbiased estimator:
The fraction is the chance that a random set of \(k\) of the \(n\) samples contains only wrong ones. With \(n = 10\) and \(c = 3\): pass@1 \(= 0.3\) and pass@5 \(= 1 - \binom{7}{5}/\binom{10}{5} = 1 - 21/252 = 0.917\). The naive \(1 - (1 - 0.3)^5 = 0.832\) is a different (biased) number, because it samples with replacement.
Best-of-N with an imperfect verifier
pass@N assumes a perfect checker. With a real verifier, what accuracy do we get if we sample \(N\) answers and return one that passes? Say each sample is correct with probability \(p\). The verifier accepts a correct answer with probability \(a\) (its recall) and a wrong answer with probability \(b\) (its false-positive rate). We return a random accepted answer, or a random answer if none is accepted. Then:
- Read the symbols
- \(s\)
- the chance that one sample is accepted
- \(q_0\)
- the chance that none of the \(N\) samples is accepted
- precision
- the chance that an accepted sample is really correct; as \(N\) grows, \(q_0 \to 0\) and the accuracy goes to this number
The lesson: with more samples, the accuracy does not go to 1. It goes to the verifier’s precision. The false-positive rate \(b\) sets the ceiling, and it matters more when the policy is weak (small \(p\)), because then most samples are wrong and a few of them slip through.
How to read Figure 5
With the defaults (\(p = 0.3\), \(a = 0.9\), \(b = 0.05\)), 16 samples give 0.883 while a perfect checker would give 0.997, and no number of samples gets past the precision 0.885. Move \(b\) to 0.2: the ceiling drops to 0.659. Move \(p\) down to 0.05 with \(b = 0.05\): the ceiling is 0.486. Then set \(b = 0\): the green curve joins the grey one. Exact checks, or very strong tests, are what make verifiers so valuable.
- input: answer \(y\), problem type, reference \(a^*\) (answer or tests), time limit
- if the required format is missing then return 0 // no final answer, no code block
- if math: \(\hat a \leftarrow\) last boxed expression; normalize both (strip spaces, units, commas; parse numbers, fractions, or symbolic expressions)
- return 1 if \(\hat a\) equals \(a^*\) (exactly, or symbolically, or within a tolerance) else 0
- if code: write the code to a sandbox with no network and limited memory
- run the hidden tests with the time limit; a timeout or crash counts as a fail
- return 1 if all tests pass else 0 // or the fraction passed, as a softer reward
- log every rejected answer that looks correct, and check samples by hand // find false negatives early
LLM-as-a-judge and rubric rewards
Most useful tasks have no exact answer: write an email, explain an idea, summarize a video. For them, a strong LLM can act as the scorer. It gets the prompt, the answer (or two answers) and instructions, and outputs a score or a preference. This is LLM-as-a-judge. Zheng et al. (2023) found that GPT-4 as a judge agreed with human preferences over 80% of the time, about the same rate at which humans agree with each other. [Certain] When the judge’s outputs are used to train the policy, this is RL from AI feedback (session 74).
There are two formats. Single-answer grading: “score this answer from 1 to 10”. Pairwise comparison: “which of answers A and B is better?”. Zheng et al. also listed the judge’s main biases: [Certain]
- Position bias: preferring the answer shown first (or second), regardless of quality.
- Verbosity bias: preferring longer answers, even when the extra text adds nothing.
- Self-enhancement bias: preferring answers written by the judge model itself, or by similar models.
A simple model of position bias: let \(d\) be the true quality gap (in logit units) between answers A and B, and \(\beta\) an extra push toward whichever answer is shown first. Then
The fix is to ask twice, with the order swapped, and average (or count a win only if both orders agree). When \(d = 0\), the swap average is exactly 0.5: the bias cancels. A low consistency rate (the fraction of pairs where both orders give the same winner) is a warning that the judge is guessing.
How to read Figure 6
With \(d = 0.5\) (A is somewhat better) and \(\beta = 1\), A wins 81.8% of the time when shown first, but only 37.8% when shown second: in that order the judge picks the worse answer most of the time. The swap average, 59.8%, is on the right side of 0.5. The two orders agree only 42.2% of the time. Set \(d = 0\): one order alone says 73.1% for A, a pure artifact, while the swap average is exactly 50%. The swap average is still pulled toward 0.5 compared with \(\sigma(d)\), so a biased judge gives weaker, but not wrong-signed, signals.
Rubric rewards
A single score from 1 to 10 is vague: the judge may weigh things differently from one answer to the next. A rubric breaks the score into explicit checks. For the prompt “Explain why the sky is blue to a 10-year-old”, a rubric might be: (1) mentions that sunlight contains many colors, weight 3; (2) says blue light is scattered more, weight 2; (3) uses no technical words without explaining them, weight 2; (4) is under 150 words, weight 1. The judge answers each check with yes or no, and the reward is the weighted fraction:
If an answer passes checks 1, 2 and 4 but fails 3, the reward is \((3 + 2 + 0 + 1)/8 = 0.75\). Some checks (like check 4) can be done by code, which removes judge noise for them. Gunjal et al. (2025, “Rubrics as Rewards”) wrote a rubric for each prompt and used it as the RL reward on medical and science questions. Their best version beat a judge that gave direct 1-to-10-style (Likert) scores by up to 31% (relative) on HealthBench, and the rubric reward worked better with small judge models. [Certain] Rubrics also make the reward easier to read: when the reward goes up, you can see which checks improved.
Generative verifiers
An LLM judge can also be trained, not only prompted. A generative verifier (GenRM, Zhang et al., 2024) is an LLM fine-tuned to answer “Is the answer correct? Yes or No”, optionally after writing a short chain of thought about the solution. Its reward is the probability of the token “Yes”:
With \(m = 1\) and no critique, this is a classifier that reuses the LM head. With \(m \gt 1\) sampled critiques, the scores are averaged, a form of majority voting for the verifier. Zhang et al. report best-of-N on GSM8K going from 73% to 93.4% with this approach. [Certain] The cost: a verifier that writes critiques is much slower than a scalar head.
Calibration and robustness
A reward model that outputs probabilities should mean what it says. It is calibrated if, among all answers it scores 0.8, about 80% are really correct. Calibration matters whenever we use the score as a number and not only to rank: when we set a threshold (“accept answers with score above 0.9”), when we combine it with other rewards, when we decide if more samples are needed, and in RL, where an overconfident reward model gives big rewards to answers that do not deserve them.
The usual measure is the expected calibration error (ECE). Put the predictions into \(M\) bins by their value (for example 0–0.1, 0.1–0.2, …), and compare, in each bin, the average prediction with the fraction that is really correct:
- Read the symbols
- \(B_m\)
- the set of answers whose predicted probability falls in bin \(m\); \(|B_m|\) is its size, and \(n\) the total number of answers
- \(\operatorname{acc}(B_m)\)
- the fraction of answers in the bin that are really correct
- \(\operatorname{conf}(B_m)\)
- the average predicted probability in the bin
Large networks are often overconfident: their probabilities are too close to 0 and 1. A one-parameter fix is temperature scaling (Guo et al., 2017): divide the logit by a temperature \(T\), fitted on held-out data by minimizing the log loss:
\(T \gt 1\) softens overconfident scores. Dividing by a positive number does not change the order of the scores, so best-of-N picks and ranking accuracy stay the same; only the probabilities change.
How to read Figure 7
At \(T = 1\), the top bin has an average prediction of 0.987 but only 89.7% of its answers are correct, and the bottom bin says 0.013 while 10.5% are correct: the bars lie below the diagonal on the right and above it on the left. ECE is 0.099 and the log loss is 0.531. Move \(T\) to 2.5: the bars move onto the diagonal, ECE drops to 0.003 and the log loss to 0.381. “Fit T” finds \(T = 2.45\), the value that minimizes the log loss on this data. Note the bins near 0 and 1 empty out as \(T\) grows: the model stops making extreme claims.
Robustness: the reward is a proxy
Calibration on a fixed test set is not enough. During RL, the policy changes, and the reward model sees answers unlike its training data. Three things go wrong, and you have met each of them already:
- Distribution shift. The reward model was trained on samples from an older policy. After some RL steps, the policy writes new kinds of answers, and the reward model’s accuracy (and calibration) on them is unknown.
- Spurious features. The reward model learns easy cues that went with good answers in its data: length, confident tone, markdown lists, phrases like “Let me double-check”. The policy then makes these cues stronger without being more correct. Length is the best-known case (session 67).
- Overoptimization. Gao et al. (2022) showed that as the policy is pushed further from its start (measured by KL), the proxy reward keeps rising while the true (“gold”) reward rises, peaks and then falls. [Certain] DeepSeek-R1 saw this in practice: with a learned helpfulness reward model, the reward kept going up while Codeforces performance went down. [Certain]
The common defenses keep the policy where the reward is trustworthy, or make the reward harder to fool:
- Read the symbols
- \(r_1, \dots, r_E\)
- an ensemble of \(E\) reward models (different seeds or data); \(\bar r\) is their mean
- \(\lambda \operatorname{Var}_e[r_e]\)
- a penalty when the ensemble members disagree, which happens on answers unlike the training data
- \(\beta \KL\)
- the usual penalty for moving far from the reference policy (session 66)
Coste et al. (2023) found that ensembles with such conservative objectives reduce overoptimization. [Likely] (Formula 73.15 shows the idea; the exact penalty in their paper may differ.) Other tools: normalize rewards per prompt (GRPO does this, session 70); add an explicit length penalty or length-controlled evaluation; combine a verifier with a reward model, so that correctness is exact and only style is learned; retrain the reward model on samples from the current policy (iterated RLHF, as in Llama 2); and stop training early, using held-out human or verifier checks, not the training reward. Static benchmarks such as RewardBench (Lambert et al., 2024) compare reward models on fixed chosen/rejected pairs; a higher score there is useful, but it does not guarantee a better policy after RL. [Likely]
Watching overoptimization happen: best-of-N against a hackable reward model
Best-of-N is the simplest optimizer of all: sample \(N\) answers and keep the one with the highest proxy score,
- Read the symbols
- \(\hat r\)
- the proxy score: a learned reward model, or the mean of an ensemble (73.15)
- \(r^\ast\)
- the true quality, which the selection never sees
- \(\text{gap}(N)\)
- how much the chosen answer’s proxy score overstates its true quality; it grows with \(N\) because the argmax prefers answers whose error happens to be large and positive
Figure 8 simulates it. Each answer has a true quality \(r^\ast \sim \mathcal{N}(0, 1)\), and the reward model sees \(\hat r = r^\ast + \text{noise}\), with noise of standard deviation \(\sigma\). A small fraction of answers are exploits: they use a trick the reward model likes (a confident tone, a format, a phrase), so their true quality is low (\(r^\ast \sim \mathcal{N}(-1.5, 0.5^2)\)) but their proxy score is high (\(\hat r \sim \mathcal{N}(H, 1)\) plus the noise). The ensemble averages the noise of several reward models, but every member is fooled by the same trick, because they learned from the same data.
- sample 256 answers: true quality \(r^\ast(y_i)\); a fraction are exploits
- score them: \(\hat r(y_i)\) from one reward model, and the ensemble mean
- for \(N = 1, 2, 4, \dots, 256\): \(\hat y_N \leftarrow \argmax_{i \le N} \hat r(y_i)\) // (73.16), also for the ensemble and for \(r^\ast\) itself
- add \(\hat r(\hat y_N)\) and \(r^\ast(\hat y_N)\) to the running means
How to read Figure 8
Press Play and let the means settle. With the defaults (\(\sigma = 0.5\), 2% exploits with \(H = 2\), seed 1), after 2,000 trials the proxy score of the pick keeps rising with \(N\): 0.01, 2.13 and 3.56 at \(N = 1, 16, 256\). The true quality rises, peaks at \(N = 32\) (1.16) and falls to 0.48 at \(N = 256\), because 52.6% of the picks at \(N = 256\) are exploits (right chart). The oracle keeps improving (2.83 at 256): the good answers are there, the scorer just cannot find them. This is the shape of formula (67.12) in session 67, produced here by a mechanism you can see. The ensemble removes some noise, so it is better at small \(N\) (1.24 at \(N = 16\)), but it is fooled by the shared exploit even more reliably, and at \(N = 256\) it is worse (0.28). Set the exploit rate to 0: there is no peak any more, and the true quality at \(N = 256\) is 2.55 (ensemble 2.75); noise alone makes best-of-N less efficient, but not harmful. Raise it to 0.05: the peak moves to \(N = 8\) and the true quality at 256 is −0.70.
Putting it together in PyTorch
The code below builds a math verifier and a code verifier, the pass@k estimator, and then trains the same small network twice on toy “solutions”: once with the ORM loss (73.2) and once with the PRM loss (73.3). Each toy solution has 6 steps; a wrong step carries a hidden signal, and every step after the first error is wrong. It runs on a CPU in under a minute.
import math, re, subprocess, sys, torch from fractions import Fraction torch.manual_seed(0) # 1) a rule-based math verifier: extract the final answer, compare exactly def extract(text): m = re.findall(r"\\boxed\{([^{}]*)\}", text) return m[-1].strip() if m else None # no boxed answer -> no reward def to_num(s): try: return Fraction(s.replace(",", "").replace(" ", "")) except (ValueError, ZeroDivisionError): return None def math_reward(text, gold): a = extract(text) if a is None: return 0.0 x, y = to_num(a), to_num(gold) return float(x is not None and x == y) # 0.5 == 1/2 == 2/4 print(math_reward(r"... so the speed is \boxed{50}", "50"), math_reward(r"\boxed{1/2}", "0.5"), math_reward("the answer is 50", "50")) # prints 1.0 1.0 0.0 (the last one is a false negative: right, but not boxed) # 2) a code verifier: run hidden unit tests in a separate process with a timeout def code_reward(src, tests, timeout=2.0): prog = src + "\n" + "\n".join(f"assert {t}" for t in tests) + "\nprint('OK')" try: r = subprocess.run([sys.executable, "-c", prog], capture_output=True, text=True, timeout=timeout) return float(r.returncode == 0 and r.stdout.strip().endswith("OK")) except subprocess.TimeoutExpired: return 0.0 good = "def add(a, b):\n return a + b" hack = "def add(a, b):\n return 5" # passes a weak test set print(code_reward(good, ["add(2, 3) == 5"]), code_reward(hack, ["add(2, 3) == 5"]), code_reward(hack, ["add(2, 3) == 5", "add(-1, 1) == 0"])) # prints 1.0 1.0 0.0 (one test: a false positive; two tests: caught) # 3) the unbiased pass@k estimator (formula 73.8) def pass_at_k(n, c, k): if n - c < k: return 1.0 return 1.0 - math.comb(n - c, k) / math.comb(n, k) print(round(pass_at_k(10, 3, 1), 4), round(pass_at_k(10, 3, 5), 4)) # 0.3 0.9167 # 4) ORM vs PRM on toy "solutions": 6 steps, each step is a feature vector. # A wrong step has a shift along a hidden direction; the solution is correct # only if no step is wrong. Same network, two different losses. S, D, N = 6, 16, 4000 bad_dir = torch.randn(D); bad_dir /= bad_dir.norm() def make(n): x = torch.randn(n, S, D) first = torch.randint(0, 2 * S, (n,)) # first error position; >= S means no error err = torch.arange(S)[None] >= first[:, None] # every step after the first error is "wrong" x[:, :, :] += 3.0 * bad_dir * (torch.arange(S)[None] == first[:, None]).float()[..., None] step_ok = (~err).float() # PRM target: prefix is still correct final_ok = (first >= S).float() # ORM target: whole solution correct return x, step_ok, final_ok, first class RM(torch.nn.Module): def __init__(s): super().__init__(); s.rnn = torch.nn.GRU(D, 32, batch_first=True); s.head = torch.nn.Linear(32, 1) def forward(s, x): return s.head(s.rnn(x)[0]).squeeze(-1) # one logit per step bce = torch.nn.functional.binary_cross_entropy_with_logits def train(kind): torch.manual_seed(1); m = RM(); opt = torch.optim.AdamW(m.parameters(), lr=3e-3) x, step_ok, final_ok, first = make(N) for it in range(300): logit = m(x) if kind == "orm": loss = bce(logit[:, -1], final_ok) # formula 73.2: last step only else: # formula 73.3: every step, mask = (torch.arange(S)[None] <= first[:, None]).float() # up to the first error loss = (bce(logit, step_ok, reduction="none") * mask).sum() / mask.sum() opt.zero_grad(); loss.backward(); opt.step() return m torch.manual_seed(2); xt, st, ft, first_t = make(2000) for kind in ["orm", "prm"]: m = train(kind) with torch.no_grad(): p = torch.sigmoid(m(xt)) score = p[:, -1] if kind == "orm" else p.min(1).values # formula 73.4 (min) for the PRM acc_final = ((score > 0.5).float() == ft).float().mean() # is the solution correct? wrong = first_t < S locate = (p[wrong].argmin(1) == first_t[wrong]).float().mean() # where is the first error? print(kind, "final acc", round(acc_final.item(), 3), "finds first error", round(locate.item(), 3)) # on our CPU run: orm final acc 0.797, finds first error 0.178 (chance is 1/6 = 0.167) # prm final acc 0.84, finds first error 0.908
Read the last two lines. Both models learn to judge the whole solution, but only the PRM learns where the error is: the ORM’s per-step outputs are no better than a random guess, because it was never trained on them. The PRM gets the location right 91% of the time, and its solution-level accuracy is also higher here, because the per-step labels give it more signal from the same data. (Exact values can differ slightly across PyTorch versions and machines.)
Where this is used
| Setting | Reward source | Why |
|---|---|---|
| RL for math reasoning (DeepSeek-R1-Zero, RLVR) | Verifier on the final answer + format reward | Exact, cannot be hacked by fooling a network; no labels to collect [Certain] |
| RL for code | Hidden unit tests in a sandbox (pass or fraction passed) | Execution is the ground truth; the risk is weak tests |
| Best-of-N and search at test time | ORM, PRM (min or product), or a generative verifier | Picks the best of many samples; PRMs can prune partial solutions (session 72) |
| Chat and writing quality in RLHF | Bradley–Terry reward model, often with a length penalty | No exact answer; human preference is the target |
| Open-ended expert answers (medicine, science) | LLM judge with a per-prompt rubric | Explicit criteria give a more stable reward than one vague score [Certain for Gunjal et al.] |
| Model evaluation (MT-Bench, Arena-style) | Pairwise LLM judge with order swapping | Cheap proxy for human votes; over 80% agreement in Zheng et al. |
| Video question answering and temporal grounding | Verifier: multiple-choice answer match, or IoU between the predicted and true time span as the reward | Gives RLVR-style rewards to video LLMs without a learned reward model [Likely] |
| Video captioning and summarization, Video-RAG answers | LLM or VLM judge with a rubric: mentions key events, no made-up objects, cites the right clip | No single correct caption; rubric checks target faithfulness [Likely] |
| Key-moment detection, action recognition labels | Verifier against annotated timestamps or class labels | The label set is fixed, so the reward is exact [Likely] |
Common mistakes
- Trusting a high training reward. The reward going up only means the policy is good at the proxy. Track a held-out signal (verifier, human check, a different judge) and stop when it stops improving.
- A strict parser that rejects correct answers. If “0.50”, “1/2” and “\(\tfrac{1}{2}\)” are not all accepted, the policy learns a format, not math. Read a sample of rejected answers by hand before a long run.
- Weak unit tests. A few tests let constant or special-cased programs pass. Use many tests, edge cases, and random inputs checked against a reference solution.
- Ranking PRM scores by product across very different lengths. The product (73.4) punishes long solutions; use the minimum, or compare solutions of similar length.
- Using only one order in pairwise judging. Position bias can flip the result. Always swap and average, and report the consistency rate.
- Treating a reward model score as a probability without checking. Measure ECE on held-out data and fit a temperature before using thresholds.
- Never retraining the reward model. After the policy moves, the reward model’s training data no longer matches what it scores. Refresh it with new policy samples, or rely on a verifier where possible.
- Summing step rewards without care. If each step gets a positive reward, the policy can earn more by writing more (or repeated) steps. Use a per-step reward with zero mean, the minimum, or only the final reward.
Interview questions
What is the difference between an outcome reward model and a process reward model?basic
Why do outcome labels give false positives, and why does that matter for RL?core
How does Math-Shepherd label steps without humans? What are the failure modes?core
Product or minimum to turn PRM step scores into one score?core
Why did DeepSeek-R1 not use neural reward models for its reasoning RL?core
Derive the accuracy limit of best-of-N with a verifier that has false positives.advanced
Explain the pass@k estimator. Why not just compute \(1-(1-\hat p)^k\)?core
What biases does an LLM judge have, and how do you reduce them?basic
Why use a rubric instead of asking the judge for a 1–10 score?core
What is reward-model calibration, how do you measure it and fix it? Does it change best-of-N?advanced
Your RL reward keeps rising but held-out quality is dropping. What is happening and what do you do?advanced
Exercises
- A PRM gives the steps of a solution 0.9, 0.8, 0.95, 0.7. Compute the product and the minimum. Which step would you show the student first?
- Monte Carlo labels with \(N = 16\): a wrong step still reaches the right answer 10% of the time. What is the probability that its hard label is 1? What is the standard deviation of its soft label?
- With \(n = 20\) samples and \(c = 4\) correct, compute pass@1 and pass@2 with formula (73.8).
- Best-of-N with \(p = 0.5\), \(a = 1\), \(b = 0.1\). What is the accuracy limit as \(N \to \infty\)? And for \(N = 1\)?
- A judge has \(\beta = 2\) and the answers are equal (\(d = 0\)). What does one ordering report? What is the consistency rate?
- Explain in two sentences why temperature scaling cannot fix a reward model that ranks answers in the wrong order.
Answerscheck
Key takeaways
- Rewards come from four sources: outcome reward models, process reward models, verifiers, and LLM judges (Figure 1). Real systems mix them.
- An ORM (73.2) learns “is the final answer right?” from automatic labels; it is cheap but accepts right answers with wrong reasoning, and it can be fooled when you sample enough.
- A PRM (73.3) scores each step and combines them by product or minimum (73.4); it beat the ORM in best-of-N on MATH (78.2% vs 72.4%). Step labels come from people (PRM800K) or Monte Carlo completions (73.5), which are noisy.
- In RL, an outcome reward gives every step the same credit, so a lucky answer praises its wrong steps; a process reward credits each step and, in the toy of Figure 4, learns a long chain much faster (73.6).
- Verifiers (73.7) give exact rewards for math and code and are the default for reasoning RL; their false positives set a ceiling on best-of-N (73.9), and pass@k (73.8) measures with them.
- LLM judges work for open-ended tasks but have position, length and self-preference biases; swap orders (73.10), use rubrics (73.11), or train generative verifiers (73.12).
- Check calibration with ECE (73.13) and fix it with temperature scaling (73.14); protect RL against overoptimization with KL, ensembles (73.15), verifiers and held-out checks. Best-of-N against a hackable reward model shows the same rise and fall as N grows (73.16, Figure 8).
Key formulas
References for this session
- Lightman et al. (2023). Let’s Verify Step by Step
- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Lambert (2025). Reinforcement Learning from Human Feedback (book)
- Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems
- Uesato et al. (2022). Solving math word problems with process- and outcome-based feedback
- Wang et al. (2023). Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
- Zhang et al. (2025). The Lessons of Developing Process Reward Models in Mathematical Reasoning
- Chen et al. (2021). Evaluating Large Language Models Trained on Code
- Zhang et al. (2024). Generative Verifiers: Reward Modeling as Next-Token Prediction
- Gunjal et al. (2025). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- Guo, Pleiss, Sun and Weinberger (2017). On Calibration of Modern Neural Networks
- Gao, Schulman and Hilton (2022). Scaling Laws for Reward Model Overoptimization
- Coste et al. (2023). Reward Model Ensembles Help Mitigate Overoptimization
- Lambert et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling
- DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Snell et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters