LLM & VLM courseBlog
73Track 4: Post-training and RL · session 21 of 25

Reward models, deeper: outcome rewards, process rewards, verifiers

RL can only be as good as its reward. This session compares the four main sources of reward for an LLM answer: outcome reward models that score the whole answer, process reward models that score every step, verifiers that check math answers and run code tests, and LLM judges with rubrics. It ends with calibration and with the ways a policy learns to fool a reward.

Time: 3 to 4 hours (reading 2 to 2.5 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 64, 67, 70 and 72

What you will learn

  • How an outcome reward model is trained from automatic correctness labels, and why it accepts right answers reached by wrong reasoning
  • How a process reward model scores each step, how PRM800K was labeled, and how product and minimum turn step scores into one score
  • How Math-Shepherd labels steps with Monte Carlo completions, and why these labels are noisy in a predictable way
  • Verifiers for math and code, the pass@k estimator, and why a verifier’s false-positive rate caps best-of-N accuracy
  • LLM-as-a-judge: position, length and self-preference biases, order swapping, rubric rewards and generative verifiers
  • Calibration of reward models (ECE, temperature scaling) and robustness: distribution shift, spurious features, overoptimization and the usual defenses

Why this matters

Every RL method in this track needs one thing: a number that says how good an answer is. In session 67 that number came from a reward model trained on human comparisons. This session looks closer at where the reward comes from, and at what goes wrong when the reward is wrong.

There are four main sources of reward for an LLM answer. An outcome reward model (ORM) reads the whole answer and gives one score. A process reward model (PRM) gives a score to every step of a reasoning chain. A verifier is a program that checks the answer: it compares a math answer with the known one, or runs unit tests on code. An LLM judge is another language model that reads the answer, often with a list of criteria (a rubric), and gives a score. Real systems mix them. DeepSeek-R1 used rule-based rewards (a verifier plus a format check) for math and code, and a learned reward model only for general chat data. [Certain]

The choice matters for two reasons. First, at test time a reward model picks the best of many samples (best-of-N, session 72); a better scorer gives a better pick. Second, during RL the policy will find and use every flaw in the reward. A reward that is 95% right is fine for ranking a few answers, but RL searches millions of answers and finds the 5%. This is reward hacking, and it is the main reason why reasoning models today rely on verifiers wherever they can.

Four ways to score an answer

All four sources answer the same question: given a prompt \(x\) and an answer \(y\), what is the reward \(r(x, y)\)? They differ in what they look at (only the end, or every step), how they are built (trained on labels, or written as code), and how they fail.

prompt x + answer y steps s1 … sK Outcome reward model (ORM)a network; reads all of y, one score Process reward model (PRM)a network; one score per step Verifiera program; checks answer or runs tests LLM judge (+ rubric)a prompted LLM; criteria, critique, score 0.83 (one number for the answer)fails: blind to lucky right answers 0.97, 0.95, 0.35, 0.90 → combinefails: step labels are costly or noisy 1 or 0 (exact)fails: weak tests, false positives “meets 3 of 4 criteria” → 0.75fails: position and length bias
Figure 1. Four sources of reward for the same answer. The two learned models (top) give soft scores and can be hacked. The verifier is exact but only exists where the answer can be checked. The judge works on open-ended answers, but has its own biases. The numbers on the right are examples, used again later in this session.

Recall the reward model from session 67. It is an LLM with a scalar head on the last token, trained on pairs (chosen answer \(y_w\), rejected answer \(y_l\)) with the Bradley–Terry model:

\[ P(y_w \succ y_l \mid x) = \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big), \qquad \mathcal{L}_{\text{BT}} = -\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big) \tag{73.1} \]

This is already an outcome reward model: it scores the whole answer. Its score has no fixed meaning; only differences matter. The reward models in this session are different in one important way: most of them predict a probability of being correct. That gives the score a meaning (“this answer is right with probability 0.8”), and it lets us ask if the model is calibrated, which we do at the end.

Outcome reward models

For math and code there is often a known final answer. Then we do not need human comparisons: we can label each sampled solution as correct or wrong automatically, and train a classifier on those labels. This is the outcome-supervised reward model. It was introduced as a “verifier” by Cobbe et al. (2021) together with the GSM8K dataset (8.5K grade-school math problems: 7.5K for training, 1K for testing). [Certain]

The recipe: fine-tune a generator, sample many solutions per training problem (Cobbe et al. used 100), mark each one correct if its final answer matches the reference, and train a model to predict that label. With \(z = 1\) for correct and \(z = 0\) for wrong, the loss is binary cross-entropy:

\[ \mathcal{L}_{\text{ORM}} = -\Big[ z \log \sigma\big(r_\phi(x, y)\big) + (1 - z) \log\Big(1 - \sigma\big(r_\phi(x, y)\big)\Big) \Big] \tag{73.2} \]
Read the symbols
\(x, y\)
the problem and one sampled solution
\(z\)
the label: 1 if the final answer of \(y\) is right, 0 if not (found by a program, not a person)
\(r_\phi(x, y)\)
the reward model’s logit, read at the last token; \(\sigma(r_\phi)\) is its predicted probability that \(y\) is correct

At test time we sample \(N\) solutions and return the one with the highest score. Cobbe et al. found that a 6B model with a verifier did about as well as a fine-tuned model 30 times larger, and that the gain grows with more training data. [Certain] They also found two details that matter in practice. A token-level verifier, which predicts “will this solution end correct?” after every token (a value function, session 65), worked better than one prediction at the end. And with more samples, performance rose up to about 400 solutions per problem and then fell: with enough tries, the generator finds wrong solutions that fool the verifier. [Certain] That is reward hacking at test time, before any RL.

Worked example: the ORM loss

The ORM says \(\sigma(r) = 0.8\) for a solution. If the solution is correct, the loss is \(-\log 0.8 = 0.223\). If it is wrong, the loss is \(-\log 0.2 = 1.609\), about 7 times larger. The gradient pushes the score of wrong solutions down hard.

The weak spot: right answer, wrong reasoning

The label \(z\) only checks the final answer. A solution with a wrong step that still reaches the right number gets \(z = 1\). Such false positives are labeled as good, so the ORM learns to accept them. Lightman et al. (2023) point out that this automatic grading is “not perfectly reliable” for this reason. [Certain] For math tests, a lucky right answer counts the same. But in RL, the policy is rewarded for a flawed method, and for tutoring or proofs the reasoning is the whole point.

Process reward models

A process reward model scores each step. We split a solution into steps \(s_1, \dots, s_K\) (in practice, usually one step per line or paragraph). After the last token of step \(k\), the PRM predicts \(p_k\): the probability that step \(k\) is correct, given the problem and the steps before it.

Lightman et al. (2023), “Let’s Verify Step by Step”, trained a PRM on PRM800K: 800K step labels from people, on 75K solutions to 12K MATH problems. [Certain] Each step was labeled positive (correct and useful), negative (wrong or unreasonable) or neutral (unclear). They labeled each solution only up to its first wrong step: for a wrong solution, this tells exactly where the mistake is, and nothing after it is needed. The PRM is trained like a language model: after each step it predicts one label token, and we maximize the log-likelihood of the right label. With two labels this is cross-entropy over steps, up to the first error:

\[ \mathcal{L}_{\text{PRM}} = -\frac{1}{K'} \sum_{k=1}^{K'} \Big[ z_k \log p_k + (1 - z_k) \log (1 - p_k) \Big], \qquad K' = \min(K, \text{first wrong step}) \tag{73.3} \]
Read the symbols
\(p_k\)
the PRM’s probability that step \(k\) is correct, read at the end of step \(k\); one forward pass gives all \(p_k\) at once
\(z_k\)
the step label: 1 for good (positive or neutral), 0 for the first wrong step
\(K'\)
the number of labeled steps: all \(K\) for a correct solution, up to and including the first error otherwise

From step scores to one solution score

To rank whole solutions we need one number per solution. Two common choices:

\[ R_{\text{prod}}(x, y) = \prod_{k=1}^{K} p_k, \qquad R_{\min}(x, y) = \min_{k} \, p_k \tag{73.4} \]

The product is “the probability that every step is correct” if step errors were independent; Lightman et al. used it. The minimum says “a solution is as good as its weakest step”; Math-Shepherd used it. In Lightman et al. the difference was small: 78.2% with the product and 77.6% with the minimum. [Certain] One trap: the product shrinks with length. Ten steps at 0.95 give \(0.95^{10} = 0.599\), but three steps at 0.95 give \(0.857\). Ranking by product favors short solutions; the minimum does not have this problem. A third choice, the score of the last step only, turns the PRM back into an outcome scorer.

Figure 2 uses the problem “A train goes 120 km in 2 hours, then 80 km in 2 hours. What is its average speed?” (answer: 50 km/h). Solution B gets the right answer by averaging the two speeds. That method is wrong in general (it works here only because the two times are equal), so a good PRM gives that step a low score. The step probabilities are made-up teaching values, not output of a real model.

Figure 2. An outcome check versus a process reward model on three solutions to the same problem. Bars show the PRM’s probability that each step is correct. Click a step to mark it as an error (its probability drops to 0.10) and click again to undo.
How to read Figure 2

Compare the three solutions under each aggregation. The outcome check (does the final answer equal 50?) gives A and B the same reward, 1. The PRM product gives A 0.858 and B 0.284, because step 3 of B is flawed. “Last step” gives B 0.90: it looks only at the end, so it misses the flaw, like an ORM. For Solution C the final answer 40 is wrong, and the PRM also says where: step 2 (2 + 2 = 5) has the lowest score. Mark more steps as errors and see that the minimum only reacts to the worst step, while the product reacts to every one.

Worked example: scoring the three solutions

Solution A, steps 0.97, 0.96, 0.95, 0.97: product \(= 0.858\), minimum \(= 0.95\). Solution B, steps 0.95, 0.95, 0.35, 0.90: product \(= 0.284\), minimum \(= 0.35\). Solution C, steps 0.97, 0.08, 0.60, 0.85: product \(= 0.040\), minimum \(= 0.08\). Both PRM rules rank \(\text{A} \gt \text{B} \gt \text{C}\). The outcome check says \(\text{A} = \text{B} = 1 \gt \text{C} = 0\). Only the PRM separates sound reasoning from a lucky answer.

Does process supervision help?

Two studies compared the two kinds of supervision. On GSM8K, Uesato et al. (2022) found that outcome supervision gives a similar final-answer error rate with less labeling, but process supervision (or a reward model that imitates it) was needed to get the reasoning right: compared with the previous best, final-answer error went from 16.8% to 12.7%, and the reasoning error among solutions with a correct final answer went from 14.0% to 3.4%. [Certain] On the harder MATH dataset, with a large GPT-4-based generator, Lightman et al. found a clear win for process supervision in best-of-N selection:

Method (best of 1,860 samples, 500 MATH test problems)% solved
Majority vote (most common final answer)69.6
Outcome reward model (ORM)72.4
Process reward model (PRM)78.2

Source: Lightman et al. (2023), main comparison of the large-scale reward models. [Certain] Note what this measures: picking one answer from many. It does not show that a PRM is a better reward for RL; that is a different question, discussed below.

Step labels without people: Monte Carlo estimation

Human step labels are slow and costly. Math-Shepherd (Wang et al., 2023) labels steps automatically. The idea: a step is good if the solution can still reach the right answer from there. So, for each prefix \(s_1, \dots, s_k\), let a “completer” model finish the solution \(N\) times, and check each final answer \(a_j\) against the reference \(a^*\):

\[ y_k^{\text{HE}} = \begin{cases} 1 & \text{if some } a_j = a^* \\ 0 & \text{otherwise} \end{cases} \qquad\qquad y_k^{\text{SE}} = \frac{1}{N} \sum_{j=1}^{N} \mathbb{1}[a_j = a^*] \tag{73.5} \]
Read the symbols
\(N\)
the number of completions per step; Math-Shepherd used \(N = 8\)
\(y_k^{\text{HE}}\)
the hard estimate: 1 if at least one completion reaches the right answer
\(y_k^{\text{SE}}\)
the soft estimate: the fraction of completions that reach it, an estimate of the step’s value \(V(s_{1:k})\)

Math-Shepherd trained its PRM on the hard estimate and used the minimum over steps (73.4). It improved Mistral-7B with step-by-step PPO from 77.9% to 84.1% on GSM8K, and to 89.1% with PRM reranking on top. [Certain] Note what this label really is: not “is this step correct?” but “can a completer still get the right answer from here?”. These two differ, and Figure 3 shows how.

true chance to finish correctly from this stepsoft estimate (fraction correct)hard label 1hard label 0
Figure 3. Monte Carlo step labels for a five-step solution whose step 4 is wrong. After the error, a completer can still reach the right answer by luck (4% of the time from step 4, 2% from step 5). Each “Sample again” draws new completions with a fixed seed.
How to read Figure 3

The grey bars are the true chance of finishing correctly, the orange dots are the soft estimates from \(N\) completions, and the colored squares are the hard labels. With \(N = 8\), the wrong step 4 gets the hard label 1 (“good”) with probability \(1 - 0.96^8 = 0.279\): more than one in four bad steps is labeled good. Raise \(N\): the soft estimate gets more accurate (its noise falls like \(1/\sqrt{N}\)), but the hard label gets worse, because more tries means more lucky recoveries. At \(N = 64\) the bad step is labeled good 92.7% of the time.

Two more problems with Monte Carlo labels. A strong completer can recover from a bad step (it fixes the error and moves on), so the step looks good. A weak completer fails even after a good step, so the step looks bad. Zhang et al. (2025, Qwen team) found that Monte Carlo labels gave PRMs with worse performance and generalization than labels from an LLM judge or from people, and that many PRMs give their lowest score mostly to the final step, so they act more like outcome scorers. They proposed keeping only steps where the Monte Carlo label and an LLM judge agree. [Certain]

Using a PRM: reranking, search and dense rewards

A PRM can be used in three ways. Reranking: score \(N\) finished solutions with (73.4) and keep the best, as in the table above. Search: because the PRM scores partial solutions, we can prune bad paths early. Snell et al. (2024) used PRM-guided beam search to spend test-time compute more efficiently than best-of-N on some problems. [Likely] Dense rewards in RL: give a reward after every step, not only at the end, so the policy gets credit for the right steps (Math-Shepherd’s step-by-step PPO).

Algorithm 1: PRM-guided beam search
  1. input: problem \(x\), policy \(\pi\), PRM, beam width \(B\), expansions per beam \(M\), max steps \(K\)
  2. beams \(\leftarrow\) \(\{\,\text{empty solution}\,\}\)
  3. for \(k = 1\) to \(K\):
  4.   candidates \(\leftarrow\) for each beam, sample \(M\) next steps from \(\pi\) // stop a step at a newline
  5.   score each candidate prefix with the PRM: \(p_k\) for its new step, solution score \(\min(p_1, \dots, p_k)\)
  6.   beams \(\leftarrow\) the \(B\) best candidates // bad paths are dropped early
  7.   move finished candidates (with a final answer) to the output list
  8. return the finished solution with the highest PRM score (or a vote weighted by the scores)

Why is the PRM not used everywhere in RL? DeepSeek-R1 lists three reasons in its “unsuccessful attempts”: it is hard to define what a “step” is in general reasoning; it is hard to know if an intermediate step is correct (models label badly, and people do not scale); and a model-based PRM “inevitably leads to reward hacking”, while retraining it costs compute and makes the pipeline complex. They concluded that a PRM is good for reranking and guided search, but its benefit in large-scale RL was small compared with its cost. [Certain] This is one reason the GRPO pipelines of session 70 use outcome rewards from verifiers.

Which step gets the credit? Outcome versus process rewards in RL

The costs above are real. So what does a dense, per-step reward buy? The answer is credit assignment: telling the policy which step was good or bad. Write a solution as actions \(a_1, \dots, a_K\) (one per step). The policy gradient with a per-step advantage \(A_t\) is

\[ \nabla_\theta J = \E\Big[\sum_{t=1}^{K} A_t\, \nabla_\theta \log \pi_\theta(a_t \mid x, a_{\lt t})\Big], \qquad A_t^{\text{ORM}} = R - b, \qquad A_t^{\text{PRM}} = s_t - b_t \tag{73.6} \]
Read the symbols
\(R\)
the outcome reward: 1 if the final answer is right, else 0 (from a verifier or an ORM)
\(s_t\)
the process reward of step \(t\): the PRM’s verdict on that step (1 = looks correct, 0 = looks wrong)
\(b, b_t\)
baselines: here the mean of the other samples in the batch, for the whole answer or for step \(t\)

With an outcome reward, every step of a solution gets the same advantage: a right answer praises its wrong steps too, and a wrong answer blames its right steps. With a process reward, each step gets its own signal. Figure 4 trains two policies on a toy reasoning chain. Each step is a yes/no action: “correct step” with probability \(p_t = \sigma(\theta_t)\). The final answer is right if every step is right; if some step is wrong, it is still right with probability “lucky guess” (the two-mistakes-that-cancel case of the ORM section). Step 3 is hard (\(p_3 = 0.30\) at the start); the others start at 0.75. In this toy the steps are independent, so a step’s own score is the right credit; in real chains a step also changes what comes after it, and the PRM reward is usually summed over the remaining steps.

Running: one iteration for both learners
  1. each learner samples 8 chains \(a_{i,1..K}\), step \(t\) correct with \(p_t = \sigma(\theta_t)\)
  2. outcome \(R_i = 1\) if the final answer is right (all steps right, or a lucky guess)
  3. the PRM scores every step: \(s_{i,t}\) // flipped with prob. “PRM label noise”
  4. ORM credit: \(A_{i,t} = R_i - \frac{1}{7}\sum_{j \ne i} R_j\) for every step
  5. PRM credit: \(A_{i,t} = s_{i,t} - \frac{1}{7}\sum_{j \ne i} s_{j,t}\) // step by step
  6. \(\theta_t \mathrel{+}= \eta \cdot \frac{1}{8}\sum_i A_{i,t}\,(a_{i,t} - p_t)\) // (73.6)
positive credit (cell color)negative creditORM learnerPRM learnerdashed: every step right
Figure 4. Outcome reward against process reward as the RL signal, on a toy chain of \(K\) yes/no steps. Both learners use the same random numbers, so at the first iteration they draw the same chains. In the two grids each row is one sampled chain, ✓ and ✗ mark right and wrong steps, the cell color is the credit \(A_{i,t}\) that the step receives, and the last column is the final answer (“lucky” = right answer from a wrong chain). The run stops after 300 iterations.
How to read Figure 4

Use speed 1 and Next line for one iteration. After line 4, look at the ORM grid: a whole row is blue or red. A “lucky” row is blue on its wrong steps too, so the ORM learner is rewarded for a wrong step; a row that failed only at step 3 gets the same negative (or zero) credit on its seven good steps. After line 5, the PRM grid is colored step by step: only the ✗ cells are red. Then Play at speed 4. With the defaults (8 steps, lucky-guess rate 0.20, seed 1) both start with a 23.2% chance of a right answer (only 4.0% from sound reasoning). After 25 iterations the PRM learner is at 74.7% and the ORM learner at 32.6%; after 50 iterations, 87.4% against 55.6%. Both get there in the end (97.8% and 96.3% at 300): the outcome reward is not wrong, only slow and noisy, because a step’s credit is mixed with the luck of all the other steps. Set the lucky-guess rate to 0: the ORM learner gets almost no signal at first, because almost every chain fails (19.6% after 25 iterations). Add PRM label noise 0.10: the PRM learner slows down a little (70.2% after 25 iterations) but still leads. A learned PRM that can be fooled is another matter; that is the reward hacking DeepSeek-R1 warned about. (Speeds 1 and 2 run one line of the algorithm at a time, 0.65 s or 0.2 s per line; speeds 3 to 5 run whole iterations, more of them per frame. “Next line” and the next-step button work at any speed.)

Verifiers for math and code

A verifier here means a program, not a network. For math it extracts the final answer (for example the text inside \boxed{...}) and checks it against the reference, after normalizing the format (0.5, 1/2 and 2/4 are the same). For code it runs the program on hidden unit tests in a sandbox. The reward is exact:

\[ r(x, y) = \mathbb{1}\big[\, \text{check}(\text{extract}(y),\ a^*_x) = \text{true} \,\big] \;+\; \lambda_{\text{fmt}}\, \mathbb{1}\big[\, y \text{ has the required format} \,\big] \tag{73.7} \]
Read the symbols
\(\text{extract}(y)\)
the part of the answer that is checked: the boxed number, or the code block
\(a^*_x\)
the reference: the known answer, or the test cases for problem \(x\)
\(\lambda_{\text{fmt}}\)
the weight of an optional format reward (for example, thinking inside think tags); DeepSeek-R1-Zero used accuracy rewards and format rewards [Certain]

This is the reward of RL with verifiable rewards (RLVR). It has no learned parameters, so the policy cannot make it give a high score to a wrong answer by finding a weak spot in a network. But a verifier can still be wrong, in two directions. A false negative rejects a correct answer: the policy wrote “\(x = 0.50\)” but the parser expected “1/2”, or the answer was right but not boxed. A false positive accepts a wrong answer: the unit tests are too weak, so a program that returns a constant passes them. With RL, false positives are the dangerous ones, because the policy learns to produce them.

pass@k: how verifiers are used to measure

Code models are measured with pass@k: the chance that at least one of \(k\) samples passes all tests. Sampling exactly \(k\) answers gives a noisy estimate. Chen et al. (2021, Codex) sample \(n \ge k\) answers, count the \(c\) correct ones, and use the unbiased estimator:

\[ \text{pass@}k = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}} \tag{73.8} \]

The fraction is the chance that a random set of \(k\) of the \(n\) samples contains only wrong ones. With \(n = 10\) and \(c = 3\): pass@1 \(= 0.3\) and pass@5 \(= 1 - \binom{7}{5}/\binom{10}{5} = 1 - 21/252 = 0.917\). The naive \(1 - (1 - 0.3)^5 = 0.832\) is a different (biased) number, because it samples with replacement.

Best-of-N with an imperfect verifier

pass@N assumes a perfect checker. With a real verifier, what accuracy do we get if we sample \(N\) answers and return one that passes? Say each sample is correct with probability \(p\). The verifier accepts a correct answer with probability \(a\) (its recall) and a wrong answer with probability \(b\) (its false-positive rate). We return a random accepted answer, or a random answer if none is accepted. Then:

\[ \text{acc}(N) = \big(1 - q_0\big)\, \underbrace{\frac{p\,a}{p\,a + (1 - p)\,b}}_{\text{precision}} + q_0\, \frac{p\,(1 - a)}{1 - s}, \qquad s = p\,a + (1 - p)\,b, \quad q_0 = (1 - s)^N \tag{73.9} \]
Read the symbols
\(s\)
the chance that one sample is accepted
\(q_0\)
the chance that none of the \(N\) samples is accepted
precision
the chance that an accepted sample is really correct; as \(N\) grows, \(q_0 \to 0\) and the accuracy goes to this number

The lesson: with more samples, the accuracy does not go to 1. It goes to the verifier’s precision. The false-positive rate \(b\) sets the ceiling, and it matters more when the policy is weak (small \(p\)), because then most samples are wrong and a few of them slip through.

pass@N (a perfect verifier)pick an accepted sample, formula (73.9)ceiling = precision
Figure 5. Best-of-N with a verifier. The horizontal axis is \(N\) from 1 to 256 on a log scale. The grey curve is what a perfect checker would give; the green curve is what this verifier gives.
How to read Figure 5

With the defaults (\(p = 0.3\), \(a = 0.9\), \(b = 0.05\)), 16 samples give 0.883 while a perfect checker would give 0.997, and no number of samples gets past the precision 0.885. Move \(b\) to 0.2: the ceiling drops to 0.659. Move \(p\) down to 0.05 with \(b = 0.05\): the ceiling is 0.486. Then set \(b = 0\): the green curve joins the grey one. Exact checks, or very strong tests, are what make verifiers so valuable.

Algorithm 2: a verifiable reward function for RL (math and code)
  1. input: answer \(y\), problem type, reference \(a^*\) (answer or tests), time limit
  2. if the required format is missing then return 0 // no final answer, no code block
  3. if math: \(\hat a \leftarrow\) last boxed expression; normalize both (strip spaces, units, commas; parse numbers, fractions, or symbolic expressions)
  4.   return 1 if \(\hat a\) equals \(a^*\) (exactly, or symbolically, or within a tolerance) else 0
  5. if code: write the code to a sandbox with no network and limited memory
  6.   run the hidden tests with the time limit; a timeout or crash counts as a fail
  7.   return 1 if all tests pass else 0 // or the fraction passed, as a softer reward
  8. log every rejected answer that looks correct, and check samples by hand // find false negatives early

LLM-as-a-judge and rubric rewards

Most useful tasks have no exact answer: write an email, explain an idea, summarize a video. For them, a strong LLM can act as the scorer. It gets the prompt, the answer (or two answers) and instructions, and outputs a score or a preference. This is LLM-as-a-judge. Zheng et al. (2023) found that GPT-4 as a judge agreed with human preferences over 80% of the time, about the same rate at which humans agree with each other. [Certain] When the judge’s outputs are used to train the policy, this is RL from AI feedback (session 74).

There are two formats. Single-answer grading: “score this answer from 1 to 10”. Pairwise comparison: “which of answers A and B is better?”. Zheng et al. also listed the judge’s main biases: [Certain]

  • Position bias: preferring the answer shown first (or second), regardless of quality.
  • Verbosity bias: preferring longer answers, even when the extra text adds nothing.
  • Self-enhancement bias: preferring answers written by the judge model itself, or by similar models.

A simple model of position bias: let \(d\) be the true quality gap (in logit units) between answers A and B, and \(\beta\) an extra push toward whichever answer is shown first. Then

\[ P(\text{A wins} \mid \text{A first}) = \sigma(d + \beta), \qquad P(\text{A wins} \mid \text{A second}) = \sigma(d - \beta), \qquad P_{\text{swap}} = \tfrac12\big[\sigma(d + \beta) + \sigma(d - \beta)\big] \tag{73.10} \]

The fix is to ask twice, with the order swapped, and average (or count a win only if both orders agree). When \(d = 0\), the swap average is exactly 0.5: the bias cancels. A low consistency rate (the fraction of pairs where both orders give the same winner) is a warning that the judge is guessing.

Figure 6. A pairwise judge with position bias, formula (73.10). Each bar is a probability that A wins (or, for the last bar, that both orders agree). The dashed line is the unbiased judge, \(\sigma(d)\).
How to read Figure 6

With \(d = 0.5\) (A is somewhat better) and \(\beta = 1\), A wins 81.8% of the time when shown first, but only 37.8% when shown second: in that order the judge picks the worse answer most of the time. The swap average, 59.8%, is on the right side of 0.5. The two orders agree only 42.2% of the time. Set \(d = 0\): one order alone says 73.1% for A, a pure artifact, while the swap average is exactly 50%. The swap average is still pulled toward 0.5 compared with \(\sigma(d)\), so a biased judge gives weaker, but not wrong-signed, signals.

Rubric rewards

A single score from 1 to 10 is vague: the judge may weigh things differently from one answer to the next. A rubric breaks the score into explicit checks. For the prompt “Explain why the sky is blue to a 10-year-old”, a rubric might be: (1) mentions that sunlight contains many colors, weight 3; (2) says blue light is scattered more, weight 2; (3) uses no technical words without explaining them, weight 2; (4) is under 150 words, weight 1. The judge answers each check with yes or no, and the reward is the weighted fraction:

\[ r_{\text{rubric}}(x, y) = \frac{\sum_{j=1}^{J} w_j\, c_j(x, y)}{\sum_{j=1}^{J} w_j}, \qquad c_j \in \{0, 1\} \tag{73.11} \]

If an answer passes checks 1, 2 and 4 but fails 3, the reward is \((3 + 2 + 0 + 1)/8 = 0.75\). Some checks (like check 4) can be done by code, which removes judge noise for them. Gunjal et al. (2025, “Rubrics as Rewards”) wrote a rubric for each prompt and used it as the RL reward on medical and science questions. Their best version beat a judge that gave direct 1-to-10-style (Likert) scores by up to 31% (relative) on HealthBench, and the rubric reward worked better with small judge models. [Certain] Rubrics also make the reward easier to read: when the reward goes up, you can see which checks improved.

Generative verifiers

An LLM judge can also be trained, not only prompted. A generative verifier (GenRM, Zhang et al., 2024) is an LLM fine-tuned to answer “Is the answer correct? Yes or No”, optionally after writing a short chain of thought about the solution. Its reward is the probability of the token “Yes”:

\[ r_{\text{GenRM}}(x, y) = \frac{1}{m} \sum_{i=1}^{m} P_\theta\big(\text{“Yes”} \mid x,\, y,\, \text{critique}_i\big), \qquad \text{critique}_i \sim P_\theta(\cdot \mid x, y) \tag{73.12} \]

With \(m = 1\) and no critique, this is a classifier that reuses the LM head. With \(m \gt 1\) sampled critiques, the scores are averaged, a form of majority voting for the verifier. Zhang et al. report best-of-N on GSM8K going from 73% to 93.4% with this approach. [Certain] The cost: a verifier that writes critiques is much slower than a scalar head.

Calibration and robustness

A reward model that outputs probabilities should mean what it says. It is calibrated if, among all answers it scores 0.8, about 80% are really correct. Calibration matters whenever we use the score as a number and not only to rank: when we set a threshold (“accept answers with score above 0.9”), when we combine it with other rewards, when we decide if more samples are needed, and in RL, where an overconfident reward model gives big rewards to answers that do not deserve them.

The usual measure is the expected calibration error (ECE). Put the predictions into \(M\) bins by their value (for example 0–0.1, 0.1–0.2, …), and compare, in each bin, the average prediction with the fraction that is really correct:

\[ \text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} \, \Big| \operatorname{acc}(B_m) - \operatorname{conf}(B_m) \Big| \tag{73.13} \]
Read the symbols
\(B_m\)
the set of answers whose predicted probability falls in bin \(m\); \(|B_m|\) is its size, and \(n\) the total number of answers
\(\operatorname{acc}(B_m)\)
the fraction of answers in the bin that are really correct
\(\operatorname{conf}(B_m)\)
the average predicted probability in the bin

Large networks are often overconfident: their probabilities are too close to 0 and 1. A one-parameter fix is temperature scaling (Guo et al., 2017): divide the logit by a temperature \(T\), fitted on held-out data by minimizing the log loss:

\[ \hat p = \sigma\!\left( \frac{r_\phi(x, y)}{T} \right), \qquad T^\star = \argmin_{T \gt 0} \; -\frac{1}{n} \sum_{i=1}^{n} \Big[ z_i \log \hat p_i + (1 - z_i) \log (1 - \hat p_i) \Big] \tag{73.14} \]

\(T \gt 1\) softens overconfident scores. Dividing by a positive number does not change the order of the scores, so best-of-N picks and ranking accuracy stay the same; only the probabilities change.

fraction really correct in each binaverage predicted probabilityperfect calibration
Figure 7. A reliability diagram for a synthetic, overconfident reward model on 2,000 answers. Its logits are 2.5 times too large. Bars show the real fraction correct in 10 bins of predicted probability; a calibrated model has bars on the diagonal. The data are fixed (no randomness), so the numbers are the same every time.
How to read Figure 7

At \(T = 1\), the top bin has an average prediction of 0.987 but only 89.7% of its answers are correct, and the bottom bin says 0.013 while 10.5% are correct: the bars lie below the diagonal on the right and above it on the left. ECE is 0.099 and the log loss is 0.531. Move \(T\) to 2.5: the bars move onto the diagonal, ECE drops to 0.003 and the log loss to 0.381. “Fit T” finds \(T = 2.45\), the value that minimizes the log loss on this data. Note the bins near 0 and 1 empty out as \(T\) grows: the model stops making extreme claims.

Robustness: the reward is a proxy

Calibration on a fixed test set is not enough. During RL, the policy changes, and the reward model sees answers unlike its training data. Three things go wrong, and you have met each of them already:

  • Distribution shift. The reward model was trained on samples from an older policy. After some RL steps, the policy writes new kinds of answers, and the reward model’s accuracy (and calibration) on them is unknown.
  • Spurious features. The reward model learns easy cues that went with good answers in its data: length, confident tone, markdown lists, phrases like “Let me double-check”. The policy then makes these cues stronger without being more correct. Length is the best-known case (session 67).
  • Overoptimization. Gao et al. (2022) showed that as the policy is pushed further from its start (measured by KL), the proxy reward keeps rising while the true (“gold”) reward rises, peaks and then falls. [Certain] DeepSeek-R1 saw this in practice: with a learned helpfulness reward model, the reward kept going up while Codeforces performance went down. [Certain]

The common defenses keep the policy where the reward is trustworthy, or make the reward harder to fool:

\[ \max_\pi \; \E_{x,\, y \sim \pi} \Big[ \bar r(x, y) - \lambda\, \operatorname{Var}_e\big[r_e(x, y)\big] \Big] - \beta\, \KL\big(\pi \,\|\, \pi_{\text{ref}}\big), \qquad \bar r = \frac{1}{E} \sum_{e=1}^{E} r_e \tag{73.15} \]
Read the symbols
\(r_1, \dots, r_E\)
an ensemble of \(E\) reward models (different seeds or data); \(\bar r\) is their mean
\(\lambda \operatorname{Var}_e[r_e]\)
a penalty when the ensemble members disagree, which happens on answers unlike the training data
\(\beta \KL\)
the usual penalty for moving far from the reference policy (session 66)

Coste et al. (2023) found that ensembles with such conservative objectives reduce overoptimization. [Likely] (Formula 73.15 shows the idea; the exact penalty in their paper may differ.) Other tools: normalize rewards per prompt (GRPO does this, session 70); add an explicit length penalty or length-controlled evaluation; combine a verifier with a reward model, so that correctness is exact and only style is learned; retrain the reward model on samples from the current policy (iterated RLHF, as in Llama 2); and stop training early, using held-out human or verifier checks, not the training reward. Static benchmarks such as RewardBench (Lambert et al., 2024) compare reward models on fixed chosen/rejected pairs; a higher score there is useful, but it does not guarantee a better policy after RL. [Likely]

Watching overoptimization happen: best-of-N against a hackable reward model

Best-of-N is the simplest optimizer of all: sample \(N\) answers and keep the one with the highest proxy score,

\[ \hat y_N = \argmax_{i \le N}\ \hat r(x, y_i), \qquad \text{gap}(N) = \E\big[\hat r(x, \hat y_N)\big] - \E\big[r^\ast(x, \hat y_N)\big] \tag{73.16} \]
Read the symbols
\(\hat r\)
the proxy score: a learned reward model, or the mean of an ensemble (73.15)
\(r^\ast\)
the true quality, which the selection never sees
\(\text{gap}(N)\)
how much the chosen answer’s proxy score overstates its true quality; it grows with \(N\) because the argmax prefers answers whose error happens to be large and positive

Figure 8 simulates it. Each answer has a true quality \(r^\ast \sim \mathcal{N}(0, 1)\), and the reward model sees \(\hat r = r^\ast + \text{noise}\), with noise of standard deviation \(\sigma\). A small fraction of answers are exploits: they use a trick the reward model likes (a confident tone, a format, a phrase), so their true quality is low (\(r^\ast \sim \mathcal{N}(-1.5, 0.5^2)\)) but their proxy score is high (\(\hat r \sim \mathcal{N}(H, 1)\) plus the noise). The ensemble averages the noise of several reward models, but every member is fooled by the same trick, because they learned from the same data.

Running: one trial of best-of-N
  1. sample 256 answers: true quality \(r^\ast(y_i)\); a fraction are exploits
  2. score them: \(\hat r(y_i)\) from one reward model, and the ensemble mean
  3. for \(N = 1, 2, 4, \dots, 256\): \(\hat y_N \leftarrow \argmax_{i \le N} \hat r(y_i)\) // (73.16), also for the ensemble and for \(r^\ast\) itself
  4. add \(\hat r(\hat y_N)\) and \(r^\ast(\hat y_N)\) to the running means
proxy score of the picktrue quality of the pick (one reward model)true quality of the ensemble’s pickoracle: pick by true qualityexploit answers
Figure 8. Best-of-N with an imperfect reward model, as a Monte Carlo simulation. Each trial samples 256 answers and replays the selection for every \(N\) on the first \(N\) of them; the curves are running means over trials (2,000 trials in total), with \(N\) on a log scale. Middle: the 256 answers of the current trial; the orange circle and square are the reward model’s picks for \(N = 16\) and \(N = 256\), the green circle is the best answer by true quality.
How to read Figure 8

Press Play and let the means settle. With the defaults (\(\sigma = 0.5\), 2% exploits with \(H = 2\), seed 1), after 2,000 trials the proxy score of the pick keeps rising with \(N\): 0.01, 2.13 and 3.56 at \(N = 1, 16, 256\). The true quality rises, peaks at \(N = 32\) (1.16) and falls to 0.48 at \(N = 256\), because 52.6% of the picks at \(N = 256\) are exploits (right chart). The oracle keeps improving (2.83 at 256): the good answers are there, the scorer just cannot find them. This is the shape of formula (67.12) in session 67, produced here by a mechanism you can see. The ensemble removes some noise, so it is better at small \(N\) (1.24 at \(N = 16\)), but it is fooled by the shared exploit even more reliably, and at \(N = 256\) it is worse (0.28). Set the exploit rate to 0: there is no peak any more, and the true quality at \(N = 256\) is 2.55 (ensemble 2.75); noise alone makes best-of-N less efficient, but not harmful. Raise it to 0.05: the peak moves to \(N = 8\) and the true quality at 256 is −0.70.

Putting it together in PyTorch

The code below builds a math verifier and a code verifier, the pass@k estimator, and then trains the same small network twice on toy “solutions”: once with the ORM loss (73.2) and once with the PRM loss (73.3). Each toy solution has 6 steps; a wrong step carries a hidden signal, and every step after the first error is wrong. It runs on a CPU in under a minute.

import math, re, subprocess, sys, torch
from fractions import Fraction
torch.manual_seed(0)

# 1) a rule-based math verifier: extract the final answer, compare exactly
def extract(text):
    m = re.findall(r"\\boxed\{([^{}]*)\}", text)
    return m[-1].strip() if m else None          # no boxed answer -> no reward
def to_num(s):
    try: return Fraction(s.replace(",", "").replace(" ", ""))
    except (ValueError, ZeroDivisionError): return None
def math_reward(text, gold):
    a = extract(text)
    if a is None: return 0.0
    x, y = to_num(a), to_num(gold)
    return float(x is not None and x == y)       # 0.5 == 1/2 == 2/4
print(math_reward(r"... so the speed is \boxed{50}", "50"),
      math_reward(r"\boxed{1/2}", "0.5"), math_reward("the answer is 50", "50"))
# prints 1.0 1.0 0.0   (the last one is a false negative: right, but not boxed)

# 2) a code verifier: run hidden unit tests in a separate process with a timeout
def code_reward(src, tests, timeout=2.0):
    prog = src + "\n" + "\n".join(f"assert {t}" for t in tests) + "\nprint('OK')"
    try:
        r = subprocess.run([sys.executable, "-c", prog], capture_output=True, text=True, timeout=timeout)
        return float(r.returncode == 0 and r.stdout.strip().endswith("OK"))
    except subprocess.TimeoutExpired:
        return 0.0
good = "def add(a, b):\n    return a + b"
hack = "def add(a, b):\n    return 5"                  # passes a weak test set
print(code_reward(good, ["add(2, 3) == 5"]), code_reward(hack, ["add(2, 3) == 5"]),
      code_reward(hack, ["add(2, 3) == 5", "add(-1, 1) == 0"]))
# prints 1.0 1.0 0.0   (one test: a false positive; two tests: caught)

# 3) the unbiased pass@k estimator (formula 73.8)
def pass_at_k(n, c, k):
    if n - c < k: return 1.0
    return 1.0 - math.comb(n - c, k) / math.comb(n, k)
print(round(pass_at_k(10, 3, 1), 4), round(pass_at_k(10, 3, 5), 4))   # 0.3 0.9167

# 4) ORM vs PRM on toy "solutions": 6 steps, each step is a feature vector.
#    A wrong step has a shift along a hidden direction; the solution is correct
#    only if no step is wrong. Same network, two different losses.
S, D, N = 6, 16, 4000
bad_dir = torch.randn(D); bad_dir /= bad_dir.norm()
def make(n):
    x = torch.randn(n, S, D)
    first = torch.randint(0, 2 * S, (n,))           # first error position; >= S means no error
    err = torch.arange(S)[None] >= first[:, None]   # every step after the first error is "wrong"
    x[:, :, :] += 3.0 * bad_dir * (torch.arange(S)[None] == first[:, None]).float()[..., None]
    step_ok = (~err).float()                         # PRM target: prefix is still correct
    final_ok = (first >= S).float()                  # ORM target: whole solution correct
    return x, step_ok, final_ok, first
class RM(torch.nn.Module):
    def __init__(s):
        super().__init__(); s.rnn = torch.nn.GRU(D, 32, batch_first=True); s.head = torch.nn.Linear(32, 1)
    def forward(s, x): return s.head(s.rnn(x)[0]).squeeze(-1)    # one logit per step
bce = torch.nn.functional.binary_cross_entropy_with_logits
def train(kind):
    torch.manual_seed(1); m = RM(); opt = torch.optim.AdamW(m.parameters(), lr=3e-3)
    x, step_ok, final_ok, first = make(N)
    for it in range(300):
        logit = m(x)
        if kind == "orm": loss = bce(logit[:, -1], final_ok)          # formula 73.2: last step only
        else:                                                          # formula 73.3: every step,
            mask = (torch.arange(S)[None] <= first[:, None]).float()   # up to the first error
            loss = (bce(logit, step_ok, reduction="none") * mask).sum() / mask.sum()
        opt.zero_grad(); loss.backward(); opt.step()
    return m
torch.manual_seed(2); xt, st, ft, first_t = make(2000)
for kind in ["orm", "prm"]:
    m = train(kind)
    with torch.no_grad():
        p = torch.sigmoid(m(xt))
        score = p[:, -1] if kind == "orm" else p.min(1).values             # formula 73.4 (min) for the PRM
        acc_final = ((score > 0.5).float() == ft).float().mean()            # is the solution correct?
        wrong = first_t < S
        locate = (p[wrong].argmin(1) == first_t[wrong]).float().mean()       # where is the first error?
    print(kind, "final acc", round(acc_final.item(), 3), "finds first error", round(locate.item(), 3))
# on our CPU run: orm final acc 0.797, finds first error 0.178 (chance is 1/6 = 0.167)
#                 prm final acc 0.84,  finds first error 0.908

Read the last two lines. Both models learn to judge the whole solution, but only the PRM learns where the error is: the ORM’s per-step outputs are no better than a random guess, because it was never trained on them. The PRM gets the location right 91% of the time, and its solution-level accuracy is also higher here, because the per-step labels give it more signal from the same data. (Exact values can differ slightly across PyTorch versions and machines.)

Where this is used

SettingReward sourceWhy
RL for math reasoning (DeepSeek-R1-Zero, RLVR)Verifier on the final answer + format rewardExact, cannot be hacked by fooling a network; no labels to collect [Certain]
RL for codeHidden unit tests in a sandbox (pass or fraction passed)Execution is the ground truth; the risk is weak tests
Best-of-N and search at test timeORM, PRM (min or product), or a generative verifierPicks the best of many samples; PRMs can prune partial solutions (session 72)
Chat and writing quality in RLHFBradley–Terry reward model, often with a length penaltyNo exact answer; human preference is the target
Open-ended expert answers (medicine, science)LLM judge with a per-prompt rubricExplicit criteria give a more stable reward than one vague score [Certain for Gunjal et al.]
Model evaluation (MT-Bench, Arena-style)Pairwise LLM judge with order swappingCheap proxy for human votes; over 80% agreement in Zheng et al.
Video question answering and temporal groundingVerifier: multiple-choice answer match, or IoU between the predicted and true time span as the rewardGives RLVR-style rewards to video LLMs without a learned reward model [Likely]
Video captioning and summarization, Video-RAG answersLLM or VLM judge with a rubric: mentions key events, no made-up objects, cites the right clipNo single correct caption; rubric checks target faithfulness [Likely]
Key-moment detection, action recognition labelsVerifier against annotated timestamps or class labelsThe label set is fixed, so the reward is exact [Likely]

Common mistakes

  • Trusting a high training reward. The reward going up only means the policy is good at the proxy. Track a held-out signal (verifier, human check, a different judge) and stop when it stops improving.
  • A strict parser that rejects correct answers. If “0.50”, “1/2” and “\(\tfrac{1}{2}\)” are not all accepted, the policy learns a format, not math. Read a sample of rejected answers by hand before a long run.
  • Weak unit tests. A few tests let constant or special-cased programs pass. Use many tests, edge cases, and random inputs checked against a reference solution.
  • Ranking PRM scores by product across very different lengths. The product (73.4) punishes long solutions; use the minimum, or compare solutions of similar length.
  • Using only one order in pairwise judging. Position bias can flip the result. Always swap and average, and report the consistency rate.
  • Treating a reward model score as a probability without checking. Measure ECE on held-out data and fit a temperature before using thresholds.
  • Never retraining the reward model. After the policy moves, the reward model’s training data no longer matches what it scores. Refresh it with new policy samples, or rely on a verifier where possible.
  • Summing step rewards without care. If each step gets a positive reward, the policy can earn more by writing more (or repeated) steps. Use a per-step reward with zero mean, the minimum, or only the final reward.

Interview questions

What is the difference between an outcome reward model and a process reward model?basic
An ORM gives one score for the whole answer, usually trained to predict if the final answer is correct (labels come from checking the final answer). A PRM gives a score to every reasoning step, trained on step-level labels up to the first wrong step. The PRM can locate errors and penalize right-answer-wrong-reasoning solutions; the ORM cannot. To rank whole solutions, a PRM’s step scores are combined, usually by product or minimum.
Why do outcome labels give false positives, and why does that matter for RL?core
A solution can reach the right final answer through wrong steps (errors that cancel, a lucky guess, a method that only works for these numbers). Checking the final answer marks it as correct. An ORM trained on such labels learns to accept flawed reasoning, and an RL policy optimized on it is rewarded for flawed reasoning, which may then fail on other problems. Process supervision gives such solutions a low score at the flawed step.
How does Math-Shepherd label steps without humans? What are the failure modes?core
For each prefix, a completer model finishes the solution \(N\) times (8 in the paper). The hard label is 1 if any completion reaches the right answer; the soft label is the fraction that do. Failure modes: a bad step can be labeled good when a completion recovers by luck or fixes the error (probability \(1-(1-q)^N\), which grows with \(N\)); a good step can be labeled bad if the completer is weak; the label measures “can still reach the answer”, not “this step is correct”. Later work (Qwen team) found such labels generalize worse than LLM-judge or human labels and recommended filtering by agreement.
Product or minimum to turn PRM step scores into one score?core
The product is the probability that all steps are correct if errors were independent; the minimum is the score of the weakest step. In Lightman et al. they performed almost the same (78.2% vs 77.6%). The product shrinks with the number of steps (0.95¹⁰ = 0.599 vs 0.95³ = 0.857), so it favors short solutions; the minimum is length-neutral but ignores everything except the worst step. Using only the last step’s score makes the PRM behave like an ORM.
Why did DeepSeek-R1 not use neural reward models for its reasoning RL?core
They observed that neural reward models (outcome or process) are susceptible to reward hacking in large-scale RL, and retraining them costs compute and complicates the pipeline. For PRMs specifically they also noted that steps are hard to define and step correctness is hard to label at scale. So for math, code and logic they used rule-based accuracy and format rewards; a learned preference reward model was used only for general data, in limited steps, and they reported reward hacking with it.
Derive the accuracy limit of best-of-N with a verifier that has false positives.advanced
Each sample is correct with probability \(p\); the verifier accepts correct ones with probability \(a\) and wrong ones with \(b\). Given that a sample is accepted, it is correct with probability \(pa/(pa+(1-p)b)\), the precision. When \(N\) grows, the chance that none is accepted, \((1-s)^N\), goes to 0, so the accuracy of “return a random accepted sample” goes to the precision, not to 1. Example: \(p = 0.3, a = 0.9, b = 0.05\) gives a ceiling of 0.885; \(b = 0.2\) gives 0.659. Weak policies (small \(p\)) suffer most.
Explain the pass@k estimator. Why not just compute \(1-(1-\hat p)^k\)?core
Sample \(n \ge k\) solutions, count \(c\) correct; pass@k \(= 1 - \binom{n-c}{k}/\binom{n}{k}\), the chance that a random subset of \(k\) of the \(n\) samples has no correct one, subtracted from 1. It is unbiased for the true pass@k. Plugging \(\hat p = c/n\) into \(1-(1-\hat p)^k\) is biased (it is a concave function of a noisy estimate, so it underestimates on average) and models sampling with replacement. For \(n=10, c=3, k=5\): 0.917 versus 0.832.
What biases does an LLM judge have, and how do you reduce them?basic
Position bias (preferring the first or second answer), verbosity bias (preferring longer answers), self-enhancement bias (preferring its own style), plus limited ability to grade math or code it cannot solve. Reduce them by swapping the order and averaging (or requiring agreement), length-controlled comparison or explicit length rules, a judge from a different model family, rubrics with concrete checks, giving the judge a reference answer, and asking for a short reasoning before the verdict. Track the consistency rate.
Why use a rubric instead of asking the judge for a 1–10 score?core
A single score mixes many criteria with weights that change from answer to answer, so it is noisy and hard to interpret. A rubric lists concrete yes/no checks with weights; the reward is the weighted fraction met. Each check is an easier question, some checks can be done by code, and when the reward changes you can see why. Gunjal et al. report gains over direct Likert-score judges and less dependence on the judge size. Risks: a rubric can miss what matters, and the policy will optimize exactly the listed checks.
What is reward-model calibration, how do you measure it and fix it? Does it change best-of-N?advanced
Calibrated means predicted probabilities match observed frequencies: of answers scored 0.8, about 80% are correct. Measure with a reliability diagram and ECE (bin predictions, average \(|\text{acc}-\text{conf}|\) weighted by bin size). Fix overconfidence with temperature scaling, fitting one \(T\) on held-out data by minimizing log loss. Dividing logits by \(T \gt 0\) keeps the order, so best-of-N picks and ranking accuracy are unchanged; thresholds, mixing with other rewards, and the size of RL rewards do change. Calibration also degrades under distribution shift, so re-check it on current policy samples.
Your RL reward keeps rising but held-out quality is dropping. What is happening and what do you do?advanced
Reward overoptimization (Goodhart): the policy exploits flaws of the proxy reward, often through length, style or formatting cues, and has moved to answers outside the reward model’s training distribution (Gao et al.; DeepSeek-R1 saw the same with a helpfulness RM). Actions: roll back to the best checkpoint by the held-out metric; raise the KL penalty; use a reward-model ensemble with a disagreement penalty; add a length penalty; replace learned rewards by verifiers where possible; collect fresh labels on current samples and retrain the RM; inspect high-reward samples by hand.

Exercises

  1. A PRM gives the steps of a solution 0.9, 0.8, 0.95, 0.7. Compute the product and the minimum. Which step would you show the student first?
  2. Monte Carlo labels with \(N = 16\): a wrong step still reaches the right answer 10% of the time. What is the probability that its hard label is 1? What is the standard deviation of its soft label?
  3. With \(n = 20\) samples and \(c = 4\) correct, compute pass@1 and pass@2 with formula (73.8).
  4. Best-of-N with \(p = 0.5\), \(a = 1\), \(b = 0.1\). What is the accuracy limit as \(N \to \infty\)? And for \(N = 1\)?
  5. A judge has \(\beta = 2\) and the answers are equal (\(d = 0\)). What does one ordering report? What is the consistency rate?
  6. Explain in two sentences why temperature scaling cannot fix a reward model that ranks answers in the wrong order.
Answerscheck
1. Product \(0.9 \times 0.8 \times 0.95 \times 0.7 = 0.479\); minimum 0.7, at step 4, so show step 4 first. 2. \(1 - 0.9^{16} = 0.815\): the bad step is labeled good most of the time. Soft label standard deviation \(\sqrt{0.1 \times 0.9 / 16} = 0.075\). 3. pass@1 \(= 1 - 16/20 = 0.2\); pass@2 \(= 1 - \binom{16}{2}/\binom{20}{2} = 1 - 120/190 = 0.368\). 4. Precision \(= 0.5/(0.5 + 0.05) = 0.909\) is the limit. For \(N = 1\): if accepted (probability 0.55) it is correct with probability 0.909; if rejected (0.45) it is always wrong because \(a = 1\); so \(0.55 \times 0.909 = 0.5 = p\), as it must be with one sample. 5. \(\sigma(2) = 0.881\) for whichever answer is shown first. Consistency \(= \sigma(2)\sigma(-2) + (1-\sigma(2))(1-\sigma(-2)) = 2 \times 0.881 \times 0.119 = 0.210\). 6. Temperature scaling divides every logit by the same positive number, so the order of scores, and therefore every ranking and best-of-N choice, stays the same. It only changes how confident the probabilities are; fixing the order needs better data or a better model.

Key takeaways

  • Rewards come from four sources: outcome reward models, process reward models, verifiers, and LLM judges (Figure 1). Real systems mix them.
  • An ORM (73.2) learns “is the final answer right?” from automatic labels; it is cheap but accepts right answers with wrong reasoning, and it can be fooled when you sample enough.
  • A PRM (73.3) scores each step and combines them by product or minimum (73.4); it beat the ORM in best-of-N on MATH (78.2% vs 72.4%). Step labels come from people (PRM800K) or Monte Carlo completions (73.5), which are noisy.
  • In RL, an outcome reward gives every step the same credit, so a lucky answer praises its wrong steps; a process reward credits each step and, in the toy of Figure 4, learns a long chain much faster (73.6).
  • Verifiers (73.7) give exact rewards for math and code and are the default for reasoning RL; their false positives set a ceiling on best-of-N (73.9), and pass@k (73.8) measures with them.
  • LLM judges work for open-ended tasks but have position, length and self-preference biases; swap orders (73.10), use rubrics (73.11), or train generative verifiers (73.12).
  • Check calibration with ECE (73.13) and fix it with temperature scaling (73.14); protect RL against overoptimization with KL, ensembles (73.15), verifiers and held-out checks. Best-of-N against a hackable reward model shows the same rise and fall as N grows (73.16, Figure 8).

Key formulas

\[ \mathcal{L}_{\text{PRM}} = -\tfrac{1}{K'}\textstyle\sum_{k \le K'} \big[z_k \log p_k + (1-z_k)\log(1-p_k)\big], \qquad R = \min_k p_k \ \text{ or } \ \prod_k p_k \]
\[ \text{pass@}k = 1 - \binom{n-c}{k}\Big/\binom{n}{k}, \qquad \text{acc}(N \to \infty) = \frac{pa}{pa + (1-p)b}, \qquad \text{ECE} = \sum_m \frac{|B_m|}{n}\big|\text{acc}(B_m) - \text{conf}(B_m)\big| \]

References for this session

  1. Lightman et al. (2023). Let’s Verify Step by Step
  2. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
  3. Lambert (2025). Reinforcement Learning from Human Feedback (book)
  4. Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems
  5. Uesato et al. (2022). Solving math word problems with process- and outcome-based feedback
  6. Wang et al. (2023). Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
  7. Zhang et al. (2025). The Lessons of Developing Process Reward Models in Mathematical Reasoning
  8. Chen et al. (2021). Evaluating Large Language Models Trained on Code
  9. Zhang et al. (2024). Generative Verifiers: Reward Modeling as Next-Token Prediction
  10. Gunjal et al. (2025). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
  11. Guo, Pleiss, Sun and Weinberger (2017). On Calibration of Modern Neural Networks
  12. Gao, Schulman and Hilton (2022). Scaling Laws for Reward Model Overoptimization
  13. Coste et al. (2023). Reward Model Ensembles Help Mitigate Overoptimization
  14. Lambert et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling
  15. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  16. Snell et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters