Evaluation of LLM systems
A model is only one part of an LLM product, and public scores do not tell you if your product works. This session compares static benchmarks, arenas and your own test set, shows how LLM judges are biased and how to correct them, and gives the statistics (intervals, paired tests, sample size) you need to trust a difference. It ends with a regression suite and a release gate you can build.
What you will learn
- What static benchmarks, arenas and your own test set each measure, and when to use which
- How Chatbot Arena turns pairwise votes into ratings with the Bradley–Terry model
- LLM-as-judge setups, and the position, length and self-preference biases with real numbers from MT-Bench and AlpacaEval
- How to check a judge against humans with Cohen’s kappa
- Wilson intervals, paired comparisons, McNemar’s test and the paired bootstrap
- How many test items you need: the sample-size formula and why pairing helps
- How to build a versioned, sliced regression suite and a per-slice release gate
Why this matters
You change one prompt, swap the model for a cheaper one, or add a retrieval step. Is the product better or worse now? Without a good evaluation, nobody can answer this question. The team ships changes on feeling, and users find the regressions.
In session 40 we scored a base model on public benchmarks. In session 149 we scored single outputs with BLEU, ROUGE and BERTScore. A real LLM system is more than a model: it has a prompt, maybe a retriever (session 152), tools and an agent loop (session 154), and output rules. Its quality depends on all of these parts together, and on the questions your users ask. This session is about measuring that system, and about making the numbers trustworthy.
Four ideas carry the session:
- Three sources of evidence: static benchmarks (fixed public test sets such as MMLU and GSM8K), arenas (people vote between two anonymous answers), and your own test set. Each one answers a different question.
- LLM-as-judge: using a strong model to grade answers. It is cheap and fast, but it has known biases: it prefers the first answer, longer answers, and sometimes its own answers.
- Statistics: every score is an estimate from a sample. You need confidence intervals, paired comparisons, and enough items to see the difference you care about.
- A regression suite: a versioned test set with graders that runs on every change and blocks a release when some part of the product gets worse.
Three sources of evidence
Before any statistics, decide what question you want to answer. The three common sources of evidence answer different questions:
Static benchmarks
A static benchmark is a fixed set of questions with known answers. MMLU has multiple-choice questions on 57 subjects, with 14,042 test questions. GSM8K has grade-school math word problems; its test split has 1,319 problems. [Certain] They are cheap to run and everybody uses the same items, so you can compare with published numbers. How to run them correctly (log-likelihood scoring, answer extraction, harness settings, for example with the LM Evaluation Harness) was the topic of session 40. Suites such as HELM go wider: it evaluated 30 models on 42 scenarios and, for its 16 core scenarios, measured 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency), not only accuracy. [Certain]
They have three problems for product work. First, contamination: test items are on the web, so they can end up in training data, and the score then measures memory. Second, saturation: when the best models score near the top, the benchmark can no longer separate them. Top models now score close to 90% on MMLU, and part of the remaining errors are wrong labels in the dataset itself. [Likely] Third, a benchmark question is not your user’s question. A model that is 3 points better on MMLU can be worse at following your output format.
Arenas
In an arena, a user types any prompt, sees the answers of two anonymous models side by side, and votes for the better one (or a tie). Chatbot Arena collected more than 240,000 such votes in its first months. [Certain] Because the prompts are new and come from real people, an arena is hard to game by leaking test data. But voters reward style: long, well-formatted, confident answers win more votes, even when they are not more correct. [Likely]
How do you turn many pairwise votes into one score per model? With the Bradley–Terry model. Each model \(i\) gets one number, its rating \(R_i\). The probability that \(i\) beats \(j\) depends only on the difference of ratings:
- Read the symbols
- \(i \succ j\)
- “the answer of model \(i\) is preferred to the answer of model \(j\)”
- \(R_i, R_j\)
- ratings on the chess-style Elo scale; only differences matter
- 400
- a scale constant from chess: a gap of 400 points means 10 to 1 odds
A gap of 0 gives 50%. A gap of 100 points gives 64.0%, a gap of 200 gives 76.0%. The ratings are fitted to all votes at once by maximum likelihood (choose the ratings that make the observed votes most probable):
Here \(W_{ij}\) is the number of votes where \(i\) beat \(j\). Classic Elo instead updates the two ratings after each single game, so the result depends on the order of the games. Chatbot Arena moved to the Bradley–Terry fit for this reason, and it reports bootstrap confidence intervals for each rating. [Likely] Figure 2 fits (155.2) to a vote table you can change.
How to read Figure 2
With the starting votes, A beats B in 60 of 100 votes and the fit gives A 1114.0 and B 1027.9; the fitted model predicts 62.1% for A against B, close to the observed 60%. Click the cell “D beats A” once: ten surprise wins for the weakest model. D rises to 893.1 and A falls to 1097.4, but B and C barely move. Every vote is used for every rating, which is why the fit does not depend on the order of the votes.
Your own test set
The third source is a set of inputs that look like your real traffic, with a way to grade each output. It is the only one that measures your system. Good places to collect items: real (anonymized) user requests sampled from logs, every bug report and failure you have seen, hard cases written by domain experts, and edge cases (empty input, very long input, other languages, attempts to break the rules). Label each item with a slice (a subgroup such as “billing questions” or “long documents”) so you can see where the system fails, not only how often.
Graders: who decides if an answer is good?
A test item needs a grader: a rule that turns an output into a score. Use the cheapest grader that is correct for the item:
| Grader | Good for | Weakness |
|---|---|---|
| Exact match, regex, number check | Short answers, math, classification labels | Misses correct answers written differently |
| Schema and rule checks | JSON output, tool calls, required fields, length limits | Checks the form, not the content |
| Execution | Code (run the unit tests), SQL (compare result tables), agent tasks (check the final state) | Needs a sandbox; tests can be incomplete |
| Reference metrics (ROUGE, BERTScore) | Quick trend checks when you have reference answers | Weak link to human judgment (session 149) |
| LLM-as-judge | Open-ended answers: helpfulness, faithfulness to sources, tone | Biases (next section); must be checked against humans |
| Human experts | The ground truth for subjective quality; checking the judge | Slow, costly, and humans also disagree |
LLM-as-judge and its biases
An LLM judge is a strong model that reads a question and one or two answers and outputs a grade. There are three common setups: single-answer grading (give a score from 1 to 10, ideally with a rubric), pairwise comparison (which of A and B is better?), and reference-guided grading (compare the answer with a correct reference, useful for math and facts). MT-Bench, a set of 80 multi-turn questions in 8 categories, was built to study these judges. With ties removed, GPT-4 as a pairwise judge agreed with human experts 85% of the time, while two humans agreed with each other 81% of the time. [Certain] So a good judge can be as consistent as a human rater. But it has biases that a human rater may not have.
Position bias
The judge prefers an answer because of where it is shown, usually first. In the MT-Bench study, each pair was judged twice with the order swapped. GPT-4 gave the same verdict in both orders only 65.0% of the time; in 30.0% of cases it preferred whichever answer came first. Claude-v1 was consistent in only 23.8% of cases and preferred the first answer in 75.0%. [Certain] The standard fix is to judge both orders: count a win only if the same answer wins in both orders, and call it a tie otherwise.
Length (verbosity) bias
The judge prefers longer answers, even when the extra text adds nothing. In the MT-Bench study, a “repetitive list” attack (the same answer with its items repeated in new words) fooled Claude-v1 and GPT-3.5 in 91.3% of cases and GPT-4 in 8.7%. [Certain] AlpacaEval, an automatic benchmark that uses an LLM judge, fixed this with a length-controlled win rate: a regression model predicts what the judge would say if both answers had the same length. This raised its Spearman correlation with Chatbot Arena rankings from 0.94 to 0.98. [Certain]
Self-preference
A judge may prefer answers written by itself or by its own model family, perhaps because their style looks “right” to it. In the MT-Bench study, GPT-4 gave its own answers a 10% higher win rate and Claude-v1 25% higher, but the authors said the data were too small to be sure. [Certain] Practical rule: do not let a model judge a contest it takes part in, or use judges from several families.
We can write a simple model of a biased pairwise judge. Let \(q_X\) be the true quality of answer \(X\). The judge sees answer \(F\) first and answer \(S\) second:
- Read the symbols
- \(\sigma\)
- the sigmoid function, \(\sigma(x) = 1/(1 + e^{-x})\), which turns any number into a probability
- \(k\)
- how sharply the judge sees quality differences (in Figure 3, \(k = 3\))
- \(\beta\)
- position bias: extra push toward the first answer; \(\beta = 0\) means no bias
- \(\gamma\)
- length bias: extra push per 100 tokens of extra length
- \(\ell_F, \ell_S\)
- lengths of the two answers in tokens
Swapping the order turns \(+\beta\) into \(-\beta\) for the same pair, so averaging the two orders cancels the position bias. It does not cancel the length bias: the longer answer is still longer in both orders. Figure 3 shows this on 300 simulated pairs.
How to read Figure 3
Set both biases to 0: all bars equal the dashed line. Now raise \(\beta\): “A shown first” goes up, “B shown first” goes down, and their average stays on the dashed line. That is why we always judge both orders. Raise \(\gamma\) with A longer: all bars move up together, and swapping cannot help. Turn on length control: the length term is removed, and the average returns to the dashed line. The “inconsistent” number is the share of questions where the two orders disagree. Track it for your judge: a high value means the judge is noisy or biased on that kind of question.
- input: question \(x\), answers \(a\) and \(b\), judge prompt with a rubric, judge model \(J\)
- \(v_1 \leftarrow J(x, \text{first} = a, \text{second} = b)\) // returns "first", "second" or "tie"
- \(v_2 \leftarrow J(x, \text{first} = b, \text{second} = a)\) // same pair, order swapped
- map both verdicts back to names: \(w_1, w_2 \in \{a, b, \text{tie}\}\)
- if \(w_1 = w_2\): return \(w_1\)
- else: log the item as inconsistent; return tie // conservative rule from the MT-Bench study
- use temperature 0, a fixed judge version, and ask for a short reason before the verdict
Other practical rules for judges: give a rubric (what earns each score), ask the judge to explain before it scores, prefer pairwise or pass/fail over 1–10 scales (scales drift and cluster), pin the judge model version, and treat the judge prompt as code that is versioned and reviewed. For RAG systems, ask the judge narrow questions such as “is every claim in the answer supported by the given passages?” (session 153).
Is the judge good enough? Check it against humans
Before you trust a judge, have humans label a sample (100–300 items is common [Likely]) and measure agreement. Raw agreement is misleading when most items pass: a judge that always says “pass” agrees 90% of the time if 90% of items pass. Cohen’s kappa corrects for the agreement you would get by chance:
- Read the symbols
- \(p_o\)
- observed agreement: the share of items where judge and human give the same label
- \(p_e\)
- agreement expected by chance if the two labeled independently
- \(p_J, p_H\)
- the share of items the judge (or the human) labels “pass”
\(\kappa = 1\) is perfect agreement, \(\kappa = 0\) is no better than chance. Values above about 0.6 are often called “substantial”. [Likely] Also look at the two kinds of disagreement separately: a judge that passes bad answers is more dangerous than one that fails good ones.
A judge and a human label 200 answers as pass or fail. Both say pass on 120, both say fail on 40. The judge says pass but the human says fail on 25; the judge says fail but the human says pass on 15.
Observed agreement \(p_o = (120 + 40)/200 = 0.800\). The judge passes \(p_J = 145/200 = 0.725\); the human passes \(p_H = 135/200 = 0.675\). Chance agreement \(p_e = 0.725 \times 0.675 + 0.275 \times 0.325 = 0.4894 + 0.0894 = 0.5788\).
\(\kappa = (0.800 - 0.5788)/(1 - 0.5788) = 0.525\). The 80% agreement sounds good, but kappa is only moderate. And the judge is too kind: 25 of the 65 answers the human failed were passed by the judge. Fix the rubric for those cases before you use this judge as a gate.
Statistics: is the difference real?
Your test set is a sample. Run the same system on a different sample of 400 questions and you get a different score. So every score needs an error bar. Session 40 gave the basic formula, \(\mathrm{SE} = \sqrt{p(1-p)/n}\), and the 95% interval \(p \pm 1.96\,\mathrm{SE}\). Product test sets add three problems: they are small, the two systems answer the same items, and you need to plan how many items to label.
Small test sets: the Wilson interval
The simple interval \(p \pm 1.96\,\mathrm{SE}\) (the Wald interval) breaks for small \(n\) or for \(p\) near 0 or 1. With 18 of 20 correct, it gives \([0.769, 1.031]\), which goes above 100%. With 20 of 20 correct, it gives \([1, 1]\): zero uncertainty, which is clearly wrong. The Wilson interval stays inside \([0, 1]\) and is accurate for small samples:
- Read the symbols
- \(\hat p\)
- the measured pass rate (\(k/n\))
- \(n\)
- number of items
- \(z\)
- 1.96 for a 95% interval
For 18 of 20, Wilson gives \([0.699, 0.972]\). For 20 of 20, it gives \([0.839, 1]\): even a perfect score on 20 items only tells you the true rate is probably above 84%.
Comparing two systems: use the pairs
When the old and new system answer the same items, most items are “easy for both” or “hard for both”. Those items tell you nothing about the difference. Only the discordant items (where exactly one system is right) carry information. A paired analysis uses this. For each item \(i\), let \(d_i = y_i^{\text{new}} - y_i^{\text{old}}\), which is \(+1\), \(0\) or \(-1\):
- Read the symbols
- \(y_i\)
- 1 if the system passes item \(i\), 0 if not
- \(b\)
- items the new system passes and the old one fails
- \(c\)
- items the old system passes and the new one fails
- \((b + c)/n\)
- the discordance rate: the share of items where the two systems disagree
A classic test that uses only the discordant items is McNemar’s test. If the two systems are equally good, \(b\) and \(c\) should be about equal, and
follows a chi-square distribution with 1 degree of freedom; values above 3.84 are significant at the 5% level. For small \(b + c\) (say under 25), use the exact binomial version instead. [Certain]
400 items. The old system passes 300 (75.0%), the new one 316 (79.0%). The new one wins \(b = 36\) items the old one failed, and loses \(c = 20\) items the old one passed.
Unpaired (wrong here): \(\mathrm{SE} = \sqrt{0.75 \cdot 0.25/400 + 0.79 \cdot 0.21/400} = 0.0297\). The gain is \(0.04\), so \(z = 1.35\) and the 95% interval is \([-1.8, +9.8]\) points. Not significant.
Paired: \(\bar d = 16/400 = 0.04\), and \((b + c)/n = 0.14\), so \(\mathrm{SE} = \sqrt{(0.14 - 0.0016)/400} = 0.0186\). Now \(z = 2.15\) and the interval is \([+0.4, +7.6]\) points. McNemar: \(\chi^2 = 16^2/56 = 4.57\), p-value 0.033 (exact binomial version: 0.044). The gain is real.
Same data, different answer. The paired SE is smaller because the 344 items where both systems agree add no noise to the difference.
If you cannot assume a formula, use the paired bootstrap: resample the \(n\) items with replacement many times, keeping each item’s old and new scores together, and compute the difference each time. The middle 95% of those differences is the interval. It works for any metric, including judge scores and averages over several samples per item.
How many items do you need?
Turn the question around: what is the smallest difference \(\delta\) you care about, and how many items \(n\) do you need to see it? With significance level 5% (\(z_{\alpha/2} = 1.96\)) and power 80% (\(z_\beta = 0.84\); power is the chance of detecting a real difference of size \(\delta\)):
- Read the symbols
- \(p_1, p_2\)
- pass rates of the old and new system (\(p_2 = p_1 + \delta\))
- \(\delta\)
- the smallest difference you want to detect
- \(r\)
- the discordance rate \((b + c)/n\); it must be at least \(\delta\)
- \((z_{\alpha/2} + z_\beta)^2\)
- \((1.96 + 0.8416)^2 = 7.85\) for 5% significance and 80% power
The \(1/\delta^2\) is the painful part: to see half the difference you need four times the items. Example: base rate 75%, \(\delta = 2\) points. Unpaired: \(n \approx 7{,}155\) items. Paired, if the two systems disagree on 10% of items: \(n \approx 1{,}955\). If they disagree on only 5%: \(n \approx 974\). Small prompt changes usually have low discordance, so pairing helps the most exactly where you need it.
How to read Figure 4
Start at the default: the paired curve reaches 80% power at about 1,955 items, the unpaired one at about 7,155. Halve \(\delta\) to 1 point: both numbers grow about four times. Lower the discordance rate: the paired curve moves left, because fewer items disagree and the difference becomes easier to see. The discordance rate cannot be below \(\delta\) (the gain itself comes from discordant items); the figure warns you if you try. A practical reading at a 75% base rate: with 300 items, a paired test (10% discordance) has about 79% power for a 5-point change, and an unpaired test has 87% power for a 10-point change. For a 2-point change, power is only 19% (paired) and 8% (unpaired).
A regression suite for your product
A regression suite is your test set, its graders and a pass/fail rule, run automatically on every change: a new prompt, a new model version, a new retriever, a new tool. Like unit tests in software, its job is to catch things that used to work and now break.
How to build it
- Start small and real. 50–200 items from real traffic and known failures are more useful than 10,000 synthetic ones. Grow it every week.
- Every bug becomes a test. When a user reports a bad answer, add it (and a few variants) with the correct grading rule.
- Slices. Tag each item: task type, language, input length, customer tier, risk level (safety, money, legal). Report every slice, not only the average.
- Freeze and version. The suite, the graders and the judge prompt have version numbers. A score is only comparable with a score from the same suite version.
- Keep a hidden part. If engineers tune prompts on the whole suite, the suite becomes a training set and stops predicting real quality. Keep a held-out part that is only run at release time.
- Measure cost and latency too. A change that gains 1 point but doubles latency or token cost may be a regression for the product (session 81).
The release gate
The average can hide a disaster. A new model can gain 3 points on common questions and lose 8 points on refunds, and the average barely moves. So the gate looks at every slice. For a slice with paired difference \(\bar d\), standard error \(\mathrm{SE}\), and a tolerance \(\tau\) (the largest drop you accept):
The first line says “we are confident the slice got worse”. The second says “we are confident it did not get worse by more than \(\tau\)” (a non-inferiority test). The third is honest: the slice is too small to know, so add items or have a person look. Safety slices often use \(\tau = 0\) and a hard rule (any new failure blocks). [Likely]
How to read Figure 5
At the defaults (\(\tau = 5\), 95%) the overall change is −0.40 points, with an interval from −2.32 to +1.52: the average says “no change”. But the refund slice dropped 7.5 points, and its whole interval is below zero, so the release is blocked. “Long documents” has only 80 items: its interval is so wide that no tolerance under 12 points can pass it. Switch to 99%: the refund interval now crosses zero, so the slice falls into “not enough evidence” — the drop is still there, but 120 items cannot prove it at that level. Small slices need more items, not looser rules.
- input: suite \(D\) (items with slices), graders \(g\), current system \(S_0\), candidate \(S_1\), tolerances \(\tau_s\), confidence \(z\)
- for each item \(x_i \in D\): \(y_i^0 \leftarrow g(x_i, S_0(x_i))\), \(y_i^1 \leftarrow g(x_i, S_1(x_i))\) // same items, same graders
- \(d_i \leftarrow y_i^1 - y_i^0\)
- for each slice \(s\): compute \(\bar d_s\) and \(\mathrm{SE}_s\) with (155.6), or a cluster bootstrap
- \(\quad\) verdict\(_s \leftarrow\) rule (155.9) with \(\tau_s\) // safety slices: \(\tau_s = 0\)
- if any verdict is “regression”: block and show the failing items
- else if any verdict is “not enough evidence”: send to human review or add items
- else: pass; also check cost and latency budgets
- store all outputs, grades and judge reasons with the suite version
After launch
Offline tests cannot see everything. After launch, add online signals: thumbs up and down, how often users retry or edit the answer, task completion, and A/B tests on real traffic (the same paired-or-not statistics apply). Sample live traffic every week, grade it, and move the surprising cases into the suite. The suite and the product then improve together.
All of it in code
The PyTorch code below fits Bradley–Terry ratings, computes a Wilson interval, runs the paired comparison and McNemar’s test, a paired bootstrap, and Cohen’s kappa for a judge. It is deterministic and runs on a CPU in about a second.
import math, torch torch.manual_seed(0) # 1) Bradley-Terry ratings from pairwise votes, formula (155.2) names = ["A", "B", "C", "D"] W = torch.tensor([[0, 60, 70, 80], # W[i, j] = votes where i beat j [40, 0, 55, 70], [30, 45, 0, 65], [20, 30, 35, 0]], dtype=torch.float64) s = torch.zeros(4, dtype=torch.float64, requires_grad=True) # one strength per model opt = torch.optim.LBFGS([s], max_iter=200, tolerance_grad=1e-12, line_search_fn="strong_wolfe") def nll(): opt.zero_grad() logp = torch.nn.functional.logsigmoid(s[:, None] - s[None, :]) # log P(i beats j) loss = -(W * logp).sum() loss.backward() return loss opt.step(nll) elo = 400 / math.log(10) * (s - s.mean()).detach() + 1000 # Elo scale, formula (155.1) print([f"{n}:{r:.1f}" for n, r in zip(names, elo.tolist())]) # A:1114.0 B:1027.9 C:981.8 D:876.2 # 2) Wilson interval, formula (155.5) def wilson(k, n, z=1.96): p = k / n; den = 1 + z * z / n c = (p + z * z / (2 * n)) / den h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / den return c - h, c + h print("wilson 18/20:", [round(v, 3) for v in wilson(18, 20)]) # [0.699, 0.972] # 3) paired comparison on the same 400 items, formulas (155.6) and (155.7) n = 400 old = torch.zeros(n); new = torch.zeros(n) old[:300] = 1 # old system passes 300 items new[:280] = 1; new[300:336] = 1 # new loses 20 of them, wins 36 others: 316 passes perm = torch.randperm(n); old, new = old[perm], new[perm] d = new - old se = d.std(unbiased=False) / math.sqrt(n) print(f"diff {d.mean():.3f} paired SE {se:.4f} z {d.mean() / se:.2f}") # 0.040 0.0186 2.15 b = int(((new == 1) & (old == 0)).sum()); c = int(((new == 0) & (old == 1)).sum()) print("McNemar chi2:", round((b - c) ** 2 / (b + c), 2)) # 4.57 # 4) paired bootstrap: resample items, keep the pairs together g = torch.Generator().manual_seed(0) idx = torch.randint(0, n, (10000, n), generator=g) boot = d[idx].mean(dim=1) lo, hi = torch.quantile(boot, torch.tensor([0.025, 0.975], dtype=boot.dtype)) print(f"bootstrap 95% CI [{lo:.3f}, {hi:.3f}]") # [0.002, 0.078] # 5) judge vs human agreement: Cohen's kappa, formula (155.4) judge = torch.tensor([1] * 145 + [0] * 55) human = torch.tensor([1] * 120 + [0] * 25 + [1] * 15 + [0] * 40) po = (judge == human).float().mean() pj, ph = judge.float().mean(), human.float().mean() pe = pj * ph + (1 - pj) * (1 - ph) print(f"agreement {po:.3f} chance {pe:.4f} kappa {(po - pe) / (1 - pe):.3f}") # 0.800 0.5788 0.525
The bootstrap interval \([0.002, 0.078]\) is close to the formula interval \([0.004, 0.076]\) from the worked example, as it should be for 400 items.
Where this is used
| Setting | Evidence and graders | Statistics and gate |
|---|---|---|
| Choosing a base or API model | Public benchmarks for the skills you need, arena ratings, then a short run on your own test set | Benchmark error bars; arena rating intervals; never decide on one benchmark alone |
| Customer-support assistant | Logged tickets, rubric-based LLM judge checked against support staff, rule checks for policy text | Per-slice gate (refunds, billing, legal); safety slice with \(\tau = 0\) |
| RAG over company documents | Retrieval recall on labeled queries, judge for “every claim supported by the passages” (session 153) | Cluster bootstrap by document |
| Code assistant and agents | Execution: unit tests, final environment state; \(\text{pass@}k\) (session 149) | Several runs per task (sampling noise); paired by task |
| Structured extraction (JSON) | Schema validation plus field-level exact match | Small \(\delta\) needs many items; track the schema error rate separately |
| Video QA, video retrieval and Video-RAG | Multiple-choice video benchmarks (for example MVBench); your own clips with time-stamped answers; judge for free-form answers | Cluster by video: many questions per clip are not independent |
| Key-moment detection and video summaries | Temporal overlap with human-marked moments; pairwise judge for summary quality, both orders | Paired comparison per video; slices by video length and domain |
| Training-time model selection (RLHF, DPO) | Pairwise judge win rate vs a reference model, with length control (session 67) | Report length-controlled and raw win rates; check that reward gains are not just length |
Common mistakes
- Reporting one number without an error bar. “81% vs 79%” on 300 items is noise. Always report the interval and \(n\).
- Unpaired tests on paired data. Both systems answered the same items; use (155.6), McNemar or a paired bootstrap. The unpaired test throws away most of your power.
- Judging in one order only. Position bias then moves the win rate by many points. Judge both orders, or at least randomize the order per item.
- Trusting the judge without checking it. Measure agreement with humans (kappa, and both error types) on your data before using a judge as a gate. Re-check when you change the judge model or prompt.
- Letting length win. If the new system writes 40% longer answers and wins the judge, check the length-controlled result before you celebrate.
- Only watching the average. Slices catch the regressions that averages hide (Figure 5).
- Tuning on the test set. After many prompt edits against the same suite, its score is optimistic. Keep a held-out part and refresh it.
- Treating clustered items as independent. Many questions about one document or video give error bars that are too small. Bootstrap by cluster.
Interview questions
What are the pros and cons of static benchmarks, arenas, and your own test set?basic
How does Chatbot Arena turn pairwise votes into a leaderboard?core
Name the main biases of an LLM judge and how to reduce each one.core
Why does swapping the order fix position bias but not length bias?advanced
How do you know if your LLM judge is good enough to use?core
Old system 75%, new system 79% on the same 400 items. Is the gain real?core
How many test items do you need to detect a 2-point improvement?core
Why is the Wald interval bad for small test sets, and what do you use instead?basic
Your test questions come in groups of ten per document. What changes in the statistics?advanced
Design a regression suite and release gate for an LLM product.advanced
The new model wins 70% of pairwise judge comparisons, but users do not notice. What could be going on?advanced
Exercises
- Model A is rated 1200 and model B 1100. What share of votes should A win? What gap gives a 90% win rate?
- A judge says “pass” for every item. Humans pass 90% of items. What are the raw agreement and Cohen’s kappa?
- Compute the Wilson 95% interval for 9 correct out of 10.
- Two systems on 1,000 shared items: \(b = 30\), \(c = 18\). Compute \(\bar d\), the paired SE, \(z\), and McNemar’s \(\chi^2\). Is the difference significant at 5%?
- Your paired test needs 1,955 items to see a 2-point gain. How many does it need for a 1-point gain, if the discordance rate stays at 10%?
- In Figure 3, set \(\beta = 0\), \(\gamma = 0.3\), and make B longer by 200 tokens. Which way do the bars move, and does order swapping help?
Answerscheck
Key takeaways
- Benchmarks and arenas help you choose a model; only your own test set, with slices, tells you whether your system works.
- Arenas turn pairwise votes into ratings with Bradley–Terry (155.1, 155.2); a 100-point gap is a 64% win rate.
- LLM judges can match human agreement, but they prefer the first answer, longer answers and sometimes their own. Judge both orders, control for length, and check the judge against humans with kappa (155.4).
- Use Wilson intervals (155.5) for small sets and paired analysis (155.6, 155.7, bootstrap) when two systems answer the same items; it can turn noise into a clear result.
- Sample size grows as \(1/\delta^2\) (155.8): 300 paired items can see a 5-point change, not a 1- or 2-point change.
- A regression suite is versioned, sliced, and gated per slice (155.9): block confident regressions, pass confident non-inferiority, and send thin evidence to review.
Key formulas
References for this session
- Hendrycks et al. (2020). Measuring Massive Multitask Language Understanding
- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Chiang et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- Liang et al. (2022). Holistic Evaluation of Language Models
- EleutherAI (code). LM Evaluation Harness
- Hendrycks et al. (2020). Measuring Massive Multitask Language Understanding
- Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems (GSM8K)
- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Chiang et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- Dubois et al. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Liang et al. (2022). Holistic Evaluation of Language Models
- EleutherAI (code). LM Evaluation Harness
- Miller (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
- Li et al. (2023). MVBench: A Comprehensive Multi-modal Video Understanding Benchmark