LLM & VLM courseBlog
40Track 3: Pretraining and scale · session 10 of 22

Evaluating base models

After pretraining you have a base model and one question: is it good? This session shows the tools to answer it: perplexity and bits per byte, the big benchmarks (MMLU, HellaSwag, GSM8K), the two ways to score them (log-likelihood and generation), how test data leaks into training, and how to run an evaluation harness so your numbers mean something.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: BeginnerBefore this: Sessions 3, 6, 8 and 9

What you will learn

  • Why perplexity can only be compared with the same tokenizer and the same text, and how bits per byte fixes the tokenizer problem
  • What MMLU, HellaSwag and GSM8K measure, how big they are, and what random guessing scores
  • Log-likelihood scoring for multiple choice (acc, acc_norm, letter vs cloze format) and generation scoring with answer extraction (exact match, pass@k)
  • How large the noise is: the standard error of an accuracy, and when a difference between two models is real
  • Contamination: how test data leaks into training, and how to detect it (n-gram overlap, Min-K% Prob, fresh test sets, canaries)
  • How to use lm-evaluation-harness correctly: fixed settings, logged samples, no chat template on base models, and a reporting checklist

Why this matters

You finished a pretraining run. The loss curve went down. Now someone asks: “Is this model better than the last one? Better than Llama?” The loss alone cannot answer that. You need evaluation: a fixed, repeatable way to turn a model into a few numbers that others can trust.

A base model is the model right after pretraining, before any instruction tuning or RLHF. It does not follow instructions. It only continues text. So we must ask our questions in a form that “continue the text” can answer: we compare the probability of possible endings, or we let the model continue a worked example and read the answer at the end.

This sounds easy. It is not. The same model can get very different scores from small changes in the prompt, in the scoring rule, or in the code. In 2023, Hugging Face found that LLaMA 65B scored 63.7% on MMLU with one implementation (HELM), 63.6% with the original code, and 48.8% with the version of the Evaluation Harness used then. Same model, same questions, a 15-point gap, and the ranking of models changed too. [Certain] (Hugging Face blog)

In session 39 we learned to train big models with limited memory. This session closes the loop: how to measure what the training produced. The same tools are used every few thousand steps during a real pretraining run, to pick data mixtures, to compare checkpoints, and to decide when to stop. They come back for chat models in session 155 and for vision models in session 192.

Three kinds of evaluation for a base model

There are three families of measurement. Each answers a different question.

  1. Intrinsic: perplexity or bits per byte. How well does the model predict held-out text? This is the training loss measured on new text. It is cheap, smooth and has low noise, so it is good for comparing checkpoints of one model.
  2. Multiple-choice benchmarks scored by log-likelihood. Each question has a few possible answers. We ask the model how probable each answer is and pick the most probable one. MMLU and HellaSwag work this way. No text is generated.
  3. Generative benchmarks. The model writes an answer, and a program checks it: exact match of a number (GSM8K), or running unit tests on code (HumanEval).
Test itemquestion + gold Promptk-shot examples Score each choicelog p(choice | prompt) Generategreedy, until stop text argmaxacc, acc_norm extractexact match mean± std. error
Figure 1. The evaluation pipeline. Every test item becomes a prompt (often with a few solved examples first). Then one of two paths: score each possible answer by its log-probability (top, blue), or let the model generate and check the answer (bottom, orange). The per-item scores are averaged, and the average has an error bar.

Perplexity, and why it is only comparable sometimes

Recall from session 3 that the perplexity of a model on a text of \(T\) tokens is the exponential of the average negative log-likelihood (NLL) per token:

\[ \mathrm{PPL} = \exp\!\left( -\frac{1}{T} \sum_{t=1}^{T} \log p_\theta(x_t \mid x_{\lt t}) \right) \tag{40.1} \]
Read the symbols
\(x_t\)
the token at position \(t\) of the test text
\(x_{\lt t}\)
all tokens before it (the context)
\(p_\theta\)
the probability the model with weights \(\theta\) gives to the true next token
\(T\)
the number of tokens. This depends on the tokenizer.

A perplexity of 16 means: on average the model is as unsure as if it had to pick uniformly among 16 tokens. Lower is better.

The trap is in \(T\). Perplexity is a number per token. Two models with different tokenizers cut the same text into a different number of tokens. A tokenizer with a big vocabulary makes fewer, longer tokens. Each token then carries more information and is harder to predict, so per-token perplexity goes up, even if the model predicts the text just as well. Comparing per-token perplexity across tokenizers compares units, not models.

The fix is to normalize by something the tokenizer cannot change: the number of UTF-8 bytes of the text. The total NLL of the whole text is a fair quantity (it is the code length of the text under the model). Divide it by the bytes and convert nats to bits:

\[ \mathrm{BPB} = \frac{1}{B \ln 2} \sum_{t=1}^{T} -\log p_\theta(x_t \mid x_{\lt t}) \tag{40.2} \]
Read the symbols
\(\mathrm{BPB}\)
bits per byte: how many bits the model needs, on average, to encode one byte of the text
\(B\)
the number of UTF-8 bytes of the test text (the same for every tokenizer)
\(\ln 2\)
converts nats (natural log) to bits (log base 2)

Combine (40.1) and (40.2). If the tokenizer produces on average \(B/T\) bytes per token, then

\[ \mathrm{PPL} = 2^{\,\mathrm{BPB} \cdot B / T} \tag{40.3} \]

So per-token perplexity grows exponentially with bytes per token. The Pile paper reported bits per byte for exactly this reason: it compares models with different tokenizers. [Certain] (Gao et al., 2020) Bits per character and “perplexity per word” are older versions of the same idea.

Worked example: the better model has the worse perplexity

A test text has \(B = 1{,}200\) bytes. Model A has a large vocabulary: 4.0 bytes per token, so \(T = 300\). Its total NLL is 831.8 nats. Model B has a small vocabulary: 3.0 bytes per token, \(T = 400\). Its total NLL is 915.0 nats.

Per token: A has \(831.8 / 300 = 2.773\) nats, so \(\mathrm{PPL}_A = e^{2.773} = 16.0\). B has \(915.0/400 = 2.287\) nats, so \(\mathrm{PPL}_B = e^{2.287} = 9.85\). B looks much better.

Per byte: \(\mathrm{BPB}_A = 831.8 / (1200 \ln 2) = 1.00\) and \(\mathrm{BPB}_B = 915.0/(1200 \ln 2) = 1.10\). A needs fewer bits for the same text. A is the better model. Check with (40.3): \(2^{1.0 \times 4} = 16\) and \(2^{1.1 \times 3} = 2^{3.3} = 9.85\).

Model A Model B
model Amodel B
Figure 2. Per-token perplexity versus bits per byte for two models on the same text. Set the tokenizer (bytes per token) and the real quality (bits per byte) of each model. The left panel shows what a per-token perplexity table would say; the right panel shows the fair comparison.
How to read Figure 2

Start with the defaults: A is better in bits per byte (1.00 vs 1.10), but B “wins” on perplexity (9.85 vs 16.0) only because its tokens are shorter. Now set both bytes-per-token sliders to the same value: the two panels then agree, because with one tokenizer perplexity and bits per byte rank models the same way. That is the rule: perplexity is comparable only with the same tokenizer and the same text.

Same tokenizer, but still not comparable

Bits per byte fixes the tokenizer, not the text. Perplexity also changes with:

  • The test data. Code and Wikipedia are easy to predict; tweets and math are harder. A model trained on more code will look better on a code-heavy test set. Only compare on the exact same documents.
  • The context each token sees. If you cut a long document into independent 512-token pieces, the first tokens of each piece have no context and get high loss. A sliding window with overlap (each token scored with a long context) gives lower perplexity. The harness calls this “rolling log-likelihood”. [Certain] Report the window and the stride.
  • Special tokens. Whether a BOS token is added, and whether EOS is scored, changes the sum (see session 9).

And low perplexity does not mean a model is useful. It is an average over every token, most of them easy (“the”, punctuation). Two models with the same perplexity can differ a lot on reasoning. That is why we also use benchmarks.

The benchmarks: MMLU, HellaSwag, GSM8K

A benchmark is a fixed set of test questions with known answers (the gold answers) and a fixed scoring rule. Three appear in almost every base-model report.

BenchmarkWhat it testsSize used for scoringFormatRandom guess
MMLU (Hendrycks et al., 2020)knowledge in 57 subjects: law, medicine, math, history…14,042 test questions [Certain]4 choices A–D, usually 5-shot (5 solved examples from the same subject first) [Certain]25%
HellaSwag (Zellers et al., 2019)common sense: pick the most plausible next sentence10,042 validation items (test labels are not public, so the validation set is what people report) [Likely]4 endings, scored by log-likelihood; wrong endings were written by a model and filtered to fool models (“adversarial filtering”) [Certain]25%
GSM8K (Cobbe et al., 2021)grade-school math word problems, 2 to 8 steps1,319 test problems (7,473 train) [Certain]generate a step-by-step solution; the final number is compared with the gold number; often 8-shot chain-of-thought [Likely]about 0%

When HellaSwag came out, humans scored above 95% and the best models below 48%. [Certain] Today strong base models are near the human level, which is called saturation: the benchmark no longer separates good models. Later benchmarks (MMLU-Pro with 10 choices, harder math sets) exist for this reason.

Scoring 1: log-likelihood for multiple choice

A base model cannot be trusted to write “B” on command. So for multiple choice we do not generate. We put the question (the context \(x\)) in front of each answer \(c\) and compute how probable the model finds that answer, token by token:

\[ s(c) = \sum_{j=1}^{|c|} \log p_\theta\big(c_j \mid x,\, c_{\lt j}\big), \qquad \hat{y} = \argmax_{c \in \mathcal{C}}\; s(c) \tag{40.4} \]
Read the symbols
\(x\)
the context: few-shot examples plus the question
\(c\), \(c_j\)
one candidate answer, and its \(j\)-th token
\(|c|\)
the number of tokens in the answer
\(\mathcal{C}\)
the set of candidate answers (4 for MMLU and HellaSwag)
\(\hat y\)
the model’s prediction. It is correct if it equals the gold answer.

One forward pass per candidate is enough (the context can even be shared with a KV cache). The accuracy, written acc, is the fraction of items where \(\hat y\) is the gold answer.

The length problem and acc_norm

Every token adds a negative number to \(s(c)\). So long answers get lower scores just for being long. A short, wrong ending like “ waited.” can beat a long, correct one. The fix is to normalize by length:

\[ s_{\text{norm}}(c) = \frac{s(c)}{\mathrm{len}(c)}, \qquad \hat{y}_{\text{norm}} = \argmax_{c \in \mathcal{C}}\; s_{\text{norm}}(c) \tag{40.5} \]

Here \(\mathrm{len}(c)\) is a length that does not depend on the tokenizer. In lm-evaluation-harness, acc_norm divides by the number of characters of the answer string. [Certain] (checked in the harness source) Dividing by tokens is another choice, but it again ties the score to the tokenizer. HellaSwag results are usually reported as acc_norm. [Likely]

Two formats: cloze and letter

  • Cloze format (“fill in the blank”): score the full answer text, as above. This works even for small models, because it only asks “which continuation is more natural?”.
  • Letter format (multiple-choice format): show all four answers labelled A–D, end with “Answer:”, and compare only the probabilities of the tokens “ A”, “ B”, “ C”, “ D”. This is how the original MMLU code scores. It needs a model that has learned the skill of reading labelled options; small base models are close to random in this format. [Likely]

The OLMES standard (Gu et al., 2024) proposes to use cloze for small base models and the letter format for larger ones, with documented prompts and normalization, so that numbers from different papers match. [Certain] (arXiv 2406.08446)

Figure 3. One HellaSwag-style item scored four ways. The per-token log-probabilities are made up for teaching (a toy, not a real model). The gold ending is B. Click a scoring rule to see which ending the model “chooses”. Each chip is one token in quotes with its log-probability; a space inside the quotes, as in “ A”, is part of the token.
How to read Figure 3

Each bar’s length is minus the score of one ending, so the shortest bar is the ending the model prefers. The token chips under each ending show the per-token log-probabilities. With the raw sum, the short ending A wins and the item counts as wrong. Dividing by characters (acc_norm) makes B win. “Letter only” shows the letter format for a small base model: the four letter probabilities are almost equal, so the choice is close to random.

The numbers from Figure 3

Ending A, “ waited.” (8 characters, 2 tokens): \(s = -2.1 - 0.6 = -2.7\). Ending B, “ set a timer for forty minutes.” (31 characters, 7 tokens): \(s = -6.2\). C: \(s = -11.1\) (26 characters). D: \(s = -8.0\) (14 characters).

Sum: A wins with \(-2.7\). Wrong. Per character: A \(= -2.7/8 = -0.338\), B \(= -6.2/31 = -0.200\), C \(= -0.427\), D \(= -0.571\). B wins. Correct. Per token: A \(-1.35\), B \(-0.886\), C \(-1.586\), D \(-2.0\); B wins here too.

Algorithm 1: log-likelihood scoring of a multiple-choice benchmark
  1. input: model \(p_\theta\), items \(\{(x_i, \mathcal{C}_i, y_i)\}_{i=1}^{n}\), \(k\) few-shot examples, a fixed prompt template
  2. for each item \(i\) do
  3.   \(\text{ctx} \leftarrow\) template(\(k\) solved examples) + template(\(x_i\)) // same examples and seed for every model
  4.   for each candidate \(c \in \mathcal{C}_i\): \(s(c) \leftarrow \sum_j \log p_\theta(c_j \mid \text{ctx}, c_{\lt j})\) // formula (40.4), one forward pass
  5.   \(\text{acc}_i \leftarrow [\argmax_c s(c) = y_i]\), \(\;\text{accnorm}_i \leftarrow [\argmax_c s(c)/\mathrm{len}(c) = y_i]\) // formula (40.5)
  6. end for
  7. return mean of acc and accnorm, each with its standard error // formula (40.7)

Scoring 2: generation and answer extraction

For GSM8K, the answer is a number that the model must compute. We show a few solved problems (each ending with the final answer), then the new problem, and let the model write. Then a small program reads the answer:

Algorithm 2: generation scoring (GSM8K style)
  1. input: model, problems \(\{(q_i, a_i)\}\), \(k\) solved examples with reasoning, stop strings (for example “Question:” or a blank line)
  2. for each problem \(i\) do
  3.   \(\text{out} \leftarrow\) greedy decoding from template(\(k\) examples) + \(q_i\), until a stop string or \(N_{\max}\) tokens // deterministic
  4.   \(\hat a \leftarrow\) number after “####” in out; if none, the last number in out // extraction rule is part of the benchmark
  5.   normalize: remove commas, “$”, a trailing “.”
  6.   \(\text{em}_i \leftarrow [\hat a = a_i]\) // exact match
  7. end for
  8. return mean exact match and its standard error

In GSM8K, every gold solution ends with a line like #### 72. [Certain] The few-shot examples teach the model to end the same way. Many errors in practice are extraction errors, not reasoning errors: the model wrote “72 apples” and the regex expected a bare number, or the model kept going and wrote a second problem because the stop string was wrong. Always read a sample of the raw outputs.

Chain-of-thought examples (step-by-step solutions in the few-shot prompt) help a lot on math. [Certain] (Wei et al., 2022) So “GSM8K 8-shot CoT” and “GSM8K 5-shot without CoT” are different benchmarks. Always report the number of shots and whether CoT was used.

pass@k for code

For code (HumanEval), the output is run against unit tests. A model gets \(k\) tries; the item counts as solved if any try passes. With \(n\) samples per problem, of which \(c\) pass, the unbiased estimate is [Certain] (Chen et al., 2021):

\[ \text{pass@}k = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}} \tag{40.6} \]

Example: \(n = 10\), \(c = 3\). Then \(\text{pass@}1 = 0.30\) and \(\text{pass@}5 = 1 - \binom{7}{5}/\binom{10}{5} = 1 - 21/252 = 0.917\). Sampling uses a temperature above 0, so fix the seed.

How noisy is a score?

A benchmark is a sample of questions. A different sample would give a slightly different score. Each item is right (1) or wrong (0), so with accuracy \(p\) on \(n\) items, the standard error (the typical size of this sampling noise) is:

\[ \mathrm{SE} = \sqrt{\frac{p\,(1 - p)}{n}}, \qquad \text{95\% interval} \approx p \pm 1.96\,\mathrm{SE} \tag{40.7} \]
Read the symbols
\(p\)
measured accuracy (a fraction between 0 and 1)
\(n\)
number of test items
\(1.96\)
for a normal distribution, 95% of the mass lies within 1.96 standard errors of the mean

To compare two models A and B on the same \(n\) items, a simple test treats the two scores as independent:

\[ z = \frac{p_A - p_B}{\sqrt{\mathrm{SE}_A^2 + \mathrm{SE}_B^2}} \tag{40.8} \]

If \(|z| \gt 1.96\), the difference is unlikely to be pure noise. Because both models answer the same questions, a paired test (look at items where exactly one model is right) is more powerful and is the better choice when you have per-item results. [Certain]

Worked example: is 52% vs 50% on GSM8K real?

\(n = 1{,}319\). At \(p = 0.5\): \(\mathrm{SE} = \sqrt{0.25/1319} = 0.0138\), so 1.38 points, and the 95% interval is about \(\pm 2.7\) points.

For 52% vs 50%: \(z = 0.02 / \sqrt{0.01376^2 + 0.01377^2} = 1.03\). Not significant. A 2-point gain on GSM8K from one training change is within noise.

MMLU has \(n = 14{,}042\); at \(p = 0.6\) the \(\mathrm{SE}\) is 0.41 points (95%: \(\pm 0.81\)). To get ±1 point at 95% when \(p = 0.5\), you need \(n = (1.96/0.01)^2 \times 0.25 = 9{,}604\) items.

Figure 4. Error bars of a benchmark score. Each dot is a model’s accuracy and each bar its 95% interval from (40.7). The readout gives the \(z\) value of (40.8) and says whether the gap is larger than the noise.
How to read Figure 4

With the GSM8K size, a 2-point gap has overlapping bars and \(z \approx 1.03\). Click “MMLU test”: the bars shrink by a factor of about 3.3 (the axis zooms in, so watch the tick labels) and the same 2-point gap is now clearly significant. Click “HumanEval”: with only 164 problems, even a 5-point gap is noise. Small benchmarks need big gaps.

Contamination: when the test leaks into training

Contamination means that test items (questions, and often answers) were in the training data. Pretraining data is a large crawl of the web (session 34), and benchmarks are on the web: on GitHub, in papers, on study sites, in blog posts that discuss them. So a model can partly remember the test. Its score then measures memory, not skill.

Contamination comes in levels:

  • Exact copy: the test question and answer appear word for word in a training document.
  • Near copy or paraphrase: the same problem with small changes in wording or numbers.
  • Format or distribution leakage: the training data contains many very similar problems (for example, synthetic data made by imitating the benchmark). This is hard to call cheating, but it still inflates the score.

Detection 1: n-gram overlap

An n-gram is a run of \(n\) consecutive words (or tokens). Split each test item into n-grams and look them up in an index of the training data. GPT-3 used 13-grams and flagged a test item as contaminated if any of its 13-grams appeared in the training set. [Certain] (Brown et al., 2020) The GPT-3 authors also reported that a bug in their filtering left some overlap in the training data. [Certain]

The choice of \(n\) is a trade-off. Large \(n\) gives few false alarms but misses paraphrases. Small \(n\) catches more, but common phrases such as “how many apples does” match innocent text.

Training document
Test item (words covered by a matching n-gram are marked)
Figure 5. An n-gram contamination check on one made-up GSM8K-style test item (24 words). Pick a training document and a value of \(n\). Words are lower-cased and punctuation is removed, as most checkers do.
How to read Figure 5

The exact copy matches at every \(n\): 12 of 12 13-grams. The paraphrase shares 3 of 17 8-grams but zero 13-grams, so a GPT-3-style 13-gram filter misses it. “Same topic only” shares no 5-grams, but at \(n = 3\) it matches 2 trigrams (“apples to his”, “to his sister”): a false alarm. No single \(n\) catches everything.

Detection 2: the model’s own probabilities

Often you cannot search the training data (closed models, or data not kept). Then you can look at the model. A text the model has seen tends to have no very surprising tokens. Min-K% Prob (Shi et al., 2023) takes the \(k\%\) of tokens with the lowest probability and averages their log-probabilities:

\[ \text{Min-K\%}(x) = \frac{1}{|S_k|} \sum_{x_t \in S_k} \log p_\theta(x_t \mid x_{\lt t}) \tag{40.9} \]

Here \(S_k\) is the set of the \(k\%\) least likely tokens of the text \(x\). A high value (no strong surprises) suggests the text was in training. This is a statistical signal, not proof. [Certain] (arXiv 2310.16789)

Detection 3: fresh, parallel test sets

The strongest check: write a new test set that matches the old one in style and difficulty but was never published. GSM1k is such a copy of GSM8K. Some models dropped by up to 8 points on it, and models that gave higher probability to GSM8K test items tended to drop more (Spearman \(r^2 = 0.36\)). [Certain] (Zhang et al., 2024)

Prevention

  • Decontaminate your training data against every benchmark you will report: remove documents with long n-gram overlap with test items. Do this before training; you cannot remove it later.
  • Canary strings. BIG-bench files contain a unique “canary” GUID string. If a model can complete the string, its training data contained the benchmark files. [Certain]
  • Hold out private test sets and newer benchmarks made after the training data cutoff.
  • Report the contamination analysis with the scores, as the GPT-3 and Llama 2 papers did. [Certain]

Using an evaluation harness correctly

An evaluation harness is a library that holds the benchmark data, prompt templates, scoring rules and metrics in one place, so every model is tested the same way. The most used one for base models is EleutherAI’s lm-evaluation-harness (lm_eval). [Certain] It has four request types: loglikelihood, multiple_choice, generate_until and loglikelihood_rolling (for perplexity). [Certain] A typical run on a base model:

pip install "lm_eval[hf]"
lm_eval --model hf \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype=bfloat16 \
    --tasks hellaswag,mmlu,gsm8k \
    --device cuda:0 --batch_size 8 \
    --output_path results/ --log_samples

The revision=step100000 part loads an intermediate checkpoint from the Hub. This is how you track a benchmark during training. [Certain] (example from the harness README) --log_samples saves every prompt and every model output, so you can read them later.

A harness removes many errors, but not all. The rules from Biderman et al. (2024), who maintain the harness, are short: share the exact code and settings, report everything that can change the score, and look at the outputs. [Certain] (arXiv 2405.14782) In practice:

  1. Pin versions. Record the harness version (or git commit), the task version and the model revision. Task definitions change over time.
  2. Fix and report every setting: number of shots, which metric (acc or acc_norm), CoT or not, maximum generation length, stop strings, dtype and seed.
  3. No chat template for a base model. A base model was never trained on role tokens. Do not pass --apply_chat_template. For instruction-tuned models, use the model’s own template. (See session 9.)
  4. Check the BOS token. Some models (Llama, Gemma) expect a BOS token at the start; scoring without it can lower the results. [Likely]
  5. Use --limit only for debugging. A score on 100 items has an SE of about 5 points.
  6. Read 20 logged samples per task before you believe a number: is the prompt what you expect? Is extraction working?
  7. Report the standard error next to every score (the harness prints it).

Here is everything above in a small runnable script. The “model” is a byte-level bigram so that it runs in a second on a CPU, but the scoring functions are the same ones a harness uses for a real LLM.

import math, re, torch
torch.manual_seed(0)

# 1) a tiny "language model": a byte-level bigram with add-one smoothing
corpus = ("the cat sat on the mat. the dog sat on the log. "
          "she put the cake in the oven and set a timer. "
          "he put the bread in the oven and waited for ten minutes. ") * 20
data = torch.tensor(list(corpus.encode("utf-8")))
counts = torch.ones(256, 256)                          # add-one smoothing
counts.index_put_((data[:-1], data[1:]), torch.ones(len(data) - 1), accumulate=True)
logp = torch.log(counts / counts.sum(1, keepdim=True)) # logp[a, b] = log p(b | a)

def loglik(context, continuation):
    """Formula (40.4): sum of log p over the continuation tokens, given everything before."""
    ids = list((context + continuation).encode("utf-8"))   # tokenize together, then split
    start = len(context.encode("utf-8"))
    return sum(logp[ids[i - 1], ids[i]].item() for i in range(start, len(ids)))

def bits_per_byte(text):
    """Formula (40.2): total NLL in bits divided by the number of predicted UTF-8 bytes."""
    ids = list(text.encode("utf-8"))
    nll = -sum(logp[ids[i - 1], ids[i]].item() for i in range(1, len(ids)))
    return nll / math.log(2) / (len(ids) - 1)

# 2) multiple choice: acc (raw sum) and acc_norm (divide by characters), formula (40.5)
def score_mc(context, choices, gold):
    ll = [loglik(context, c) for c in choices]
    norm = [l / len(c) for l, c in zip(ll, choices)]
    pick = max(range(len(ll)), key=ll.__getitem__)
    pick_norm = max(range(len(ll)), key=norm.__getitem__)
    return int(pick == gold), int(pick_norm == gold), ll, norm

ctx = "she put the cake in the oven and"
choices = [" waited.", " set a timer.", " flew away.", " ate a fox."]
acc, acc_norm, ll, norm = score_mc(ctx, choices, gold=1)
print("sum:     ", [round(x, 2) for x in ll])
print("per char:", [round(x, 2) for x in norm])
print("acc:", acc, " acc_norm:", acc_norm)

# 3) generation scoring: extract the final number, then exact match
def extract_answer(text):
    m = re.search(r"####\s*(-?[\d,\.]+)", text)
    nums = [m.group(1)] if m else re.findall(r"-?\d[\d,]*\.?\d*", text)
    return nums[-1].replace(",", "").rstrip(".") if nums else None

print(extract_answer("3 boxes x 12 = 36, 36 - 9 = 27.\n#### 27"), extract_answer("So Tom has 27 apples."))

# 4) bits per byte of a held-out sentence
print("bits per byte:", round(bits_per_byte("the dog sat on the mat."), 3))

# 5) standard error of an accuracy, formula (40.7)
p, n = 0.50, 1319
print("GSM8K standard error (points):", round(100 * math.sqrt(p * (1 - p) / n), 2))

# Output:
# sum:      [-25.41, -34.31, -51.73, -33.11]
# per char: [-3.18, -2.64, -4.7, -3.01]
# acc: 0  acc_norm: 1
# 27 27
# bits per byte: 2.921
# GSM8K standard error (points): 1.38

Notice that the toy reproduces the length problem from Figure 3: the raw sum picks the short “ waited.”, and acc_norm picks the gold “ set a timer.”.

Where this is used

SettingWhat is measuredTypical method
Monitoring a pretraining runvalidation bits per byte on a fixed held-out set, plus a few cheap benchmarks every N stepsloglikelihood_rolling and HellaSwag-style log-likelihood tasks; smooth signals for early checkpoints
Choosing a data mixture or tokenizersmall “proxy” models trained on each option (session 35)bits per byte (tokenizers differ, so never per-token perplexity), plus benchmarks above chance
Base-model release reports (Llama 3, Qwen)MMLU, HellaSwag, GSM8K, code and many morea fixed harness, with shots, CoT and metric reported per task, and a contamination analysis [Likely]
Quantization and compressionhow much quality is lost at 4 bits (session 105)perplexity on the same text with the same tokenizer (valid here), plus a few benchmarks
Video question answering (Video-MME, MVBench)understanding of video clipsmultiple choice; the model is asked for a letter, so letter-format issues apply; also check for leakage of test videos into training [Likely]
Video retrieval, action recognition, key-moment detectionRecall@K, top-1 accuracy, mAP on fixed test splitssame rules: fixed split, error bars, no overlap between training clips and test clips (near-duplicate detection from session 209 finds re-uploads)

Common mistakes

  • Comparing per-token perplexity across tokenizers. Use bits per byte (40.2), on the exact same text.
  • Comparing numbers from different papers as if they were one table. Shots, prompt, acc vs acc_norm, letter vs cloze, and CoT all change the score, sometimes by more than 10 points (the MMLU example above).
  • Ignoring the noise. A 1-point gain on GSM8K (SE 1.4 points) is not a result. Report the standard error and use a paired test.
  • Using a chat template on a base model, or forgetting the template on a chat model. Both lower scores for reasons unrelated to skill.
  • Tokenizing the context and the answer separately, which breaks the space at the boundary and distorts \(s(c)\).
  • Not reading the outputs. Bad stop strings and a strict regex produce “wrong” answers that are actually right.
  • Tuning on the test set. Choosing prompts, checkpoints or data mixtures by their test score turns the test set into a training signal. Keep a separate validation set for decisions.
  • Skipping decontamination and reporting a benchmark that is in your training data.

Interview questions

Model A has perplexity 9 and model B has perplexity 14 on the same text. Is A better?basic
Only if they use the same tokenizer and the same evaluation setup (context window, BOS handling). Perplexity is per token, and a tokenizer with longer tokens gives higher per-token perplexity for the same quality: \(\mathrm{PPL} = 2^{\mathrm{BPB}\cdot B/T}\) (40.3). With different tokenizers, compare bits per byte: total NLL divided by \(B \ln 2\).
How is MMLU scored for a base model that cannot follow instructions?basic
No free generation is needed. The prompt has 5 solved examples from the same subject, then the question with options A–D and “Answer:”. In the letter format the model’s log-probabilities of the tokens “ A”, “ B”, “ C”, “ D” are compared and the highest is the prediction. In the cloze format the full answer texts are scored by log-likelihood instead. Accuracy over 14,042 test questions; chance is 25%.
What is acc_norm and why does it exist?core
The log-likelihood of an answer is a sum of negative numbers, one per token, so longer answers are penalized for length. acc_norm divides each answer’s log-likelihood by its length before taking the argmax. In lm-evaluation-harness the length is the number of characters, which does not depend on the tokenizer. It is the usual metric for HellaSwag, where endings have very different lengths.
Log-likelihood scoring versus generation: what are the pros and cons?core
Log-likelihood: cheap (one forward pass per choice), deterministic, works for weak base models, no parsing. But it only works when answers are a short list, and it measures ranking, not the ability to produce the answer. Generation: tests the real skill (multi-step reasoning, code), works for open answers. But it is slower, depends on decoding settings, stop strings and the extraction regex, and is noisier.
Your new checkpoint scores 52% on GSM8K, the old one 50%. Did it improve?core
Not shown. With \(n = 1319\), \(\mathrm{SE} = \sqrt{0.25/1319} \approx 1.4\) points; the 95% interval is about \(\pm 2.7\) points. \(z = 0.02/\sqrt{0.0138^2 + 0.0138^2} \approx 1.03 \lt 1.96\). A paired test on per-item results is more powerful; better still, add more evaluation data or average several related benchmarks.
What is data contamination and how do you detect it?core
Test items (often with answers) present in the training data, so the score measures memory. Detection: (1) n-gram overlap between test items and the training corpus (GPT-3 used 13-grams); (2) when the data is not available, membership signals from the model such as Min-K% Prob or unusually low loss on test items compared with similar fresh items; (3) a fresh parallel test set (GSM1k for GSM8K): a large drop indicates leakage; (4) canary strings (BIG-bench).
Why can a 13-gram filter miss contamination, and why not just use 3-grams?advanced
A paraphrase with changed wording or numbers can share no 13-gram with the original while the problem is the same (in Figure 5, 0 of 12). With 3-grams, common phrases (“to his sister”) match unrelated documents and flag most of the test set. So n-gram filters catch exact and near copies; paraphrase leakage needs embedding similarity, model-based checks or fresh test sets.
A small model scores 26% on MMLU. What do you conclude and what would you use instead?core
That is chance level (25%), so MMLU gives no signal at this size, especially in letter format, which small models cannot use. Use bits per byte on held-out text, and cloze-format tasks (HellaSwag, ARC in cloze form) whose scores rise smoothly. OLMES recommends cloze for small models for this reason.
The same model gets 63.7% and 48.8% on MMLU in two tools. How can that happen?advanced
The implementations differ in the scoring rule (only the letter token vs the full “letter + answer” sequence vs generated text), in the prompt wording, and in normalization. Each is a different measurement. This happened with LLaMA 65B in 2023. The lesson: only compare models evaluated with the same code, prompts and settings, and report them.
How would you set up evaluation during a pretraining run?advanced
A fixed held-out set from the training distribution plus some out-of-domain text, scored as bits per byte (with a fixed window and stride); a few cheap log-likelihood benchmarks that are above chance at the current scale; every N steps on the saved checkpoint, with the harness pinned to a version. Decontaminate the training data against all reported benchmarks first. Keep a validation split for decisions (data mixture, learning rate) and use test splits only for the final report, with standard errors.

Exercises

  1. A text has 2,000 bytes. Model A (5 bytes per token) has total NLL 1,250 nats. Model B (3.5 bytes per token) has total NLL 1,300 nats. Compute the per-token perplexity and the bits per byte of each. Which model is better?
  2. In Figure 3, which ending wins if we use the raw sum but drop the last token (“.”) of every ending? Does this fix the length problem?
  3. What is the 95% interval of an accuracy of 70% on HumanEval (164 problems)?
  4. With \(n = 20\) samples per problem and \(c = 2\) correct, compute pass@1 and pass@10.
  5. In Figure 5, set \(n = 5\). How many 5-grams of the test item match the paraphrase? Name one rule that flags the paraphrase but not the “same topic” document.
  6. You want to evaluate a base model on GSM8K with lm_eval. List three settings you must report with the score.
Answerscheck
1. A: \(T = 400\), \(1250/400 = 3.125\) nats, \(\mathrm{PPL} = e^{3.125} = 22.8\); \(\mathrm{BPB} = 1250/(2000 \ln 2) = 0.902\). B: \(T = 571.4\), \(1300/571.4 = 2.275\) nats, \(\mathrm{PPL} = 9.73\); \(\mathrm{BPB} = 0.938\). A is better (fewer bits per byte), although its perplexity looks more than twice as bad. 2. Sums without the “.”: A \(= -2.1\), B \(= -5.9\), C \(= -10.9\), D \(= -7.7\). A still wins; the problem is length, not punctuation, so it is not fixed. 3. \(\mathrm{SE} = \sqrt{0.7 \times 0.3/164} = 0.0358\); interval \(70 \pm 7.0\) points, about 63% to 77%. 4. \(\text{pass@}1 = 1 - \binom{18}{1}/\binom{20}{1} = 2/20 = 0.10\); \(\text{pass@}10 = 1 - \binom{18}{10}/\binom{20}{10} = 1 - 43758/184756 = 0.763\). 5. 6 of 20 (30%). For example “flag if at least 25% of the 5-grams match”: the paraphrase is flagged (30%), the same-topic document is not (0 of 20). 6. Number of shots and whether chain-of-thought examples are used; the metric and extraction rule (strict “####” match or flexible last number); decoding (greedy, max tokens, stop strings) and the harness/task version. Also: no chat template, dtype, model revision.

Key takeaways

  • Perplexity (40.1) is per token, so it depends on the tokenizer: \(\mathrm{PPL} = 2^{\mathrm{BPB}\cdot B/T}\) (40.3). Compare models with different tokenizers in bits per byte (40.2), on the same text with the same window.
  • Multiple-choice benchmarks (MMLU, HellaSwag) are scored by log-likelihood (40.4); acc_norm (40.5) removes the length bias; letter format needs a capable model, cloze works for small ones.
  • Generative benchmarks (GSM8K, code) depend on prompts, decoding, stop strings and answer extraction; \(\text{pass@}k\) (40.6) for sampled code.
  • Every score has noise: \(\mathrm{SE} = \sqrt{p(1-p)/n}\) (40.7), about 1.4 points on GSM8K. Use (40.8) or a paired test before claiming a gain.
  • Contamination makes scores measure memory. Detect it with n-gram overlap, Min-K% Prob (40.9) and fresh test sets (GSM1k); prevent it by decontaminating training data.
  • Use a harness with pinned versions, fixed and reported settings, no chat template for base models, logged samples you actually read, and error bars.

Key formulas

\[ \mathrm{BPB} = \frac{\sum_t -\log p_\theta(x_t \mid x_{\lt t})}{B \ln 2}, \quad \mathrm{PPL} = 2^{\mathrm{BPB}\cdot B/T}, \quad s_{\text{norm}}(c) = \frac{\sum_j \log p_\theta(c_j \mid x, c_{\lt j})}{\mathrm{len}(c)} \]
\[ \mathrm{SE} = \sqrt{\frac{p(1-p)}{n}}, \quad z = \frac{p_A - p_B}{\sqrt{\mathrm{SE}_A^2 + \mathrm{SE}_B^2}}, \quad \text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} \]

References for this session

  1. Hendrycks et al. (2020). Measuring Massive Multitask Language Understanding
  2. Liang et al. (2022). Holistic Evaluation of Language Models
  3. EleutherAI (code). LM Evaluation Harness
  4. Hendrycks et al. (2020). Measuring Massive Multitask Language Understanding
  5. Liang et al. (2022). Holistic Evaluation of Language Models
  6. EleutherAI (code). LM Evaluation Harness
  7. Zellers et al. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?
  8. Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems (GSM8K)
  9. Zhang et al. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k)
  10. Fourrier et al. (Hugging Face) (2023). What's going on with the Open LLM Leaderboard? (blog)
  11. Gu et al. (2024). OLMES: A Standard for Language Model Evaluations
  12. Biderman et al. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models
  13. Shi et al. (2023). Detecting Pretraining Data from Large Language Models (Min-K% Prob)
  14. Gao et al. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling
  15. Brown et al. (2020). Language Models are Few-Shot Learners
  16. Chen et al. (2021). Evaluating Large Language Models Trained on Code
  17. Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  18. Srivastava et al. (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (BIG-bench)