LLM & VLM courseBlog
3Track 1: Foundations · session 3 of 15

Math you need, part 2: probability, cross-entropy, KL divergence and perplexity

An LLM does not output a word. It outputs a probability for every possible token. To understand how it learns, how we measure it, and how we compare it with another model, we need a few ideas from probability and information theory: likelihood, entropy, cross-entropy, KL divergence, perplexity and calibration. This session builds each one from zero.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 1 and 2

What you will learn

  • Probability distributions, conditional probability and Bayes’ rule, with a key-moment detection example
  • Maximum likelihood, and why training an LLM is the same as minimizing cross-entropy
  • Entropy: how to measure surprise and uncertainty, in nats and bits
  • Cross-entropy and KL divergence, why KL is not symmetric, and where each direction is used (distillation, RLHF, GRPO, TensorRT calibration)
  • Perplexity, and why bits per byte is the fair way to compare models with different tokenizers
  • Calibration: when a model’s 90% is really 90%, and how temperature scaling fixes it

Why this matters

At every step, an LLM gives a probability to each of its 128,256 tokens. Everything we do with it, training, measuring, comparing, aligning and serving, is built on a handful of ideas about probabilities.

You have already seen some of them in Session 1: the loss was \(-\log p\), and perplexity was \(e^{\text{loss}}\). This session explains where they come from and adds the rest of the toolkit. These ideas appear all over the course:

IdeaWhere you will meet it
Maximum likelihood, cross-entropyThe training loss of every LLM, VLM and classifier
EntropyHow sure the model is; sampling and temperature; uncertainty for hallucination detection
KL divergenceDistillation (session 59), the KL penalty in RLHF (session 66), DPO (session 68), GRPO (session 70), VAEs (session 223), INT8 calibration in TensorRT
Perplexity, bits per byteEvaluating base models (session 40) and filtering pretraining data (session 34)
Bayes’ rule, calibrationSetting thresholds for rare events (key moments, brand safety), trusting a VLM’s “yes” or “no”

Probability: the basic rules

A random variable \(X\) is a quantity whose value we do not know in advance: the next token, the label of a video clip, whether a goal happens. A probability distribution gives each possible value a number between 0 and 1, and all the numbers add up to 1:

\[ p(x) \ge 0 \ \text{ for every } x, \qquad \sum_{x} p(x) = 1 \tag{3.1} \]

For an LLM, \(X\) is the next token and \(p\) is the softmax output: a list of 128,256 numbers that sum to 1. A distribution over a finite list like this is called categorical.

Joint, conditional and the chain rule

The joint probability \(p(x, y)\) is the chance that both things happen. The conditional probability \(p(y \mid x)\) is the chance of \(y\) when we already know \(x\):

\[ p(y \mid x) = \frac{p(x, y)}{p(x)}, \qquad p(x, y) = p(x)\, p(y \mid x) \tag{3.2} \]

Apply the second form again and again and you get the chain rule from Session 1: \(p(x_1, \dots, x_T) = \prod_t p(x_t \mid x_{\lt t})\). Two events are independent when knowing one tells you nothing about the other: \(p(y \mid x) = p(y)\).

Bayes’ rule

Often we know \(p(\text{evidence} \mid \text{cause})\) but we want \(p(\text{cause} \mid \text{evidence})\). Bayes’ rule turns one into the other:

\[ p(y \mid x) = \frac{p(x \mid y)\, p(y)}{p(x)} = \frac{p(x \mid y)\, p(y)}{\sum_{y'} p(x \mid y')\, p(y')} \tag{3.3} \]
Read the symbols
\(p(y)\)
the prior: how common the cause is before we look
\(p(x \mid y)\)
the likelihood: how often we see this evidence when the cause is true
\(p(y \mid x)\)
the posterior: what we should believe after seeing the evidence
Worked example: does a crowd roar mean a goal?

In a football match, say 2% of 10-second clips contain a goal. When there is a goal, the crowd roars 90% of the time. But the crowd also roars in 10% of clips without a goal (near misses, fouls, songs). A clip has a roar. How likely is a goal?

\[ p(\text{goal} \mid \text{roar}) = \frac{0.9 \times 0.02}{0.9 \times 0.02 + 0.1 \times 0.98} = \frac{0.018}{0.116} = 0.155 \]

Only about 16%. Out of 1,000 clips, 18 have a goal and a roar, but 98 have a roar without a goal. The rare cause is swamped by the common false alarms. This is the base-rate effect, and it is the main reason key-moment detectors produce too many false alarms when the event is rare.

goal and roargoal, no roar (missed)roar, no goal (false alarm)nothing
Figure 1. Bayes’ rule with 1,000 clips, one dot each. The posterior is the green dots divided by all the dots that roar (green plus orange).
How to read Figure 1

Move the false-alarm rate to 0: every roar is a goal, and the posterior becomes 100%. Now make goals rarer: even a very good detector produces mostly false alarms. In practice you fix this by adding more evidence (for example the score changing on the scoreboard) or by raising the decision threshold.

Expectation

The expected value of a function \(f\) under \(p\) is its average, weighted by probability:

\[ \E_{x \sim p}\left[f(x)\right] = \sum_x p(x)\, f(x) \;\approx\; \frac{1}{N} \sum_{n=1}^{N} f(x_n), \quad x_n \sim p \tag{3.4} \]

The right side is the Monte Carlo estimate: if we cannot sum over every possible value (for example, every possible sentence), we draw samples and average. RL for LLMs (Track 4) is built on such sample averages.

Maximum likelihood: how models learn from data

We have data \(x_1, \dots, x_N\) and a model \(p_\theta\) with parameters \(\theta\). Which \(\theta\) is best? The maximum likelihood answer: choose the \(\theta\) that makes the data we saw as probable as possible. If the examples are independent, their joint probability is a product, and we take the log as in Session 1:

\[ \hat\theta = \argmax_\theta \prod_{n=1}^{N} p_\theta(x_n) = \argmax_\theta \sum_{n=1}^{N} \log p_\theta(x_n) = \argmin_\theta \Big( -\frac{1}{N} \sum_{n=1}^{N} \log p_\theta(x_n) \Big) \tag{3.5} \]

The quantity in the last bracket is the negative log-likelihood (NLL). Minimizing NLL is the same as maximizing likelihood.

Worked example: a biased coin

We toss a coin 10 times and see 7 heads. Model: heads with probability \(\theta\). The log-likelihood is

\[ \ell(\theta) = 7 \log \theta + 3 \log (1 - \theta) \]

Set the derivative to zero: \(\dfrac{7}{\theta} - \dfrac{3}{1 - \theta} = 0\), so \(\theta = 0.7\). The best guess is simply the fraction of heads. At \(\theta = 0.7\), \(\ell = -6.109\). In general, for \(h\) heads in \(n\) tosses, \(\hat\theta = h/n\). For a categorical variable, the same rule gives: the maximum likelihood probability of each value is its frequency in the data.

Figure 2. The likelihood of the data for every \(\theta\), divided by its largest value so the peak is 1: \(\exp\big(\ell(\theta) - \ell(\hat\theta)\big)\) with \(\ell(\theta) = h\log\theta + (n-h)\log(1-\theta)\). The peak is always at \(\theta = h/n\).
How to read Figure 2

Keep \(h/n = 0.7\) but increase \(n\) (try 10, then 100): the peak stays at 0.7 but becomes narrower. More data makes the estimate more certain. Set \(h = 0\): the maximum is at \(\theta = 0\), so the model says heads are impossible. That is overfitting to small data, and it is why LLMs never give exactly zero probability (softmax cannot), and why classical n-gram models needed smoothing.

From maximum likelihood to the LLM loss

An LLM is a categorical distribution over the next token for every context. Maximum likelihood over a corpus of \(T\) tokens is exactly the loss from Session 1:

\[ \mathcal{L}(\theta) = -\frac{1}{T} \sum_{t=1}^{T} \log p_\theta(x_t \mid x_{\lt t}) \tag{3.6} \]

Why is it called cross-entropy? To answer that we first need entropy.

Information and entropy

Surprise

Seeing an event with probability \(p\) carries information, or surprise, equal to

\[ I(x) = -\log p(x) \tag{3.7} \]

A sure event (\(p = 1\)) carries 0 surprise. A rare event carries a lot. And surprises add up: two independent events with probabilities \(p\) and \(q\) have joint probability \(pq\), and \(-\log(pq) = -\log p - \log q\). The log is the only function with this property, which is why it is everywhere.

Units: with the natural log (\(\ln\)) the unit is the nat; with \(\log_2\) it is the bit. To convert: \(1 \text{ nat} = 1/\ln 2 \approx 1.443\) bits. PyTorch losses are in nats.

Entropy: average surprise

The entropy of a distribution is the expected surprise:

\[ H(p) = \E_{x \sim p}\left[-\log p(x)\right] = -\sum_x p(x) \log p(x) \tag{3.8} \]

It measures uncertainty. It is 0 when one outcome is certain, and largest when all \(K\) outcomes are equally likely, where it equals \(\log K\).

Worked examples

A fair coin: \(H = -2 \times 0.5 \log_2 0.5 = 1\) bit. The distribution \((\tfrac12, \tfrac14, \tfrac18, \tfrac18)\): \(H = \tfrac12 \cdot 1 + \tfrac14 \cdot 2 + 2 \cdot \tfrac18 \cdot 3 = 1.75\) bits \(= 1.213\) nats. A uniform guess over Llama 3’s 128,256 tokens: \(\ln 128{,}256 = 11.76\) nats \(= 16.97\) bits. That is the loss of an untrained model, as we saw in Session 1.

A helpful number is \(e^{H}\) (or \(2^{H}\) in bits): the effective number of choices. A distribution with entropy \(\ln 5\) is “as uncertain as” a fair choice among 5 options, however its probabilities are spread.

Figure 3. Entropy of a next-token distribution over 5 tokens. The sliders set unnormalized weights; the bars show the normalized probabilities (blue) and the surprise \(-\log_2 p\) of each token in bits (orange). Entropy is the probability-weighted average of the orange bars.
How to read Figure 3

Make one token very likely: its surprise goes to 0 and the entropy drops toward 0. The rare tokens have huge surprise, but they are rarely seen, so they add little to the average. Press “Uniform”: entropy reaches its maximum \(\log_2 5 = 2.32\) bits and the effective number of choices is exactly 5.

Cross-entropy

Now suppose the true distribution is \(p\) (the data), but we predict with \(q\) (the model). Each outcome \(x\) still happens with probability \(p(x)\), but our surprise is measured with our own prediction, \(-\log q(x)\). The average surprise is the cross-entropy:

\[ H(p, q) = \E_{x \sim p}\left[-\log q(x)\right] = -\sum_x p(x) \log q(x) \tag{3.9} \]

In training, the “true” distribution for one position is one-hot: probability 1 on the token that really came next, \(c\). Then the sum has a single term:

\[ H(\text{one-hot}_c,\ q) = -\log q(c) \tag{3.10} \]

That is exactly the per-token loss. Averaged over all positions, it is formula (3.6). So the LLM loss is the cross-entropy between the data and the model, and minimizing it is maximum likelihood. Same thing, three names.

Label smoothing

Sometimes we use a soft target instead of one-hot: keep \(1 - \varepsilon\) on the true class and spread \(\varepsilon\) evenly over all \(K\) classes. With \(K = 4\) and \(\varepsilon = 0.1\), the target becomes \((0.925, 0.025, 0.025, 0.025)\). This stops the model from pushing logits to infinity and often improves calibration in classifiers. In PyTorch: F.cross_entropy(logits, y, label_smoothing=0.1). LLM pretraining usually does not use it. [Likely]

KL divergence: the extra cost of a wrong model

Cross-entropy is never smaller than entropy: predicting with the wrong distribution can only add surprise. The extra amount is the Kullback–Leibler (KL) divergence:

\[ \KL(p \,\|\, q) = \sum_x p(x) \log \frac{p(x)}{q(x)} = H(p, q) - H(p) \tag{3.11} \]

So we have the identity that explains everything:

\[ \underbrace{H(p, q)}_{\text{training loss}} = \underbrace{H(p)}_{\text{fixed by the data}} + \underbrace{\KL(p \,\|\, q)}_{\text{what training can reduce}} \tag{3.12} \]

The entropy of the data does not depend on the model. So minimizing cross-entropy is the same as minimizing \(\KL(\text{data} \,\|\, \text{model})\). This also tells you that the loss can never reach 0 on real text: natural language has its own entropy. A loss curve that flattens is often approaching that floor.

Worked example

\(p = (\tfrac12, \tfrac14, \tfrac18, \tfrac18)\), and the model guesses uniform \(q = (\tfrac14, \tfrac14, \tfrac14, \tfrac14)\). Then \(H(p) = 1.75\) bits, \(H(p, q) = 2\) bits, so \(\KL(p \,\|\, q) = 0.25\) bits (0.173 nats). Every outcome costs, on average, a quarter bit more than it needs to.

Three properties

  1. Never negative: \(\KL(p \,\|\, q) \ge 0\). Proof idea: \(-\log\) is convex, so by Jensen’s inequality
    \[ \KL(p\,\|\,q) = \E_p\!\left[-\log \frac{q}{p}\right] \ge -\log \E_p\!\left[\frac{q}{p}\right] = -\log \sum_x q(x) = -\log 1 = 0 \]
  2. Zero only when equal: \(\KL(p \,\|\, q) = 0\) exactly when \(p = q\).
  3. Not symmetric: \(\KL(p \,\|\, q) \ne \KL(q \,\|\, p)\) in general. So it is not a true distance. Example: \(p = (0.9, 0.1)\), \(q = (0.5, 0.5)\) gives \(\KL(p\|q) = 0.368\) but \(\KL(q\|p) = 0.511\) nats.

Figure 4 runs formulas (3.8), (3.9) and (3.11) in your browser, one token at a time. Each step adds one term to each of the three sums, and the panel shows the term with the real numbers put in.

Model q (drag the sliders; they set weights, q is the weights divided by their sum)

  1. input: data \(p\), model \(q\) over 5 tokens
  2. \(\text{CE} \leftarrow 0,\ H \leftarrow 0,\ \text{KL} \leftarrow 0\)
  3. for each token \(x\):
  4. \(\text{CE} \leftarrow \text{CE} - p(x)\ln q(x)\) // (3.9)
  5. \(H \leftarrow H - p(x)\ln p(x)\) // (3.8)
  6. \(\text{KL} \leftarrow \text{KL} + p(x)\ln\frac{p(x)}{q(x)}\) // (3.11)
  7. return CE, \(H\), KL // check CE = H + KL (3.12)
Figure 4. Cross-entropy, entropy and KL for two distributions over 5 tokens, computed term by term. Top: data \(p\) (grey) and model \(q\) (colored). Middle rows: the term each token adds to the cross-entropy and to the KL (red when negative). Bottom: the three running sums. The last step checks identity (3.12): \(H(p, q) = H(p) + \KL(p\|q)\).
How to read Figure 4

Press Play with the one-hot data. Only “shot” has \(p \gt 0\), so only the first step adds anything: \(-1 \times \ln 0.441 = 0.818\) nats. The other four tokens add 0, because \(0 \times \ln q = 0\). That is ordinary training: the loss looks only at the true token. \(H(p) = 0\), so the cross-entropy and the KL are the same number.

Switch to the soft teacher and play again. Now every token adds to all three sums. The KL terms of “ball”, “penalty”, “cross” and “cake” are negative (\(q\) gives them more than \(p\) does), so the running KL falls from 0.184 to 0.066. One term can be negative; the full sum never is. The loss ends at 1.124 and can only fall to \(H(p) = 1.058\), never to 0. That is knowledge distillation: the student learns the teacher’s whole distribution, including which wrong answers are “almost right”. With label smoothing the floor is \(H(p) = 0.390\). Press “Set q equal to p” to reach it: the KL becomes 0.

Forward and reverse KL

Because KL is not symmetric, the direction matters, and the two directions behave very differently when the model is simpler than the target:

  • Forward KL, \(\KL(p \,\|\, q)\), averages over \(p\). Wherever \(p\) has mass and \(q\) is near zero, \(\log(p/q)\) explodes. So \(q\) must cover every mode of \(p\), even if it puts mass in between. This is “mode-covering”. Maximum likelihood and standard distillation use this direction.
  • Reverse KL, \(\KL(q \,\|\, p)\), averages over \(q\). Now \(q\) is punished for putting mass where \(p\) is small, but not for ignoring parts of \(p\). So \(q\) picks one mode and fits it tightly. This is “mode-seeking”. RLHF’s KL penalty, MiniLLM-style distillation (session 119) and variational inference use this direction.

KL is a sum (or, for continuous values, an integral) of the terms \(p(x)\log\frac{p(x)}{q(x)}\). Plot that term against \(x\) and the KL is the area under it: green where \(p\) is above \(q\), red (negative) where \(q\) is above \(p\). Figure 5 fits a one-peaked \(q\) to a two-peaked \(p\) by gradient descent on this area, for either direction.

  1. input: target \(p\) on 400 grid points, start \(\mu, \sigma\), step size \(\eta\); \(s = \ln\sigma\)
  2. repeat 60 times:
  3. \(q \leftarrow \mathcal{N}(\mu, \sigma^2)\) on the grid, normalized
  4. \(L \leftarrow \sum_i \Delta x\; p_i \ln\frac{p_i}{q_i}\) // forward; reverse swaps p and q
  5. \(g_\mu, g_s \leftarrow\) slopes of \(L\) by finite differences
  6. \(\mu \leftarrow \mu - \eta\, g_\mu,\ \ s \leftarrow s - \eta\, g_s\) // step length capped at 0.5
Figure 5. Fitting a single bell curve \(q\) (colored) to a two-peaked target \(p\) (grey) by gradient descent. Dashed: the starting \(q\). Bottom panel: the KL term at every \(x\) for the chosen direction; the green area minus the red area is the KL. Each step is one iteration of the loop, with the real loss, slopes and update.
How to read Figure 5

Press Play with forward KL from \(\mu_0 = 1, \sigma_0 = 1\). In the first steps \(\sigma\) grows fast: the green area sits under the far-left peak, where \(q\) is almost zero and \(\ln(p/q)\) is huge. After 60 steps \(q\) is a wide bell between the peaks (\(\mu = 0.06\), \(\sigma = 2.91\), KL = 0.591). It covers both modes but puts mass in the empty valley; the red area there is where \(q\) is above \(p\).

Now switch to reverse KL. From the same start, \(q\) slides onto the right peak and stops at \(\mu = 2.50\), \(\sigma = 0.70\), KL = 0.693 (that is \(\ln 2\): \(q\) is one of the two halves of \(p\)). The term \(q\ln(q/p)\) is zero wherever \(q\) is zero, so the left peak costs nothing. Set \(\mu_0 = 0\): by symmetry \(q\) cannot choose a side, and it gets stuck in a wide bell with KL = 1.261. Reverse KL has more than one local minimum, and where you start decides which one you reach. With \(\eta = 1\) or 2 the reverse fit jumps between two points and never settles, while the forward fit with \(\eta = 2\) reaches its minimum faster: the step size that works depends on the shape of the loss. Think of a small student copying a large teacher: forward KL makes it hedge between all the teacher’s answers; reverse KL makes it confidently give one of them.

KL between two Gaussians

For two 1-D normal distributions there is a closed form, used in every VAE:

\[ \KL\big(\mathcal{N}(\mu_1, \sigma_1^2) \,\|\, \mathcal{N}(\mu_2, \sigma_2^2)\big) = \log\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} - \frac12 \tag{3.13} \]

Example: \(\KL(\mathcal{N}(0, 1) \,\|\, \mathcal{N}(1, 2^2)) = \log 2 + \tfrac{1 + 1}{8} - \tfrac12 = 0.443\) nats.

Estimating KL from samples (used in RLHF and GRPO)

For two LLMs we cannot sum over all possible texts. We sample texts \(y\) from the policy \(\pi\) and average a per-sample estimate. With the ratio \(r = \pi_{\text{ref}}(y) / \pi(y)\):

\[ k_1 = -\log r, \qquad k_3 = (r - 1) - \log r \tag{3.14} \]

Both average to \(\KL(\pi \,\|\, \pi_{\text{ref}})\). \(k_1\) is simple but its single samples can be negative and noisy. \(k_3\) is never negative and has lower variance; GRPO uses this form in its KL penalty. [Certain]

Perplexity, and comparing models fairly

Perplexity is the exponential of the average cross-entropy per token:

\[ \operatorname{PPL} = \exp\Big( -\frac{1}{T} \sum_{t=1}^{T} \log q(x_t \mid x_{\lt t}) \Big) = \Big( \prod_{t=1}^{T} \frac{1}{q(x_t \mid x_{\lt t})} \Big)^{1/T} \tag{3.15} \]

The second form shows what it is: the geometric mean of \(1/q\), the average “branching factor”. For a sentence where the model gave the correct tokens probabilities \(0.5, 0.2, 0.8, 0.1\): the losses are \(0.693, 1.609, 0.223, 2.303\), their mean is \(1.207\), and \(\operatorname{PPL} = e^{1.207} = 3.34\). The geometric mean of the probabilities is \(0.299\), and \(1/0.299 = 3.34\). Same number.

Why perplexity does not compare across tokenizers

Perplexity is “per token”. If one tokenizer cuts text into more, smaller tokens, each token is easier to predict, so per-token perplexity looks better even when the model is worse. The fair measure normalizes by something both models share, such as the number of bytes (or characters) of the raw text:

\[ \text{bits per byte} = \frac{\sum_t -\ln q(x_t \mid x_{\lt t})}{(\ln 2) \times \text{number of bytes}} \tag{3.16} \]
Worked example: the perplexity trap

The same 4,000-byte text. Model A uses 1,000 tokens with average loss 2.0: perplexity \(e^{2.0} = 7.39\). Model B uses 1,300 tokens with average loss 1.6: perplexity \(e^{1.6} = 4.95\). B looks much better.

Now total surprise: A spends \(1000 \times 2.0 = 2000\) nats, B spends \(1300 \times 1.6 = 2080\) nats. Bits per byte: A \(= 2000 / (0.693 \times 4000) = 0.721\), B \(= 0.750\). A is the better model. B only looked better because it divided by more tokens.

Probability the model gave each correct token → its loss in nats

  1. input: \(q_t = q(x_t \mid x_{\lt t})\) for the true tokens
  2. \(S \leftarrow 0\) (total nats), \(T \leftarrow 0\)
  3. for each token \(t\):
  4. \(\ell_t \leftarrow -\ln q_t\) // (3.10)
  5. \(S \leftarrow S + \ell_t,\ \ T \leftarrow T + 1\)
  6. perplexity so far \(\leftarrow e^{S/T}\)
  7. return \(\operatorname{PPL} = e^{S/T} = \big(\prod_t 1/q_t\big)^{1/T}\) // (3.15)
Figure 6. Perplexity of one sentence, one token at a time. Each slider is the probability the model gave to the correct token. Top row: that probability as a filled bar. Middle: the loss \(-\ln q_t\) of each token; the dashed line is the mean loss so far. Bottom: the perplexity so far, \(e^{\text{mean loss}}\).
How to read Figure 6

Press Play. The first token, “The”, gets only \(q = 0.05\) (it starts the sentence, so many words were possible): its loss is 2.996 nats and the perplexity after one token is 20.0. Each later token is easier, and the perplexity falls: 8.16, 4.81, 3.86, 3.54, and finally 3.04 after six tokens. The geometric mean of the six probabilities is 0.329, and \(1/0.329 = 3.04\): the same number. “The” alone is 45% of the total loss. Now drag the slider of “penalty” down to 0.01: one surprising token raises the perplexity of the whole sentence.

Calibration: is 90% really 90%?

A model is calibrated if, among all the predictions where it says “90% sure”, about 90% are correct. Modern deep networks are usually overconfident: their 90% is right less often than 90% of the time. This matters whenever you act on a probability: a threshold for alerts, a “yes/no” answer from a VLM, or deciding when to say “I don’t know”.

We measure it with the expected calibration error. Put the predictions into \(M\) bins by confidence, and compare each bin’s average confidence with its accuracy:

\[ \operatorname{ECE} = \sum_{b=1}^{M} \frac{|B_b|}{N} \,\big|\operatorname{acc}(B_b) - \operatorname{conf}(B_b)\big| \tag{3.17} \]

The simplest fix is temperature scaling: divide all logits by one number \(T\), chosen on a validation set to minimize the NLL. It does not change which class wins (the order of logits stays the same), so accuracy is unchanged; only the confidence moves.

\[ T^{*} = \argmin_{T \gt 0} \; -\frac{1}{N} \sum_{n=1}^{N} \log \softmax\!\left(\mathbf{z}_n / T\right)_{y_n} \tag{3.18} \]
Figure 7. A reliability diagram for 3,000 synthetic predictions over 10 classes from an overconfident model. Bars: accuracy in each confidence bin. Diagonal: perfect calibration. Gaps below the diagonal mean overconfidence.
How to read Figure 7

At \(T = 1\) the bars sit below the diagonal: the model says 90% but is right much less often. Raise \(T\): confidence drops and the bars move onto the line; ECE and NLL both fall. “Fit T” finds the best value on a grid. Accuracy never changes, because dividing by \(T\) keeps the order of the logits. In this toy, the data was generated so that the true best \(T\) is 3.

Where these ideas are used

ApplicationIdeaHow
Training any LLM or VLMCross-entropy (3.6)Maximum likelihood on next tokens; for VLMs only on the answer text
Action recognition with rare classesWeighted cross-entropy, focal lossGive rare classes more weight, or multiply the loss by \((1 - q_c)^{\gamma}\) so easy, common examples count less (session 203)
Key-moment detectionBayes’ rule (3.3), calibrationRare events need calibrated scores and thresholds chosen with the base rate in mind (session 207)
Yes/no questions to a VLMCalibration (3.17–3.18)Read \(p(\text{“yes”})\) from the logits; calibrate with temperature before thresholding
DistillationForward or reverse KLMatch the teacher’s full distribution, not just its top answer
RLHF, DPO, GRPOKL to a reference modelKeep the tuned model close to where it started, so it does not drift or exploit the reward
Pretraining data filteringPerplexityText with very high perplexity under a reference model is often noise; very low can mean repeated boilerplate
Hallucination detectionEntropyHigh entropy over meanings of sampled answers suggests the model is guessing (session 156)

Pseudo-code

Algorithm 1: cross-entropy from logits, with label smoothing and padding
  1. input: logits \(Z \in \R^{N \times K}\), labels \(y \in \{1..K\}^N\) (pad label = ignore), smoothing \(\varepsilon\)
  2. \(\log P \leftarrow Z - \operatorname{LSE}(Z)\) row by row // stable log-softmax (Session 2)
  3. keep only rows whose label is not padding; let \(N'\) be how many
  4. for each kept row \(n\) do
  5. \(\text{nll}_n \leftarrow -\log P_{n, y_n}\) // formula (3.10)
  6. \(\text{smooth}_n \leftarrow -\frac{1}{K}\sum_k \log P_{n,k}\) // cross-entropy with the uniform part
  7. \(\ell_n \leftarrow (1 - \varepsilon)\,\text{nll}_n + \varepsilon\,\text{smooth}_n\)
  8. return \(\frac{1}{N'}\sum_n \ell_n\) // average over real tokens only
Algorithm 2: KL between a teacher and a student, from logits
  1. input: teacher logits \(\mathbf{a}\), student logits \(\mathbf{b}\), temperature \(\tau\)
  2. \(\log p \leftarrow \operatorname{logsoftmax}(\mathbf{a}/\tau)\), \(\;\log q \leftarrow \operatorname{logsoftmax}(\mathbf{b}/\tau)\)
  3. forward: \(\KL(p\|q) \leftarrow \sum_k e^{\log p_k} (\log p_k - \log q_k)\) // formula (3.11)
  4. reverse: \(\KL(q\|p) \leftarrow \sum_k e^{\log q_k} (\log q_k - \log p_k)\)
  5. return the chosen direction \(\times\, \tau^2\) // \(\tau^2\) keeps gradient size stable (Hinton et al.)
Algorithm 3: fit the temperature on a validation set
  1. input: validation logits \(\{\mathbf{z}_n\}\), labels \(\{y_n\}\)
  2. for \(T\) in a grid from 0.5 to 5 do
  3. \(\text{NLL}(T) \leftarrow -\frac{1}{N}\sum_n \log \softmax(\mathbf{z}_n / T)_{y_n}\)
  4. \(T^* \leftarrow\) the \(T\) with the smallest NLL // formula (3.18); or optimize \(\log T\) with gradient descent
  5. return \(T^*\) // at test time use \(\softmax(\mathbf{z}/T^*)\)
Algorithm 4: perplexity and bits per byte of a corpus
  1. input: documents, model, tokenizer
  2. \(S \leftarrow 0\) (total nats), \(n_{\text{tok}} \leftarrow 0\), \(n_{\text{byte}} \leftarrow 0\)
  3. for each document \(d\) do
  4. \(S \leftarrow S + \sum_t -\log q(x_t \mid x_{\lt t})\) // sum, do not average yet
  5. \(n_{\text{tok}} \leftarrow n_{\text{tok}} + |\text{tokens}(d)|\), \(\;n_{\text{byte}} \leftarrow n_{\text{byte}} + |\text{utf8}(d)|\)
  6. return \(\operatorname{PPL} = e^{S / n_{\text{tok}}}\) and \(\text{BPB} = S / (n_{\text{byte}} \ln 2)\) // formulas (3.15), (3.16)

A PyTorch script that checks the main claims:

import math, torch
import torch.nn.functional as F
torch.manual_seed(0)

# 1) H(p, q) = H(p) + KL(p || q), formula (3.12), in nats
p = torch.tensor([0.5, 0.25, 0.125, 0.125]); q = torch.full((4,), 0.25)
H = -(p * p.log()).sum(); CE = -(p * q.log()).sum(); KL = (p * (p / q).log()).sum()
print(H.item(), CE.item(), KL.item(), (H + KL - CE).abs().item())   # 1.213, 1.386, 0.173, ~0

# 2) F.kl_div trap: input must be LOG q, target is p, and use reduction="batchmean"
logits_s, logits_t = torch.randn(8, 10), torch.randn(8, 10)
log_q, p_t = F.log_softmax(logits_s, -1), F.softmax(logits_t, -1)
kl_right = F.kl_div(log_q, p_t, reduction="batchmean")             # = mean over rows of KL(p_t || q)
kl_manual = (p_t * (p_t.log() - log_q)).sum(-1).mean()
print(kl_right.item(), kl_manual.item())                           # equal

# 3) label smoothing is built into cross_entropy
y = torch.randint(0, 10, (8,))
print(F.cross_entropy(logits_s, y).item(), F.cross_entropy(logits_s, y, label_smoothing=0.1).item())

# 4) perplexity and bits per byte from a token-level loss; -100 marks padding
logits = torch.randn(2, 6, 50); targets = torch.randint(0, 50, (2, 6)); targets[1, 4:] = -100
nll_sum = F.cross_entropy(logits.reshape(-1, 50), targets.reshape(-1), ignore_index=-100, reduction="sum")
n_tok = (targets != -100).sum(); n_bytes = 40                     # bytes of the raw text (example value)
print("PPL", math.exp(nll_sum.item() / n_tok.item()), "BPB", nll_sum.item() / (n_bytes * math.log(2)))

# 5) Gaussian KL, formula (3.13)
P, Q = torch.distributions.Normal(0., 1.), torch.distributions.Normal(1., 2.)
print(torch.distributions.kl_divergence(P, Q).item())              # 0.443

Common mistakes

  • Wrong arguments to F.kl_div. Its first argument must be log-probabilities of \(q\), and the second is \(p\) (probabilities). It computes \(\KL(p\|q)\). Passing plain probabilities, or swapping them, gives wrong numbers without any error.
  • reduction="mean" in F.kl_div. It divides by the number of elements (\(\text{batch} \times \text{classes}\)), not by the batch. Use "batchmean" to get the real KL per example.
  • Counting padding in the loss. Padded positions must use ignore_index (default −100), or the loss and perplexity are wrong and depend on batch padding.
  • Comparing perplexity across tokenizers. Use bits per byte (3.16), or evaluate both models on the same tokenization.
  • Averaging perplexities. Average token losses first (weighted by token count), then exponentiate.
  • Mixing nats and bits. PyTorch uses natural logs. Divide by \(\ln 2 = 0.693\) to get bits.
  • Trusting softmax confidence as a probability. Check calibration on held-out data before thresholding, especially for rare events.
  • Ignoring the base rate. A detector with 90% recall and 10% false-alarm rate is mostly false alarms when the event happens in 2% of clips (Figure 1).

Interview questions

Why is the LLM training loss called cross-entropy, and how is it related to maximum likelihood?basic
For each position the target is one-hot on the true next token, and the cross-entropy \(H(\text{one-hot}, q) = -\log q(\text{true token})\). Averaging over positions gives the negative log-likelihood of the corpus. Minimizing it is maximum likelihood estimation. Also, since \(H(p,q) = H(p) + \KL(p\|q)\) and \(H(p)\) is fixed by the data, it also minimizes \(\KL(\text{data}\|\text{model})\).
Define entropy, cross-entropy and KL, and give the identity that connects them.basic
\(H(p) = -\sum p\log p\) (average surprise under \(p\)); \(H(p,q) = -\sum p\log q\) (average surprise when reality is \(p\) but you predict \(q\)); \(\KL(p\|q) = \sum p\log(p/q)\). Identity: \(H(p,q) = H(p) + \KL(p\|q)\). KL is the extra cost of using the wrong distribution.
Why is KL divergence non-negative, and why is it not a distance?core
By Jensen’s inequality with the convex function \(-\log\): \(\E_p[-\log(q/p)] \ge -\log\E_p[q/p] = -\log 1 = 0\), with equality only when \(p = q\). It is not a distance because it is not symmetric and does not satisfy the triangle inequality. Symmetric alternatives include Jensen–Shannon divergence.
Forward or reverse KL for distilling a large LLM into a small one?advanced
Forward \(\KL(p_{\text{teacher}}\|q_{\text{student}})\) is mode-covering: the student must put mass everywhere the teacher does, and a small student may spread mass into regions the teacher considers unlikely, which can produce low-quality generations. Reverse \(\KL(q\|p)\) is mode-seeking: the student concentrates on the teacher’s high-probability outputs, which often gives better generation quality for small students (the motivation of MiniLLM). Forward KL with soft labels is standard and simple for classification-style distillation. In practice, choose by evaluating generation quality and diversity; on-policy methods (GKD) also help.
Why does RLHF add a KL penalty to a reference model? Which direction is it?core
The reward model is imperfect; without a leash, the policy drifts far from the SFT model and exploits the reward (reward hacking), often losing fluency. The penalty \(\beta\,\KL(\pi\|\pi_{\text{ref}})\) is the reverse direction, estimated on samples from the policy itself (for example with \(k_1\) or \(k_3\) from formula 3.14). It also has a clean interpretation: the optimal policy is \(\pi^*(y) \propto \pi_{\text{ref}}(y)\,e^{r(y)/\beta}\), which is what DPO builds on.
What is perplexity, and why can you not compare it across tokenizers?core
Perplexity is \(\exp\) of the average per-token cross-entropy, the geometric mean of \(1/q\). It is per token, and tokenizers split the same text into different numbers of tokens, so per-token numbers are not comparable. Compare total log-likelihood per byte or per character (bits per byte), or evaluate with the same tokenizer.
A model’s loss stops decreasing at 1.8 nats per token. Is it broken?advanced
Not necessarily. The loss is \(H(\text{data}) + \KL(\text{data}\|\text{model})\); natural text has non-zero entropy, so the loss has a floor above 0. The plateau may be that floor for this model size and data (scaling laws model it as an irreducible term \(E\)). Check validation vs training loss, learning-rate schedule, data quality and repetition before concluding it is a bug.
What is calibration and how do you fix an overconfident classifier?core
A model is calibrated if predictions with confidence \(c\) are correct about a fraction \(c\) of the time; measure with reliability diagrams and ECE (formula 3.17). Temperature scaling divides logits by a single \(T\), fit on validation NLL (3.18); it keeps accuracy and ranking and usually fixes most of the gap. Other options: label smoothing during training, Platt scaling, isotonic regression, or ensembles.
Your key-moment detector has 90% recall and a 10% false-positive rate, and goals occur in 2% of clips. What precision do you expect?core
Bayes: precision \(= 0.9\cdot0.02 / (0.9\cdot0.02 + 0.1\cdot0.98) = 0.018/0.116 \approx 15.5\%\). Most alerts are false. Fixes: lower the false-positive rate (more evidence, such as scoreboard OCR or audio and video together), raise the threshold, or report scores rather than binary alerts.
What does label smoothing do to the target and to the logits?core
It replaces one-hot targets with \((1-\varepsilon)\) on the true class and \(\varepsilon/K\) on every class. The optimal logits become finite instead of growing without bound, which regularizes the model and often improves calibration. It can hurt distillation, since it removes information about which wrong classes are similar (Müller et al.).
How do you compute KL between two LLMs’ output distributions in practice?advanced
Per token, the exact KL over the vocabulary is cheap: take both log-softmaxes and compute \(\sum_k p_k(\log p_k - \log q_k)\), averaged over positions. For whole sequences you cannot sum over all texts, so sample sequences from one model and average per-token log-ratios (Monte Carlo, formula 3.4); in RL use the \(k_1\) or \(k_3\) estimators (3.14). In PyTorch, mind the F.kl_div argument order and use batchmean.
What is the entropy of a uniform distribution over the Llama 3 vocabulary, and why does it matter?basic
\(\ln 128{,}256 \approx 11.76\) nats (16.97 bits). It is the expected initial loss of a randomly initialized model with near-uniform outputs. If your first training steps show a loss far above this, something is wrong, for example a bad initialization, wrong labels or a broken loss.

Exercises

  1. Compute \(H(p)\) in bits for \(p = (0.25, 0.25, 0.25, 0.25)\) and for \(p = (1, 0, 0, 0)\).
  2. A model gives the true token probability 0.05. What is the loss in nats and in bits?
  3. Show that if \(q\) is uniform over \(K\) classes, then \(\KL(p\|q) = \log K - H(p)\).
  4. Three documents have average losses 2.0, 2.0 and 3.0 nats with 100, 100 and 800 tokens. What is the corpus perplexity? What wrong number do you get by averaging the three perplexities?
  5. In Figure 1, keep recall at 90% and goals at 2%. What false-alarm rate gives a precision of 50%?
  6. Compute \(\KL(\mathcal{N}(0, 1)\|\mathcal{N}(0, 1))\) and \(\KL(\mathcal{N}(2, 1)\|\mathcal{N}(0, 1))\) with formula (3.13).
Answerscheck
1. 2 bits; 0 bits. 2. \(-\ln 0.05 = 3.00\) nats \(= 4.32\) bits. 3. \(\KL(p\|q) = \sum p\log p - \sum p\log(1/K) = -H(p) + \log K\). 4. Weighted mean loss \(= (200 + 200 + 2400)/1000 = 2.8\), so \(\operatorname{PPL} = e^{2.8} = 16.4\). The wrong average: \((e^2 + e^2 + e^3)/3 = (7.39 + 7.39 + 20.09)/3 = 11.6\). 5. Precision \(= 0.018/(0.018 + 0.98f) = 0.5\) gives \(0.98f = 0.018\), so \(f \approx 1.84\%\). 6. 0, and \(0 + (1 + 4)/2 - 1/2 = 2\) nats.

Key takeaways

  • Maximum likelihood (3.5) = minimizing negative log-likelihood = minimizing cross-entropy (3.9–3.10). The LLM loss is all three.
  • Entropy (3.8) is average surprise; \(e^{H}\) is the effective number of choices. A uniform guess over the vocabulary costs \(\ln|V|\).
  • \(H(p,q) = H(p) + \KL(p\|q)\) (3.12): training reduces only the KL part; the data’s entropy is a floor.
  • KL is not symmetric: forward is mode-covering (likelihood, standard distillation), reverse is mode-seeking (RLHF penalty, MiniLLM). Sample estimators (3.14) are used in RL.
  • Perplexity is per token; compare models with different tokenizers using bits per byte (3.16). Check calibration (3.17) before trusting probabilities; temperature scaling (3.18) fixes most overconfidence.

Key formulas

\[ H(p) = -\sum_x p\log p, \quad H(p,q) = -\sum_x p\log q = H(p) + \KL(p\|q), \quad \operatorname{PPL} = e^{\mathcal{L}}, \quad p(y\mid x) = \frac{p(x\mid y)\,p(y)}{p(x)} \]

References for this session

  1. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 3: N-gram Language Models
  2. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 1: Foundations of Machine Learning
  3. Guo et al. (2017). On Calibration of Modern Neural Networks
  4. Hinton et al. (2015). Distilling the Knowledge in a Neural Network
  5. Schulman (2020). Approximating KL Divergence (blog)
  6. Szegedy et al. (2015). Rethinking the Inception Architecture for Computer Vision
  7. Lin et al. (2017). Focal Loss for Dense Object Detection
  8. Müller et al. (2019). When Does Label Smoothing Help?
  9. NVIDIA (docs). TensorRT documentation