Math you need, part 2: probability, cross-entropy, KL divergence and perplexity
An LLM does not output a word. It outputs a probability for every possible token. To understand how it learns, how we measure it, and how we compare it with another model, we need a few ideas from probability and information theory: likelihood, entropy, cross-entropy, KL divergence, perplexity and calibration. This session builds each one from zero.
What you will learn
- Probability distributions, conditional probability and Bayes’ rule, with a key-moment detection example
- Maximum likelihood, and why training an LLM is the same as minimizing cross-entropy
- Entropy: how to measure surprise and uncertainty, in nats and bits
- Cross-entropy and KL divergence, why KL is not symmetric, and where each direction is used (distillation, RLHF, GRPO, TensorRT calibration)
- Perplexity, and why bits per byte is the fair way to compare models with different tokenizers
- Calibration: when a model’s 90% is really 90%, and how temperature scaling fixes it
Why this matters
At every step, an LLM gives a probability to each of its 128,256 tokens. Everything we do with it, training, measuring, comparing, aligning and serving, is built on a handful of ideas about probabilities.
You have already seen some of them in Session 1: the loss was \(-\log p\), and perplexity was \(e^{\text{loss}}\). This session explains where they come from and adds the rest of the toolkit. These ideas appear all over the course:
| Idea | Where you will meet it |
|---|---|
| Maximum likelihood, cross-entropy | The training loss of every LLM, VLM and classifier |
| Entropy | How sure the model is; sampling and temperature; uncertainty for hallucination detection |
| KL divergence | Distillation (session 59), the KL penalty in RLHF (session 66), DPO (session 68), GRPO (session 70), VAEs (session 223), INT8 calibration in TensorRT |
| Perplexity, bits per byte | Evaluating base models (session 40) and filtering pretraining data (session 34) |
| Bayes’ rule, calibration | Setting thresholds for rare events (key moments, brand safety), trusting a VLM’s “yes” or “no” |
Probability: the basic rules
A random variable \(X\) is a quantity whose value we do not know in advance: the next token, the label of a video clip, whether a goal happens. A probability distribution gives each possible value a number between 0 and 1, and all the numbers add up to 1:
For an LLM, \(X\) is the next token and \(p\) is the softmax output: a list of 128,256 numbers that sum to 1. A distribution over a finite list like this is called categorical.
Joint, conditional and the chain rule
The joint probability \(p(x, y)\) is the chance that both things happen. The conditional probability \(p(y \mid x)\) is the chance of \(y\) when we already know \(x\):
Apply the second form again and again and you get the chain rule from Session 1: \(p(x_1, \dots, x_T) = \prod_t p(x_t \mid x_{\lt t})\). Two events are independent when knowing one tells you nothing about the other: \(p(y \mid x) = p(y)\).
Bayes’ rule
Often we know \(p(\text{evidence} \mid \text{cause})\) but we want \(p(\text{cause} \mid \text{evidence})\). Bayes’ rule turns one into the other:
- Read the symbols
- \(p(y)\)
- the prior: how common the cause is before we look
- \(p(x \mid y)\)
- the likelihood: how often we see this evidence when the cause is true
- \(p(y \mid x)\)
- the posterior: what we should believe after seeing the evidence
In a football match, say 2% of 10-second clips contain a goal. When there is a goal, the crowd roars 90% of the time. But the crowd also roars in 10% of clips without a goal (near misses, fouls, songs). A clip has a roar. How likely is a goal?
Only about 16%. Out of 1,000 clips, 18 have a goal and a roar, but 98 have a roar without a goal. The rare cause is swamped by the common false alarms. This is the base-rate effect, and it is the main reason key-moment detectors produce too many false alarms when the event is rare.
How to read Figure 1
Move the false-alarm rate to 0: every roar is a goal, and the posterior becomes 100%. Now make goals rarer: even a very good detector produces mostly false alarms. In practice you fix this by adding more evidence (for example the score changing on the scoreboard) or by raising the decision threshold.
Expectation
The expected value of a function \(f\) under \(p\) is its average, weighted by probability:
The right side is the Monte Carlo estimate: if we cannot sum over every possible value (for example, every possible sentence), we draw samples and average. RL for LLMs (Track 4) is built on such sample averages.
Maximum likelihood: how models learn from data
We have data \(x_1, \dots, x_N\) and a model \(p_\theta\) with parameters \(\theta\). Which \(\theta\) is best? The maximum likelihood answer: choose the \(\theta\) that makes the data we saw as probable as possible. If the examples are independent, their joint probability is a product, and we take the log as in Session 1:
The quantity in the last bracket is the negative log-likelihood (NLL). Minimizing NLL is the same as maximizing likelihood.
We toss a coin 10 times and see 7 heads. Model: heads with probability \(\theta\). The log-likelihood is
Set the derivative to zero: \(\dfrac{7}{\theta} - \dfrac{3}{1 - \theta} = 0\), so \(\theta = 0.7\). The best guess is simply the fraction of heads. At \(\theta = 0.7\), \(\ell = -6.109\). In general, for \(h\) heads in \(n\) tosses, \(\hat\theta = h/n\). For a categorical variable, the same rule gives: the maximum likelihood probability of each value is its frequency in the data.
How to read Figure 2
Keep \(h/n = 0.7\) but increase \(n\) (try 10, then 100): the peak stays at 0.7 but becomes narrower. More data makes the estimate more certain. Set \(h = 0\): the maximum is at \(\theta = 0\), so the model says heads are impossible. That is overfitting to small data, and it is why LLMs never give exactly zero probability (softmax cannot), and why classical n-gram models needed smoothing.
From maximum likelihood to the LLM loss
An LLM is a categorical distribution over the next token for every context. Maximum likelihood over a corpus of \(T\) tokens is exactly the loss from Session 1:
Why is it called cross-entropy? To answer that we first need entropy.
Information and entropy
Surprise
Seeing an event with probability \(p\) carries information, or surprise, equal to
A sure event (\(p = 1\)) carries 0 surprise. A rare event carries a lot. And surprises add up: two independent events with probabilities \(p\) and \(q\) have joint probability \(pq\), and \(-\log(pq) = -\log p - \log q\). The log is the only function with this property, which is why it is everywhere.
Units: with the natural log (\(\ln\)) the unit is the nat; with \(\log_2\) it is the bit. To convert: \(1 \text{ nat} = 1/\ln 2 \approx 1.443\) bits. PyTorch losses are in nats.
Entropy: average surprise
The entropy of a distribution is the expected surprise:
It measures uncertainty. It is 0 when one outcome is certain, and largest when all \(K\) outcomes are equally likely, where it equals \(\log K\).
A fair coin: \(H = -2 \times 0.5 \log_2 0.5 = 1\) bit. The distribution \((\tfrac12, \tfrac14, \tfrac18, \tfrac18)\): \(H = \tfrac12 \cdot 1 + \tfrac14 \cdot 2 + 2 \cdot \tfrac18 \cdot 3 = 1.75\) bits \(= 1.213\) nats. A uniform guess over Llama 3’s 128,256 tokens: \(\ln 128{,}256 = 11.76\) nats \(= 16.97\) bits. That is the loss of an untrained model, as we saw in Session 1.
A helpful number is \(e^{H}\) (or \(2^{H}\) in bits): the effective number of choices. A distribution with entropy \(\ln 5\) is “as uncertain as” a fair choice among 5 options, however its probabilities are spread.
How to read Figure 3
Make one token very likely: its surprise goes to 0 and the entropy drops toward 0. The rare tokens have huge surprise, but they are rarely seen, so they add little to the average. Press “Uniform”: entropy reaches its maximum \(\log_2 5 = 2.32\) bits and the effective number of choices is exactly 5.
Cross-entropy
Now suppose the true distribution is \(p\) (the data), but we predict with \(q\) (the model). Each outcome \(x\) still happens with probability \(p(x)\), but our surprise is measured with our own prediction, \(-\log q(x)\). The average surprise is the cross-entropy:
In training, the “true” distribution for one position is one-hot: probability 1 on the token that really came next, \(c\). Then the sum has a single term:
That is exactly the per-token loss. Averaged over all positions, it is formula (3.6). So the LLM loss is the cross-entropy between the data and the model, and minimizing it is maximum likelihood. Same thing, three names.
Label smoothing
Sometimes we use a soft target instead of one-hot: keep \(1 - \varepsilon\) on the true class and spread \(\varepsilon\) evenly over all \(K\) classes. With \(K = 4\) and \(\varepsilon = 0.1\), the target becomes \((0.925, 0.025, 0.025, 0.025)\). This stops the model from pushing logits to infinity and often improves calibration in classifiers. In PyTorch: F.cross_entropy(logits, y, label_smoothing=0.1). LLM pretraining usually does not use it. [Likely]
KL divergence: the extra cost of a wrong model
Cross-entropy is never smaller than entropy: predicting with the wrong distribution can only add surprise. The extra amount is the Kullback–Leibler (KL) divergence:
So we have the identity that explains everything:
The entropy of the data does not depend on the model. So minimizing cross-entropy is the same as minimizing \(\KL(\text{data} \,\|\, \text{model})\). This also tells you that the loss can never reach 0 on real text: natural language has its own entropy. A loss curve that flattens is often approaching that floor.
\(p = (\tfrac12, \tfrac14, \tfrac18, \tfrac18)\), and the model guesses uniform \(q = (\tfrac14, \tfrac14, \tfrac14, \tfrac14)\). Then \(H(p) = 1.75\) bits, \(H(p, q) = 2\) bits, so \(\KL(p \,\|\, q) = 0.25\) bits (0.173 nats). Every outcome costs, on average, a quarter bit more than it needs to.
Three properties
- Never negative: \(\KL(p \,\|\, q) \ge 0\). Proof idea: \(-\log\) is convex, so by Jensen’s inequality\[ \KL(p\,\|\,q) = \E_p\!\left[-\log \frac{q}{p}\right] \ge -\log \E_p\!\left[\frac{q}{p}\right] = -\log \sum_x q(x) = -\log 1 = 0 \]
- Zero only when equal: \(\KL(p \,\|\, q) = 0\) exactly when \(p = q\).
- Not symmetric: \(\KL(p \,\|\, q) \ne \KL(q \,\|\, p)\) in general. So it is not a true distance. Example: \(p = (0.9, 0.1)\), \(q = (0.5, 0.5)\) gives \(\KL(p\|q) = 0.368\) but \(\KL(q\|p) = 0.511\) nats.
Figure 4 runs formulas (3.8), (3.9) and (3.11) in your browser, one token at a time. Each step adds one term to each of the three sums, and the panel shows the term with the real numbers put in.
Model q (drag the sliders; they set weights, q is the weights divided by their sum)
- input: data \(p\), model \(q\) over 5 tokens
- \(\text{CE} \leftarrow 0,\ H \leftarrow 0,\ \text{KL} \leftarrow 0\)
- for each token \(x\):
- \(\text{CE} \leftarrow \text{CE} - p(x)\ln q(x)\) // (3.9)
- \(H \leftarrow H - p(x)\ln p(x)\) // (3.8)
- \(\text{KL} \leftarrow \text{KL} + p(x)\ln\frac{p(x)}{q(x)}\) // (3.11)
- return CE, \(H\), KL // check CE = H + KL (3.12)
How to read Figure 4
Press Play with the one-hot data. Only “shot” has \(p \gt 0\), so only the first step adds anything: \(-1 \times \ln 0.441 = 0.818\) nats. The other four tokens add 0, because \(0 \times \ln q = 0\). That is ordinary training: the loss looks only at the true token. \(H(p) = 0\), so the cross-entropy and the KL are the same number.
Switch to the soft teacher and play again. Now every token adds to all three sums. The KL terms of “ball”, “penalty”, “cross” and “cake” are negative (\(q\) gives them more than \(p\) does), so the running KL falls from 0.184 to 0.066. One term can be negative; the full sum never is. The loss ends at 1.124 and can only fall to \(H(p) = 1.058\), never to 0. That is knowledge distillation: the student learns the teacher’s whole distribution, including which wrong answers are “almost right”. With label smoothing the floor is \(H(p) = 0.390\). Press “Set q equal to p” to reach it: the KL becomes 0.
Forward and reverse KL
Because KL is not symmetric, the direction matters, and the two directions behave very differently when the model is simpler than the target:
- Forward KL, \(\KL(p \,\|\, q)\), averages over \(p\). Wherever \(p\) has mass and \(q\) is near zero, \(\log(p/q)\) explodes. So \(q\) must cover every mode of \(p\), even if it puts mass in between. This is “mode-covering”. Maximum likelihood and standard distillation use this direction.
- Reverse KL, \(\KL(q \,\|\, p)\), averages over \(q\). Now \(q\) is punished for putting mass where \(p\) is small, but not for ignoring parts of \(p\). So \(q\) picks one mode and fits it tightly. This is “mode-seeking”. RLHF’s KL penalty, MiniLLM-style distillation (session 119) and variational inference use this direction.
KL is a sum (or, for continuous values, an integral) of the terms \(p(x)\log\frac{p(x)}{q(x)}\). Plot that term against \(x\) and the KL is the area under it: green where \(p\) is above \(q\), red (negative) where \(q\) is above \(p\). Figure 5 fits a one-peaked \(q\) to a two-peaked \(p\) by gradient descent on this area, for either direction.
- input: target \(p\) on 400 grid points, start \(\mu, \sigma\), step size \(\eta\); \(s = \ln\sigma\)
- repeat 60 times:
- \(q \leftarrow \mathcal{N}(\mu, \sigma^2)\) on the grid, normalized
- \(L \leftarrow \sum_i \Delta x\; p_i \ln\frac{p_i}{q_i}\) // forward; reverse swaps p and q
- \(g_\mu, g_s \leftarrow\) slopes of \(L\) by finite differences
- \(\mu \leftarrow \mu - \eta\, g_\mu,\ \ s \leftarrow s - \eta\, g_s\) // step length capped at 0.5
How to read Figure 5
Press Play with forward KL from \(\mu_0 = 1, \sigma_0 = 1\). In the first steps \(\sigma\) grows fast: the green area sits under the far-left peak, where \(q\) is almost zero and \(\ln(p/q)\) is huge. After 60 steps \(q\) is a wide bell between the peaks (\(\mu = 0.06\), \(\sigma = 2.91\), KL = 0.591). It covers both modes but puts mass in the empty valley; the red area there is where \(q\) is above \(p\).
Now switch to reverse KL. From the same start, \(q\) slides onto the right peak and stops at \(\mu = 2.50\), \(\sigma = 0.70\), KL = 0.693 (that is \(\ln 2\): \(q\) is one of the two halves of \(p\)). The term \(q\ln(q/p)\) is zero wherever \(q\) is zero, so the left peak costs nothing. Set \(\mu_0 = 0\): by symmetry \(q\) cannot choose a side, and it gets stuck in a wide bell with KL = 1.261. Reverse KL has more than one local minimum, and where you start decides which one you reach. With \(\eta = 1\) or 2 the reverse fit jumps between two points and never settles, while the forward fit with \(\eta = 2\) reaches its minimum faster: the step size that works depends on the shape of the loss. Think of a small student copying a large teacher: forward KL makes it hedge between all the teacher’s answers; reverse KL makes it confidently give one of them.
KL between two Gaussians
For two 1-D normal distributions there is a closed form, used in every VAE:
Example: \(\KL(\mathcal{N}(0, 1) \,\|\, \mathcal{N}(1, 2^2)) = \log 2 + \tfrac{1 + 1}{8} - \tfrac12 = 0.443\) nats.
Estimating KL from samples (used in RLHF and GRPO)
For two LLMs we cannot sum over all possible texts. We sample texts \(y\) from the policy \(\pi\) and average a per-sample estimate. With the ratio \(r = \pi_{\text{ref}}(y) / \pi(y)\):
Both average to \(\KL(\pi \,\|\, \pi_{\text{ref}})\). \(k_1\) is simple but its single samples can be negative and noisy. \(k_3\) is never negative and has lower variance; GRPO uses this form in its KL penalty. [Certain]
Perplexity, and comparing models fairly
Perplexity is the exponential of the average cross-entropy per token:
The second form shows what it is: the geometric mean of \(1/q\), the average “branching factor”. For a sentence where the model gave the correct tokens probabilities \(0.5, 0.2, 0.8, 0.1\): the losses are \(0.693, 1.609, 0.223, 2.303\), their mean is \(1.207\), and \(\operatorname{PPL} = e^{1.207} = 3.34\). The geometric mean of the probabilities is \(0.299\), and \(1/0.299 = 3.34\). Same number.
Why perplexity does not compare across tokenizers
Perplexity is “per token”. If one tokenizer cuts text into more, smaller tokens, each token is easier to predict, so per-token perplexity looks better even when the model is worse. The fair measure normalizes by something both models share, such as the number of bytes (or characters) of the raw text:
The same 4,000-byte text. Model A uses 1,000 tokens with average loss 2.0: perplexity \(e^{2.0} = 7.39\). Model B uses 1,300 tokens with average loss 1.6: perplexity \(e^{1.6} = 4.95\). B looks much better.
Now total surprise: A spends \(1000 \times 2.0 = 2000\) nats, B spends \(1300 \times 1.6 = 2080\) nats. Bits per byte: A \(= 2000 / (0.693 \times 4000) = 0.721\), B \(= 0.750\). A is the better model. B only looked better because it divided by more tokens.
Probability the model gave each correct token → its loss in nats
- input: \(q_t = q(x_t \mid x_{\lt t})\) for the true tokens
- \(S \leftarrow 0\) (total nats), \(T \leftarrow 0\)
- for each token \(t\):
- \(\ell_t \leftarrow -\ln q_t\) // (3.10)
- \(S \leftarrow S + \ell_t,\ \ T \leftarrow T + 1\)
- perplexity so far \(\leftarrow e^{S/T}\)
- return \(\operatorname{PPL} = e^{S/T} = \big(\prod_t 1/q_t\big)^{1/T}\) // (3.15)
How to read Figure 6
Press Play. The first token, “The”, gets only \(q = 0.05\) (it starts the sentence, so many words were possible): its loss is 2.996 nats and the perplexity after one token is 20.0. Each later token is easier, and the perplexity falls: 8.16, 4.81, 3.86, 3.54, and finally 3.04 after six tokens. The geometric mean of the six probabilities is 0.329, and \(1/0.329 = 3.04\): the same number. “The” alone is 45% of the total loss. Now drag the slider of “penalty” down to 0.01: one surprising token raises the perplexity of the whole sentence.
Calibration: is 90% really 90%?
A model is calibrated if, among all the predictions where it says “90% sure”, about 90% are correct. Modern deep networks are usually overconfident: their 90% is right less often than 90% of the time. This matters whenever you act on a probability: a threshold for alerts, a “yes/no” answer from a VLM, or deciding when to say “I don’t know”.
We measure it with the expected calibration error. Put the predictions into \(M\) bins by confidence, and compare each bin’s average confidence with its accuracy:
The simplest fix is temperature scaling: divide all logits by one number \(T\), chosen on a validation set to minimize the NLL. It does not change which class wins (the order of logits stays the same), so accuracy is unchanged; only the confidence moves.
How to read Figure 7
At \(T = 1\) the bars sit below the diagonal: the model says 90% but is right much less often. Raise \(T\): confidence drops and the bars move onto the line; ECE and NLL both fall. “Fit T” finds the best value on a grid. Accuracy never changes, because dividing by \(T\) keeps the order of the logits. In this toy, the data was generated so that the true best \(T\) is 3.
Where these ideas are used
| Application | Idea | How |
|---|---|---|
| Training any LLM or VLM | Cross-entropy (3.6) | Maximum likelihood on next tokens; for VLMs only on the answer text |
| Action recognition with rare classes | Weighted cross-entropy, focal loss | Give rare classes more weight, or multiply the loss by \((1 - q_c)^{\gamma}\) so easy, common examples count less (session 203) |
| Key-moment detection | Bayes’ rule (3.3), calibration | Rare events need calibrated scores and thresholds chosen with the base rate in mind (session 207) |
| Yes/no questions to a VLM | Calibration (3.17–3.18) | Read \(p(\text{“yes”})\) from the logits; calibrate with temperature before thresholding |
| Distillation | Forward or reverse KL | Match the teacher’s full distribution, not just its top answer |
| RLHF, DPO, GRPO | KL to a reference model | Keep the tuned model close to where it started, so it does not drift or exploit the reward |
| Pretraining data filtering | Perplexity | Text with very high perplexity under a reference model is often noise; very low can mean repeated boilerplate |
| Hallucination detection | Entropy | High entropy over meanings of sampled answers suggests the model is guessing (session 156) |
Pseudo-code
- input: logits \(Z \in \R^{N \times K}\), labels \(y \in \{1..K\}^N\) (pad label = ignore), smoothing \(\varepsilon\)
- \(\log P \leftarrow Z - \operatorname{LSE}(Z)\) row by row // stable log-softmax (Session 2)
- keep only rows whose label is not padding; let \(N'\) be how many
- for each kept row \(n\) do
- \(\text{nll}_n \leftarrow -\log P_{n, y_n}\) // formula (3.10)
- \(\text{smooth}_n \leftarrow -\frac{1}{K}\sum_k \log P_{n,k}\) // cross-entropy with the uniform part
- \(\ell_n \leftarrow (1 - \varepsilon)\,\text{nll}_n + \varepsilon\,\text{smooth}_n\)
- return \(\frac{1}{N'}\sum_n \ell_n\) // average over real tokens only
- input: teacher logits \(\mathbf{a}\), student logits \(\mathbf{b}\), temperature \(\tau\)
- \(\log p \leftarrow \operatorname{logsoftmax}(\mathbf{a}/\tau)\), \(\;\log q \leftarrow \operatorname{logsoftmax}(\mathbf{b}/\tau)\)
- forward: \(\KL(p\|q) \leftarrow \sum_k e^{\log p_k} (\log p_k - \log q_k)\) // formula (3.11)
- reverse: \(\KL(q\|p) \leftarrow \sum_k e^{\log q_k} (\log q_k - \log p_k)\)
- return the chosen direction \(\times\, \tau^2\) // \(\tau^2\) keeps gradient size stable (Hinton et al.)
- input: validation logits \(\{\mathbf{z}_n\}\), labels \(\{y_n\}\)
- for \(T\) in a grid from 0.5 to 5 do
- \(\text{NLL}(T) \leftarrow -\frac{1}{N}\sum_n \log \softmax(\mathbf{z}_n / T)_{y_n}\)
- \(T^* \leftarrow\) the \(T\) with the smallest NLL // formula (3.18); or optimize \(\log T\) with gradient descent
- return \(T^*\) // at test time use \(\softmax(\mathbf{z}/T^*)\)
- input: documents, model, tokenizer
- \(S \leftarrow 0\) (total nats), \(n_{\text{tok}} \leftarrow 0\), \(n_{\text{byte}} \leftarrow 0\)
- for each document \(d\) do
- \(S \leftarrow S + \sum_t -\log q(x_t \mid x_{\lt t})\) // sum, do not average yet
- \(n_{\text{tok}} \leftarrow n_{\text{tok}} + |\text{tokens}(d)|\), \(\;n_{\text{byte}} \leftarrow n_{\text{byte}} + |\text{utf8}(d)|\)
- return \(\operatorname{PPL} = e^{S / n_{\text{tok}}}\) and \(\text{BPB} = S / (n_{\text{byte}} \ln 2)\) // formulas (3.15), (3.16)
A PyTorch script that checks the main claims:
import math, torch import torch.nn.functional as F torch.manual_seed(0) # 1) H(p, q) = H(p) + KL(p || q), formula (3.12), in nats p = torch.tensor([0.5, 0.25, 0.125, 0.125]); q = torch.full((4,), 0.25) H = -(p * p.log()).sum(); CE = -(p * q.log()).sum(); KL = (p * (p / q).log()).sum() print(H.item(), CE.item(), KL.item(), (H + KL - CE).abs().item()) # 1.213, 1.386, 0.173, ~0 # 2) F.kl_div trap: input must be LOG q, target is p, and use reduction="batchmean" logits_s, logits_t = torch.randn(8, 10), torch.randn(8, 10) log_q, p_t = F.log_softmax(logits_s, -1), F.softmax(logits_t, -1) kl_right = F.kl_div(log_q, p_t, reduction="batchmean") # = mean over rows of KL(p_t || q) kl_manual = (p_t * (p_t.log() - log_q)).sum(-1).mean() print(kl_right.item(), kl_manual.item()) # equal # 3) label smoothing is built into cross_entropy y = torch.randint(0, 10, (8,)) print(F.cross_entropy(logits_s, y).item(), F.cross_entropy(logits_s, y, label_smoothing=0.1).item()) # 4) perplexity and bits per byte from a token-level loss; -100 marks padding logits = torch.randn(2, 6, 50); targets = torch.randint(0, 50, (2, 6)); targets[1, 4:] = -100 nll_sum = F.cross_entropy(logits.reshape(-1, 50), targets.reshape(-1), ignore_index=-100, reduction="sum") n_tok = (targets != -100).sum(); n_bytes = 40 # bytes of the raw text (example value) print("PPL", math.exp(nll_sum.item() / n_tok.item()), "BPB", nll_sum.item() / (n_bytes * math.log(2))) # 5) Gaussian KL, formula (3.13) P, Q = torch.distributions.Normal(0., 1.), torch.distributions.Normal(1., 2.) print(torch.distributions.kl_divergence(P, Q).item()) # 0.443
Common mistakes
- Wrong arguments to
F.kl_div. Its first argument must be log-probabilities of \(q\), and the second is \(p\) (probabilities). It computes \(\KL(p\|q)\). Passing plain probabilities, or swapping them, gives wrong numbers without any error. reduction="mean"inF.kl_div. It divides by the number of elements (\(\text{batch} \times \text{classes}\)), not by the batch. Use"batchmean"to get the real KL per example.- Counting padding in the loss. Padded positions must use
ignore_index(default −100), or the loss and perplexity are wrong and depend on batch padding. - Comparing perplexity across tokenizers. Use bits per byte (3.16), or evaluate both models on the same tokenization.
- Averaging perplexities. Average token losses first (weighted by token count), then exponentiate.
- Mixing nats and bits. PyTorch uses natural logs. Divide by \(\ln 2 = 0.693\) to get bits.
- Trusting softmax confidence as a probability. Check calibration on held-out data before thresholding, especially for rare events.
- Ignoring the base rate. A detector with 90% recall and 10% false-alarm rate is mostly false alarms when the event happens in 2% of clips (Figure 1).
Interview questions
Why is the LLM training loss called cross-entropy, and how is it related to maximum likelihood?basic
Define entropy, cross-entropy and KL, and give the identity that connects them.basic
Why is KL divergence non-negative, and why is it not a distance?core
Forward or reverse KL for distilling a large LLM into a small one?advanced
Why does RLHF add a KL penalty to a reference model? Which direction is it?core
What is perplexity, and why can you not compare it across tokenizers?core
A model’s loss stops decreasing at 1.8 nats per token. Is it broken?advanced
What is calibration and how do you fix an overconfident classifier?core
Your key-moment detector has 90% recall and a 10% false-positive rate, and goals occur in 2% of clips. What precision do you expect?core
What does label smoothing do to the target and to the logits?core
How do you compute KL between two LLMs’ output distributions in practice?advanced
F.kl_div argument order and use batchmean.What is the entropy of a uniform distribution over the Llama 3 vocabulary, and why does it matter?basic
Exercises
- Compute \(H(p)\) in bits for \(p = (0.25, 0.25, 0.25, 0.25)\) and for \(p = (1, 0, 0, 0)\).
- A model gives the true token probability 0.05. What is the loss in nats and in bits?
- Show that if \(q\) is uniform over \(K\) classes, then \(\KL(p\|q) = \log K - H(p)\).
- Three documents have average losses 2.0, 2.0 and 3.0 nats with 100, 100 and 800 tokens. What is the corpus perplexity? What wrong number do you get by averaging the three perplexities?
- In Figure 1, keep recall at 90% and goals at 2%. What false-alarm rate gives a precision of 50%?
- Compute \(\KL(\mathcal{N}(0, 1)\|\mathcal{N}(0, 1))\) and \(\KL(\mathcal{N}(2, 1)\|\mathcal{N}(0, 1))\) with formula (3.13).
Answerscheck
Key takeaways
- Maximum likelihood (3.5) = minimizing negative log-likelihood = minimizing cross-entropy (3.9–3.10). The LLM loss is all three.
- Entropy (3.8) is average surprise; \(e^{H}\) is the effective number of choices. A uniform guess over the vocabulary costs \(\ln|V|\).
- \(H(p,q) = H(p) + \KL(p\|q)\) (3.12): training reduces only the KL part; the data’s entropy is a floor.
- KL is not symmetric: forward is mode-covering (likelihood, standard distillation), reverse is mode-seeking (RLHF penalty, MiniLLM). Sample estimators (3.14) are used in RL.
- Perplexity is per token; compare models with different tokenizers using bits per byte (3.16). Check calibration (3.17) before trusting probabilities; temperature scaling (3.18) fixes most overconfidence.
Key formulas
References for this session
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 3: N-gram Language Models
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 1: Foundations of Machine Learning
- Guo et al. (2017). On Calibration of Modern Neural Networks
- Hinton et al. (2015). Distilling the Knowledge in a Neural Network
- Schulman (2020). Approximating KL Divergence (blog)
- Szegedy et al. (2015). Rethinking the Inception Architecture for Computer Vision
- Lin et al. (2017). Focal Loss for Dense Object Detection
- Müller et al. (2019). When Does Label Smoothing Help?
- NVIDIA (docs). TensorRT documentation