Knowledge distillation
Knowledge distillation trains a small student model to behave like a large teacher model. The student learns from the teacher's full probability distribution (logit distillation) or from the text the teacher writes (sequence-level distillation), which is how small open models and small reasoning models are made today. We also see why a student sometimes cannot copy its teacher.
What you will learn
- Soft targets: why the teacher's full distribution carries more information than a one-hot label
- Temperature in the softmax, the KD loss, its gradient (q - p)/T and why the loss is multiplied by T squared
- Forward KL (mode-covering) versus reverse KL (mode-seeking) for a student that is too small
- Sequence-level distillation (Kim and Rush): training on the teacher's outputs, and on-policy distillation against exposure bias
- How reasoning traces are distilled into small models with rejection sampling and SFT, with the DeepSeek-R1 numbers
- The capacity gap: why a bigger teacher or a longer trace can give a worse student, and what to do about it
Why this matters
In session 58 we made fine-tuning cheap: the big model stays frozen in 4 bits and we train small adapters. But the model we serve is still big. Every user request pays for all of its parameters. Knowledge distillation attacks that cost directly: a large, strong model (the teacher) trains a small model (the student) so that the student behaves like the teacher at a fraction of the price.
The student is trained not only on the “correct answer” but on what the teacher says: its full probability distribution over the next token (logit distillation), or the text it writes (sequence-level distillation). This extra signal is much richer than a single correct token, and that is why a distilled student usually beats the same small model trained from scratch on the same data.
Distillation is everywhere in real systems:
- Gemma 2 trained its 2B and 9B models “with knowledge distillation instead of next token prediction”, on 2 and 8 trillion tokens, which is more than 50 times the compute-optimal number of tokens. [Certain]
- Llama 3.2 1B and 3B were pruned from Llama 3.1 8B, and logits from Llama 3.1 8B and 70B were used as targets during pretraining. [Certain]
- DeepSeek-R1-Distill: six small models (1.5B to 70B) were made by plain SFT on about 800k samples written by DeepSeek-R1. The 7B student reaches 55.5% on AIME 2024, more than QwQ-32B-Preview (50.0%). [Certain]
- DistilBERT is 40% smaller than BERT, about 60% faster, and keeps about 97% of its language-understanding score. [Certain]
This session builds the idea from the original soft-target trick of Hinton et al. (2015) to how reasoning models are distilled today, and ends with the main limit: a student cannot copy a teacher that is too far above it. The inference track goes deeper into white-box and black-box methods (session 119, session 120).
Soft targets and temperature
A normal training label is one-hot: probability 1 on the correct token, 0 on all others. It says what is right, but nothing about how wrong the other options are. A trained teacher knows more. After “The cat sat on the”, it gives high probability to “mat”, some to “floor” and “sofa”, and almost none to “banana”. These small numbers tell the student which words are similar and which mistakes are reasonable. Hinton et al. called this dark knowledge and the teacher’s distribution soft targets. [Certain]
The problem: a confident teacher puts most mass on one token, and the useful information hides in tiny probabilities (0.001 versus 0.000001) that barely affect a cross-entropy loss. The fix is to soften the softmax with a temperature \(T\):
- Read the symbols
- \(z_i\)
- the logit (raw score before the softmax) for token \(i\)
- \(V\)
- the vocabulary size, the number of possible next tokens
- \(T\)
- the temperature: \(T = 1\) is the normal softmax, \(T \gt 1\) makes the distribution flatter, \(T \lt 1\) makes it sharper
- \(p_i(T)\)
- the probability of token \(i\) at temperature \(T\)
Dividing by \(T\) shrinks the gaps between logits. A gap of 3 between “roof” (2.0) and “banana” (−1.0) in the example below means a probability ratio of \(e^{3} \approx 20\) at \(T = 1\), but only \(e^{3/4} \approx 2.1\) at \(T = 4\). The order of the tokens never changes; only the contrast does. Both teacher and student use the same \(T\) during distillation. After training, the student is used at \(T = 1\). [Certain]
Teacher logits for “The cat sat on the ___”: mat 6.0, floor 4.5, sofa 4.0, bed 3.5, roof 2.0, banana −1.0. With formula (59.1):
\(T = 1\): (0.685, 0.153, 0.093, 0.056, 0.013, 0.001). Entropy 0.988 nats; the five non-top tokens share 31.5% of the mass; sofa is 148 times more likely than banana.
\(T = 2\): (0.436, 0.206, 0.160, 0.125, 0.059, 0.013). Entropy 1.465 nats; non-top mass 56.4%; sofa/banana ratio 12.2.
\(T = 4\): (0.297, 0.204, 0.180, 0.159, 0.109, 0.052). Entropy 1.680 nats; non-top mass 70.3%; ratio 3.5. The ranking is the same at every \(T\), but at \(T = 4\) the student can actually see that “roof” is a better mistake than “banana”.
How to read Figure 1
Press “Sweep T” to watch the temperature rise from 0.5 to 10 while every bar and number is recomputed with formula (59.1). At \(T = 1\) the teacher bar for “mat” is tall and “banana” is invisible: the soft target is close to the one-hot label. Move \(T\) to 4: the teacher spreads out and the small tokens become visible. Now watch the readout. The raw KL shrinks as \(T\) grows (both distributions get flat, so they look alike), and the largest gradient shrinks roughly like \(1/T^2\). After multiplying by \(T^2\), the gradient stays in the same range (about 0.25 at \(T = 1\) and \(T = 2\), 0.21 at \(T = 4\)). That is why the loss is scaled by \(T^2\). Very low \(T\) (below 1) makes both distributions sharper than normal and the soft targets lose their extra information.
Logit distillation: the KL loss
In logit distillation (also called word-level or token-level distillation, or white-box distillation, because we must see the teacher’s logits) the student matches the teacher’s softened distribution. Hinton et al. combine two terms: a normal cross-entropy with the true label, and a KL divergence to the teacher at temperature \(T\) (session 3 introduced KL):
- Read the symbols
- \(p(T)\)
- the teacher distribution at temperature \(T\), formula (59.1) with the teacher logits \(v_i\)
- \(q(T)\)
- the student distribution at temperature \(T\), with the student logits \(z_i\); \(q(1)\) is the normal student softmax
- \(y\)
- the true (hard) label, if the data has one
- \(\alpha\)
- the weight of the distillation term, between 0 and 1; \(\alpha = 1\) means pure distillation with no labels
- \(\KL(p \| q)\)
- \(\sum_i p_i \log (p_i / q_i)\): how many extra nats the student pays for using \(q\) where the teacher uses \(p\). It equals the cross-entropy \(-\sum_i p_i \log q_i\) minus the teacher entropy, which does not depend on the student, so both give the same gradient
- \(T^2\)
- a scale factor that keeps the soft-target gradient the same size when \(T\) changes (see below)
Where the \(T^2\) comes from
The gradient of the KL term with respect to one student logit is simple:
“Student probability minus teacher probability”, as for any softmax with cross-entropy, times \(1/T\) from the chain rule through \(z_i/T\). When \(T\) is large compared with the logits, \(e^{x} \approx 1 + x\), and with logits that have zero mean over the vocabulary Hinton et al. show:
- Read the symbols
- \(z_i\), \(v_i\)
- the student and teacher logits for token \(i\), each shifted to have mean 0 over the vocabulary
- \(V\)
- the vocabulary size (\(N\) in the original paper)
Two lessons. First, the gradient shrinks like \(1/T^2\), so without the \(T^2\) factor in (59.2) a higher temperature would silently weaken the distillation term against the hard-label term. [Certain] Second, at high temperature distillation becomes logit matching: the loss behaves like \(\frac{1}{2}(z_i - v_i)^2\), so the student copies the teacher’s logits, including the very negative ones. At lower temperatures the student cares less about matching very negative logits. Hinton et al. found that intermediate temperatures often work best, which suggests that the most negative logits are noisy and are better ignored. [Certain]
Use the teacher of Figure 1 and student logits (5.0, 4.8, 2.0, 3.0, 1.0, 0.5). At \(T = 1\) the student gives “mat” 0.492, so the hard-label loss is \(-\log 0.492 = 0.710\). The KL to the teacher is 0.196 at \(T = 1\), 0.076 at \(T = 2\), and 0.026 at \(T = 4\). Multiplied by \(T^2\): 0.196, 0.303 and 0.415.
The largest gradient (59.3) is on “floor” at \(T = 1\): \(q - p = 0.403 - 0.153 = 0.250\). At \(T = 2\) the largest gradient is 0.062, at \(T = 4\) only 0.013. Times \(T^2\): 0.250, 0.249 and 0.212, the same size. At \(T = 20\), the exact gradient times \(T^2\) for “sofa” is −0.251 and the approximation (59.4) gives −0.258: the logit-matching limit is close.
The KD step in motion
Put (59.2) and (59.3) together. The hard-label term gives the usual softmax gradient \(q_i(1) - y_i\), and the KD term gives \(T^2 \cdot \frac{1}{T}(q_i(T) - p_i(T))\). So one gradient step on a single example moves every student logit by
Here \(y_i\) is 1 for the labeled token and 0 for the others, and \(\eta\) is the learning rate. Figure 2 runs exactly this loop on the six-token example: the teacher is frozen, and the student’s six logits are the only parameters. Each step has four lines: forward pass, loss, gradient, update. The box next to the pseudo-code shows each formula with the current numbers in it.
- \(p \leftarrow \softmax(v / T)\) // teacher, frozen, (59.1)
- \(q_T \leftarrow \softmax(z / T)\), \(q_1 \leftarrow \softmax(z)\) // student
- \(\mathcal{L} \leftarrow (1 - \alpha)\,(-\ln q_1[y]) + \alpha\, T^2 \KL(p \,\|\, q_T)\) // (59.2)
- \(g \leftarrow (1 - \alpha)\,(q_1 - e_y) + \alpha\, T\,(q_T - p)\) // (59.3)
- \(z \leftarrow z - \eta\, g\); go back to line 2
How to read Figure 2
Press Step a few times. Line 1 computes the teacher’s soft targets once. In step 1 at the default setting (\(T = 4\), \(\alpha = 0.5\), \(\eta = 1\)) the loss is 0.5625 and the largest push is on “mat”: \(g = 0.5 \cdot (0.492 - 1) + 0.5 \cdot 4 \cdot (0.269 - 0.297) = -0.310\), so its logit goes up from 5.0 to 5.310, while “floor” (\(g = 0.305\)) goes down. After 40 steps the KL at \(T = 4\) falls from 0.0260 to 0.0049. Now uncheck the \(T^2\) box: the KD gradient is 16 times smaller, the hard label wins, the student puts 0.953 on “mat” (the teacher has 0.685), and the KL after 40 steps is 0.0425. Set \(\alpha = 1\) (pure distillation): with \(T^2\) the student copies the teacher almost exactly (KL 0.0000, “mat” 0.686); without it, the KL is still 0.0130 after 40 steps. At \(T = 1\) the checkbox changes nothing, because \(T^2 = 1\).
Why soft targets help: a tiny student trained live
Hinton et al. showed how much information soft targets carry with a striking MNIST test. They removed every example of the digit 3 from the transfer set (the data used for distillation). The distilled student had never seen a 3. Still, after one bias correction it got 98.6% of the test 3s right, because the teacher’s soft targets on other digits (a 3-like 8, a 3-like 5) had told it what a 3 looks like. [Certain] Without the correction it made 206 test errors, 133 of them on the 1,010 threes, because the learned bias for class 3 was too low. [Certain]
Figure 3 shows the same effect at a size you can watch. The teacher is a fixed linear classifier over points \(x \in [-1, 1]^2\) with three classes A, B and C: logits \(v(x) = W x + c\). Two students with the same form \(z(x) = A x + b\) start from zero and see the same few transfer points. One is trained on hard labels (the teacher’s top class, one-hot), the other on the teacher’s soft targets with the loss \(T^2\,\KL\) of (59.2) with \(\alpha = 1\). The student’s form can represent the teacher exactly, so there is no capacity gap; the only difference is how much each training point tells the student.
- teacher logits for every transfer point: \(v_n = W x_n + c\) // frozen
- student logits: \(z_n = A x_n + b\) // both start at 0
- hard: \(\ell = -\ln \softmax(z_n)[y_n]\); soft: \(\ell = T^2 \KL(p_n \,\|\, q_n)\) // (59.2)
- \(\partial \ell / \partial z_n\): hard \((q_n - e_y)/N\); soft \(T\,(q_n - p_n)/N\) // (59.3)
- \(A \leftarrow A - \eta \sum_n \partial z_n\, x_n^{\top}\), \(b \leftarrow b - \eta \sum_n \partial z_n\)
- evaluate both students on a \(20 \times 20\) grid: agreement with the teacher’s class
How to read Figure 3
Press Play. With the default 6 points and \(T = 2\), the soft-target student reaches 95% agreement with the teacher at step 42 and ends at 98.8%; the hard-label student ends at 83.0%, because six one-hot labels leave the borders free to sit anywhere between the points. Watch the formula box: point 1 has hard target (0, 0, 1) but soft target with mass on all three classes, so it pushes on every logit. Now check “Leave class C out”: no transfer point is labeled C. The hard-label student never predicts C (67.0%), but the soft-target student still finds the C region (99.8%), because the soft targets of A and B points near the border say how close C is. This is Hinton’s “mythical digit” in two dimensions. Move \(N\) to 20: the hard student catches up (98.8% against 99.0%). Soft targets help most when data is scarce.
Distilling a language model, token by token
For an LLM, the “classification” is the next token, and there is one at every position. Token-level distillation applies (59.2) at every position of every training sequence \(x\):
- Read the symbols
- \(x_{\lt t}\)
- the tokens before position \(t\), taken from the training data (the same prefix goes into teacher and student)
- \(p(\cdot \mid x_{\lt t};T)\)
- the teacher’s full next-token distribution at temperature \(T\): a vector of \(V\) numbers per position
- \(q_\theta\)
- the student with parameters \(\theta\), the only thing that is trained
- \(N\)
- the number of positions that get a loss (with an SFT loss mask, only the assistant tokens; see session 54)
Gemma 2 used exactly this kind of objective during pretraining: minimize \(\sum_x -P_T(x \mid x_c) \log P_S(x \mid x_c)\), the cross-entropy between teacher and student next-token distributions. [Certain] Three practical points decide whether it is possible:
- Same tokenizer. The KL compares two vectors over the same vocabulary. Teacher and student must share the tokenizer, which is why logit distillation is usually done inside one model family (Gemma 27B to 9B, Llama 70B to 3B). Cross-tokenizer methods exist but are harder. [Likely]
- Access to logits. You need the teacher’s weights or an API that returns full log-probabilities. A chat API that returns only text allows only sequence-level distillation (next section).
- Cost. Either run the teacher forward at every training step (online), or precompute its outputs once and store them (offline). Storing full distributions is impossible at scale, so offline pipelines keep only the top-\(k\) tokens per position. [Likely]
Take a vocabulary of 128,256 tokens (Llama 3) and a distillation set of 1 billion tokens. Full logits in bf16 (2 bytes) need \(10^9 \times 128{,}256 \times 2 \approx 256.5\) TB. Keeping only the top 20 tokens per position, with a 4-byte token id and a 2-byte value each, needs \(10^9 \times 20 \times 6 = 120\) GB: over 2,000 times less. The cost is a small bias: the student never sees the tail of the distribution, and the kept probabilities must be renormalized or the missing mass treated as one extra bucket.
Forward KL or reverse KL?
\(\KL(p \| q)\) is not symmetric. The two directions push a limited student in different ways:
- Forward KL is mode-covering. Wherever the teacher has mass (\(p_i \gt 0\)) and the student has almost none, \(\log(p_i/q_i)\) explodes. So the student spreads out to cover everything the teacher might say, even if it must put mass between the teacher’s modes. This is what (59.2) and (59.5) use.
- Reverse KL is mode-seeking. The average is over the student’s own outputs. Putting mass where the teacher has almost none is punished hard; ignoring a teacher mode costs nothing. So the student picks one mode and fits it well.
For a small generative student, covering everything can mean generating text the teacher would never write. MiniLLM (Gu et al.) therefore replaces forward KL with reverse KL, to stop the student from “overestimating the low-probability regions of the teacher distribution”, and reports gains from 120M to 13B parameters. [Certain] Reverse KL is harder to optimize because the average is over the student’s own samples, so it needs sampling and policy-gradient-like methods. [Certain] Which direction is better depends on the task and on the decoding method; the GKD paper treats the choice as a hyper-parameter. [Likely]
How to read Figure 4
At distance 0 the teacher has one mode and both students agree. At the default distance 6 the teacher puts only 0.6% of its mass in the gap \(|x| \lt 0.5\). The forward-KL student is wide (\(\sigma = 3.10\)) and sits between the modes, with 11.9% of its mass in the gap. The reverse-KL student sits on the heavier mode (\(\mu = -2.99\), \(\sigma = 1.02\)) and ignores the other one. Now lower the distance slowly: below about 5 (with weight 0.6) the modes overlap enough that covering both is cheaper even for reverse KL, and the two students look alike. For a language model, “between the modes” is a mix of two good answers that is itself not a good answer. Set the weight to 0.5 and the distance to 7: the reverse-KL student still picks one side, and which side depends on tiny numerical differences.
Sequence-level distillation: train on the teacher’s outputs
Token-level distillation teaches the student, position by position, to predict like the teacher on given prefixes from the data. But a generative model is judged on whole sequences that it writes itself. Kim and Rush (2016) asked: why not distill at the level of sequences? [Certain]
Write the student as \(q_\theta(y \mid x)\) over whole output sequences \(y\) for an input \(x\). Kim and Rush compare two losses. Word-level KD (Word-KD) is (59.5) on the gold prefix:
Sequence-level KD (Seq-KD) matches the teacher’s distribution over all sequences, which is a sum over exponentially many \(y\). They approximate the teacher distribution by its mode \(\hat{y}\), found with beam search:
- Read the symbols
- \(y\), \(y_{\lt t}\)
- an output sequence, and its first \(t-1\) tokens
- \(p(y \mid x)\)
- the teacher’s probability of the whole sequence: the product of its next-token probabilities
- \(\hat{y}\)
- the teacher’s best output found by beam search: a proxy for the mode of \(p(y \mid x)\)
So Seq-KD is very simple in practice: (1) train the teacher, (2) run beam search with it over the training inputs, (3) train the student with ordinary cross-entropy on these outputs instead of the original references. [Certain] This is exactly “SFT on teacher outputs”, and it needs no logits at all: it works with a different tokenizer and with a teacher behind an API. Today it is usually called black-box distillation or simply training on synthetic data. They also defined Seq-Inter: train on the sequence in the teacher’s beam that is closest (by BLEU) to the human reference, a mix of “what the teacher would say” and “what is correct”. [Certain]
Why does the mode work when it is a single sequence out of billions? Kim and Rush measured that the teacher’s beam output already carries a lot of its mass: on average 1.3% of the whole distribution for English→German. [Certain] And training on it makes the student peaked: after Seq-KD the student puts 16.9% of its mass on its own best output, against 0.9% for the baseline student. [Certain] A peaked student does not need beam search: greedy decoding finds almost the same answer.
Teacher: 4-layer LSTM with 1000 units, 221M parameters, BLEU 17.7 (greedy) and 19.5 (beam 5). Student: 2 × 500, 84M parameters. [Certain]
| Student 2 × 500 trained with | BLEU, greedy | BLEU, beam 5 | Mass on own best output |
|---|---|---|---|
| Original references (baseline) | 14.7 | 17.6 | 0.9% |
| Word-KD (59.7) | 15.4 | 17.7 | 1.0% |
| Seq-KD (59.8) | 18.9 | 19.0 | 16.9% |
| Seq-KD, then Seq-Inter | 18.9 | 19.3 | 7.6% |
Seq-KD adds 4.2 BLEU with greedy decoding (18.9 − 14.7) and 1.4 with beam search. The greedy student (18.9) is better than the greedy teacher (17.7). The student with greedy decoding ran at 1051.3 words per second on a GPU against 101.9 for the teacher with beam search, about 10 times faster. After pruning 80% of its weights it had 17M parameters, 13 times fewer than the teacher, at BLEU 19.1 versus 19.5. [Certain]
On-policy distillation: let the student write
Both Word-KD and Seq-KD train the student on prefixes it did not write. At inference, the student conditions on its own earlier tokens, and once it makes a small mistake it is in a state it never saw in training. This train–test mismatch is called exposure bias. On-policy distillation fixes it: sample sequences from the student, and ask the teacher for its next-token distribution at every prefix of those samples. GKD (Agarwal et al.) trains on such student-generated sequences with teacher feedback, allows other divergences than forward KL, and can be combined with RL fine-tuning. [Certain] Gemma 2’s post-training used it too: “distillation from the teacher on the student’s distribution”. [Certain]
- Read the symbols
- \(y \sim q_\theta\)
- the sequences are sampled from the student, not taken from the data or the teacher
- \(D\)
- a divergence per position: forward KL, reverse KL, or a mix (GKD uses a generalized Jensen–Shannon divergence)
In practice the gradient is not taken through the sampling: the sampled tokens are treated as fixed data, and only the per-token divergence is differentiated. [Likely] The cost is a teacher forward pass on fresh student samples at every step, so it is more expensive than offline Seq-KD.
How to read Figure 5
Hard labels train one path, the reference, with one-hot targets. Word-KD walks the same reference path but gives each of its two positions the teacher’s full distribution (blue). Seq-KD greedy follows the teacher’s best first token, “Thank”, and gets “Thank you” (0.180), which is not the teacher’s mode: beam search keeps 3 candidates and finds “Thanks !” (0.280). On-policy uses the student’s own samples as prefixes, and the teacher grades every position: press “Sample again” to see that the student is corrected on paths such as “Cheers mate” that no reference would contain.
Distilling reasoning traces into small models
Reasoning models such as DeepSeek-R1 write a long chain of thought (a reasoning trace: the intermediate steps) before the final answer (session 72). They get this skill from large-scale reinforcement learning with verifiable rewards (session 70), which is expensive and works best on strong base models. Distillation offers a shortcut: let the strong reasoner write traces, keep the correct ones, and run SFT on a small model. This is sequence-level distillation (59.8) with sampling instead of beam search, plus a filter.
DeepSeek-R1 did exactly this. [Certain]
- Data. About 600k reasoning samples, made by sampling several responses per prompt from an R1 checkpoint and keeping only correct ones (rule-based checks, or DeepSeek-V3 as a judge), after removing traces with mixed languages, long paragraphs and code blocks; plus about 200k non-reasoning samples (writing, factual QA, translation). About 800k in total.
- Students. Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-14B, Qwen2.5-32B, Llama-3.1-8B and Llama-3.3-70B-Instruct, fine-tuned on the 800k samples.
- Only SFT. No RL stage for the students, “even though incorporating RL could substantially boost model performance”.
- Read the symbols
- \(x\)
- a prompt, for example a math problem with a known answer
- \(r\), \(a\)
- the teacher’s reasoning trace and its final answer, sampled together
- \(\text{verify}\)
- a checker: compare with the reference answer, run unit tests for code, or ask a judge model; this is rejection sampling: wrong samples are thrown away
- \(q_\theta(r, a \mid x)\)
- the student’s probability of the whole trace and answer; the loss is the SFT loss, with the prompt masked
| Model (AIME 2024 and MATH-500, pass@1) | AIME 2024 | MATH-500 | GPQA Diamond |
|---|---|---|---|
| DeepSeek-R1-Distill-Qwen-1.5B | 28.9 | 83.9 | 33.8 |
| DeepSeek-R1-Distill-Qwen-7B | 55.5 | 92.8 | 49.1 |
| DeepSeek-R1-Distill-Qwen-14B | 69.7 | 93.9 | 59.1 |
| DeepSeek-R1-Distill-Qwen-32B | 72.6 | 94.3 | 62.1 |
| DeepSeek-R1-Distill-Llama-8B | 50.4 | 89.1 | 49.0 |
| DeepSeek-R1-Distill-Llama-70B | 70.0 | 94.5 | 65.2 |
| QwQ-32B-Preview (comparison) | 50.0 | 90.6 | 54.5 |
| GPT-4o-0513 (comparison) | 9.3 | 74.6 | 49.9 |
Numbers from Table 5 of the DeepSeek-R1 paper. [Certain] The most important experiment is the comparison with RL on the same base. DeepSeek ran large-scale RL (over 10K steps on math, code and STEM data) on Qwen-32B-Base and got DeepSeek-R1-Zero-Qwen-32B: 47.0 on AIME 2024, about the same as QwQ-32B-Preview (50.0). The distilled Qwen-32B gets 72.6, and is better on every benchmark they report. [Certain] Their conclusion: distilling a strong model into a small one is cheap and effective, while RL on the small model needs enormous compute and may still not reach distillation; but going beyond the teacher still needs stronger base models and larger-scale RL. [Certain]
Rejection sampling keeps a prompt only if at least one of \(k\) teacher samples is correct. With an independent success chance \(s\) per sample:
For an easy prompt with \(s = 0.6\) and \(k = 4\): kept with probability \(1 - 0.4^4 = 0.974\), and 2.4 correct traces on average. For a hard prompt with \(s = 0.3\): kept with probability \(1 - 0.7^4 = 0.760\), 1.2 traces. So the filter quietly removes hard prompts and gives easy prompts more traces. Many pipelines keep at most one or two traces per prompt, or sample more for hard prompts, to fight this bias. [Likely]
Practical details that matter for reasoning distillation:
- Very long targets. Traces of thousands to tens of thousands of tokens need long-context SFT, packing with document masks, and answers that are never cut (session 54). A cut trace loses its final answer.
- Format. Traces are often wrapped in special tags (for example a think block, then a summary). The student must learn the same template that inference will use.
- Correct is not the same as clean. A trace can reach the right answer with wrong steps. Answer checks do not catch this; judges and step checks catch some of it. [Likely]
- Licenses and terms of use. Some providers forbid using their outputs to train competing models; check before distilling from an API. [Likely]
When the student cannot copy the teacher: the capacity gap
If teachers help, should we always use the biggest teacher? No. Several studies found that a stronger teacher can give a worse student:
- Mirzadeh et al. (2019) found that student performance degrades when the size gap between teacher and student is large, and proposed a teacher assistant: distill from the big teacher into a medium model, then from the medium model into the student. [Certain]
- Cho and Hariharan (2019): “larger models do not often make better teachers”. They explain it with mismatched capacity, find that a sequence of distillation steps does not fix it in their setting, and show that a teacher stopped early in its training (less sharp, easier to copy) helps. [Certain]
- Li et al. (2025), Small Models Struggle to Learn from Strong Reasoners: students of 3B parameters or less do not consistently gain from long chains of thought or from larger teachers; they learn better from shorter, simpler traces. Their Mix Distillation mixes long and short traces, or traces from larger and smaller teachers. [Certain]
- Hinton et al.: with a very small MNIST student (30 units per layer), temperatures 2.5 to 4 worked clearly better than higher or lower ones; with 300 or more units, all temperatures above 8 gave similar results. [Certain] A small student cannot use all the fine detail that a high temperature reveals.
The common reason: the student’s family of functions cannot contain the teacher. The best the student can do is the closest point in its own family, and the distance to that point is a floor that no amount of data removes:
- Read the symbols
- \(\min_\theta\)
- the best student in its architecture, after perfect optimization on unlimited data
- \(\varepsilon_{\text{gap}}\)
- the capacity gap: 0 only if the student family can represent the teacher
When \(\varepsilon_{\text{gap}}\) is large, forward KL makes the student spread mass between teacher modes (Figure 4), which for text means bland or mixed-up outputs. Long reasoning traces make it worse: the student must reproduce a long, precise chain, and one wrong step breaks it. The cures all reduce what the student has to copy: a closer teacher (teacher assistant, early-stopped teacher), simpler targets (shorter traces, Mix Distillation), a moderate temperature, mode-seeking or on-policy objectives (MiniLLM, GKD), or simply a bigger student.
How to read Figure 6
At rank 12 the student can represent the teacher, and the remaining forward KL is almost 0. Lower the rank: the KL floor (59.12) rises, the capacity gap. Data is not the problem, since the student sees the exact teacher distribution for every context. Compare the objectives at rank 2 or 3: the forward-KL student has higher entropy than the teacher (it covers), the reverse-KL student has lower entropy (it picks modes), and each one is better on the KL it was trained on. Move the context slider to see single distributions.
Putting it together
- input: teacher \(p\) (frozen), student \(q_\theta\) with the same tokenizer, data \(\mathcal{D}\), temperature \(T\), weight \(\alpha\)
- for each batch of token sequences \(x\) with loss mask \(m\):
- with no gradient: \(v \leftarrow\) teacher logits on \(x\) // shape (B, L, V)
- \(z \leftarrow\) student logits on \(x\)
- \(\ell_{\text{KD}} \leftarrow T^2 \cdot \KL(\softmax(v/T)\,\|\,\softmax(z/T))\) per position // (59.5)
- \(\ell_{\text{CE}} \leftarrow\) next-token cross-entropy of \(z\) with the data tokens // optional hard labels
- loss \(\leftarrow \sum m \cdot \big((1-\alpha)\,\ell_{\text{CE}} + \alpha\,\ell_{\text{KD}}\big) / \sum m\) // (59.2)
- loss.backward(); optimizer step on \(\theta\) only
- offline variant: precompute top-\(k\) teacher tokens and log-probs per position once; renormalize over those \(k\)
- input: prompts \(\{x\}\) with checkable answers, teacher \(p\) (API or weights), samples per prompt \(k\)
- for each prompt \(x\):
- sample \(k\) outputs \((r_j, a_j) \sim p(\cdot \mid x)\) at the teacher’s recommended temperature
- keep those with verify\((x, a_j) = 1\) and a clean format // rejection sampling, (59.10)
- keep at most \(c\) traces per prompt // limit the easy-prompt bias, (59.11)
- add non-reasoning chats so the student stays a general assistant
- decontaminate against evaluation sets; deduplicate
- SFT the student on the result: prompt masked, long rows, no cut traces // session 54
- optional: on-policy distillation (59.9) or RL on the student afterwards
The PyTorch code below runs on a CPU in a few seconds. It checks the gradient formula (59.3) and the \(T^2\) effect on the worked example, then trains a rank-3 student to copy a full-rank bigram teacher in three ways: hard labels sampled from the teacher, the teacher’s greedy outputs (sequence-level distillation with one-token sequences), and the full teacher distribution (logit distillation). Last, it shows the capacity gap (59.12) by changing the student rank.
import torch, torch.nn.functional as F torch.manual_seed(0) # 1) the distillation loss, formula (59.2): hard-label CE + T^2 * KL at temperature T def kd_loss(s_logits, t_logits, labels=None, T=2.0, alpha=0.5): log_q = F.log_softmax(s_logits / T, dim=-1) # student at temperature T p = F.softmax(t_logits / T, dim=-1) # teacher soft targets kl = F.kl_div(log_q, p, reduction="batchmean") # KL(p || q), averaged over rows if labels is None: return kl * T * T ce = F.cross_entropy(s_logits, labels) # hard labels at T = 1 return (1 - alpha) * ce + alpha * T * T * kl # 2) check the gradient, formula (59.3): dKL/dz = (q - p) / T (per row) z_t = torch.tensor([[6.0, 4.5, 4.0, 3.5, 2.0, -1.0]]) # teacher logits (worked example) z_s = torch.tensor([[5.0, 4.8, 2.0, 3.0, 1.0, 0.5]], requires_grad=True) for T in [1.0, 2.0, 4.0]: z_s.grad = None (kd_loss(z_s, z_t, T=T) / (T * T)).backward() # plain KL, without the T^2 manual = (F.softmax(z_s / T, -1) - F.softmax(z_t / T, -1)) / T print(T, torch.allclose(z_s.grad, manual, atol=1e-6), round(z_s.grad.abs().max().item() * T * T, 4)) # 1.0 True 0.2498 / 2.0 True 0.2492 / 4.0 True 0.2122 : the T^2 factor keeps the gradient size steady # 3) a teacher and a smaller student: bigram language models over V tokens V, r = 12, 3 teacher = torch.randn(V, V) * 2.0 # full-rank teacher logits: row = previous token class Student(torch.nn.Module): # rank-r student: cannot copy the teacher exactly def __init__(s, r=r): super().__init__() s.E, s.W = torch.nn.Embedding(V, r), torch.nn.Linear(r, V) def forward(s, ctx): return s.W(s.E(ctx)) ctx = torch.randint(0, V, (240,)) # the transfer set: 240 contexts hard = torch.multinomial(F.softmax(teacher[ctx], -1), 1).squeeze(1) # one sampled next token each mode = teacher[ctx].argmax(-1) # sequence-level KD with greedy decoding def train(kind, steps=400, rank=r): torch.manual_seed(1); m = Student(rank); opt = torch.optim.Adam(m.parameters(), lr=0.05) for _ in range(steps): out = m(ctx) if kind == "hard": loss = F.cross_entropy(out, hard) if kind == "mode": loss = F.cross_entropy(out, mode) if kind == "logit": loss = kd_loss(out, teacher[ctx], T=1.0) opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): # evaluate on ALL contexts at T = 1 p, lq = F.softmax(teacher, -1), F.log_softmax(m(torch.arange(V)), -1) kl = (p * (p.log() - lq)).sum(-1).mean().item() agree = (lq.argmax(-1) == teacher.argmax(-1)).float().mean().item() ent = -(lq.exp() * lq).sum(-1).mean().item() return round(kl, 3), round(agree, 3), round(ent, 3) H_t = -(F.softmax(teacher, -1) * F.log_softmax(teacher, -1)).sum(-1).mean().item() print("teacher entropy", round(H_t, 3)) # 1.244 for kind in ["hard", "mode", "logit"]: print(kind, train(kind)) # (KL to teacher, top-1 agreement, entropy) # hard (1.237, 0.917, 1.049) one sample per context: noisy # mode (4.109, 1.0, 0.004) copies the top token perfectly, but collapses to one-hot # logit (0.113, 0.917, 1.364) closest to the teacher; entropy above the teacher's (it covers) # 4) the capacity gap (59.12): the KL floor shrinks only when the student gets bigger for rank in [1, 3, 6, 12]: print(rank, train("logit", rank=rank)[0]) # 0.505, 0.113, 0.011, 0.0
The output shows the three regimes of this session in one place. Hard labels give the student a noisy view of the teacher. Training on the teacher’s top choice copies the decisions perfectly (agreement 1.0) but throws away all uncertainty, which is why sequence-level students become peaked. Logit distillation gets closest in KL, but a rank-3 student still keeps a floor of 0.113 nats that only a bigger student removes. (This tiny bigram setting is a stand-in: on real LLMs, sequence-level distillation often beats token-level, as Kim and Rush found, because generation quality is what is measured, not KL. [Likely])
Where this is used
| Setting | What is distilled | Method and details |
|---|---|---|
| Small open LLMs (Gemma 2 2B and 9B, Llama 3.2 1B and 3B) | A larger model of the same family | Token-level KD during pretraining (59.5), same tokenizer; often after pruning (session 115) |
| Small reasoning models (DeepSeek-R1-Distill) | Long correct reasoning traces of a strong reasoner | Rejection sampling and SFT (59.10); RL on top is optional |
| Chat assistants from API teachers (Alpaca-style data) | Teacher answers to many prompts | Black-box sequence-level KD (59.8); quality filtering matters (session 54) |
| Encoders for search and classification (DistilBERT) | A large encoder’s outputs | Soft targets plus extra losses on hidden states (session 33) |
| Draft models for speculative decoding | The target model’s next-token distribution | A draft that agrees with the target more often is accepted more often (session 128) [Likely] |
| Video classification and action recognition on device | A large video model’s soft labels per clip | Logit KD with temperature (59.2); soft labels carry the similarity between actions (“jogging” is close to “running”) [Likely] |
| Video retrieval and fingerprinting | A big embedding model’s similarity scores | The student matches the teacher’s similarity distribution over a batch, a KL over candidates (session 60) [Likely] |
| Video-LLMs: dense captions, key-moment detection, Video-RAG | A strong captioner’s or VLM’s outputs on many clips | Sequence-level KD as synthetic captions and timestamps, then SFT of a small Video-LLM (session 205) [Likely] |
Common mistakes
- Forgetting the \(T^2\) factor. When you raise \(T\), the soft-target gradient shrinks by \(1/T^2\) (59.4) and the hard-label term takes over without anyone noticing.
- Applying the temperature to only one side, or evaluating the student at the training temperature. Teacher and student both use \(T\) in the KD term; the student is deployed at \(T = 1\).
- Wrong KL argument order in code. PyTorch’s
F.kl_div(input, target)expects the student log-probabilities as input and the teacher probabilities as target, and computes \(\KL(\text{target}\,\|\,\text{input})\). Passing probabilities instead of log-probabilities, or swapping them, gives a wrong loss with no error. Usereduction="batchmean", not"mean", which also divides by the vocabulary size. [Certain] - Mismatched tokenizers or chat templates in logit KD. The two distributions must be over the same tokens at the same positions. With different tokenizers, use sequence-level distillation.
- Distilling wrong answers. Teacher outputs are not ground truth. Without a correctness filter, the student copies the teacher’s errors, often more confidently. Verify answers where possible.
- Choosing the biggest teacher by default. A teacher far above the student can give a worse student (capacity gap). Try a closer teacher, shorter traces, or a teacher assistant.
- Benchmark contamination. Teachers trained on the internet may reproduce test questions and answers. Decontaminate the distillation set against your evaluation sets.
- Expecting the student to beat the teacher on hard skills. Distillation copies; it rarely exceeds the teacher on the distilled skill. To go further you need RL or better data.
Interview questions
What is knowledge distillation, and why does it beat training the small model on labels alone?basic
Why do we use a temperature, and why multiply the KD loss by \(T^2\)?core
Show that high-temperature distillation is logit matching.advanced
Forward KL or reverse KL for distilling a generative LLM?advanced
Compare word-level and sequence-level distillation. When would you choose each?core
What is exposure bias, and how does on-policy distillation address it?core
How were the DeepSeek-R1-Distill models made, and what did the RL comparison show?core
Why can a larger teacher produce a worse student? What do you do about it?advanced
You must distill a 70B model into a 3B model for a chat product. Walk through the design.advanced
How would you distill a large video model into a small on-device action recognizer?core
Exercises
- Teacher logits are (2, 0, −2). Compute the teacher distribution at \(T = 1\) and \(T = 2\). How does the ratio between the first and the last probability change?
- In a KD loss with \(T = 3\), you forget the \(T^2\) factor. Roughly by how much is the distillation gradient weakened compared with the hard-label gradient?
- Teacher \(p = (0.5, 0.5)\), student \(q = (0.9, 0.1)\). Compute \(\KL(p\|q)\) and \(\KL(q\|p)\) in nats. Which direction punishes this student more, and why?
- In Figure 5’s tree, why does greedy decoding of the teacher not find the most likely sequence? What is the probability of each?
- A distillation set has 2 billion positions and you store the top 32 teacher tokens per position with 4-byte ids and 2-byte values. How much storage? What is wrong with simply dropping the other tokens?
- For a prompt where the teacher is correct 20% of the time, how many samples \(k\) do you need so the prompt is kept with probability at least 0.9?
Answerscheck
Key takeaways
- A teacher’s full distribution (soft targets) carries much more information than a one-hot label; the temperature (59.1) makes the small probabilities visible, and the \(T^2\) factor (59.2) keeps the soft-target gradient the right size.
- The KD gradient is \((q - p)/T\) (59.3); at high temperature distillation becomes matching the centered logits (59.4).
- Logit (token-level) KD (59.5) needs logits and the same tokenizer; it is used inside model families (Gemma 2, Llama 3.2). Forward KL covers modes, reverse KL picks them (59.6).
- Sequence-level KD (59.8) is SFT on teacher outputs: black-box, cross-tokenizer, makes students peaked so greedy decoding works (Kim and Rush: +4.2 BLEU greedy). On-policy distillation (59.9) fixes exposure bias.
- Reasoning distillation is rejection sampling plus SFT (59.10): DeepSeek-R1-Distill-Qwen-32B reached 72.6 on AIME 2024 against 47.0 for RL on the same base.
- The capacity gap (59.12): a student cannot copy a teacher outside its family; bigger teachers or longer traces can hurt small students. Use closer teachers, simpler targets or bigger students.
Key formulas
References for this session
- Hinton et al. (2015). Distilling the Knowledge in a Neural Network
- Kim & Rush (2016). Sequence-Level Knowledge Distillation
- DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Raschka, Build a Reasoning Model (From Scratch), chapter 8: Distilling reasoning models
- Gu et al. (2023). MiniLLM: Knowledge Distillation of Large Language Models
- Agarwal et al. (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD)
- Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size
- Meta AI (2024). Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
- Sanh et al. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- Mirzadeh et al. (2019). Improved Knowledge Distillation via Teacher Assistant
- Cho & Hariharan (2019). On the Efficacy of Knowledge Distillation
- Li et al. (2025). Small Models Struggle to Learn from Strong Reasoners