LLM & VLM courseBlog
59Track 4: Post-training and RL · session 7 of 25

Knowledge distillation

Knowledge distillation trains a small student model to behave like a large teacher model. The student learns from the teacher's full probability distribution (logit distillation) or from the text the teacher writes (sequence-level distillation), which is how small open models and small reasoning models are made today. We also see why a student sometimes cannot copy its teacher.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: BeginnerBefore this: Sessions 3, 12 and 54

What you will learn

  • Soft targets: why the teacher's full distribution carries more information than a one-hot label
  • Temperature in the softmax, the KD loss, its gradient (q - p)/T and why the loss is multiplied by T squared
  • Forward KL (mode-covering) versus reverse KL (mode-seeking) for a student that is too small
  • Sequence-level distillation (Kim and Rush): training on the teacher's outputs, and on-policy distillation against exposure bias
  • How reasoning traces are distilled into small models with rejection sampling and SFT, with the DeepSeek-R1 numbers
  • The capacity gap: why a bigger teacher or a longer trace can give a worse student, and what to do about it

Why this matters

In session 58 we made fine-tuning cheap: the big model stays frozen in 4 bits and we train small adapters. But the model we serve is still big. Every user request pays for all of its parameters. Knowledge distillation attacks that cost directly: a large, strong model (the teacher) trains a small model (the student) so that the student behaves like the teacher at a fraction of the price.

The student is trained not only on the “correct answer” but on what the teacher says: its full probability distribution over the next token (logit distillation), or the text it writes (sequence-level distillation). This extra signal is much richer than a single correct token, and that is why a distilled student usually beats the same small model trained from scratch on the same data.

Distillation is everywhere in real systems:

  • Gemma 2 trained its 2B and 9B models “with knowledge distillation instead of next token prediction”, on 2 and 8 trillion tokens, which is more than 50 times the compute-optimal number of tokens. [Certain]
  • Llama 3.2 1B and 3B were pruned from Llama 3.1 8B, and logits from Llama 3.1 8B and 70B were used as targets during pretraining. [Certain]
  • DeepSeek-R1-Distill: six small models (1.5B to 70B) were made by plain SFT on about 800k samples written by DeepSeek-R1. The 7B student reaches 55.5% on AIME 2024, more than QwQ-32B-Preview (50.0%). [Certain]
  • DistilBERT is 40% smaller than BERT, about 60% faster, and keeps about 97% of its language-understanding score. [Certain]

This session builds the idea from the original soft-target trick of Hinton et al. (2015) to how reasoning models are distilled today, and ends with the main limit: a student cannot copy a teacher that is too far above it. The inference track goes deeper into white-box and black-box methods (session 119, session 120).

Soft targets and temperature

A normal training label is one-hot: probability 1 on the correct token, 0 on all others. It says what is right, but nothing about how wrong the other options are. A trained teacher knows more. After “The cat sat on the”, it gives high probability to “mat”, some to “floor” and “sofa”, and almost none to “banana”. These small numbers tell the student which words are similar and which mistakes are reasonable. Hinton et al. called this dark knowledge and the teacher’s distribution soft targets. [Certain]

The problem: a confident teacher puts most mass on one token, and the useful information hides in tiny probabilities (0.001 versus 0.000001) that barely affect a cross-entropy loss. The fix is to soften the softmax with a temperature \(T\):

\[ p_i(T) = \frac{\exp(z_i / T)}{\sum_{j=1}^{V} \exp(z_j / T)} \tag{59.1} \]
Read the symbols
\(z_i\)
the logit (raw score before the softmax) for token \(i\)
\(V\)
the vocabulary size, the number of possible next tokens
\(T\)
the temperature: \(T = 1\) is the normal softmax, \(T \gt 1\) makes the distribution flatter, \(T \lt 1\) makes it sharper
\(p_i(T)\)
the probability of token \(i\) at temperature \(T\)

Dividing by \(T\) shrinks the gaps between logits. A gap of 3 between “roof” (2.0) and “banana” (−1.0) in the example below means a probability ratio of \(e^{3} \approx 20\) at \(T = 1\), but only \(e^{3/4} \approx 2.1\) at \(T = 4\). The order of the tokens never changes; only the contrast does. Both teacher and student use the same \(T\) during distillation. After training, the student is used at \(T = 1\). [Certain]

Worked example: the same teacher at three temperatures

Teacher logits for “The cat sat on the ___”: mat 6.0, floor 4.5, sofa 4.0, bed 3.5, roof 2.0, banana −1.0. With formula (59.1):

\(T = 1\): (0.685, 0.153, 0.093, 0.056, 0.013, 0.001). Entropy 0.988 nats; the five non-top tokens share 31.5% of the mass; sofa is 148 times more likely than banana.

\(T = 2\): (0.436, 0.206, 0.160, 0.125, 0.059, 0.013). Entropy 1.465 nats; non-top mass 56.4%; sofa/banana ratio 12.2.

\(T = 4\): (0.297, 0.204, 0.180, 0.159, 0.109, 0.052). Entropy 1.680 nats; non-top mass 70.3%; ratio 3.5. The ranking is the same at every \(T\), but at \(T = 4\) the student can actually see that “roof” is a better mistake than “banana”.

one-hot label (“mat”)teacher at temperature \(T\)student at temperature \(T\)
Figure 1. Six candidate next tokens for “The cat sat on the ___”. Teacher logits (6.0, 4.5, 4.0, 3.5, 2.0, −1.0) and student logits (5.0, 4.8, 2.0, 3.0, 1.0, 0.5) are fixed; only the temperature changes. The readout gives the teacher entropy, the mass outside the top token, the KL divergence at this temperature, and the largest gradient on a student logit, with and without the \(T^2\) factor of formula (59.2).
How to read Figure 1

Press “Sweep T” to watch the temperature rise from 0.5 to 10 while every bar and number is recomputed with formula (59.1). At \(T = 1\) the teacher bar for “mat” is tall and “banana” is invisible: the soft target is close to the one-hot label. Move \(T\) to 4: the teacher spreads out and the small tokens become visible. Now watch the readout. The raw KL shrinks as \(T\) grows (both distributions get flat, so they look alike), and the largest gradient shrinks roughly like \(1/T^2\). After multiplying by \(T^2\), the gradient stays in the same range (about 0.25 at \(T = 1\) and \(T = 2\), 0.21 at \(T = 4\)). That is why the loss is scaled by \(T^2\). Very low \(T\) (below 1) makes both distributions sharper than normal and the soft targets lose their extra information.

Logit distillation: the KL loss

In logit distillation (also called word-level or token-level distillation, or white-box distillation, because we must see the teacher’s logits) the student matches the teacher’s softened distribution. Hinton et al. combine two terms: a normal cross-entropy with the true label, and a KL divergence to the teacher at temperature \(T\) (session 3 introduced KL):

\[ \mathcal{L} = (1 - \alpha)\, \mathrm{CE}\!\left(y,\ q(1)\right) \;+\; \alpha\, T^2\, \KL\!\left(p(T)\ \|\ q(T)\right) \tag{59.2} \]
Read the symbols
\(p(T)\)
the teacher distribution at temperature \(T\), formula (59.1) with the teacher logits \(v_i\)
\(q(T)\)
the student distribution at temperature \(T\), with the student logits \(z_i\); \(q(1)\) is the normal student softmax
\(y\)
the true (hard) label, if the data has one
\(\alpha\)
the weight of the distillation term, between 0 and 1; \(\alpha = 1\) means pure distillation with no labels
\(\KL(p \| q)\)
\(\sum_i p_i \log (p_i / q_i)\): how many extra nats the student pays for using \(q\) where the teacher uses \(p\). It equals the cross-entropy \(-\sum_i p_i \log q_i\) minus the teacher entropy, which does not depend on the student, so both give the same gradient
\(T^2\)
a scale factor that keeps the soft-target gradient the same size when \(T\) changes (see below)

Where the \(T^2\) comes from

The gradient of the KL term with respect to one student logit is simple:

\[ \frac{\partial}{\partial z_i} \KL\!\left(p(T)\,\|\,q(T)\right) = \frac{1}{T}\left(q_i(T) - p_i(T)\right) \tag{59.3} \]

“Student probability minus teacher probability”, as for any softmax with cross-entropy, times \(1/T\) from the chain rule through \(z_i/T\). When \(T\) is large compared with the logits, \(e^{x} \approx 1 + x\), and with logits that have zero mean over the vocabulary Hinton et al. show:

\[ \frac{\partial}{\partial z_i} \KL \;\approx\; \frac{1}{V T^2}\,\left(z_i - v_i\right) \tag{59.4} \]
Read the symbols
\(z_i\), \(v_i\)
the student and teacher logits for token \(i\), each shifted to have mean 0 over the vocabulary
\(V\)
the vocabulary size (\(N\) in the original paper)

Two lessons. First, the gradient shrinks like \(1/T^2\), so without the \(T^2\) factor in (59.2) a higher temperature would silently weaken the distillation term against the hard-label term. [Certain] Second, at high temperature distillation becomes logit matching: the loss behaves like \(\frac{1}{2}(z_i - v_i)^2\), so the student copies the teacher’s logits, including the very negative ones. At lower temperatures the student cares less about matching very negative logits. Hinton et al. found that intermediate temperatures often work best, which suggests that the most negative logits are noisy and are better ignored. [Certain]

Worked example: the loss and its gradient

Use the teacher of Figure 1 and student logits (5.0, 4.8, 2.0, 3.0, 1.0, 0.5). At \(T = 1\) the student gives “mat” 0.492, so the hard-label loss is \(-\log 0.492 = 0.710\). The KL to the teacher is 0.196 at \(T = 1\), 0.076 at \(T = 2\), and 0.026 at \(T = 4\). Multiplied by \(T^2\): 0.196, 0.303 and 0.415.

The largest gradient (59.3) is on “floor” at \(T = 1\): \(q - p = 0.403 - 0.153 = 0.250\). At \(T = 2\) the largest gradient is 0.062, at \(T = 4\) only 0.013. Times \(T^2\): 0.250, 0.249 and 0.212, the same size. At \(T = 20\), the exact gradient times \(T^2\) for “sofa” is −0.251 and the approximation (59.4) gives −0.258: the logit-matching limit is close.

The KD step in motion

Put (59.2) and (59.3) together. The hard-label term gives the usual softmax gradient \(q_i(1) - y_i\), and the KD term gives \(T^2 \cdot \frac{1}{T}(q_i(T) - p_i(T))\). So one gradient step on a single example moves every student logit by

\[ \frac{\partial \mathcal{L}}{\partial z_i} = (1-\alpha)\big(q_i(1) - y_i\big) \;+\; \alpha\, T\, \big(q_i(T) - p_i(T)\big), \qquad z_i \leftarrow z_i - \eta\, \frac{\partial \mathcal{L}}{\partial z_i} \]

Here \(y_i\) is 1 for the labeled token and 0 for the others, and \(\eta\) is the learning rate. Figure 2 runs exactly this loop on the six-token example: the teacher is frozen, and the student’s six logits are the only parameters. Each step has four lines: forward pass, loss, gradient, update. The box next to the pseudo-code shows each formula with the current numbers in it.

teacher \(p(T)\), logits \(v\)student \(q(T)\), logits \(z\)logit goes uplogit goes downdashed curve: same run with the other \(T^2\) setting
  1. \(p \leftarrow \softmax(v / T)\) // teacher, frozen, (59.1)
  2. \(q_T \leftarrow \softmax(z / T)\),  \(q_1 \leftarrow \softmax(z)\) // student
  3. \(\mathcal{L} \leftarrow (1 - \alpha)\,(-\ln q_1[y]) + \alpha\, T^2 \KL(p \,\|\, q_T)\) // (59.2)
  4. \(g \leftarrow (1 - \alpha)\,(q_1 - e_y) + \alpha\, T\,(q_T - p)\) // (59.3)
  5. \(z \leftarrow z - \eta\, g\); go back to line 2
Figure 2. Logit distillation of one example, trained in your browser with plain gradient descent for 40 steps. Teacher logits \(v = (6.0, 4.5, 4.0, 3.5, 2.0, -1.0)\), starting student logits \(z = (5.0, 4.8, 2.0, 3.0, 1.0, 0.5)\), hard label “mat”. Under each token: the teacher logit, the student logit, and (on lines 4 and 5) the gradient \(g_i\). The arrows show the direction each student logit will move. Right: KL\((p \,\|\, q_T)\) after each step; the dashed grey curve is the same run with the \(T^2\) checkbox flipped.
How to read Figure 2

Press Step a few times. Line 1 computes the teacher’s soft targets once. In step 1 at the default setting (\(T = 4\), \(\alpha = 0.5\), \(\eta = 1\)) the loss is 0.5625 and the largest push is on “mat”: \(g = 0.5 \cdot (0.492 - 1) + 0.5 \cdot 4 \cdot (0.269 - 0.297) = -0.310\), so its logit goes up from 5.0 to 5.310, while “floor” (\(g = 0.305\)) goes down. After 40 steps the KL at \(T = 4\) falls from 0.0260 to 0.0049. Now uncheck the \(T^2\) box: the KD gradient is 16 times smaller, the hard label wins, the student puts 0.953 on “mat” (the teacher has 0.685), and the KL after 40 steps is 0.0425. Set \(\alpha = 1\) (pure distillation): with \(T^2\) the student copies the teacher almost exactly (KL 0.0000, “mat” 0.686); without it, the KL is still 0.0130 after 40 steps. At \(T = 1\) the checkbox changes nothing, because \(T^2 = 1\).

Why soft targets help: a tiny student trained live

Hinton et al. showed how much information soft targets carry with a striking MNIST test. They removed every example of the digit 3 from the transfer set (the data used for distillation). The distilled student had never seen a 3. Still, after one bias correction it got 98.6% of the test 3s right, because the teacher’s soft targets on other digits (a 3-like 8, a 3-like 5) had told it what a 3 looks like. [Certain] Without the correction it made 206 test errors, 133 of them on the 1,010 threes, because the learned bias for class 3 was too low. [Certain]

Figure 3 shows the same effect at a size you can watch. The teacher is a fixed linear classifier over points \(x \in [-1, 1]^2\) with three classes A, B and C: logits \(v(x) = W x + c\). Two students with the same form \(z(x) = A x + b\) start from zero and see the same few transfer points. One is trained on hard labels (the teacher’s top class, one-hot), the other on the teacher’s soft targets with the loss \(T^2\,\KL\) of (59.2) with \(\alpha = 1\). The student’s form can represent the teacher exactly, so there is no capacity gap; the only difference is how much each training point tells the student.

class Aclass Bclass Chard-label studentsoft-target studentdashed lines: teacher borders
  1. teacher logits for every transfer point: \(v_n = W x_n + c\) // frozen
  2. student logits: \(z_n = A x_n + b\) // both start at 0
  3. hard: \(\ell = -\ln \softmax(z_n)[y_n]\);  soft: \(\ell = T^2 \KL(p_n \,\|\, q_n)\) // (59.2)
  4. \(\partial \ell / \partial z_n\): hard \((q_n - e_y)/N\);  soft \(T\,(q_n - p_n)/N\) // (59.3)
  5. \(A \leftarrow A - \eta \sum_n \partial z_n\, x_n^{\top}\),  \(b \leftarrow b - \eta \sum_n \partial z_n\)
  6. evaluate both students on a \(20 \times 20\) grid: agreement with the teacher’s class
Figure 3. Hard labels versus soft targets, trained in your browser. Teacher: \(W\) has rows (3, 1), (−2.5, 2.5), (−0.5, −3.5) and \(c = (0, 0.3, -0.3)\). The transfer points are drawn with a seeded random generator. Both students use full-batch gradient descent with \(\eta = 1\) for 150 steps. The colored cells are each student’s predicted class on a 20 × 20 grid; strong cells disagree with the teacher. A hard-label point is drawn in its class color; a soft-target point is a small pie chart of \(p(T)\). Point 1 (dashed ring) is the one shown in the formula box.
How to read Figure 3

Press Play. With the default 6 points and \(T = 2\), the soft-target student reaches 95% agreement with the teacher at step 42 and ends at 98.8%; the hard-label student ends at 83.0%, because six one-hot labels leave the borders free to sit anywhere between the points. Watch the formula box: point 1 has hard target (0, 0, 1) but soft target with mass on all three classes, so it pushes on every logit. Now check “Leave class C out”: no transfer point is labeled C. The hard-label student never predicts C (67.0%), but the soft-target student still finds the C region (99.8%), because the soft targets of A and B points near the border say how close C is. This is Hinton’s “mythical digit” in two dimensions. Move \(N\) to 20: the hard student catches up (98.8% against 99.0%). Soft targets help most when data is scarce.

Distilling a language model, token by token

For an LLM, the “classification” is the next token, and there is one at every position. Token-level distillation applies (59.2) at every position of every training sequence \(x\):

\[ \mathcal{L}_{\text{tok-KD}} = \frac{T^2}{N} \sum_{t=1}^{N} \KL\!\Big( p\big(\cdot \mid x_{\lt t};\, T\big) \;\Big\|\; q_\theta\big(\cdot \mid x_{\lt t};\, T\big) \Big) \tag{59.5} \]
Read the symbols
\(x_{\lt t}\)
the tokens before position \(t\), taken from the training data (the same prefix goes into teacher and student)
\(p(\cdot \mid x_{\lt t};T)\)
the teacher’s full next-token distribution at temperature \(T\): a vector of \(V\) numbers per position
\(q_\theta\)
the student with parameters \(\theta\), the only thing that is trained
\(N\)
the number of positions that get a loss (with an SFT loss mask, only the assistant tokens; see session 54)

Gemma 2 used exactly this kind of objective during pretraining: minimize \(\sum_x -P_T(x \mid x_c) \log P_S(x \mid x_c)\), the cross-entropy between teacher and student next-token distributions. [Certain] Three practical points decide whether it is possible:

  • Same tokenizer. The KL compares two vectors over the same vocabulary. Teacher and student must share the tokenizer, which is why logit distillation is usually done inside one model family (Gemma 27B to 9B, Llama 70B to 3B). Cross-tokenizer methods exist but are harder. [Likely]
  • Access to logits. You need the teacher’s weights or an API that returns full log-probabilities. A chat API that returns only text allows only sequence-level distillation (next section).
  • Cost. Either run the teacher forward at every training step (online), or precompute its outputs once and store them (offline). Storing full distributions is impossible at scale, so offline pipelines keep only the top-\(k\) tokens per position. [Likely]
Worked example: why nobody stores full teacher logits

Take a vocabulary of 128,256 tokens (Llama 3) and a distillation set of 1 billion tokens. Full logits in bf16 (2 bytes) need \(10^9 \times 128{,}256 \times 2 \approx 256.5\) TB. Keeping only the top 20 tokens per position, with a 4-byte token id and a 2-byte value each, needs \(10^9 \times 20 \times 6 = 120\) GB: over 2,000 times less. The cost is a small bias: the student never sees the tail of the distribution, and the kept probabilities must be renormalized or the missing mass treated as one extra bucket.

Forward KL or reverse KL?

\(\KL(p \| q)\) is not symmetric. The two directions push a limited student in different ways:

\[ \underbrace{\KL(p\,\|\,q_\theta) = \sum_i p_i \log \frac{p_i}{q_{\theta,i}}}_{\text{forward: average over the teacher}} \qquad \underbrace{\KL(q_\theta\,\|\,p) = \sum_i q_{\theta,i} \log \frac{q_{\theta,i}}{p_i}}_{\text{reverse: average over the student}} \tag{59.6} \]
  • Forward KL is mode-covering. Wherever the teacher has mass (\(p_i \gt 0\)) and the student has almost none, \(\log(p_i/q_i)\) explodes. So the student spreads out to cover everything the teacher might say, even if it must put mass between the teacher’s modes. This is what (59.2) and (59.5) use.
  • Reverse KL is mode-seeking. The average is over the student’s own outputs. Putting mass where the teacher has almost none is punished hard; ignoring a teacher mode costs nothing. So the student picks one mode and fits it well.

For a small generative student, covering everything can mean generating text the teacher would never write. MiniLLM (Gu et al.) therefore replaces forward KL with reverse KL, to stop the student from “overestimating the low-probability regions of the teacher distribution”, and reports gains from 120M to 13B parameters. [Certain] Reverse KL is harder to optimize because the average is over the student’s own samples, so it needs sampling and policy-gradient-like methods. [Certain] Which direction is better depends on the task and on the decoding method; the GKD paper treats the choice as a hyper-parameter. [Likely]

teacher \(p\) (two modes)student fitted with forward KLstudent fitted with reverse KL
Figure 4. A student that is too simple for its teacher. The teacher is a mixture of two Gaussians with standard deviation 1 at \(\pm d/2\). The student can only be one Gaussian \(\mathcal{N}(\mu, \sigma^2)\). The best student under forward KL is found exactly (it matches the teacher’s mean and variance); under reverse KL it is found by a grid search over \(\mu\) and \(\sigma\) on a grid of 401 points from −10 to 10.
How to read Figure 4

At distance 0 the teacher has one mode and both students agree. At the default distance 6 the teacher puts only 0.6% of its mass in the gap \(|x| \lt 0.5\). The forward-KL student is wide (\(\sigma = 3.10\)) and sits between the modes, with 11.9% of its mass in the gap. The reverse-KL student sits on the heavier mode (\(\mu = -2.99\), \(\sigma = 1.02\)) and ignores the other one. Now lower the distance slowly: below about 5 (with weight 0.6) the modes overlap enough that covering both is cheaper even for reverse KL, and the two students look alike. For a language model, “between the modes” is a mix of two good answers that is itself not a good answer. Set the weight to 0.5 and the distance to 7: the reverse-KL student still picks one side, and which side depends on tiny numerical differences.

Sequence-level distillation: train on the teacher’s outputs

Token-level distillation teaches the student, position by position, to predict like the teacher on given prefixes from the data. But a generative model is judged on whole sequences that it writes itself. Kim and Rush (2016) asked: why not distill at the level of sequences? [Certain]

Write the student as \(q_\theta(y \mid x)\) over whole output sequences \(y\) for an input \(x\). Kim and Rush compare two losses. Word-level KD (Word-KD) is (59.5) on the gold prefix:

\[ \mathcal{L}_{\text{Word-KD}} = -\sum_{t=1}^{|y|} \sum_{k=1}^{V} p\big(k \mid x, y_{\lt t}\big)\, \log q_\theta\big(k \mid x, y_{\lt t}\big) \tag{59.7} \]

Sequence-level KD (Seq-KD) matches the teacher’s distribution over all sequences, which is a sum over exponentially many \(y\). They approximate the teacher distribution by its mode \(\hat{y}\), found with beam search:

\[ \mathcal{L}_{\text{Seq-KD}} = -\sum_{y} p(y \mid x)\, \log q_\theta(y \mid x) \;\approx\; -\log q_\theta(\hat{y} \mid x), \qquad \hat{y} = \text{beam search}\big(p(\cdot \mid x)\big) \tag{59.8} \]
Read the symbols
\(y\), \(y_{\lt t}\)
an output sequence, and its first \(t-1\) tokens
\(p(y \mid x)\)
the teacher’s probability of the whole sequence: the product of its next-token probabilities
\(\hat{y}\)
the teacher’s best output found by beam search: a proxy for the mode of \(p(y \mid x)\)

So Seq-KD is very simple in practice: (1) train the teacher, (2) run beam search with it over the training inputs, (3) train the student with ordinary cross-entropy on these outputs instead of the original references. [Certain] This is exactly “SFT on teacher outputs”, and it needs no logits at all: it works with a different tokenizer and with a teacher behind an API. Today it is usually called black-box distillation or simply training on synthetic data. They also defined Seq-Inter: train on the sequence in the teacher’s beam that is closest (by BLEU) to the human reference, a mix of “what the teacher would say” and “what is correct”. [Certain]

Why does the mode work when it is a single sequence out of billions? Kim and Rush measured that the teacher’s beam output already carries a lot of its mass: on average 1.3% of the whole distribution for English→German. [Certain] And training on it makes the student peaked: after Seq-KD the student puts 16.9% of its mass on its own best output, against 0.9% for the baseline student. [Certain] A peaked student does not need beam search: greedy decoding finds almost the same answer.

Worked example: the numbers of Kim and Rush (WMT 2014 English→German)

Teacher: 4-layer LSTM with 1000 units, 221M parameters, BLEU 17.7 (greedy) and 19.5 (beam 5). Student: 2 × 500, 84M parameters. [Certain]

Student 2 × 500 trained withBLEU, greedyBLEU, beam 5Mass on own best output
Original references (baseline)14.717.60.9%
Word-KD (59.7)15.417.71.0%
Seq-KD (59.8)18.919.016.9%
Seq-KD, then Seq-Inter18.919.37.6%

Seq-KD adds 4.2 BLEU with greedy decoding (18.9 − 14.7) and 1.4 with beam search. The greedy student (18.9) is better than the greedy teacher (17.7). The student with greedy decoding ran at 1051.3 words per second on a GPU against 101.9 for the teacher with beam search, about 10 times faster. After pruning 80% of its weights it had 17M parameters, 13 times fewer than the teacher, at BLEU 19.1 versus 19.5. [Certain]

On-policy distillation: let the student write

Both Word-KD and Seq-KD train the student on prefixes it did not write. At inference, the student conditions on its own earlier tokens, and once it makes a small mistake it is in a state it never saw in training. This train–test mismatch is called exposure bias. On-policy distillation fixes it: sample sequences from the student, and ask the teacher for its next-token distribution at every prefix of those samples. GKD (Agarwal et al.) trains on such student-generated sequences with teacher feedback, allows other divergences than forward KL, and can be combined with RL fine-tuning. [Certain] Gemma 2’s post-training used it too: “distillation from the teacher on the student’s distribution”. [Certain]

\[ \mathcal{L}_{\text{on-policy}}(\theta) = \E_{x \sim \mathcal{D}}\ \E_{y \sim q_\theta(\cdot \mid x)} \Big[\ \sum_{t=1}^{|y|} D\Big(p(\cdot \mid x, y_{\lt t})\ \Big\|\ q_\theta(\cdot \mid x, y_{\lt t})\Big) \Big] \tag{59.9} \]
Read the symbols
\(y \sim q_\theta\)
the sequences are sampled from the student, not taken from the data or the teacher
\(D\)
a divergence per position: forward KL, reverse KL, or a mix (GKD uses a generalized Jensen–Shannon divergence)

In practice the gradient is not taken through the sampling: the sampled tokens are treated as fixed data, and only the per-token divergence is differentiated. [Likely] The cost is a teacher forward pass on fresh student samples at every step, so it is more expensive than offline Seq-KD.

trained pathposition gets the full teacher distributionnot used
Figure 5. A teacher that translates the German “Danke” into two tokens. Edge labels are the teacher’s next-token probabilities; leaf numbers are whole-sequence probabilities (their product). The human reference is “Thanks a lot”. The on-policy student has its own distribution (first token: Thank 0.30, Thanks 0.30, Cheers 0.40) and samples are drawn with a seeded random generator.
How to read Figure 5

Hard labels train one path, the reference, with one-hot targets. Word-KD walks the same reference path but gives each of its two positions the teacher’s full distribution (blue). Seq-KD greedy follows the teacher’s best first token, “Thank”, and gets “Thank you” (0.180), which is not the teacher’s mode: beam search keeps 3 candidates and finds “Thanks !” (0.280). On-policy uses the student’s own samples as prefixes, and the teacher grades every position: press “Sample again” to see that the student is corrected on paths such as “Cheers mate” that no reference would contain.

Distilling reasoning traces into small models

Reasoning models such as DeepSeek-R1 write a long chain of thought (a reasoning trace: the intermediate steps) before the final answer (session 72). They get this skill from large-scale reinforcement learning with verifiable rewards (session 70), which is expensive and works best on strong base models. Distillation offers a shortcut: let the strong reasoner write traces, keep the correct ones, and run SFT on a small model. This is sequence-level distillation (59.8) with sampling instead of beam search, plus a filter.

DeepSeek-R1 did exactly this. [Certain]

  • Data. About 600k reasoning samples, made by sampling several responses per prompt from an R1 checkpoint and keeping only correct ones (rule-based checks, or DeepSeek-V3 as a judge), after removing traces with mixed languages, long paragraphs and code blocks; plus about 200k non-reasoning samples (writing, factual QA, translation). About 800k in total.
  • Students. Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-14B, Qwen2.5-32B, Llama-3.1-8B and Llama-3.3-70B-Instruct, fine-tuned on the 800k samples.
  • Only SFT. No RL stage for the students, “even though incorporating RL could substantially boost model performance”.
\[ \mathcal{D}_{\text{distill}} = \Big\{ (x, r, a) \;:\; (r, a) \sim p(\cdot \mid x),\ \ \text{verify}(x, a) = 1 \Big\}, \qquad \mathcal{L}(\theta) = -\!\!\sum_{(x, r, a) \in \mathcal{D}_{\text{distill}}}\!\! \log q_\theta(r, a \mid x) \tag{59.10} \]
Read the symbols
\(x\)
a prompt, for example a math problem with a known answer
\(r\), \(a\)
the teacher’s reasoning trace and its final answer, sampled together
\(\text{verify}\)
a checker: compare with the reference answer, run unit tests for code, or ask a judge model; this is rejection sampling: wrong samples are thrown away
\(q_\theta(r, a \mid x)\)
the student’s probability of the whole trace and answer; the loss is the SFT loss, with the prompt masked
Model (AIME 2024 and MATH-500, pass@1)AIME 2024MATH-500GPQA Diamond
DeepSeek-R1-Distill-Qwen-1.5B28.983.933.8
DeepSeek-R1-Distill-Qwen-7B55.592.849.1
DeepSeek-R1-Distill-Qwen-14B69.793.959.1
DeepSeek-R1-Distill-Qwen-32B72.694.362.1
DeepSeek-R1-Distill-Llama-8B50.489.149.0
DeepSeek-R1-Distill-Llama-70B70.094.565.2
QwQ-32B-Preview (comparison)50.090.654.5
GPT-4o-0513 (comparison)9.374.649.9

Numbers from Table 5 of the DeepSeek-R1 paper. [Certain] The most important experiment is the comparison with RL on the same base. DeepSeek ran large-scale RL (over 10K steps on math, code and STEM data) on Qwen-32B-Base and got DeepSeek-R1-Zero-Qwen-32B: 47.0 on AIME 2024, about the same as QwQ-32B-Preview (50.0). The distilled Qwen-32B gets 72.6, and is better on every benchmark they report. [Certain] Their conclusion: distilling a strong model into a small one is cheap and effective, while RL on the small model needs enormous compute and may still not reach distillation; but going beyond the teacher still needs stronger base models and larger-scale RL. [Certain]

Worked example: how many samples per prompt?

Rejection sampling keeps a prompt only if at least one of \(k\) teacher samples is correct. With an independent success chance \(s\) per sample:

\[ P(\text{prompt kept}) = 1 - (1 - s)^k, \qquad \E[\text{correct traces per prompt}] = k\,s \tag{59.11} \]

For an easy prompt with \(s = 0.6\) and \(k = 4\): kept with probability \(1 - 0.4^4 = 0.974\), and 2.4 correct traces on average. For a hard prompt with \(s = 0.3\): kept with probability \(1 - 0.7^4 = 0.760\), 1.2 traces. So the filter quietly removes hard prompts and gives easy prompts more traces. Many pipelines keep at most one or two traces per prompt, or sample more for hard prompts, to fight this bias. [Likely]

Practical details that matter for reasoning distillation:

  • Very long targets. Traces of thousands to tens of thousands of tokens need long-context SFT, packing with document masks, and answers that are never cut (session 54). A cut trace loses its final answer.
  • Format. Traces are often wrapped in special tags (for example a think block, then a summary). The student must learn the same template that inference will use.
  • Correct is not the same as clean. A trace can reach the right answer with wrong steps. Answer checks do not catch this; judges and step checks catch some of it. [Likely]
  • Licenses and terms of use. Some providers forbid using their outputs to train competing models; check before distilling from an API. [Likely]

When the student cannot copy the teacher: the capacity gap

If teachers help, should we always use the biggest teacher? No. Several studies found that a stronger teacher can give a worse student:

  • Mirzadeh et al. (2019) found that student performance degrades when the size gap between teacher and student is large, and proposed a teacher assistant: distill from the big teacher into a medium model, then from the medium model into the student. [Certain]
  • Cho and Hariharan (2019): “larger models do not often make better teachers”. They explain it with mismatched capacity, find that a sequence of distillation steps does not fix it in their setting, and show that a teacher stopped early in its training (less sharp, easier to copy) helps. [Certain]
  • Li et al. (2025), Small Models Struggle to Learn from Strong Reasoners: students of 3B parameters or less do not consistently gain from long chains of thought or from larger teachers; they learn better from shorter, simpler traces. Their Mix Distillation mixes long and short traces, or traces from larger and smaller teachers. [Certain]
  • Hinton et al.: with a very small MNIST student (30 units per layer), temperatures 2.5 to 4 worked clearly better than higher or lower ones; with 300 or more units, all temperatures above 8 gave similar results. [Certain] A small student cannot use all the fine detail that a high temperature reveals.

The common reason: the student’s family of functions cannot contain the teacher. The best the student can do is the closest point in its own family, and the distance to that point is a floor that no amount of data removes:

\[ \min_{\theta}\ \E_{x}\Big[\KL\big(p(\cdot \mid x)\ \|\ q_\theta(\cdot \mid x)\big)\Big] \;=\; \varepsilon_{\text{gap}} \;\ge\; 0 \tag{59.12} \]
Read the symbols
\(\min_\theta\)
the best student in its architecture, after perfect optimization on unlimited data
\(\varepsilon_{\text{gap}}\)
the capacity gap: 0 only if the student family can represent the teacher

When \(\varepsilon_{\text{gap}}\) is large, forward KL makes the student spread mass between teacher modes (Figure 4), which for text means bland or mixed-up outputs. Long reasoning traces make it worse: the student must reproduce a long, precise chain, and one wrong step breaks it. The cures all reduce what the student has to copy: a closer teacher (teacher assistant, early-stopped teacher), simpler targets (shorter traces, Mix Distillation), a moderate temperature, mode-seeking or on-policy objectives (MiniLLM, GKD), or simply a bigger student.

teacherstudent (forward KL)student (reverse KL)
Figure 6. Real training in your browser. The teacher is a seeded bigram language model over 12 tokens with full-rank logits (12 × 12, entries Gaussian with standard deviation 2). The student’s logits are \(EW + b\) with \(E\) of size \(12 \times r\): a rank-\(r\) model, so small \(r\) means low capacity. Each student is trained for 400 full-batch Adam steps (learning rate 0.05) on all 12 contexts. Right: the KL that remains after training, for every rank and both objectives.
How to read Figure 6

At rank 12 the student can represent the teacher, and the remaining forward KL is almost 0. Lower the rank: the KL floor (59.12) rises, the capacity gap. Data is not the problem, since the student sees the exact teacher distribution for every context. Compare the objectives at rank 2 or 3: the forward-KL student has higher entropy than the teacher (it covers), the reverse-KL student has lower entropy (it picks modes), and each one is better on the KL it was trained on. Move the context slider to see single distributions.

Putting it together

Algorithm 1: logit distillation of a language model (online teacher)
  1. input: teacher \(p\) (frozen), student \(q_\theta\) with the same tokenizer, data \(\mathcal{D}\), temperature \(T\), weight \(\alpha\)
  2. for each batch of token sequences \(x\) with loss mask \(m\):
  3.   with no gradient: \(v \leftarrow\) teacher logits on \(x\) // shape (B, L, V)
  4.   \(z \leftarrow\) student logits on \(x\)
  5.   \(\ell_{\text{KD}} \leftarrow T^2 \cdot \KL(\softmax(v/T)\,\|\,\softmax(z/T))\) per position // (59.5)
  6.   \(\ell_{\text{CE}} \leftarrow\) next-token cross-entropy of \(z\) with the data tokens // optional hard labels
  7.   loss \(\leftarrow \sum m \cdot \big((1-\alpha)\,\ell_{\text{CE}} + \alpha\,\ell_{\text{KD}}\big) / \sum m\) // (59.2)
  8.   loss.backward(); optimizer step on \(\theta\) only
  9. offline variant: precompute top-\(k\) teacher tokens and log-probs per position once; renormalize over those \(k\)
Algorithm 2: distilling reasoning traces (sequence-level, black-box)
  1. input: prompts \(\{x\}\) with checkable answers, teacher \(p\) (API or weights), samples per prompt \(k\)
  2. for each prompt \(x\):
  3.   sample \(k\) outputs \((r_j, a_j) \sim p(\cdot \mid x)\) at the teacher’s recommended temperature
  4.   keep those with verify\((x, a_j) = 1\) and a clean format // rejection sampling, (59.10)
  5.   keep at most \(c\) traces per prompt // limit the easy-prompt bias, (59.11)
  6. add non-reasoning chats so the student stays a general assistant
  7. decontaminate against evaluation sets; deduplicate
  8. SFT the student on the result: prompt masked, long rows, no cut traces // session 54
  9. optional: on-policy distillation (59.9) or RL on the student afterwards

The PyTorch code below runs on a CPU in a few seconds. It checks the gradient formula (59.3) and the \(T^2\) effect on the worked example, then trains a rank-3 student to copy a full-rank bigram teacher in three ways: hard labels sampled from the teacher, the teacher’s greedy outputs (sequence-level distillation with one-token sequences), and the full teacher distribution (logit distillation). Last, it shows the capacity gap (59.12) by changing the student rank.

import torch, torch.nn.functional as F
torch.manual_seed(0)

# 1) the distillation loss, formula (59.2): hard-label CE + T^2 * KL at temperature T
def kd_loss(s_logits, t_logits, labels=None, T=2.0, alpha=0.5):
    log_q = F.log_softmax(s_logits / T, dim=-1)          # student at temperature T
    p = F.softmax(t_logits / T, dim=-1)                  # teacher soft targets
    kl = F.kl_div(log_q, p, reduction="batchmean")       # KL(p || q), averaged over rows
    if labels is None:
        return kl * T * T
    ce = F.cross_entropy(s_logits, labels)               # hard labels at T = 1
    return (1 - alpha) * ce + alpha * T * T * kl

# 2) check the gradient, formula (59.3): dKL/dz = (q - p) / T  (per row)
z_t = torch.tensor([[6.0, 4.5, 4.0, 3.5, 2.0, -1.0]])   # teacher logits (worked example)
z_s = torch.tensor([[5.0, 4.8, 2.0, 3.0, 1.0, 0.5]], requires_grad=True)
for T in [1.0, 2.0, 4.0]:
    z_s.grad = None
    (kd_loss(z_s, z_t, T=T) / (T * T)).backward()        # plain KL, without the T^2
    manual = (F.softmax(z_s / T, -1) - F.softmax(z_t / T, -1)) / T
    print(T, torch.allclose(z_s.grad, manual, atol=1e-6), round(z_s.grad.abs().max().item() * T * T, 4))
# 1.0 True 0.2498 / 2.0 True 0.2492 / 4.0 True 0.2122 : the T^2 factor keeps the gradient size steady

# 3) a teacher and a smaller student: bigram language models over V tokens
V, r = 12, 3
teacher = torch.randn(V, V) * 2.0                        # full-rank teacher logits: row = previous token
class Student(torch.nn.Module):                          # rank-r student: cannot copy the teacher exactly
    def __init__(s, r=r):
        super().__init__()
        s.E, s.W = torch.nn.Embedding(V, r), torch.nn.Linear(r, V)
    def forward(s, ctx): return s.W(s.E(ctx))

ctx = torch.randint(0, V, (240,))                        # the transfer set: 240 contexts
hard = torch.multinomial(F.softmax(teacher[ctx], -1), 1).squeeze(1)  # one sampled next token each
mode = teacher[ctx].argmax(-1)                           # sequence-level KD with greedy decoding

def train(kind, steps=400, rank=r):
    torch.manual_seed(1); m = Student(rank); opt = torch.optim.Adam(m.parameters(), lr=0.05)
    for _ in range(steps):
        out = m(ctx)
        if kind == "hard":  loss = F.cross_entropy(out, hard)
        if kind == "mode":  loss = F.cross_entropy(out, mode)
        if kind == "logit": loss = kd_loss(out, teacher[ctx], T=1.0)
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():                                # evaluate on ALL contexts at T = 1
        p, lq = F.softmax(teacher, -1), F.log_softmax(m(torch.arange(V)), -1)
        kl = (p * (p.log() - lq)).sum(-1).mean().item()
        agree = (lq.argmax(-1) == teacher.argmax(-1)).float().mean().item()
        ent = -(lq.exp() * lq).sum(-1).mean().item()
    return round(kl, 3), round(agree, 3), round(ent, 3)

H_t = -(F.softmax(teacher, -1) * F.log_softmax(teacher, -1)).sum(-1).mean().item()
print("teacher entropy", round(H_t, 3))                  # 1.244
for kind in ["hard", "mode", "logit"]:
    print(kind, train(kind))                             # (KL to teacher, top-1 agreement, entropy)
# hard  (1.237, 0.917, 1.049)   one sample per context: noisy
# mode  (4.109, 1.0,   0.004)   copies the top token perfectly, but collapses to one-hot
# logit (0.113, 0.917, 1.364)   closest to the teacher; entropy above the teacher's (it covers)

# 4) the capacity gap (59.12): the KL floor shrinks only when the student gets bigger
for rank in [1, 3, 6, 12]:
    print(rank, train("logit", rank=rank)[0])            # 0.505, 0.113, 0.011, 0.0

The output shows the three regimes of this session in one place. Hard labels give the student a noisy view of the teacher. Training on the teacher’s top choice copies the decisions perfectly (agreement 1.0) but throws away all uncertainty, which is why sequence-level students become peaked. Logit distillation gets closest in KL, but a rank-3 student still keeps a floor of 0.113 nats that only a bigger student removes. (This tiny bigram setting is a stand-in: on real LLMs, sequence-level distillation often beats token-level, as Kim and Rush found, because generation quality is what is measured, not KL. [Likely])

Where this is used

SettingWhat is distilledMethod and details
Small open LLMs (Gemma 2 2B and 9B, Llama 3.2 1B and 3B)A larger model of the same familyToken-level KD during pretraining (59.5), same tokenizer; often after pruning (session 115)
Small reasoning models (DeepSeek-R1-Distill)Long correct reasoning traces of a strong reasonerRejection sampling and SFT (59.10); RL on top is optional
Chat assistants from API teachers (Alpaca-style data)Teacher answers to many promptsBlack-box sequence-level KD (59.8); quality filtering matters (session 54)
Encoders for search and classification (DistilBERT)A large encoder’s outputsSoft targets plus extra losses on hidden states (session 33)
Draft models for speculative decodingThe target model’s next-token distributionA draft that agrees with the target more often is accepted more often (session 128) [Likely]
Video classification and action recognition on deviceA large video model’s soft labels per clipLogit KD with temperature (59.2); soft labels carry the similarity between actions (“jogging” is close to “running”) [Likely]
Video retrieval and fingerprintingA big embedding model’s similarity scoresThe student matches the teacher’s similarity distribution over a batch, a KL over candidates (session 60) [Likely]
Video-LLMs: dense captions, key-moment detection, Video-RAGA strong captioner’s or VLM’s outputs on many clipsSequence-level KD as synthetic captions and timestamps, then SFT of a small Video-LLM (session 205) [Likely]

Common mistakes

  • Forgetting the \(T^2\) factor. When you raise \(T\), the soft-target gradient shrinks by \(1/T^2\) (59.4) and the hard-label term takes over without anyone noticing.
  • Applying the temperature to only one side, or evaluating the student at the training temperature. Teacher and student both use \(T\) in the KD term; the student is deployed at \(T = 1\).
  • Wrong KL argument order in code. PyTorch’s F.kl_div(input, target) expects the student log-probabilities as input and the teacher probabilities as target, and computes \(\KL(\text{target}\,\|\,\text{input})\). Passing probabilities instead of log-probabilities, or swapping them, gives a wrong loss with no error. Use reduction="batchmean", not "mean", which also divides by the vocabulary size. [Certain]
  • Mismatched tokenizers or chat templates in logit KD. The two distributions must be over the same tokens at the same positions. With different tokenizers, use sequence-level distillation.
  • Distilling wrong answers. Teacher outputs are not ground truth. Without a correctness filter, the student copies the teacher’s errors, often more confidently. Verify answers where possible.
  • Choosing the biggest teacher by default. A teacher far above the student can give a worse student (capacity gap). Try a closer teacher, shorter traces, or a teacher assistant.
  • Benchmark contamination. Teachers trained on the internet may reproduce test questions and answers. Decontaminate the distillation set against your evaluation sets.
  • Expecting the student to beat the teacher on hard skills. Distillation copies; it rarely exceeds the teacher on the distilled skill. To go further you need RL or better data.

Interview questions

What is knowledge distillation, and why does it beat training the small model on labels alone?basic
A large trained teacher produces targets for a smaller student: its probability distribution over classes or next tokens (logit KD) or its generated outputs (sequence-level KD). A one-hot label gives about \(\log V\) bits of information at most and says nothing about which wrong answers are close. The teacher’s distribution tells the student, at every position, how similar the alternatives are, and it is consistent and noise-free for the student’s purposes. Hinton et al. showed the regularizing effect clearly: with only 3% of the speech data, a model trained on hard targets overfit and reached 44.5% test frame accuracy, while the same model on soft targets reached 57.0%, close to the 58.9% of the baseline trained on all data.
Why do we use a temperature, and why multiply the KD loss by \(T^2\)?core
A confident teacher puts nearly all mass on one token; the information about the other tokens is in tiny probabilities that hardly affect cross-entropy. Dividing logits by \(T \gt 1\) flattens both distributions so those differences matter in the loss; the ranking does not change. The gradient of the KL with respect to a student logit is \((q_i - p_i)/T\), and at high \(T\) it is about \((z_i - v_i)/(V T^2)\): it shrinks like \(1/T^2\). Multiplying by \(T^2\) keeps the relative weight of the soft and hard terms roughly fixed when you tune \(T\). In the worked example the largest gradient times \(T^2\) is 0.250, 0.249 and 0.212 for \(T = 1, 2, 4\).
Show that high-temperature distillation is logit matching.advanced
Start from \(\partial \KL/\partial z_i = (q_i - p_i)/T\) with \(q_i = e^{z_i/T}/\sum_j e^{z_j/T}\). For large \(T\), \(e^{z/T} \approx 1 + z/T\), so \(q_i \approx (1 + z_i/T)/(V + \sum_j z_j/T)\). If the logits have zero mean, \(\sum_j z_j = 0\) and \(q_i \approx (1 + z_i/T)/V\); the same for \(p_i\) with \(v_i\). Then the gradient is \(\frac{1}{T}\cdot\frac{z_i - v_i}{V T} = \frac{z_i - v_i}{V T^2}\), the gradient of \(\frac{1}{2 V T^2}\sum_i (z_i - v_i)^2\). So the student regresses onto the teacher’s (centered) logits. At moderate \(T\) the very negative logits get less weight, which often helps because they are noisy.
Forward KL or reverse KL for distilling a generative LLM?advanced
Forward \(\KL(p\|q)\) averages over the teacher: the student is punished wherever the teacher has mass and the student does not, so a limited student covers all modes and puts mass between them. Reverse \(\KL(q\|p)\) averages over the student: it is punished for putting mass where the teacher has none, so it picks modes. For a small generative student, mode covering can produce text the teacher would never produce; MiniLLM uses reverse KL for this reason. Reverse KL needs samples from the student (on-policy), so it is harder to optimize. GKD treats the divergence (forward, reverse, generalized JSD) as a choice tuned per task. A good answer mentions that the student’s capacity decides how much the direction matters: a student that can represent the teacher reaches the same solution with both.
Compare word-level and sequence-level distillation. When would you choose each?core
Word-level (token-level) KD matches the teacher’s full next-token distribution at every position of given prefixes (59.7); it needs the logits and the same tokenizer and gives a dense signal per token. Sequence-level KD runs the teacher (beam search or sampling) and trains the student with plain cross-entropy on those outputs (59.8); it needs only text, works across tokenizers and with API teachers, and makes the student peaked so greedy decoding works. Kim and Rush found Seq-KD much better for translation: +4.2 BLEU greedy versus +0.7 for Word-KD on the same student. Choose logit KD inside one model family with white-box access (pretraining-time KD like Gemma 2); choose sequence-level for black-box teachers, reasoning traces, or different tokenizers. They can be combined.
What is exposure bias, and how does on-policy distillation address it?core
In both word- and sequence-level KD the student is trained on prefixes written by the data or the teacher, but at inference it conditions on its own tokens. After its first mistake it is in states it never trained on, and errors compound over long outputs. On-policy distillation (GKD) samples sequences from the current student and asks the teacher for its distribution at every prefix of those samples (59.9), so the student is corrected exactly in the states it actually visits. The cost is fresh student generation plus a teacher forward pass each step. It is closely related to imitation learning methods such as DAgger. [Likely]
How were the DeepSeek-R1-Distill models made, and what did the RL comparison show?core
About 800k samples were collected: about 600k reasoning samples by rejection sampling from an R1 checkpoint (several samples per prompt, only correct and readable ones kept) and about 200k non-reasoning samples. Six open base models (Qwen2.5-Math-1.5B and 7B, Qwen2.5-14B and 32B, Llama-3.1-8B, Llama-3.3-70B-Instruct) were fine-tuned with SFT only, no RL. The 7B student scored 55.5% on AIME 2024, the 32B 72.6%. Large-scale RL directly on Qwen-32B-Base gave only 47.0%. Conclusion: for small models, distilling a strong reasoner is far cheaper and better than RL from scratch, but surpassing the teacher needs stronger bases and more RL.
Why can a larger teacher produce a worse student? What do you do about it?advanced
The student can only reach the closest function in its own family; if the teacher is far outside it, the remaining divergence (59.12) is large and the student wastes capacity on details it cannot represent (with forward KL, by smearing mass between modes). Very sharp, long or complex targets, such as long reasoning traces for a 1B model, make this worse. Evidence: Mirzadeh et al. (performance drops with large gaps), Cho and Hariharan (larger models are often not better teachers; early-stopped teachers help), and Li et al. 2025 (students of 3B or less learn better from short traces). Fixes: a teacher assistant of intermediate size, an early-stopped or smaller teacher, shorter or mixed traces (Mix Distillation), moderate temperature, mode-seeking or on-policy objectives, or a larger student.
You must distill a 70B model into a 3B model for a chat product. Walk through the design.advanced
First, the access: if both share a tokenizer and you have the weights, combine token-level KD on a large general corpus (online, or offline with top-k logits) with sequence-level SFT on teacher answers to your product prompts; otherwise only sequence-level. Use the student’s chat template and a loss mask. Pick \(T\) around 1 to 2 and \(\alpha\) by validation; keep \(T^2\). Filter teacher outputs for correctness and safety, decontaminate, deduplicate. Watch the capacity gap: if 70B traces are too long or hard, try an intermediate teacher or shorter answers. Then on-policy distillation or preference tuning to fix exposure bias, and evaluate on generation quality and on what the 3B must actually do, not only on KL or perplexity.
How would you distill a large video model into a small on-device action recognizer?core
Run the teacher on many clips (unlabeled clips are fine) to get class logits; train the student on the KD loss (59.2) with a moderate temperature plus cross-entropy on the labeled subset. Soft labels transfer the similarity structure between actions (running versus jogging), and they also help with label noise in video datasets. Match the inputs: the student often sees fewer frames or lower resolution, so the teacher should see the same clip (possibly more frames) and the student is trained to match its prediction. Expect a capacity gap if the student is tiny; an intermediate teacher can help. For retrieval, distill the teacher’s similarity distribution over a batch instead of class logits.

Exercises

  1. Teacher logits are (2, 0, −2). Compute the teacher distribution at \(T = 1\) and \(T = 2\). How does the ratio between the first and the last probability change?
  2. In a KD loss with \(T = 3\), you forget the \(T^2\) factor. Roughly by how much is the distillation gradient weakened compared with the hard-label gradient?
  3. Teacher \(p = (0.5, 0.5)\), student \(q = (0.9, 0.1)\). Compute \(\KL(p\|q)\) and \(\KL(q\|p)\) in nats. Which direction punishes this student more, and why?
  4. In Figure 5’s tree, why does greedy decoding of the teacher not find the most likely sequence? What is the probability of each?
  5. A distillation set has 2 billion positions and you store the top 32 teacher tokens per position with 4-byte ids and 2-byte values. How much storage? What is wrong with simply dropping the other tokens?
  6. For a prompt where the teacher is correct 20% of the time, how many samples \(k\) do you need so the prompt is kept with probability at least 0.9?
Answerscheck
1. \(T = 1\): \(e^{2}, e^{0}, e^{-2}\) normalized gives (0.867, 0.117, 0.016); ratio \(e^{4} \approx 54.6\). \(T = 2\): logits (1, 0, −1) give (0.665, 0.245, 0.090); ratio \(e^{2} \approx 7.4\). 2. The soft-target gradient scales like \(1/T^2 = 1/9\), so it is about 9 times weaker than intended. 3. \(\KL(p\|q) = 0.5 \ln(0.5/0.9) + 0.5 \ln(0.5/0.1) = -0.294 + 0.805 = 0.511\). \(\KL(q\|p) = 0.9 \ln(1.8) + 0.1 \ln(0.2) = 0.529 - 0.161 = 0.368\). Forward KL is larger here: the teacher puts 0.5 on the second option, where the student has only 0.1, and forward KL is the direction that punishes missing teacher mass. 4. Greedy picks the most likely first token, “Thank” (0.45), and then “you” (0.40): 0.180. But “Thanks” (0.35) followed by “!” (0.80) gives 0.280. A locally best first step can lead to a worse whole sequence; beam search with 3 candidates keeps “Thanks” and finds it. 5. \(2 \times 10^9 \times 32 \times 6 = 384\) GB. Dropping the tail makes the kept probabilities sum to less than 1; you must renormalize them or keep one extra “rest” bucket, and the student never learns the tail of the distribution. 6. Need \(1 - 0.8^k \ge 0.9\), so \(0.8^k \le 0.1\), \(k \ge \ln 0.1/\ln 0.8 = 10.3\): \(k = 11\) (\(1 - 0.8^{11} = 0.914\)).

Key takeaways

  • A teacher’s full distribution (soft targets) carries much more information than a one-hot label; the temperature (59.1) makes the small probabilities visible, and the \(T^2\) factor (59.2) keeps the soft-target gradient the right size.
  • The KD gradient is \((q - p)/T\) (59.3); at high temperature distillation becomes matching the centered logits (59.4).
  • Logit (token-level) KD (59.5) needs logits and the same tokenizer; it is used inside model families (Gemma 2, Llama 3.2). Forward KL covers modes, reverse KL picks them (59.6).
  • Sequence-level KD (59.8) is SFT on teacher outputs: black-box, cross-tokenizer, makes students peaked so greedy decoding works (Kim and Rush: +4.2 BLEU greedy). On-policy distillation (59.9) fixes exposure bias.
  • Reasoning distillation is rejection sampling plus SFT (59.10): DeepSeek-R1-Distill-Qwen-32B reached 72.6 on AIME 2024 against 47.0 for RL on the same base.
  • The capacity gap (59.12): a student cannot copy a teacher outside its family; bigger teachers or longer traces can hurt small students. Use closer teachers, simpler targets or bigger students.

Key formulas

\[ p_i(T) = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}, \qquad \mathcal{L} = (1-\alpha)\,\mathrm{CE}(y, q) + \alpha\,T^2\,\KL\big(p(T)\,\|\,q(T)\big), \qquad \frac{\partial \KL}{\partial z_i} = \frac{q_i - p_i}{T} \]
\[ \mathcal{L}_{\text{Seq-KD}} \approx -\log q_\theta(\hat{y} \mid x), \qquad \mathcal{L}_{\text{on-policy}} = \E_{y \sim q_\theta}\sum_t D\big(p(\cdot \mid y_{\lt t})\,\|\,q_\theta(\cdot \mid y_{\lt t})\big), \qquad P(\text{kept}) = 1-(1-s)^k \]

References for this session

  1. Hinton et al. (2015). Distilling the Knowledge in a Neural Network
  2. Kim & Rush (2016). Sequence-Level Knowledge Distillation
  3. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  4. Raschka, Build a Reasoning Model (From Scratch), chapter 8: Distilling reasoning models
  5. Gu et al. (2023). MiniLLM: Knowledge Distillation of Large Language Models
  6. Agarwal et al. (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD)
  7. Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size
  8. Meta AI (2024). Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
  9. Sanh et al. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
  10. Mirzadeh et al. (2019). Improved Knowledge Distillation via Teacher Assistant
  11. Cho & Hariharan (2019). On the Efficacy of Knowledge Distillation
  12. Li et al. (2025). Small Models Struggle to Learn from Strong Reasoners