Distillation for LLMs 1: white-box (logits and features)
White-box distillation means the student can read the teacher’s insides: its next-token probabilities, its hidden states and its attention maps. This session goes deep into the choice of divergence (forward KL, reverse KL, generalized JSD), on-policy training with MiniLLM and GKD, feature and layer distillation, and the capacity gap between teacher and student.
What you will learn
- What a white-box teacher gives you (logits, hidden states, attention maps), and what each signal costs
- The gradients of forward KL, reverse KL and generalized JSD with respect to the student logits, and why they cover or seek modes
- Why a small student with reverse KL can collapse onto one mode, and how initialization matters
- MiniLLM: reverse KL over whole sequences as a policy-gradient problem, with single-step decomposition, teacher-mixed sampling and length normalization
- GKD: on-policy distillation with a student data fraction and any divergence, and how it relates to MiniLLM
- Feature distillation (FitNets, TinyBERT, MiniLM, DistilBERT, Minitron): projections, layer mapping and when intermediate losses help
- The capacity gap: why the best teacher is not the biggest one, and what to do about it
Why this matters
In session 118 we made a model smaller by changing its shape: fewer layers, smaller matrices, low-rank factors. Every such cut loses quality, and the cheapest way to win it back is to let the original model teach the smaller one. When we own the teacher’s weights, we can read everything it computes: the full next-token distribution at every position, the hidden state after every layer, every attention map. This is white-box distillation. This session is about the choices that decide whether it works.
Session 59 introduced the basics: soft targets, the temperature, the \(T^2\) factor, forward versus reverse KL, sequence-level distillation and on-policy distillation. Here we go one level deeper, the level that interviews for model-compression and inference teams expect:
- Which divergence? The gradients of forward KL, reverse KL and the generalized Jensen–Shannon divergence are different, and with a small student they lead to different models.
- Which data? The student can learn on fixed text, on teacher text, or on its own samples. MiniLLM and GKD both train on the student’s own outputs, for different reasons.
- Which signals? Logits only, or also hidden states and attention maps (feature distillation), and how to match a 4-layer student to a 12-layer teacher.
- Which teacher? The biggest teacher is often not the best one: the capacity gap.
Real systems that use white-box distillation:
- Minitron (NVIDIA) pruned Nemotron-4 15B to 8B and 4B and retrained them with distillation only. Each model used 94B training tokens, against 3.8T for training Nemotron-3 8B from scratch; the paper reports “up to 40× fewer training tokens”. [Certain]
- Gemma 2 small models and Llama 3.2 1B and 3B were trained with teacher logits as targets (session 59). [Certain]
- TinyBERT, MiniLM and DistilBERT distilled hidden states and attention, not only logits. The 4-layer TinyBERT is 7.5 times smaller and 9.4 times faster than BERT-base and keeps more than 96.8% of its GLUE score. [Certain]
The next session, session 120, covers the black-box case, where you only see the teacher’s text.
What a white-box teacher gives you
Run the teacher and the student on the same tokens. At every position the teacher produces three kinds of signals, and each one has a price.
| Signal | Size per position | Needs | Used by |
|---|---|---|---|
| Next-token distribution (logits) | \(V\) numbers (vocabulary size) | Same tokenizer | Hinton KD, MiniLLM, GKD, Gemma 2, Minitron |
| Hidden state of a layer | \(d\) numbers per matched layer | A map between layers, and a projection if the widths differ | FitNets, TinyBERT, DistilBERT, Minitron (optional) |
| Attention map of a layer | \(h \times L\) numbers per matched layer (\(L\) = sequence length), \(h L^2\) per sequence | The same number of heads (or re-grouped heads) | TinyBERT, MiniLM |
From the public config files: Llama 3.1 8B has 32 layers of width 4096; Llama 3.2 1B has 16 layers of width 2048, 32 query heads; both use a vocabulary of 128,256 tokens. [Certain] Counting the weights from these configs gives 8,030,261,248 parameters for the 8B model and 1,235,814,400 for the 1B model (its embedding and output matrix are tied).
Logits: 128,256 numbers per position. Hidden states: 4096 per layer, 131,072 for all 32 layers. Attention maps: at 4096 tokens one layer of the 1B model has \(32 \times 4096^2 = 536{,}870{,}912\) attention weights per sequence. FlashAttention never writes this matrix to memory (session 124), so attention-map losses would force the slow path. This is one reason attention distillation is used for short-context encoders but rarely for long-context LLMs. [Likely]
Compute: a teacher forward pass costs about \(2 N_t\) FLOPs per token and student training about \(6 N_s\). Here \(2 \times 8.03\text{B} = 16.1\) GFLOP against \(6 \times 1.236\text{B} = 7.4\) GFLOP: the teacher’s forward pass costs 2.2 times the student’s whole training step. Online white-box distillation is not cheap.
The logit signal is by far the most used one. So we start there, with the question that matters most: which divergence should compare the two distributions?
Three divergences and their gradients
Write \(p = p(\cdot \mid c)\) for the teacher’s next-token distribution in a context \(c\), and \(q_\theta = q_\theta(\cdot \mid c)\) for the student’s. A general token-level white-box loss is
- Read the symbols
- \(c\)
- a context: the prompt plus the tokens before the current position
- \(\mathcal{C}\)
- where the contexts come from: fixed data, teacher samples, or student samples. This choice is the second half of the story (MiniLLM, GKD)
- \(D\)
- a divergence: a number that is 0 when the two distributions are equal and positive otherwise
Two choices hide inside (119.1): the divergence \(D\) and the context source \(\mathcal{C}\). We take \(D\) first. The student’s probabilities come from its logits \(z\) through a softmax, \(q_i = e^{z_i} / \sum_j e^{z_j}\). Training only ever sees the gradient with respect to \(z\), so the gradient is the best way to see what each divergence does.
Forward KL
Forward KL, \(\KL(p \| q) = \sum_i p_i \log(p_i / q_i)\), averages over the teacher. Its gradient is the familiar “student minus teacher” (session 59 derived it):
Look at a token the teacher likes (\(p_i = 0.4\)) but the student almost ignores (\(q_i = 0.05\)): the gradient is \(-0.35\), a strong push up. Every teacher mode that the student misses gets pulled in with force \(p_i - q_i\). That is mode covering.
Reverse KL
Reverse KL, \(\KL(q \| p) = \sum_i q_i \log(q_i / p_i)\), averages over the student. Use \(\partial q_j / \partial z_i = q_j(\delta_{ij} - q_i)\) and the chain rule:
- Read the symbols
- \(q_i\) in front
- the gradient on token \(i\) is weighted by the student’s own probability: tokens the student does not use get almost no gradient
- \(\log(q_i/p_i)\)
- how much more the student likes token \(i\) than the teacher does; large and positive on “junk” tokens where \(p_i\) is tiny
- \(\KL(q\|p)\)
- the average of \(\log(q_i/p_i)\) under \(q\); subtracting it makes the gradients sum to zero, as for every softmax
Two consequences. First, a teacher mode the student has dropped (\(q_i \approx 0\)) gets a gradient near 0: reverse KL does not care much about missing modes. Second, a token where the student has mass but the teacher has almost none has a huge \(\log(q_i / p_i)\): reverse KL punishes junk hard. That is mode seeking, and for a generator it means “say less, but say only what the teacher would say”.
Teacher \(p = (0.45, 0.45, 0.10)\): two good tokens and one weak token.
Student A drops a mode, \(q = (0.80, 0.10, 0.10)\). \(\KL(p\|q) = 0.418\), \(\KL(q\|p) = 0.310\). The push on the missing token 2 is \(-0.350\) under forward KL (119.2) and only \(-0.181\) under reverse KL (119.3).
Student B puts mass on the weak token, \(q = (0.45, 0.10, 0.45)\). Here both divergences are 0.526. The gradient that pushes token 3 down is \(+0.350\) under forward KL and \(+0.440\) under reverse KL.
So per unit of loss, forward KL cares most about missing modes, and reverse KL cares most about extra mass on weak tokens.
Generalized Jensen–Shannon divergence
GKD (Agarwal et al.) proposes a divergence that moves smoothly between the two, the generalized JSD. Both distributions are compared with a mixture of the two:
- Read the symbols
- \(\beta\)
- a number strictly between 0 and 1; \(\beta = 0.5\) is the classic Jensen–Shannon divergence
- \(m\)
- the mixture distribution; it is never zero where \(p\) or \(q\) is non-zero, so both KL terms stay finite
Each KL term is bounded: \(\KL(p\|m) \le \log(1/\beta)\) because \(m_i \ge \beta p_i\). So the whole divergence is at most \(-\beta\log\beta - (1-\beta)\log(1-\beta) \le \log 2 \approx 0.693\) nats. Unlike forward and reverse KL, one bad token can never make the loss explode. A short Taylor expansion gives the two limits (the GKD paper states the first one): [Certain]
So small \(\beta\) behaves like forward KL and \(\beta\) near 1 like reverse KL. TRL’s GKDTrainer uses this convention too: beta=0.0 is forward KL, beta=1.0 reverse KL, default 0.5. [Certain] The gradient has a clean form. With respect to the student probabilities, \(\partial D / \partial q_j = (1-\beta)\log(q_j / m_j)\), and through the softmax:
The first factor is the same softmax “chain rule” for all three. Only \(g\) changes. Put \(g_j = -p_j/q_j\) and you get back (119.2); put \(g_j = \log(q_j/p_j)+1\) and you get (119.3). Figure 1 computes all three on a real-looking example.
How to read Figure 1
Start with “Student drops coffee” and forward KL: the push on “coffee” is +0.330, the biggest one, and “tea” is pushed down by 0.400. Switch to reverse KL: the push on “coffee” falls to +0.122, because it is weighted by the student’s small \(q = 0.05\). Now load “Student likes junk” (0.10 on “sand”): forward KL pushes “sand” down by 0.095, reverse KL by 0.263, almost three times as hard. JSD with \(\beta = 0.5\) is in between, with smaller gradients overall because the divergence is bounded. Move \(\beta\) towards 0.05 or 0.95 and watch the pattern of bars become the forward or the reverse pattern (only the overall size shrinks, by about \(\beta\) or \(1 - \beta\), as formula (119.5) says).
A student that is too small: covering, seeking and collapse
If the student can represent the teacher exactly, all three divergences have the same minimum: \(q = p\), with loss 0. The choice of divergence only matters when the student cannot copy the teacher, which is the normal case for a small student and a hard teacher. Then each divergence picks a different “best wrong answer”.
Figure 2 makes this visible with a toy. Twelve tokens are placed on a line so that neighbors are similar (think of a one-dimensional embedding). The teacher has two modes: around token 2 (weight about 0.6) and around token 8 (about 0.4), with only 0.5% of its mass on the junk tokens 5 and 11. The student’s logits are a polynomial in the token position, so its capacity is limited:
- One bump: logits \(a\,u + b\,u^2\) (\(u\) is the scaled position). This is a discretized Gaussian: it can make only one hill.
- Two bumps: a degree-4 polynomial, enough for two hills.
- Full: 12 free logits, enough to copy the teacher exactly.
The figure trains the student for 1500 Adam steps (learning rate 0.05) on the \(\mathrm{JSD}(\beta)\) of (119.4), with \(\beta = 0\) and \(\beta = 1\) meaning exact forward and reverse KL.
How to read Figure 2
Default (one bump, start left, \(\beta = 0\)): forward KL makes a wide hill over both modes, with 13.1% of the student’s mass on junk tokens, 26 times the teacher’s 0.5%. Slide \(\beta\) up: at \(\beta = 0.5\) and above the student jumps onto the left mode, junk mass drops to 0.3%, but the right mode is ignored (forward KL above 7). This is the trade: quality of what is said versus coverage of what could be said. Now choose “Start in the middle” with \(\beta = 1\): the student stays wide (reverse KL 0.663 instead of 0.515), because reverse KL has a poor local minimum and gradient descent from the middle finds it. Choose “Two bumps”, start left, \(\beta = 1\): although the student could cover both modes, it stays on one (reverse KL 0.514 against about 0.061 when it covers both). Choose “Full”: every \(\beta\) reaches zero divergence.
Three lessons carry over to real LLMs:
- The divergence matters only through the capacity gap. With “Full” capacity the choice is irrelevant. With a small student it decides what kind of errors you get: smeared probability (forward) or missing options (reverse).
- Reverse KL can collapse. Mode seeking can turn into mode collapse: the student drops valid answers it could have learned, and its outputs lose diversity. [Certain for the toy; Likely as a general risk]
- Reverse KL depends on the starting point. Its loss surface has bad local minima, so the student must start from something sensible. MiniLLM first fine-tunes the student on the ground-truth responses before the reverse-KL phase. [Certain]
Which one wins in practice depends on the task. MiniLLM reports gains with reverse KL for instruction following. Minitron, distilling a pruned model during continued pretraining, found forward KL better: LM loss 2.155 versus 2.665 with reverse KL on its depth-pruned 8B model. [Certain] A rough rule: when the student must keep the teacher’s full knowledge (pretraining-style data), cover modes; when it must produce good answers (generation), seek them. [Likely]
MiniLLM: reverse KL over whole sequences
For a language model the “distribution” we really care about is over whole responses \(y\) for a prompt \(x\). MiniLLM (Gu et al., ICLR 2024) minimizes the reverse KL at the sequence level: [Certain]
The second equality is the chain rule of KL: the sequence-level divergence equals the sum of per-token divergences, along prefixes sampled from the student. The expectation is over the student’s own outputs, so we cannot simply differentiate a fixed dataset loss. The paper treats it like reinforcement learning (session 64): the student is a policy, and the per-token “reward” is \(r_t = \log \frac{p(y_t \mid y_{\lt t}, x)}{q_\theta(y_t \mid y_{\lt t}, x)}\), large when the teacher likes the token more than the student does.
- Read the symbols
- \(r_t\)
- the per-token reward: log of teacher probability over student probability for the sampled token
- \(R_t\)
- the return: the sum of rewards from step \(t\) to the end. A token that leads to a continuation the teacher likes gets a high return
- \(-1\)
- comes from differentiating the \(\log q_\theta\) inside the reward; its expectation is zero, so it acts like a constant baseline
- \(\nabla \log q_\theta(y_t \mid \cdot)\)
- the direction that makes token \(y_t\) more likely in that context
So a sampled token with positive \(R_t - 1\) is made more likely, and one with negative value less likely. Plain REINFORCE on (119.8) is very noisy. MiniLLM adds three fixes. [Certain]
1. Single-step decomposition. Split the return into the current step and the future:
The inner expectation \(\E_{y_t \sim q_\theta(t)}[r_t] = -\KL(q_\theta(t) \,\|\, p(t))\) is computed exactly by summing over the vocabulary, not by sampling. So the single-step part is just the gradient of the per-token reverse KL at each sampled prefix, with no sampling noise. Only the long-term part needs the policy-gradient estimate.
2. Teacher-mixed sampling. Early in training a student can “hack the reward”: produce degenerate text that it gives low probability, which still gets acceptable \(r_t\). MiniLLM samples from a mix of teacher and student and corrects with an importance weight:
with \(\alpha = 0.2\) in all experiments. The exact weight is a product over all earlier steps; the paper keeps only the current factor to reduce variance. [Certain]
3. Length normalization. Long responses add up many negative \(r_t\), so the raw return prefers short answers. MiniLLM averages the future rewards instead: \(R^{\text{Norm}}_{t+1} = \frac{1}{|y| - t - 1}\sum_{t'=t+1}^{|y|} r_{t'}\) (the paper’s normalizer). [Certain]
The long-term part is optimized with a PPO-style clipped ratio (session 66), and a language-modeling loss on a pretraining corpus is added so the student keeps its general abilities. [Certain]
- input: teacher \(p\), pretrained student \(q_{\theta_0}\), prompt–response data \(\mathcal{D}\), pretraining corpus \(\mathcal{D}_{\text{PT}}\), \(\alpha = 0.2\), clip \(\epsilon\)
- fine-tune the student on the ground-truth responses of \(\mathcal{D}\); keep the checkpoint with the lowest validation loss // a sensible start for reverse KL
- repeat until convergence:
- sample prompts from \(\mathcal{D}\); generate responses from the mixture \(\tilde{p}\) of (119.10)
- run the teacher and the student on every prefix: get \(p(\cdot \mid y_{\lt t}, x)\), \(q_\theta(\cdot \mid y_{\lt t}, x)\)
- \((\nabla\mathcal{L})_{\text{Single}}\): exact per-token reverse KL over the vocabulary, weighted by \(w_t\)
- \((\nabla\mathcal{L})^{\text{Norm}}_{\text{Long}}\): length-normalized returns, PPO clip on the ratio \(q_\theta / \tilde{p}\)
- \(\nabla \mathcal{L}_{\text{PT}}\): next-token loss on a batch from \(\mathcal{D}_{\text{PT}}\)
- \(\theta \leftarrow \theta - \eta\,\big[(\nabla\mathcal{L})_{\text{Single}} + (\nabla\mathcal{L})^{\text{Norm}}_{\text{Long}} + \nabla\mathcal{L}_{\text{PT}}\big]\)
- return \(q_\theta\)
Figure 3 shows the pieces of (119.7)–(119.10) on a response of two tokens, small enough to compute every number exactly.
How to read Figure 3
The global line is always the same: the sequence-level reverse KL is 0.6807 nats, and it equals the per-step sum of (119.7): 0.5569 from the first token plus 0.1238 from the second. Most of it comes from “Bad”. Pick “Fine thanks”: both rewards are positive (\(r_1 = +0.511\), \(r_2 = +0.560\)), the return \(R_1 = 1.070\), and the gradient makes this path more likely. Pick “Bad .”: \(r_1 = -2.079\), so \(R_1 = -1.897\) and “Bad” is pushed down, even though the second token “.” has a small positive reward. Move \(\alpha\) from 0 to 1: the chance to sample “Bad” first falls from 0.40 to 0.05 (0.33 at \(\alpha = 0.2\)), and the importance weight \(w_1 = q/\tilde{p}\) for “Bad” grows (1.212 at \(\alpha = 0.2\)) to correct for sampling it less often.
Teacher GPT-2 1.5B (27.6 Rouge-L). Students trained on about 12.5k databricks-dolly-15k examples. [Certain]
| Student | SFT, no KD | Token KD | SeqKD | MiniLLM |
|---|---|---|---|---|
| GPT-2 120M | 23.3 | 22.8 | 22.7 | 24.6 |
| GPT-2 340M | 25.5 | 25.0 | 25.3 | 25.4 |
| GPT-2 760M | 25.4 | 25.9 | 25.6 | 26.4 |
MiniLLM wins for 120M (+1.3 over SFT) and 760M (+1.0), but at 340M it is 0.1 below plain SFT. Gains of one or two points are typical, and not every size gains. Read distillation papers per row, not only from the abstract.
GKD: on-policy distillation with any divergence
Token-level KD trains on prefixes from the dataset; at test time the student continues its own prefixes. When it makes an early mistake it lands in contexts it never trained on (exposure bias, session 59). GKD (Agarwal et al.) fixes the context source \(\mathcal{C}\) of (119.1): with probability \(\lambda\) a batch uses responses generated by the student, otherwise the fixed dataset: [Certain]
- Read the symbols
- \(\lambda\)
- the student data fraction: 0 means only fixed data, 1 means only the student’s own samples
- \(D(p\|q_\theta)(y \mid x)\)
- the token-level divergence (forward KL, reverse KL or \(\mathrm{JSD}(\beta)\)) summed over the positions of response \(y\)
- \((X, Y)\)
- the fixed data: ground-truth responses, or teacher-generated ones
A key detail: GKD does not backpropagate through the student’s sampling. [Certain] The samples are treated as fixed data; only the per-token divergence is differentiated. Compare with (119.9): with reverse KL as \(D\) and \(\lambda = 1\), the GKD gradient is exactly MiniLLM’s single-step term, without the long-term policy-gradient term. GKD is simpler and has no reward variance; MiniLLM also credits a token for the future it leads to. (This follows from the two formulas.)
| Method | Divergence \(D\) | Student fraction \(\lambda\) |
|---|---|---|
| Supervised (token-level) KD | forward KL | 0 |
| SeqKD | (SFT on teacher outputs) | 0, with teacher-written data |
| On-policy KD | forward KL | 1 |
| ImitKD | forward KL | 0.5 |
| f-distill | total variation | 0.5 |
| GKD in general | any, often \(\mathrm{JSD}(\beta)\) | any |
The GKD paper lists these as special cases. [Certain] Its teacher was a fine-tuned T5-XL (about 3B), with students T5-small (77M), T5-base (250M) and T5-large (800M). On-policy GKD gave relative gains (over the gains of baseline KD) of 2.1× on summarization, 1.7× on translation and 1.9× on arithmetic reasoning, averaged over student sizes. [Certain] The best \(\beta\) changed with the task and the decoding method, while on-policy data (high \(\lambda\)) helped consistently. [Certain] TRL’s defaults are \(\lambda = 0.5\), \(\beta = 0.5\) and sampling temperature 0.9. [Certain]
- input: teacher \(p\), student \(q_\theta\), fixed data \((X, Y)\), fraction \(\lambda\), divergence \(D\) (for example \(\mathrm{JSD}(\beta)\))
- draw \(u \sim \text{Uniform}(0,1)\)
- if \(u \le \lambda\): take prompts \(x\) from \(X\) and sample \(y \sim q_\theta(\cdot \mid x)\) // no gradient through sampling
- else: take pairs \((x, y)\) from \((X, Y)\)
- run the teacher (no gradient) and the student on \((x, y)\): logits at every response position
- \(\mathcal{L} \leftarrow\) mean over response tokens of \(D\big(p(\cdot \mid x, y_{\lt t}) \,\|\, q_\theta(\cdot \mid x, y_{\lt t})\big)\) // formula (119.4) or (119.2)/(119.3)
- \(\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}\)
The price of on-policy training is generation. Sampling a response token by token is much slower than one teacher forward pass over a fixed batch, so on-policy distillation steps are dominated by decoding. Production setups use a fast inference engine for the sampling and share weights with the trainer, as in RL training (session 75). [Likely]
Feature and layer distillation
Logits are the last layer. A white-box teacher also shows how it got there. Feature distillation asks the student’s internal states to look like the teacher’s. The idea is older than LLMs: FitNets (Romero et al., 2014) trained a deeper and thinner student with “hints”, the teacher’s middle-layer outputs. [Certain]
Because the student is usually narrower, its hidden state cannot be compared to the teacher’s directly. A small learned map, the regressor or projection, translates it:
- Read the symbols
- \(\mathbf{h}^S_m\)
- the student’s hidden state after its layer \(m\) (a row vector of width \(d_S\))
- \(\mathbf{h}^T_{g(m)}\)
- the teacher’s hidden state after layer \(g(m)\), width \(d_T\)
- \(g(m)\)
- the layer map: which teacher layer student layer \(m\) must imitate
- \(W\)
- the projection; trained together with the student and thrown away afterwards
TinyBERT: embeddings, hidden states, attention and logits
TinyBERT (Jiao et al.) distills BERT-base (12 layers, width 768, 109M parameters) into TinyBERT4 (4 layers, width 312, feed-forward 1200, 12 heads, 14.5M parameters). [Certain] It uses four losses: embeddings, hidden states, attention, and the prediction layer:
- Read the symbols
- \(A_i\)
- the attention scores of head \(i\) (an \(L \times L\) matrix). TinyBERT uses the scores before the softmax, which converged faster and worked better
- \(H\), \(W_h\)
- hidden states of one layer (\(L \times d\)) and the projection from width 312 to 768; the embedding loss uses its own \(W_e\)
- \(t\)
- temperature of the soft cross-entropy; \(t = 1\) worked well
The layer map was uniform, \(g(m) = 3m\): student layer 1 copies teacher layer 3, layer 2 copies layer 6, and so on, with \(g(0) = 0\) for the embeddings. In the ablation the average dev score was 75.6 for uniform, 71.3 for “bottom” (\(g(m) = m\)) and 70.9 for “top” (\(g(m) = m + N - M\)). [Certain] Training has two stages: general distillation on Wikipedia with the pretrained BERT as teacher, then task-specific distillation from a fine-tuned BERT on augmented task data. [Certain]
MiniLM: distill relations, not vectors
MiniLM (Wang et al.) avoids projections altogether. It distills only the last layer’s self-attention, as distributions, plus a “value relation”: [Certain]
Attention rows and value relations are \(|x| \times |x|\) matrices whatever the hidden width, so the student can have any width without a projection. The paper reports that its student keeps more than 99% of the teacher’s accuracy on SQuAD 2.0 and several GLUE tasks while using 50% of the teacher’s Transformer parameters and computation; its smaller 6-layer, 384-wide model (about 22M parameters) runs 5.3 times faster than BERT-base. [Certain] It still uses 12 student heads, the same as the teacher. [Certain]
DistilBERT and Minitron: when do intermediate losses help?
DistilBERT initializes the student from the teacher “by taking one layer out of two”, and adds a cosine loss that aligns the directions of student and teacher hidden states to the soft-target and masked-LM losses. In the ablation, removing the cosine loss cost 1.46 GLUE points, and random initialization instead of teacher layers cost 3.69. [Certain] Initialization from the teacher is itself a form of white-box transfer.
Minitron is the modern LLM data point. Its loss is \(\mathcal{L} = \mathcal{L}_{\text{CLM}} + \mathcal{L}_{\text{logits}} + \alpha\, \mathcal{L}_{\text{is}}\), where \(\mathcal{L}_{\text{is}}\) sums intermediate-state losses, and the weight is set at every step:
so the intermediate term never dominates the logit term. [Certain] Its findings: intermediate-state losses help when depth is pruned a lot; when depth is kept (width pruning), logit-only distillation is best. For Minitron 8B, MMLU was 58.3 with logits only and 58.0 with logits plus intermediate states. Among intermediate losses, cosine similarity worked best. Temperature 1.0 was best, and the logit KL alone slightly beat logit KL plus the ground-truth cross-entropy. [Certain]
How to read Figure 4
TinyBERT₄ with uniform mapping matches teacher layers 3, 6, 9 and 12; the five projections have \(5 \times 312 \times 768 = 1{,}198{,}080\) parameters, about 8% of the 14.5M student, and they are thrown away after training. “Bottom” matches layers 1–4 and the student never sees what the top of the teacher computes; “top” matches 9–12 and the student never sees the lower layers. Switch to the Llama shape: 17 projections of \(2048 \times 4096\) are 142.6M parameters, 11.5% of the 1.24B student, and one layer’s attention map at 4096 tokens has 537 million entries. That is why LLM-scale distillation usually stays with logits, or uses cheap cosine losses on a few layers.
The capacity gap: the best teacher is not the biggest
The capacity gap is the distance between what the teacher computes and what the student can represent. Figure 2 showed its simplest form: when the student is too small, no divergence makes the loss zero, and the divergence only chooses which errors to keep. Real experiments show a more surprising effect: a stronger teacher can produce a worse student.
- Mirzadeh et al. found that student accuracy drops when the teacher is too large, and proposed a teacher assistant: distill the large teacher into a medium model first, then the medium model into the student. [Certain] MiniLM reports that a teacher assistant also helps for large Transformers. [Certain]
- Cho and Hariharan found that larger models are often not better teachers, and that a teacher stopped early in training can be a better teacher. [Certain]
- Zhang et al. studied language models below 3B parameters (GPT-2 and Pythia) and found that the best teacher size grows linearly with the student size. Their fit is about
where \(\mathbb{S}\) is the student size and \(\mathbb{T}^\star\) the best teacher size (fitted line \(2.498\,\mathbb{S} - 11.498\)). They used it to distill a 3B student (MiniMA) from LLaMA2-7B. [Certain] Treat the constant as a rule of thumb for that setting, not a law of nature. [Likely]
Why does a bigger teacher hurt? A bigger teacher has sharper and more complex distributions, and more of its knowledge lies outside the student’s reach. With forward KL the student spends capacity smearing mass over modes it cannot fit; the loss at the end of training is high, and the gradient signal is dominated by things the student cannot learn. [Likely] What helps, in white-box terms:
- A closer teacher (or a teacher assistant chain), or an earlier teacher checkpoint.
- Student-centered training: on-policy data (GKD) shows the teacher the contexts the student actually reaches, so the feedback is about the student’s real mistakes.
- A divergence that ignores what cannot be fit: reverse KL or \(\mathrm{JSD}(\beta)\) with larger \(\beta\), as GKD suggests when “the student lacks the expressivity”. [Certain]
- A larger temperature or top-\(k\) truncation of teacher logits changes how much of the tail the student must copy. [Likely]
- Better initialization: start from the teacher’s own weights (pruning plus distillation, as in Minitron and DistilBERT), so the student begins close to the teacher.
Putting it together in code
The code below is a complete, deterministic white-box distillation loop on toy models. It implements the generalized JSD of (119.4) with forward and reverse KL as the two ends, a GKD-style mix of fixed and on-policy batches with \(\lambda = 0.5\), and a hidden-state loss (119.12) with a projection and the uniform layer map \(g(m) = 3m\). The mixture is computed in log space with logsumexp, so tiny probabilities do not underflow.
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) def token_divergence(s_logits, t_logits, beta=0.5, T=1.0): """Per-token generalized JSD(beta) between teacher p and student q. beta=0 -> forward KL(p||q), beta=1 -> reverse KL(q||p). Shapes: (..., V).""" lq = F.log_softmax(s_logits / T, -1) # student log-probs lp = F.log_softmax(t_logits / T, -1) # teacher log-probs if beta == 0.0: return (lp.exp() * (lp - lq)).sum(-1) # forward KL if beta == 1.0: return (lq.exp() * (lq - lp)).sum(-1) # reverse KL # log of the mixture m = beta*p + (1-beta)*q, computed stably lm = torch.logsumexp(torch.stack([lp + math.log(beta), lq + math.log(1 - beta)]), 0) kl_pm = (lp.exp() * (lp - lm)).sum(-1) kl_qm = (lq.exp() * (lq - lm)).sum(-1) return beta * kl_pm + (1 - beta) * kl_qm class TinyLM(nn.Module): # a stand-in for a causal LM def __init__(s, V, d, L): super().__init__() s.emb = nn.Embedding(V, d) s.blocks = nn.ModuleList(nn.Linear(d, d) for _ in range(L)) s.head = nn.Linear(d, V) def forward(s, x): # x: (B, N) token ids h = s.emb(x).cumsum(1) / torch.arange(1, x.shape[1] + 1).view(1, -1, 1) hs = [h] # causal "mixing": running mean for blk in s.blocks: h = h + torch.tanh(blk(h)); hs.append(h) return s.head(h), hs # logits (B, N, V), hidden states V, B, N = 16, 4, 6 teacher, student = TinyLM(V, 32, 6).eval(), TinyLM(V, 16, 2) proj = nn.Linear(16, 32, bias=False) # W_h: student width -> teacher width opt = torch.optim.AdamW(list(student.parameters()) + list(proj.parameters()), lr=3e-3) x = torch.randint(0, V, (B, N)) # a fixed batch of "data" prefixes @torch.no_grad() def sample_student(prompt, n_new, gen): # on-policy data: the student writes seq = prompt for _ in range(n_new): logits, _ = student(seq) nxt = torch.multinomial(F.softmax(logits[:, -1], -1), 1, generator=gen) seq = torch.cat([seq, nxt], 1) return seq gen = torch.Generator().manual_seed(1) for step in range(301): on_policy = step % 2 == 1 # lambda = 0.5: every other batch seq = sample_student(x[:, :2], N - 2, gen) if on_policy else x with torch.no_grad(): t_logits, t_hs = teacher(seq) # teacher reads the same tokens s_logits, s_hs = student(seq) loss_logit = token_divergence(s_logits, t_logits, beta=0.5).mean() # feature loss: student layer m matches teacher layer g(m) = 3m loss_feat = sum(F.mse_loss(proj(s_hs[m]), t_hs[3 * m]) for m in range(3)) / 3 loss = loss_logit + 0.1 * loss_feat opt.zero_grad(); loss.backward(); opt.step() if step % 100 == 0: with torch.no_grad(): s_l, _ = student(x); t_l, _ = teacher(x) fk = token_divergence(s_l, t_l, 0.0).mean().item() rk = token_divergence(s_l, t_l, 1.0).mean().item() print(f"step {step:3d} JSD {loss_logit.item():.4f} feat {loss_feat.item():.4f} " f"FKL {fk:.4f} RKL {rk:.4f}")
Output with PyTorch 2.8 on CPU (it is the same on every run):
step 0 JSD 0.0836 feat 1.2670 FKL 0.3411 RKL 0.3380 step 100 JSD 0.0035 feat 0.2499 FKL 0.0138 RKL 0.0138 step 200 JSD 0.0018 feat 0.1454 FKL 0.0069 RKL 0.0071 step 300 JSD 0.0010 feat 0.1007 FKL 0.0039 RKL 0.0039
Both KL directions fall together here because this student is big enough for this random teacher: with no real capacity gap, every divergence ends at the same place, as in Figure 2 with “Full”. In a real setup, replace TinyLM with two causal LMs that share a tokenizer, mask the prompt tokens out of the loss, and sample on-policy responses with a fast inference engine.
Where this is used
| Setting | White-box signal | Typical choice |
|---|---|---|
| Recovering a pruned or factorized LLM (Minitron, session 115, session 118) | Logits; hidden states if depth was cut | Forward KL at \(T = 1\), teacher-initialized student, dynamic weight (119.15) |
| Small models of a family during pretraining (Gemma 2, Llama 3.2) | Teacher next-token distributions, often top-\(k\) | Token-level forward KL on the pretraining stream |
| Instruction-following students (MiniLLM, GKD) | Logits on the student’s own responses | Reverse KL or \(\mathrm{JSD}(\beta)\), on-policy fraction \(\lambda\) near 1 |
| Small encoders for search and classification (TinyBERT, MiniLM, DistilBERT) | Hidden states, attention, value relations | Uniform layer map, projections or relation losses |
| Draft models for speculative decoding | The target model’s next-token distribution on the draft’s own outputs | On-policy distillation raises the acceptance rate (session 128) [Likely] |
| Video classification and action recognition on device | A large video model’s class logits and frame features | Logit KD plus a cosine loss on pooled clip features [Likely] |
| Video retrieval and fingerprinting | The teacher’s similarity distribution over a batch of clips | KL between similarity rows, a relation loss like (119.14) with no projection [Likely] |
| Small Video-LLMs for captions, key-moment detection and Video-RAG (session 205) | A large Video-LLM’s token distributions on the same frames and prompt | Token-level KD when tokenizers match; otherwise black-box (session 120) [Likely] |
Common mistakes
- Mixing up the direction convention. “\(\KL(\text{teacher}\,\|\,\text{student})\)” is forward KL; in GKD/TRL,
beta=0is forward andbeta=1is reverse. Check the code, not the name. PyTorch’sF.kl_div(input, target)computes \(\KL(\text{target} \| \text{input})\) withinputas log-probabilities. - Computing the JSD mixture in probability space. \(\log(\beta p + (1-\beta) q)\) underflows for tiny probabilities in bf16; use
logsumexpof the log-probabilities as in the code above. - Starting reverse-KL training from a bad student. Reverse KL has poor local minima (Figure 2, “start in the middle”). Fine-tune first, as MiniLLM does.
- Backpropagating through sampled tokens by accident, or assuming on-policy GKD is a policy gradient. GKD treats samples as data; MiniLLM adds an explicit policy-gradient term for the future.
- Forgetting the loss mask. Prompt tokens, padding and template tokens should not get a distillation loss; otherwise the student wastes capacity copying the teacher’s guesses about the prompt.
- Feature losses with the wrong scale. An MSE on hidden states can be thousands of times larger than the logit KL and take over training. Normalize (cosine loss) or weight dynamically as in (119.15).
- Mapping layers without thought. Copying only the bottom or only the top teacher layers was clearly worse in TinyBERT (71.3 and 70.9 against 75.6 for uniform).
- Choosing the biggest teacher by default. Try a teacher about 2.5 times the student size or a teacher assistant, and measure.
Interview questions
What makes distillation “white-box”, and which signals can you use?basic
Derive the gradient of forward and reverse KL with respect to the student logits.core
Teacher (0.5, 0.5), student (0.9, 0.1). Which divergence pushes harder to fix the student, and how hard?core
What is the generalized \(\mathrm{JSD}(\beta)\) of GKD, what are its limits, and why might you prefer it?advanced
Why does MiniLLM need policy gradients, and what are its three tricks?advanced
Explain exposure bias and how GKD addresses it. What do \(\lambda\) and \(\beta\) control?core
How is on-policy GKD with reverse KL related to MiniLLM?advanced
How do you distill hidden states from a 768-wide, 12-layer teacher into a 312-wide, 4-layer student?core
Why is attention-map distillation rare for long-context LLMs?advanced
When do intermediate-state losses help in LLM distillation, according to Minitron?core
Why can a bigger teacher give a worse student, and what would you do?advanced
You have Llama 3.1 8B and want a 1B model for serving. Design the white-box distillation.advanced
Exercises
- Teacher \(p = (0.5, 0.5)\), student \(q = (0.9, 0.1)\). Compute the forward-KL and reverse-KL gradients on both student logits.
- Show with a first-order expansion that \(D_{\mathrm{JSD}(\beta)}(p\|q)/\beta \to \KL(p\|q)\) when \(\beta \to 0\).
- A 24-layer teacher of width 1024 is distilled into a 6-layer student of width 512 with uniform mapping. List \(g(m)\) for \(m = 0..6\) and count the projection parameters (one per student layer plus the embeddings).
- In Figure 3, for the sampled response “Bad .”, compute \(r_1\), \(r_2\) and \(R_1\). Is “Bad” made more or less likely?
- With teacher-mixed sampling at \(\alpha = 0.2\), what is the probability of sampling “Bad” first in Figure 3, and its importance weight \(w_1\)?
- By the fitted law (119.16), what teacher size is best for a 1B student? Name one reason not to trust this number blindly.
Answerscheck
2. With \(m = q + \beta(p - q)\): \(\KL(p\|m) \to \KL(p\|q)\), so the first term is \(\beta\KL(p\|q) + O(\beta^2)\). The second term: \(\KL(q\|m) = -\sum q_i \log(1 + \beta(p_i - q_i)/q_i) = -\beta\sum(p_i - q_i) + O(\beta^2) = O(\beta^2)\), because both distributions sum to 1. Divide by \(\beta\) and let \(\beta \to 0\).
3. \(g(m) = 4m\): 0, 4, 8, 12, 16, 20, 24. Projections: \(7 \times 512 \times 1024 = 3{,}670{,}016\) parameters.
4. \(r_1 = \ln(0.05/0.40) = -2.079\), \(r_2 = \ln(0.6/0.5) = +0.182\), \(R_1 = -1.897\). The return is far below 1, so “Bad” is made less likely; “.” after “Bad” gets a small push up.
5. \(\tilde{p} = 0.2 \times 0.05 + 0.8 \times 0.40 = 0.33\); \(w_1 = 0.40/0.33 = 1.212\). The student samples its favorite mistake less often, and each such sample counts a bit more.
6. About 2.5B. The law was fitted on GPT-2 and Pythia students below 3B with a particular data setup; the constant may change with data, divergence, training length and model family, so measure for your case.
Key takeaways
- White-box distillation can use logits, hidden states and attention maps; logits are the main signal, and they need the same tokenizer and a teacher forward pass (for 8B → 1B it costs about 2.2 times the student training step).
- All divergences share the softmax chain rule (119.6). Forward KL’s gradient \(q - p\) pulls in missing modes; reverse KL’s gradient is weighted by \(q_i\) and punishes junk hard (119.3). \(\mathrm{JSD}(\beta)\) moves between them and is bounded by \(\log 2\).
- The divergence matters only through the capacity gap. A small student covers modes with forward KL and seeks (or collapses onto) modes with reverse KL; reverse KL needs a good starting point.
- MiniLLM minimizes sequence-level reverse KL with policy gradients, plus single-step decomposition, teacher-mixed sampling (\(\alpha = 0.2\)) and length normalization. Gains are real but small and not universal.
- GKD trains on the student’s own samples with fraction \(\lambda\) and any divergence, without gradients through sampling; with reverse KL and \(\lambda = 1\) it is MiniLLM’s single-step term.
- Feature distillation needs a layer map (uniform works best) and projections or relation losses. For LLMs, Minitron found intermediate losses help only when depth is cut a lot.
- The best teacher is not the biggest: use a closer teacher (about 2.5 times the student for small LMs), a teacher assistant, on-policy data, and initialization from the teacher.
Key formulas
References for this session
- Hinton et al. (2015). Distilling the Knowledge in a Neural Network
- Gu et al. (2023). MiniLLM: Knowledge Distillation of Large Language Models
- Agarwal et al. (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD)
- Kim & Rush (2016). Sequence-Level Knowledge Distillation
- Hinton et al. (2015). Distilling the Knowledge in a Neural Network
- Gu et al. (2023). MiniLLM: Knowledge Distillation of Large Language Models
- Agarwal et al. (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD)
- Kim & Rush (2016). Sequence-Level Knowledge Distillation
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Romero et al. (2014). FitNets: Hints for Thin Deep Nets
- Jiao et al. (2019). TinyBERT: Distilling BERT for Natural Language Understanding
- Wang et al. (2020). MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- Sanh et al. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- Muralidharan et al. (2024). Compact Language Models via Pruning and Knowledge Distillation (Minitron)
- Mirzadeh et al. (2019). Improved Knowledge Distillation via Teacher Assistant
- Cho & Hariharan (2019). On the Efficacy of Knowledge Distillation
- Zhang et al. (2023). Towards the Law of Capacity Gap in Distilling Language Models