Distillation for LLMs 2: black-box (outputs only)
Often the best teacher is behind an API and you only see the text it writes. This session shows why fine-tuning on teacher text is still real distillation, how instructions, long prompts and chain-of-thought rationales are distilled, and how DeepSeek-R1-style reasoning traces are filtered so that a small model learns good reasoning and not lucky guesses.
What you will learn
- Why SFT on teacher samples is a Monte Carlo estimate of forward-KL distillation, how its error falls with the number of samples, and what top-k logprobs add
- How instruction-following data is distilled from API models: Self-Instruct and Alpaca, Orca explanations, LaMini-LM and Lion, and the false promise of imitation
- Context distillation and in-context learning distillation: moving a long prompt into the weights, and when it pays off
- Distilling step-by-step: rationales as a second training task, short answers at test time
- Fine-tune-CoT and R1-style reasoning distillation with rejection sampling, plus the s1 data-selection approach
- Why an answer check lets flawed reasoning through, how to compute how often, and a practical filter stack
Why this matters
In session 119 the student could read the teacher’s full probability distribution at every position, and sometimes its hidden states. That is white-box distillation. It needs the teacher’s weights, or at least its logits. Very often you do not have them. The strongest teachers are behind an API: you send a prompt and you get back text. Sometimes you also get the log-probabilities of the few most likely tokens, but never the full distribution, the hidden states or the gradients.
Black-box distillation is distillation with only the teacher’s outputs. In its simplest form it is very plain: collect prompts, let the teacher answer, throw away bad answers, and fine-tune the student on what is left with the normal SFT loss (session 54). The survey of Zhou et al. (Section 5.2.4) defines black-box KD exactly this way: methods that “only use the final results obtained by the teacher models” [Certain], and it sorts them into three groups: distilling in-context learning, distilling chain-of-thought reasoning, and distilling instruction following [Certain]. This session follows that map and adds the newest member: distilling long reasoning traces into small models, as DeepSeek-R1 did.
This plain recipe built many well-known models. Stanford Alpaca was LLaMA 7B fine-tuned on 52K answers written by text-davinci-003, for less than $500 of API calls [Certain]. Orca (13B) learned from explanations written by ChatGPT and GPT-4 [Certain]. The six DeepSeek-R1-Distill models (1.5B to 70B) are only SFT on about 800k samples written by DeepSeek-R1 models, with no RL [Certain]. A 32B student trained this way beat RL on the same 32B base by 25.6 points on AIME 2024 [Certain].
Black-box distillation is also a key tool for inference cost. A small model that does one task well is cheaper per token than a large general model. A model that has internalized a long system prompt does not pay for that prompt on every request. And a student that learned from rationales can answer directly, without writing the rationale. These are three different ways to buy speed, and we will compute each one.
What a black box lets you see
Write the teacher as \(p(y \mid x)\) and the student as \(q_\theta(y \mid x)\), where \(x\) is the prompt and \(y\) a whole answer. The ideal goal is the same as in white-box distillation: make the student’s distribution close to the teacher’s. With forward KL over whole sequences:
- Read the symbols
- \(p(y \mid x)\)
- the teacher’s probability of the whole answer \(y\): the product of its next-token probabilities
- \(q_\theta(y \mid x)\)
- the student’s probability of the same answer; \(\theta\) are the student’s weights
- \(H(p)\)
- the entropy of the teacher; it does not depend on \(\theta\), so we can ignore it when we train
- \(\E_{y \sim p}\)
- an average over answers drawn from the teacher
The important fact is in the last term: to minimize the forward KL, we only need samples from the teacher, not its probabilities. Draw \(K\) answers per prompt and average:
This is the ordinary SFT loss on teacher text. So “SFT on teacher outputs” is not a heuristic: it is a Monte Carlo estimate of forward-KL distillation [Certain]. This is sequence-level KD (Kim and Rush) with sampling instead of beam search, as we saw in session 59. Two consequences follow. First, it works across tokenizers: the student re-tokenizes the text with its own tokenizer. Second, it is noisy: one sample tells the student “this answer is possible”, but not how likely it is, nor what the other good answers were.
How noisy? Think of one position with \(V\) possible answers. With \(K\) samples, the counts give an estimate \(\hat{p}\) of \(p\). A classic result of statistics (the chi-square limit of the log-likelihood ratio) gives, for large \(K\):
- Read the symbols
- \(\hat{p}_K\)
- the distribution estimated from \(K\) teacher samples (counts divided by \(K\), with a little smoothing so that no probability is zero)
- \(V_{\text{eff}}\)
- the number of answers that have real probability; for an LLM token it is much smaller than the vocabulary, because the distribution is peaked
The error falls only like \(1/K\): to halve it you must pay for twice as many samples. Many APIs also return the log-probabilities of the top few tokens at each position (for example up to 20 per position [Likely]). This is a partial white box. A simple way to use it is to keep the top-\(k\) probabilities exactly and spread the missing mass over the rest:
and to train with soft cross-entropy on \(\tilde{p}\) at every position, as in white-box KD. Because LLM distributions are peaked, a small \(k\) already holds most of the mass.
A teacher has six possible replies with probabilities \(p = (0.42, 0.23, 0.14, 0.09, 0.07, 0.05)\), entropy 1.530 nats. With top-2 logprobs, (120.4) gives \(\tilde{p} = (0.42, 0.23, 0.0875, 0.0875, 0.0875, 0.0875)\) and \(\KL(p \,\|\, \tilde{p}) = 0.0247\). With top-1 the KL is 0.0835, with top-3 it is 0.0058.
How many plain text samples give the same error? From (120.3) with \(V = 6\): \(K \approx 5 / (2 \times 0.0247) \approx 101\). So in this example one call with top-2 logprobs carries about as much information as 100 sampled answers. Simulation agrees with (120.3) once \(K\) is large: averaged over 200 draws with add-half smoothing, the KL is 0.0399 at \(K = 64\) (formula: 0.0391) and 0.0089 at \(K = 256\) (formula: 0.0098). At small \(K\) the smoothing dominates and the formula does not apply.
How to read Figure 2
Grey bars are the true teacher, which the student never sees. Colored bars are the best the student can learn from what the API returned. In text mode, move \(K\) from 1 to 256 and press “Draw again” a few times: at small \(K\) the bars jump around and rare replies get almost nothing; the readout shows the KL of this draw, the average over 200 draws, and formula (120.3). In top-\(k\) mode there is no randomness: the readout shows the KL and how many text samples would give the same error.
The same comparison with real training is in the PyTorch code below: a student trained on 64 smoothed samples per prompt reaches about the same KL as one trained on a single call with top-5 logprobs. In practice, people use plain text far more often, for three reasons: many APIs give no logprobs for reasoning tokens, logprobs are per token of the teacher’s tokenizer (hard to match to the student’s tokens), and the main value is often not the distribution but the content: explanations, rationales and reasoning that the student could not produce on its own.
Distilling instruction following
The first big wave of black-box distillation (2023) taught base models to follow instructions. The hard part was not the teacher’s answers; it was the prompts. Human-written instructions are expensive, so the teacher writes them too. Self-Instruct (Wang et al.) starts from 175 human-written seed tasks: the model is shown a few seeds and asked for new instructions, then for answers; near-duplicates and invalid items are filtered out, and the pool grows [Certain]. Alpaca applied this with text-davinci-003 as the teacher: 52K demonstrations from the 175 seeds, under $500 for data, then fine-tuning LLaMA 7B for 3 hours on 8 A100 80GB GPUs, under $100 [Certain].
- Read the symbols
- \(G\)
- the instruction generator: usually the teacher itself, prompted with a few seed tasks and examples from the pool so far
- \(y_j \sim p\)
- the teacher’s answer to the new instruction
- \(\text{keep}\)
- filters: remove near-duplicates (Self-Instruct uses ROUGE-L overlap with the pool), empty or too long items, and tasks the model cannot do with text
Later work changed what the teacher writes and which prompts it gets:
- Explanations, not only answers (Orca). Orca asks the teacher to explain: each FLAN-v2 query is sent with one of 16 hand-written system messages, such as “You are an AI assistant. Provide a detailed answer so user don’t need to search outside to understand the answer.” [Certain]. The 13B student first learns from 5M ChatGPT answers, then from 1M GPT-4 answers (about 1.5× longer): ChatGPT acts as a teacher assistant that closes part of the capacity gap [Certain]. Orca reports more than 100% better than Vicuna-13B on Big-Bench Hard and 42% better on AGIEval [Certain].
- Diversity at scale (LaMini-LM). 2.58M instructions from existing and newly generated sources, answered by gpt-3.5-turbo, used to train a herd of small encoder-decoder and decoder-only models [Certain].
- Hard prompts (Lion). Adversarial distillation: the teacher judges where the student’s answer is worse than its own, and generates new hard instructions like those. A loop of imitation, discrimination and generation with only 70k training samples; Lion-13B beats Vicuna-13B by 55.4% on BBH and 16.7% on AGIEval [Certain].
- input: teacher API \(p\), seed tasks, student \(q_\theta\), target size \(n\)
- while \(|\mathcal{D}| \lt n\):
- sample a few seeds and pool items; ask the teacher for new instructions // \(G\) in (120.5)
- drop near-duplicates (overlap with the pool), invalid and unsafe instructions
- ask the teacher to answer, with a system message that asks for explanations // Orca style
- filter answers: format, length, refusals, judge score; add the rest to \(\mathcal{D}\)
- (optional, Lion) find items where the student is worse than the teacher; generate more like them
- fine-tune the student on \(\mathcal{D}\) with the SFT loss (120.2), prompt tokens masked, in the student’s chat template
The false promise: style is easy, knowledge is not
Gudibande et al. tested this idea hard. They fine-tuned imitation models of ChatGPT on bases from 1.5B to 13B parameters, with 0.3M to 150M tokens of imitation data [Certain]. Crowd workers rated the outputs as competitive with ChatGPT. But on targeted benchmarks the imitation models closed little to none of the gap from their base model to ChatGPT, on tasks not heavily covered in the imitation data [Certain]. The models were “adept at mimicking ChatGPT’s style but not its factuality” [Certain]. Their conclusion: the best lever is a better base model, not more imitation [Certain].
This fits (120.2): the loss rewards the student for giving high probability to the teacher’s tokens. Tone, format and confident structure are easy to copy and appear in every answer. Facts the student does not know cannot be learned from a few thousand answers; the student learns to sound as if it knew them, which is a recipe for hallucination. Black-box distillation works best when the knowledge is already in the student, or the task is narrow enough that the data covers it.
Distilling in-context learning and long prompts
Many production prompts are long: a system prompt with rules, a persona, tool descriptions, and a few worked examples (few-shot demonstrations, see session 143). The model then does the task well, but every request pays for those tokens in prefill compute and in KV-cache memory (session 80). Compressing the prompt is one fix (session 88). Another is to move the prompt into the weights.
Context distillation (Snell, Klein and Zhong) does this with the model as its own teacher. The teacher is the model with the context \(C\) (instructions, examples, even a scratch pad). The student is the same model without it. On synthetic task inputs, the teacher answers with the context; the student is fine-tuned to give the same final answer from the input alone [Certain]:
- Read the symbols
- \(C\)
- the context to internalize: a system prompt, few-shot examples, or step-by-step instructions
- \(p(\cdot \mid C, x)\)
- the teacher: the original model reading the context and the input
- \(q_\theta(\cdot \mid x)\)
- the student: usually initialized from the same model, but it never sees \(C\)
The paper reports that it works for abstract instructions, for step-by-step reasoning (8-digit addition), and for concrete examples; on SPIDER text-to-SQL it beat direct gradient descent on the same examples by 9% [Certain]. Since the teacher is the same model, it can return logits, so (120.6) can also use a token-level KL; the sampled form shown here works even when the teacher is an API.
A related idea transfers the ability to learn in context, not one fixed context. In-context learning distillation (Huang et al.) trains a small model on few-shot prompts with targets from a large teacher, combining an in-context learning objective with a language-modeling objective, in two settings: Meta-ICT and Multitask-ICT [Certain]. Multitask-ICT worked better for multitask few-shot learning, at more compute, and the two objectives were complementary [Certain]:
(The exact form of each term differs per setting in the paper; this is the structure [Likely].)
When does internalizing a prompt pay?
Let the context have \(C\) tokens, the user input \(u\) tokens and the output \(o\) tokens. A forward pass costs about \(2N\) FLOPs per token for a model with \(N\) parameters. Removing the context saves \(2NC\) per request. Distilling costs, once, \(M\) teacher calls over \(C + u + o\) tokens plus training on \(u + o\) tokens for \(E\) epochs at about \(6N\) per token. The break-even number of requests is:
\(N\) cancels: the break-even does not depend on the model size (when teacher and student are the same model). It ignores the attention cost, which grows with context length and makes the saving even larger for long contexts.
Llama-3.1-8B has 32 layers, 8 KV heads of size 128, hidden size 4096, MLP width 14336 and a vocabulary of 128,256, which gives 8,030,261,248 parameters. Its KV cache in bf16 is \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes (128 KiB) per token.
With \(C = 1500\), \(u = 200\), \(o = 300\): a request costs \(2N \times 2000 = 3.21 \times 10^{13}\) FLOPs with the prompt and \(8.03 \times 10^{12}\) without it, a saving of 75%. The context alone holds \(1500 \times 128\) KiB \(= 187.5\) MiB of KV cache per request (prefix caching can share it across requests, but only on the same server and while it stays in memory). With \(M = 20{,}000\) training examples and \(E = 2\) epochs, (120.8) gives \(R^{*} = 20{,}000 \times (2000 + 3000) / 1500 \approx 66{,}667\) requests. A service with a million requests a day pays this back in under two hours of traffic.
How to read Figure 3
The saving per request is the blue part of the top bar. Make the context long and the answers short (a classifier or router with many rules) and the saving is close to 100%, with a small \(R^{*}\). Make the outputs long and the saving shrinks. Raising \(M\) or \(E\) raises the one-time cost and so \(R^{*}\), but not the saving per request.
- input: model \(p\), context \(C\) (system prompt, rules, few-shot examples), a source of task inputs
- generate \(M\) diverse task inputs \(x\) (from logs, or ask the model to invent them)
- for each \(x\): teacher answers \(y \sim p(\cdot \mid C, x)\); if \(C\) asks for a scratch pad, keep only the final answer
- filter: drop refusals, wrong formats, failed checks
- student \(q_\theta \leftarrow p\); fine-tune on pairs \((x, y)\) with loss (120.6), no \(C\) in the input // LoRA is enough for one prompt
- evaluate the student without \(C\) against the teacher with \(C\) on held-out inputs
Distilling chain-of-thought: Distilling step-by-step
Chain-of-thought prompting (session 72) makes a big model write a rationale (the steps of the reasoning) before the answer. The rationale is useful for distillation: it tells the student why the answer is right, not only what it is. But a student that must write a long rationale at test time is slow: every rationale token is a decode step. Distilling step-by-step (Hsieh et al.) gets the benefit of rationales in training without paying for them at inference.
The recipe: prompt the teacher (540B PaLM) with about 10 few-shot chain-of-thought examples per task, each a triplet (input, rationale, label); on every training input the teacher writes a rationale \(\hat{r}\) and a label \(\hat{y}\) [Certain]. Then train the student (T5 at 220M, 770M and 11B) on two tasks, marked by prefixes in the input [Certain]:
- Read the symbols
- \(f\)
- the student model; \(\ell\) is the token-level cross-entropy over the target
- \(\texttt{[label]}\), \(\texttt{[rationale]}\)
- task prefixes: the same input with a different prefix asks for the label or for the rationale
- \(y_i\)
- the label: the human label if there is one, otherwise the teacher’s label \(\hat{y}_i\)
- \(\hat{r}_i\)
- the teacher’s rationale for input \(x_i\)
- \(\lambda\)
- the weight of the rationale task
At test time only the [label] task is used, so the student outputs a short answer, with no rationale and no extra decode cost [Certain]. The rationale task acts like an auxiliary loss: it shapes the shared encoder so that the label head can use the same reasoning features.
Why two tasks and not one target “rationale, then label”? The paper tested that single-task version too. It was worse, sometimes worse than no rationale at all, and the single-task student must also generate the rationale at test time:
| 220M T5, rationales from PaLM 540B | e-SNLI | ANLI | CQA | SVAMP |
|---|---|---|---|---|
| Standard fine-tuning (labels only) | 88.38 | 43.58 | 62.19 | 62.63 |
| Single-task: target = rationale + label | 88.88 | 43.50 | 61.37 | 63.00 |
| Multi-task (120.9) | 89.51 | 49.58 | 63.29 | 65.50 |
| Multi-task, rationales from GPT-NeoX 20B | 89.12 | 48.15 | 63.25 | 63.00 |
Numbers from the paper [Certain]. The multi-task gain over standard fine-tuning is 6.0 points on ANLI and 2.87 on SVAMP; a weaker teacher (20B) still helps, but less. The headline results: on e-SNLI, 12.5% of the training data was enough to beat standard fine-tuning on 100%; the 220M student (over 2000× smaller) beat few-shot PaLM 540B on e-SNLI; the 770M student beat few-shot PaLM on ANLI using 80% of the data, where standard fine-tuning could not match it with 100% [Certain].
- input: teacher API, about 10 hand-written (input, rationale, label) examples, inputs \(x_1..x_N\) (labeled or not)
- for each \(x_i\): prompt the teacher with the examples and \(x_i\); parse rationale \(\hat{r}_i\) and label \(\hat{y}_i\)
- if a human label \(y_i\) exists, use it as the label target; otherwise use \(\hat{y}_i\)
- build two training items per input: (
[label]\(x_i\) → \(y_i\)) and ([rationale]\(x_i\) → \(\hat{r}_i\)) - train the student on both with (120.9) // shared weights, mixed in every batch
- inference: feed
[label]\(x\) only // no rationale tokens to decode
Fine-tune-CoT: more rationales per question, and a filter
Ho et al. (“Large Language Models Are Reasoning Teachers”) distill rationales that the student does output. The teacher (text-davinci-002) is prompted zero-shot with “Let’s think step by step.”, and a sample is kept only if its final answer equals the gold answer [Certain]. Their diverse reasoning draws \(D\) different rationales per question at temperature 0.7 (\(D = 64\) for MultiArith and SVAMP, 8 for the others) [Certain]. The 6.7B student beat the 175B teacher on some tasks (Shuffled Objects 64.44 vs 50.22, Coin Flip 98.67 vs 92.67) but not on others (MultiArith 53.33 vs 78.89) [Certain]. They also warn about the filter: on Date Understanding, 27.6% of the teacher completions with a correct answer contained reasoning errors [Certain]. We now study that problem.
Reasoning distillation into small models, and filtering bad traces
Reasoning models such as DeepSeek-R1 write long traces (thousands of tokens) inside a think block before the answer. Session 59 showed the DeepSeek-R1 recipe briefly; here we look at the part that decides quality: which traces to keep. The general dataset is:
- Read the symbols
- \(r, a\)
- the reasoning trace and the final answer, sampled together from the teacher
- \(\text{verify}\)
- the outcome check: compare with the reference answer, run unit tests, or ask a judge model
- \(\text{check}_j\)
- other filters, each 1 (pass) or 0 (drop): format (think block closed), language, length, readability, duplicates, benchmark contamination
DeepSeek-R1 followed this pattern: for each prompt it sampled several responses and kept only correct ones, using rules or DeepSeek-V3 as a judge that sees the reference and the prediction; it filtered out traces with mixed languages, long paragraphs and code blocks; about 600k reasoning samples plus 200k non-reasoning samples [Certain]. The students (Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-14B, Qwen2.5-32B, Llama-3.1-8B, Llama-3.3-70B-Instruct) got “only SFT” [Certain].
The opposite end is s1: only 1,000 questions (s1K), chosen for difficulty, diversity and quality, with reasoning traces; Qwen2.5-32B-Instruct fine-tuned on them exceeds o1-preview on competition math by up to 27% [Certain]. s1 adds budget forcing: when the model tries to stop thinking, append “Wait” to make it continue; this lifted AIME24 from 50% to 57% [Certain]. LIMA (session 54) made the same point for chat: a small, carefully chosen set can be enough when the base model already has the knowledge. The filter matters more than the count.
How to read Figure 4
Compare the two 32B bars: same base, distillation versus RL. Then compare student sizes: on AIME the 7B student (55.5) already beats RL at 32B (47.0), and on MATH-500 the 7B student (92.8) is already above RL at 32B (91.6). Switch benchmarks to see that the gap is largest on AIME and LiveCodeBench, the hardest sets.
The answer check is not a reasoning check
The most common filter checks only the final answer. A trace with a wrong step can still end at the right answer: by luck, by a second error that cancels the first, or by a guess. Let \(s\) be the share of teacher traces whose reasoning is sound, \(a\) the chance that a sound trace ends with the right answer, and \(g\) the chance that an unsound trace still gives the right answer. Among the traces the answer filter keeps, the share that is sound (the precision of the filter) is, by Bayes’ rule:
The guess rate \(g\) depends on the answer format: near 0 for a free-form number, \(1/m\) for \(m\)-option multiple choice, 0.5 for yes/no. A step-level judge (a model or a process reward model, session 73) that passes sound traces with probability \(\rho\) and flawed ones with probability \(\phi\) gives:
Let 70% of teacher traces be sound (\(s = 0.7\)) and assume \(a = 1\). For free-form math answers (\(g = 0.02\)) the filter keeps 70.6% of traces and 99.15% of the kept ones are sound. For 4-option multiple choice (\(g = 0.25\)) it keeps 77.5% and only 90.3% are sound: almost one kept trace in ten teaches a wrong reasoning step. For yes/no (\(g = 0.5\)) it is 82.4%.
Add a judge with \(\rho = 0.9\), \(\phi = 0.3\) to the multiple-choice case: precision rises to 96.6%, and 65.3% of traces are kept. The judge throws away 10% of good traces to remove 70% of bad ones that survived the answer check. This is why Fine-tune-CoT found 27.6% flawed traces among correct ones on Date Understanding, a multiple-choice task.
How to read Figure 5
Each square is one trace. The answer filter cannot see the difference between green and orange squares: it keeps every orange trace that guessed right. Pick an answer format with the buttons, then lower \(s\) to model a weaker teacher: precision drops fast when \(g\) is large. Tick the judge and lower \(\phi\) to see how many orange squares a good step check removes, and how many green ones it costs.
A practical filter stack for reasoning traces, from cheap to expensive:
- Format. The think block is closed, the answer is in the expected place (for example a boxed number), the trace was not cut by the length limit. A cut trace teaches the student never to finish.
- Outcome. The answer matches the reference, or the code passes unit tests. This is the core filter, with the weakness of (120.11).
- Language and readability. Drop traces that switch language or are unreadable; DeepSeek-R1 dropped mixed languages, long paragraphs and code blocks [Certain].
- Length. Very long traces are expensive to train on and can teach the student to ramble; some pipelines keep the shortest correct trace per prompt [Likely].
- Duplicates and contamination. Hash-based exact dedup, n-gram near-dedup, and removal of anything close to the evaluation sets.
- Step check. A judge or a process reward model scores steps; slow but it attacks the false positives of (120.11) (session 73).
Two more ideas from the self-training family use the same filter. STaR generates rationales, and when the answer is wrong it tries again with the correct answer given as a hint (rationalization); it fine-tunes on all rationales that led to correct answers and repeats [Certain]. And rejection sampling has a hidden bias: prompts that the teacher rarely solves are kept less often, so the kept set is easier than the prompt set (session 59, formula 59.11). Sample more for hard prompts, or cap the traces per prompt.
- input: teacher reasoning model, prompts with checkable answers (math, code with tests, science), non-reasoning SFT data, student base model
- for each prompt \(x\): sample \(k\) traces \((r, a)\) at temperature about 0.6 to 1 // more samples for hard prompts
- drop traces that fail format, verify, language, length, dedup checks (120.10)
- (optional) score the rest with a step judge (120.12); keep at most one or two per prompt
- decontaminate against all evaluation sets; mix in non-reasoning data so that chat skills survive
- SFT the student on the result in the same think template it will use at inference; long context, packing with document masks, never cut a trace
- (optional) RL with verifiable rewards on top (session 70) to go beyond pure imitation
Why is distillation so much better than RL for the small model (Figure 4)? RL must discover good reasoning by trial and reward; a small model rarely finds a correct long solution, so the reward signal is sparse. Distillation gives it the correct long solution directly. DeepSeek’s conclusion was that distilling strong models into small ones gives excellent results, while small models relying on large-scale RL need huge compute and may not even reach distillation; going beyond the teacher still needs stronger bases and larger-scale RL [Certain]. The capacity gap still applies: very small students can learn better from shorter traces or from a mid-sized teacher (session 59).
Code: samples versus logprobs, and a trace filter
The first script is a tiny, deterministic version of Figure 2 with real training. A hidden teacher has 50 possible answers for each of 32 prompts. The student is a table of logits, trained with Adam on what each kind of API returns. We report the KL from the teacher to the trained student.
import torch torch.manual_seed(0) C, V = 32, 50 # 32 prompts, 50 possible answers each p = (2.5 * torch.randn(C, V)).softmax(-1) # the hidden teacher def api_samples(K, seed): # black box: K sampled answers per prompt g = torch.Generator().manual_seed(seed) return torch.multinomial(p, K, replacement=True, generator=g) # (C, K) def api_topk(k): # black box with top-k logprobs return p.topk(k, dim=-1) # values (C, k), indices (C, k) def fit(target): # target: (C, V) soft labels W = torch.zeros(C, V, requires_grad=True) opt = torch.optim.Adam([W], lr=0.1) for _ in range(400): loss = -(target * W.log_softmax(-1)).sum(-1).mean() opt.zero_grad(); loss.backward(); opt.step() q = W.detach().softmax(-1) return (p * (p.log() - q.log())).sum(-1).mean().item() # KL(p || q) def from_samples(K, eps=0.0): # SFT on text = one-hot targets, averaged y = api_samples(K, seed=K) t = torch.zeros(C, V).scatter_add_(1, y, torch.ones(C, K)) / K return (1 - eps) * t + eps / V # eps > 0: label smoothing def from_topk(k): # exact top-k, rest of the mass spread v, i = api_topk(k) rest = (1 - v.sum(-1, keepdim=True)) / (V - k) return torch.full((C, V), 0.0).add_(rest).scatter_(1, i, v) print("teacher entropy (nats):", round(-(p * p.log()).sum(-1).mean().item(), 3)) for K in [1, 4, 16, 64, 256]: print(f"text, K={K:3d} samples KL {fit(from_samples(K)):7.3f}" f" with smoothing 0.1 KL {fit(from_samples(K, 0.1)):6.3f}") for k in [1, 5, 20]: print(f"one call, top-{k:2d} logprobs KL {fit(from_topk(k)):6.3f}") print(f"full distribution (white-box) KL {fit(p):6.3f}")
teacher entropy (nats): 1.767 text, K= 1 samples KL 4.205 with smoothing 0.1 KL 2.117 text, K= 4 samples KL 2.451 with smoothing 0.1 KL 1.114 text, K= 16 samples KL 0.983 with smoothing 0.1 KL 0.413 text, K= 64 samples KL 0.360 with smoothing 0.1 KL 0.159 text, K=256 samples KL 0.098 with smoothing 0.1 KL 0.076 one call, top- 1 logprobs KL 0.763 one call, top- 5 logprobs KL 0.154 one call, top-20 logprobs KL 0.013 full distribution (white-box) KL 0.000
Plain samples overfit: with \(K = 1\), the student puts almost all its mass on one answer and the KL is 4.2 nats, more than the teacher’s entropy. Label smoothing halves it, and the error falls roughly like \(1/K\) as in (120.3). One call with top-5 logprobs (KL 0.154) is about as good as 64 smoothed samples (0.159), and top-20 (0.013) is almost white-box. (Here 50 answers are a toy “vocabulary”; for real LLMs the effective support per token is small, so top-k is even more effective.)
The second script is a minimal filter stack for reasoning traces, formula (120.10): format, answer, language, length and duplicates.
import re, hashlib def final_answer(trace): m = re.findall(r"\\boxed\{([^{}]*)\}", trace) return m[-1].strip() if m else None def keep(trace, gold, max_tokens=16000): reasons = [] if "</think>" not in trace: reasons.append("no end of thinking") # cut or wrong format if final_answer(trace) != gold: reasons.append("wrong or missing answer") letters = [c for c in trace if c.isalpha()] if letters and sum(not c.isascii() for c in letters) / len(letters) > 0.05: reasons.append("mixed language") # English task, other script if len(trace.split()) > max_tokens: # crude token count reasons.append("too long") return reasons seen = set() def dedup(prompt, trace): h = hashlib.sha1((prompt + trace).encode()).hexdigest() if h in seen: return False seen.add(h); return True gold = "12" traces = [ "<think>3 boxes of 4 apples: 3*4 = 12.</think> The answer is \\boxed{12}", "<think>3 boxes of 4 apples: 3+4 = 7, hmm, so 12?</think> \\boxed{12}", "<think>3 boxes, 4 apples each, 3*4 = 13.</think> \\boxed{13}", "<think>每盒4个, 3盒, 3*4 = 12.</think> \\boxed{12}", "<think>3 boxes of 4 apples: 3*4 = 12. Let me check again: 3*4", "<think>3 boxes of 4 apples: 3*4 = 12.</think> The answer is \\boxed{12}", ] for t in traces: r = keep(t, gold) if not r and not dedup("Q1", t): r = ["duplicate"] print("KEEP" if not r else "drop", "|", ", ".join(r) or "-", "|", t[:44])
KEEP | - | <think>3 boxes of 4 apples: 3*4 = 12.</think
KEEP | - | <think>3 boxes of 4 apples: 3+4 = 7, hmm, so
drop | wrong or missing answer | <think>3 boxes, 4 apples each, 3*4 = 13.</th
drop | mixed language | <think>每盒4个, 3盒, 3*4 = 12.</think> \boxed{12
drop | no end of thinking, wrong or missing answer | <think>3 boxes of 4 apples: 3*4 = 12. Let me
drop | duplicate | <think>3 boxes of 4 apples: 3*4 = 12.</think
Look at the second line: “3+4 = 7, hmm, so 12?” has a wrong step and a correct boxed answer, and it passes every check. That is the false positive of (120.11). Only a step judge would catch it.
Where this is used
| Setting | Black-box method | Why |
|---|---|---|
| A small chat or instruction model from an API teacher | Self-Instruct style data (120.5), explanation system messages (Orca), hard-prompt mining (Lion) | No access to logits; text is all you need for SFT |
| A narrow production task (classification, extraction, routing) | Distilling step-by-step (120.9): rationales in training, short labels at test time | Small model, short outputs, better accuracy per labeled example |
| A long system prompt or many few-shot examples on every request | Context distillation (120.6), break-even (120.8) | Removes prefill compute and KV memory of the prompt |
| Small reasoning models (1.5B to 70B) | R1-style rejection sampling plus SFT (120.10), maybe RL after | Beats RL on the same small base at a fraction of the cost |
| A data-efficient reasoning model | s1-style selection of about 1,000 hard, diverse, high-quality traces | Quality of the filter matters more than the count |
| Video captioning and video QA models | A strong VLM writes dense captions and QA pairs for clips; a video model is trained on them (LLaVA-Video-178K is synthetic data of this kind) | Human video annotation is very expensive; captions from a teacher are text, so black-box works [Likely] |
| Video retrieval, key-moment detection and Video-RAG | Teacher writes captions or timestamps of key moments; a small model or text embedder is trained to match them | Index long archives cheaply with a small model at ingestion time [Likely] |
| Draft models for speculative decoding | Train the draft on the target model’s own generated text | A draft that imitates the target is accepted more often (session 128) [Likely] |
Common mistakes
- Expecting knowledge to transfer through style. Imitation data teaches format and tone fast; facts the student lacks do not appear. Human raters are fooled, targeted benchmarks are not (the false promise).
- Trusting the answer check. Correct answer does not mean correct reasoning, especially for multiple choice or yes/no (120.11). Add a step judge, or prefer free-form answers when you build the prompt set.
- Training on cut traces. If the teacher hit its length limit, the trace has no answer and no end of thinking. Keeping it teaches the student to never finish.
- One sample per prompt with no smoothing. The student becomes over-confident on whatever the teacher said once. Use several samples, label smoothing, or top-k logprobs if available (120.4).
- Mismatch of chat template or think format. The student must be trained in the template it will be served with (session 9); a teacher’s tags copied into the wrong template break stopping and parsing.
- Contaminated prompts. Prompts generated by a teacher, or collected from the web, often contain benchmark questions. Decontaminate before training, not after the scores look too good.
- Ignoring the terms of use. Some API terms forbid training competing models on outputs [Likely]. Check before spending money on data.
- A student too small for the traces. Very long traces from a huge teacher can be worse for a 1B student than shorter ones from a mid-sized teacher (the capacity gap).
Interview questions
You only have API access to a strong model. How do you build a small model for your task?basic
Why is SFT on teacher samples a form of distillation and not just data augmentation?core
What do top-k logprobs buy you, and why are they often not used?core
Explain Distilling step-by-step. Why does the student not output the rationale?core
What is context distillation and when does it pay off?core
Why did imitation models of ChatGPT look good to raters but fail benchmarks?core
Your answer-checked reasoning dataset still teaches wrong steps. Why, and how much?advanced
Why did distillation beat RL for Qwen-32B in DeepSeek-R1?advanced
How do s1 and DeepSeek-R1 differ in data strategy?advanced
What is the bias of rejection sampling, and how do you fix it?advanced
Exercises
- The teacher of Figure 2 returns top-2 logprobs. Write \(\tilde{p}\) from (120.4) and compute \(\KL(p \,\|\, \tilde{p})\).
- With \(V_{\text{eff}} = 6\), how many text samples does (120.3) need for an expected KL of 0.01?
- Context distillation: \(C = 3000\), \(u = 100\), \(o = 400\), \(M = 50{,}000\), \(E = 1\). Compute \(R^{*}\) from (120.8).
- A yes/no task with \(s = 0.5\) and \(a = 1\). What is the precision of the answer filter? And with a judge, \(\rho = 0.9\), \(\phi = 0.3\)?
- A teacher solves a hard prompt with probability 0.2 per sample. With \(k = 8\) samples, what is the chance the prompt is kept, and how many correct traces do you expect?
- You sample 50,000 prompts × 8 traces × 6,000 output tokens. At a hypothetical price of $1 per million output tokens, what does it cost? If 40% of the traces survive the filters, what is the cost per kept trace?
Answerscheck
Key takeaways
- Black-box distillation uses only the teacher’s outputs. SFT on teacher samples is a Monte Carlo estimate of forward-KL distillation (120.1, 120.2); its error falls like \(1/K\) (120.3). Top-k logprobs, if available, are worth many samples (120.4).
- Instruction-following distillation needs prompts as much as answers: Self-Instruct and Alpaca generate them (120.5), Orca asks for explanations, Lion mines hard prompts. Imitation copies style easily, knowledge hardly.
- Context distillation moves a long prompt into the weights (120.6); it saves the prompt’s prefill and KV cache on every request, with a break-even (120.8) that does not depend on model size.
- Distilling step-by-step trains on rationales as a second task (120.9) and answers without them at test time: better accuracy with less data and no decode cost.
- Reasoning distillation (R1 style) is rejection sampling plus SFT (120.10). It beat RL on the same 32B base (72.6 vs 47.0 on AIME 2024). The filter decides the quality: an answer check passes flawed traces at the rate of (120.11); step judges help (120.12).
Key formulas
Next, session 121 keeps one model but lets easy tokens and easy inputs use fewer layers: another way to spend less compute where the task is simple.
References for this session
- Hsieh et al. (2023). Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
- DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Kim & Rush (2016). Sequence-Level Knowledge Distillation
- Wang et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions
- Taori et al. (2023). Alpaca: A Strong, Replicable Instruction-Following Model
- Mukherjee, Mitra, Jawahar et al. (2023). Orca: Progressive Learning from Complex Explanation Traces of GPT-4
- Wu, Waheed, Zhang et al. (2023). LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions
- Chen et al. (2023). Symbolic Discovery of Optimization Algorithms (Lion)
- Gudibande, Wallace, Snell et al. (2023). The False Promise of Imitating Proprietary LLMs
- Snell, Klein and Zhong (2022). Learning by Distilling Context
- Huang, Chen, Yu and McKeown (2022). In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models
- Ho, Schmid and Yun (2022). Large Language Models Are Reasoning Teachers
- Zelikman, Wu, Mu and Goodman (2022). STaR: Bootstrapping Reasoning With Reasoning
- Muennighoff, Yang, Shi et al. (2025). s1: Simple test-time scaling
- Zhou et al. (2023). LIMA: Less Is More for Alignment
- Zhang et al. (2024). Video Instruction Tuning With Synthetic Data
- Agarwal et al. (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD)