LLM & VLM courseBlog
120Track 5: LLM inference and serving · session 43 of 65

Distillation for LLMs 2: black-box (outputs only)

Often the best teacher is behind an API and you only see the text it writes. This session shows why fine-tuning on teacher text is still real distillation, how instructions, long prompts and chain-of-thought rationales are distilled, and how DeepSeek-R1-style reasoning traces are filtered so that a small model learns good reasoning and not lucky guesses.

Time: 2.5 to 3.5 hours (reading 2 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 59 and 119 (knowledge distillation, white-box distillation), session 54 (SFT), basic probability (KL divergence, Bayes' rule)

What you will learn

  • Why SFT on teacher samples is a Monte Carlo estimate of forward-KL distillation, how its error falls with the number of samples, and what top-k logprobs add
  • How instruction-following data is distilled from API models: Self-Instruct and Alpaca, Orca explanations, LaMini-LM and Lion, and the false promise of imitation
  • Context distillation and in-context learning distillation: moving a long prompt into the weights, and when it pays off
  • Distilling step-by-step: rationales as a second training task, short answers at test time
  • Fine-tune-CoT and R1-style reasoning distillation with rejection sampling, plus the s1 data-selection approach
  • Why an answer check lets flawed reasoning through, how to compute how often, and a practical filter stack

Why this matters

In session 119 the student could read the teacher’s full probability distribution at every position, and sometimes its hidden states. That is white-box distillation. It needs the teacher’s weights, or at least its logits. Very often you do not have them. The strongest teachers are behind an API: you send a prompt and you get back text. Sometimes you also get the log-probabilities of the few most likely tokens, but never the full distribution, the hidden states or the gradients.

Black-box distillation is distillation with only the teacher’s outputs. In its simplest form it is very plain: collect prompts, let the teacher answer, throw away bad answers, and fine-tune the student on what is left with the normal SFT loss (session 54). The survey of Zhou et al. (Section 5.2.4) defines black-box KD exactly this way: methods that “only use the final results obtained by the teacher models” [Certain], and it sorts them into three groups: distilling in-context learning, distilling chain-of-thought reasoning, and distilling instruction following [Certain]. This session follows that map and adds the newest member: distilling long reasoning traces into small models, as DeepSeek-R1 did.

This plain recipe built many well-known models. Stanford Alpaca was LLaMA 7B fine-tuned on 52K answers written by text-davinci-003, for less than $500 of API calls [Certain]. Orca (13B) learned from explanations written by ChatGPT and GPT-4 [Certain]. The six DeepSeek-R1-Distill models (1.5B to 70B) are only SFT on about 800k samples written by DeepSeek-R1 models, with no RL [Certain]. A 32B student trained this way beat RL on the same 32B base by 25.6 points on AIME 2024 [Certain].

Black-box distillation is also a key tool for inference cost. A small model that does one task well is cheaper per token than a large general model. A model that has internalized a long system prompt does not pay for that prompt on every request. And a student that learned from rationales can answer directly, without writing the rationale. These are three different ways to buy speed, and we will compute each one.

Figure 1. The black-box distillation pipeline. Only text crosses the API boundary (dashed). Every box can be changed: where the prompts come from (human, seed tasks, generated by the teacher), what the teacher is asked to write (answer, explanation, rationale, long reasoning trace), which filter is used (answer check, unit tests, judge model, format, length), and how the student is trained (plain SFT, multi-task SFT, SFT then RL).

What a black box lets you see

Write the teacher as \(p(y \mid x)\) and the student as \(q_\theta(y \mid x)\), where \(x\) is the prompt and \(y\) a whole answer. The ideal goal is the same as in white-box distillation: make the student’s distribution close to the teacher’s. With forward KL over whole sequences:

\[ \KL\big(p \,\|\, q_\theta\big) = \E_{y \sim p(\cdot \mid x)}\big[\log p(y \mid x) - \log q_\theta(y \mid x)\big] = -H(p) - \E_{y \sim p}\big[\log q_\theta(y \mid x)\big] \tag{120.1} \]
Read the symbols
\(p(y \mid x)\)
the teacher’s probability of the whole answer \(y\): the product of its next-token probabilities
\(q_\theta(y \mid x)\)
the student’s probability of the same answer; \(\theta\) are the student’s weights
\(H(p)\)
the entropy of the teacher; it does not depend on \(\theta\), so we can ignore it when we train
\(\E_{y \sim p}\)
an average over answers drawn from the teacher

The important fact is in the last term: to minimize the forward KL, we only need samples from the teacher, not its probabilities. Draw \(K\) answers per prompt and average:

\[ \mathcal{L}(\theta) = -\frac{1}{K}\sum_{k=1}^{K} \log q_\theta\big(y^{(k)} \mid x\big), \qquad y^{(k)} \sim p(\cdot \mid x) \tag{120.2} \]

This is the ordinary SFT loss on teacher text. So “SFT on teacher outputs” is not a heuristic: it is a Monte Carlo estimate of forward-KL distillation [Certain]. This is sequence-level KD (Kim and Rush) with sampling instead of beam search, as we saw in session 59. Two consequences follow. First, it works across tokenizers: the student re-tokenizes the text with its own tokenizer. Second, it is noisy: one sample tells the student “this answer is possible”, but not how likely it is, nor what the other good answers were.

How noisy? Think of one position with \(V\) possible answers. With \(K\) samples, the counts give an estimate \(\hat{p}\) of \(p\). A classic result of statistics (the chi-square limit of the log-likelihood ratio) gives, for large \(K\):

\[ \E\big[\KL(p \,\|\, \hat{p}_K)\big] \;\approx\; \frac{V_{\text{eff}} - 1}{2K} \tag{120.3} \]
Read the symbols
\(\hat{p}_K\)
the distribution estimated from \(K\) teacher samples (counts divided by \(K\), with a little smoothing so that no probability is zero)
\(V_{\text{eff}}\)
the number of answers that have real probability; for an LLM token it is much smaller than the vocabulary, because the distribution is peaked

The error falls only like \(1/K\): to halve it you must pay for twice as many samples. Many APIs also return the log-probabilities of the top few tokens at each position (for example up to 20 per position [Likely]). This is a partial white box. A simple way to use it is to keep the top-\(k\) probabilities exactly and spread the missing mass over the rest:

\[ \tilde{p}(v) = \begin{cases} p(v) & v \in \text{top-}k \\[2pt] \dfrac{1 - \sum_{u \in \text{top-}k} p(u)}{V - k} & \text{otherwise} \end{cases} \tag{120.4} \]

and to train with soft cross-entropy on \(\tilde{p}\) at every position, as in white-box KD. Because LLM distributions are peaked, a small \(k\) already holds most of the mass.

Worked example: one call with logprobs against many samples

A teacher has six possible replies with probabilities \(p = (0.42, 0.23, 0.14, 0.09, 0.07, 0.05)\), entropy 1.530 nats. With top-2 logprobs, (120.4) gives \(\tilde{p} = (0.42, 0.23, 0.0875, 0.0875, 0.0875, 0.0875)\) and \(\KL(p \,\|\, \tilde{p}) = 0.0247\). With top-1 the KL is 0.0835, with top-3 it is 0.0058.

How many plain text samples give the same error? From (120.3) with \(V = 6\): \(K \approx 5 / (2 \times 0.0247) \approx 101\). So in this example one call with top-2 logprobs carries about as much information as 100 sampled answers. Simulation agrees with (120.3) once \(K\) is large: averaged over 200 draws with add-half smoothing, the KL is 0.0399 at \(K = 64\) (formula: 0.0391) and 0.0089 at \(K = 256\) (formula: 0.0098). At small \(K\) the smoothing dominates and the formula does not apply.

teacher \(p\) (hidden)what the student can recover
Figure 2. The six-reply teacher of the worked example. In “Text samples only” the student sees \(K\) sampled replies and the recovered distribution is the smoothed count (add one half to each count). In “Top-k logprobs” it sees one call with the top-\(k\) probabilities, filled in with (120.4). The samples are drawn with a seeded random generator.
How to read Figure 2

Grey bars are the true teacher, which the student never sees. Colored bars are the best the student can learn from what the API returned. In text mode, move \(K\) from 1 to 256 and press “Draw again” a few times: at small \(K\) the bars jump around and rare replies get almost nothing; the readout shows the KL of this draw, the average over 200 draws, and formula (120.3). In top-\(k\) mode there is no randomness: the readout shows the KL and how many text samples would give the same error.

The same comparison with real training is in the PyTorch code below: a student trained on 64 smoothed samples per prompt reaches about the same KL as one trained on a single call with top-5 logprobs. In practice, people use plain text far more often, for three reasons: many APIs give no logprobs for reasoning tokens, logprobs are per token of the teacher’s tokenizer (hard to match to the student’s tokens), and the main value is often not the distribution but the content: explanations, rationales and reasoning that the student could not produce on its own.

Distilling instruction following

The first big wave of black-box distillation (2023) taught base models to follow instructions. The hard part was not the teacher’s answers; it was the prompts. Human-written instructions are expensive, so the teacher writes them too. Self-Instruct (Wang et al.) starts from 175 human-written seed tasks: the model is shown a few seeds and asked for new instructions, then for answers; near-duplicates and invalid items are filtered out, and the pool grows [Certain]. Alpaca applied this with text-davinci-003 as the teacher: 52K demonstrations from the 175 seeds, under $500 for data, then fine-tuning LLaMA 7B for 3 hours on 8 A100 80GB GPUs, under $100 [Certain].

\[ \mathcal{D} = \Big\{ (x_j, y_j) \;:\; x_j \sim G\big(\cdot \mid \text{seeds}, \mathcal{D}\big),\;\; y_j \sim p(\cdot \mid x_j),\;\; \text{keep}(x_j, y_j) = 1 \Big\} \tag{120.5} \]
Read the symbols
\(G\)
the instruction generator: usually the teacher itself, prompted with a few seed tasks and examples from the pool so far
\(y_j \sim p\)
the teacher’s answer to the new instruction
\(\text{keep}\)
filters: remove near-duplicates (Self-Instruct uses ROUGE-L overlap with the pool), empty or too long items, and tasks the model cannot do with text

Later work changed what the teacher writes and which prompts it gets:

  • Explanations, not only answers (Orca). Orca asks the teacher to explain: each FLAN-v2 query is sent with one of 16 hand-written system messages, such as “You are an AI assistant. Provide a detailed answer so user don’t need to search outside to understand the answer.” [Certain]. The 13B student first learns from 5M ChatGPT answers, then from 1M GPT-4 answers (about 1.5× longer): ChatGPT acts as a teacher assistant that closes part of the capacity gap [Certain]. Orca reports more than 100% better than Vicuna-13B on Big-Bench Hard and 42% better on AGIEval [Certain].
  • Diversity at scale (LaMini-LM). 2.58M instructions from existing and newly generated sources, answered by gpt-3.5-turbo, used to train a herd of small encoder-decoder and decoder-only models [Certain].
  • Hard prompts (Lion). Adversarial distillation: the teacher judges where the student’s answer is worse than its own, and generates new hard instructions like those. A loop of imitation, discrimination and generation with only 70k training samples; Lion-13B beats Vicuna-13B by 55.4% on BBH and 16.7% on AGIEval [Certain].
Algorithm 1: Black-box instruction distillation
  1. input: teacher API \(p\), seed tasks, student \(q_\theta\), target size \(n\)
  2. while \(|\mathcal{D}| \lt n\):
  3.   sample a few seeds and pool items; ask the teacher for new instructions // \(G\) in (120.5)
  4.   drop near-duplicates (overlap with the pool), invalid and unsafe instructions
  5.   ask the teacher to answer, with a system message that asks for explanations // Orca style
  6.   filter answers: format, length, refusals, judge score; add the rest to \(\mathcal{D}\)
  7.   (optional, Lion) find items where the student is worse than the teacher; generate more like them
  8. fine-tune the student on \(\mathcal{D}\) with the SFT loss (120.2), prompt tokens masked, in the student’s chat template

The false promise: style is easy, knowledge is not

Gudibande et al. tested this idea hard. They fine-tuned imitation models of ChatGPT on bases from 1.5B to 13B parameters, with 0.3M to 150M tokens of imitation data [Certain]. Crowd workers rated the outputs as competitive with ChatGPT. But on targeted benchmarks the imitation models closed little to none of the gap from their base model to ChatGPT, on tasks not heavily covered in the imitation data [Certain]. The models were “adept at mimicking ChatGPT’s style but not its factuality” [Certain]. Their conclusion: the best lever is a better base model, not more imitation [Certain].

This fits (120.2): the loss rewards the student for giving high probability to the teacher’s tokens. Tone, format and confident structure are easy to copy and appear in every answer. Facts the student does not know cannot be learned from a few thousand answers; the student learns to sound as if it knew them, which is a recipe for hallucination. Black-box distillation works best when the knowledge is already in the student, or the task is narrow enough that the data covers it.

Distilling in-context learning and long prompts

Many production prompts are long: a system prompt with rules, a persona, tool descriptions, and a few worked examples (few-shot demonstrations, see session 143). The model then does the task well, but every request pays for those tokens in prefill compute and in KV-cache memory (session 80). Compressing the prompt is one fix (session 88). Another is to move the prompt into the weights.

Context distillation (Snell, Klein and Zhong) does this with the model as its own teacher. The teacher is the model with the context \(C\) (instructions, examples, even a scratch pad). The student is the same model without it. On synthetic task inputs, the teacher answers with the context; the student is fine-tuned to give the same final answer from the input alone [Certain]:

\[ \mathcal{L}_{\text{ctx}}(\theta) = \E_{x}\ \E_{y \sim p(\cdot \mid C,\, x)}\big[-\log q_\theta(y \mid x)\big] \tag{120.6} \]
Read the symbols
\(C\)
the context to internalize: a system prompt, few-shot examples, or step-by-step instructions
\(p(\cdot \mid C, x)\)
the teacher: the original model reading the context and the input
\(q_\theta(\cdot \mid x)\)
the student: usually initialized from the same model, but it never sees \(C\)

The paper reports that it works for abstract instructions, for step-by-step reasoning (8-digit addition), and for concrete examples; on SPIDER text-to-SQL it beat direct gradient descent on the same examples by 9% [Certain]. Since the teacher is the same model, it can return logits, so (120.6) can also use a token-level KL; the sampled form shown here works even when the teacher is an API.

A related idea transfers the ability to learn in context, not one fixed context. In-context learning distillation (Huang et al.) trains a small model on few-shot prompts with targets from a large teacher, combining an in-context learning objective with a language-modeling objective, in two settings: Meta-ICT and Multitask-ICT [Certain]. Multitask-ICT worked better for multitask few-shot learning, at more compute, and the two objectives were complementary [Certain]:

\[ \mathcal{L}_{\text{ICL-KD}} = \mathcal{L}_{\text{ICT}}\big(\text{demos}, x \to \text{teacher output}\big) + \alpha\, \mathcal{L}_{\text{LM}} \tag{120.7} \]

(The exact form of each term differs per setting in the paper; this is the structure [Likely].)

When does internalizing a prompt pay?

Let the context have \(C\) tokens, the user input \(u\) tokens and the output \(o\) tokens. A forward pass costs about \(2N\) FLOPs per token for a model with \(N\) parameters. Removing the context saves \(2NC\) per request. Distilling costs, once, \(M\) teacher calls over \(C + u + o\) tokens plus training on \(u + o\) tokens for \(E\) epochs at about \(6N\) per token. The break-even number of requests is:

\[ R^{*} = \frac{M\big[(C + u + o) + 3E\,(u + o)\big]}{C} \tag{120.8} \]

\(N\) cancels: the break-even does not depend on the model size (when teacher and student are the same model). It ignores the attention cost, which grows with context length and makes the saving even larger for long contexts.

Worked example: a 1,500-token system prompt on Llama-3.1-8B

Llama-3.1-8B has 32 layers, 8 KV heads of size 128, hidden size 4096, MLP width 14336 and a vocabulary of 128,256, which gives 8,030,261,248 parameters. Its KV cache in bf16 is \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes (128 KiB) per token.

With \(C = 1500\), \(u = 200\), \(o = 300\): a request costs \(2N \times 2000 = 3.21 \times 10^{13}\) FLOPs with the prompt and \(8.03 \times 10^{12}\) without it, a saving of 75%. The context alone holds \(1500 \times 128\) KiB \(= 187.5\) MiB of KV cache per request (prefix caching can share it across requests, but only on the same server and while it stays in memory). With \(M = 20{,}000\) training examples and \(E = 2\) epochs, (120.8) gives \(R^{*} = 20{,}000 \times (2000 + 3000) / 1500 \approx 66{,}667\) requests. A service with a million requests a day pays this back in under two hours of traffic.

context tokensinput tokensoutput tokens
Figure 3. Context distillation for Llama-3.1-8B. Top bar: tokens processed per request with the long prompt; bottom bar: after the prompt is distilled into the weights. The readout gives FLOPs per request (\(2N\) per token), the KV cache of the context, and the break-even request count \(R^{*}\) from (120.8).
How to read Figure 3

The saving per request is the blue part of the top bar. Make the context long and the answers short (a classifier or router with many rules) and the saving is close to 100%, with a small \(R^{*}\). Make the outputs long and the saving shrinks. Raising \(M\) or \(E\) raises the one-time cost and so \(R^{*}\), but not the saving per request.

Algorithm 2: Context distillation
  1. input: model \(p\), context \(C\) (system prompt, rules, few-shot examples), a source of task inputs
  2. generate \(M\) diverse task inputs \(x\) (from logs, or ask the model to invent them)
  3. for each \(x\): teacher answers \(y \sim p(\cdot \mid C, x)\); if \(C\) asks for a scratch pad, keep only the final answer
  4. filter: drop refusals, wrong formats, failed checks
  5. student \(q_\theta \leftarrow p\); fine-tune on pairs \((x, y)\) with loss (120.6), no \(C\) in the input // LoRA is enough for one prompt
  6. evaluate the student without \(C\) against the teacher with \(C\) on held-out inputs

Distilling chain-of-thought: Distilling step-by-step

Chain-of-thought prompting (session 72) makes a big model write a rationale (the steps of the reasoning) before the answer. The rationale is useful for distillation: it tells the student why the answer is right, not only what it is. But a student that must write a long rationale at test time is slow: every rationale token is a decode step. Distilling step-by-step (Hsieh et al.) gets the benefit of rationales in training without paying for them at inference.

The recipe: prompt the teacher (540B PaLM) with about 10 few-shot chain-of-thought examples per task, each a triplet (input, rationale, label); on every training input the teacher writes a rationale \(\hat{r}\) and a label \(\hat{y}\) [Certain]. Then train the student (T5 at 220M, 770M and 11B) on two tasks, marked by prefixes in the input [Certain]:

\[ \mathcal{L} = \underbrace{\frac{1}{N}\sum_{i=1}^{N} \ell\big(f(\texttt{[label]}\,x_i),\ y_i\big)}_{\mathcal{L}_{\text{label}}} \;+\; \lambda\, \underbrace{\frac{1}{N}\sum_{i=1}^{N} \ell\big(f(\texttt{[rationale]}\,x_i),\ \hat{r}_i\big)}_{\mathcal{L}_{\text{rationale}}} \tag{120.9} \]
Read the symbols
\(f\)
the student model; \(\ell\) is the token-level cross-entropy over the target
\(\texttt{[label]}\), \(\texttt{[rationale]}\)
task prefixes: the same input with a different prefix asks for the label or for the rationale
\(y_i\)
the label: the human label if there is one, otherwise the teacher’s label \(\hat{y}_i\)
\(\hat{r}_i\)
the teacher’s rationale for input \(x_i\)
\(\lambda\)
the weight of the rationale task

At test time only the [label] task is used, so the student outputs a short answer, with no rationale and no extra decode cost [Certain]. The rationale task acts like an auxiliary loss: it shapes the shared encoder so that the label head can use the same reasoning features.

Why two tasks and not one target “rationale, then label”? The paper tested that single-task version too. It was worse, sometimes worse than no rationale at all, and the single-task student must also generate the rationale at test time:

220M T5, rationales from PaLM 540Be-SNLIANLICQASVAMP
Standard fine-tuning (labels only)88.3843.5862.1962.63
Single-task: target = rationale + label88.8843.5061.3763.00
Multi-task (120.9)89.5149.5863.2965.50
Multi-task, rationales from GPT-NeoX 20B89.1248.1563.2563.00

Numbers from the paper [Certain]. The multi-task gain over standard fine-tuning is 6.0 points on ANLI and 2.87 on SVAMP; a weaker teacher (20B) still helps, but less. The headline results: on e-SNLI, 12.5% of the training data was enough to beat standard fine-tuning on 100%; the 220M student (over 2000× smaller) beat few-shot PaLM 540B on e-SNLI; the 770M student beat few-shot PaLM on ANLI using 80% of the data, where standard fine-tuning could not match it with 100% [Certain].

Algorithm 3: Distilling step-by-step
  1. input: teacher API, about 10 hand-written (input, rationale, label) examples, inputs \(x_1..x_N\) (labeled or not)
  2. for each \(x_i\): prompt the teacher with the examples and \(x_i\); parse rationale \(\hat{r}_i\) and label \(\hat{y}_i\)
  3. if a human label \(y_i\) exists, use it as the label target; otherwise use \(\hat{y}_i\)
  4. build two training items per input: ([label] \(x_i\) → \(y_i\)) and ([rationale] \(x_i\) → \(\hat{r}_i\))
  5. train the student on both with (120.9) // shared weights, mixed in every batch
  6. inference: feed [label] \(x\) only // no rationale tokens to decode

Fine-tune-CoT: more rationales per question, and a filter

Ho et al. (“Large Language Models Are Reasoning Teachers”) distill rationales that the student does output. The teacher (text-davinci-002) is prompted zero-shot with “Let’s think step by step.”, and a sample is kept only if its final answer equals the gold answer [Certain]. Their diverse reasoning draws \(D\) different rationales per question at temperature 0.7 (\(D = 64\) for MultiArith and SVAMP, 8 for the others) [Certain]. The 6.7B student beat the 175B teacher on some tasks (Shuffled Objects 64.44 vs 50.22, Coin Flip 98.67 vs 92.67) but not on others (MultiArith 53.33 vs 78.89) [Certain]. They also warn about the filter: on Date Understanding, 27.6% of the teacher completions with a correct answer contained reasoning errors [Certain]. We now study that problem.

Reasoning distillation into small models, and filtering bad traces

Reasoning models such as DeepSeek-R1 write long traces (thousands of tokens) inside a think block before the answer. Session 59 showed the DeepSeek-R1 recipe briefly; here we look at the part that decides quality: which traces to keep. The general dataset is:

\[ \mathcal{D}_{\text{R}} = \Big\{ (x, r, a) \;:\; (r, a) \sim p(\cdot \mid x),\ \ \text{verify}(x, a) \cdot \prod_{j} \text{check}_j(x, r, a) = 1 \Big\} \tag{120.10} \]
Read the symbols
\(r, a\)
the reasoning trace and the final answer, sampled together from the teacher
\(\text{verify}\)
the outcome check: compare with the reference answer, run unit tests, or ask a judge model
\(\text{check}_j\)
other filters, each 1 (pass) or 0 (drop): format (think block closed), language, length, readability, duplicates, benchmark contamination

DeepSeek-R1 followed this pattern: for each prompt it sampled several responses and kept only correct ones, using rules or DeepSeek-V3 as a judge that sees the reference and the prediction; it filtered out traces with mixed languages, long paragraphs and code blocks; about 600k reasoning samples plus 200k non-reasoning samples [Certain]. The students (Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-14B, Qwen2.5-32B, Llama-3.1-8B, Llama-3.3-70B-Instruct) got “only SFT” [Certain].

The opposite end is s1: only 1,000 questions (s1K), chosen for difficulty, diversity and quality, with reasoning traces; Qwen2.5-32B-Instruct fine-tuned on them exceeds o1-preview on competition math by up to 27% [Certain]. s1 adds budget forcing: when the model tries to stop thinking, append “Wait” to make it continue; this lifted AIME24 from 50% to 57% [Certain]. LIMA (session 54) made the same point for chat: a small, carefully chosen set can be enough when the base model already has the knowledge. The filter matters more than the count.

distilled from DeepSeek-R1 (SFT only)RL on Qwen-32B baseQwQ-32B-Preview
Figure 4. Pass@1 scores from Tables 5 and 6 of the DeepSeek-R1 paper. The six distilled students use SFT only. DeepSeek-R1-Zero-Qwen-32B is the same Qwen 32B base trained with large-scale RL instead.
How to read Figure 4

Compare the two 32B bars: same base, distillation versus RL. Then compare student sizes: on AIME the 7B student (55.5) already beats RL at 32B (47.0), and on MATH-500 the 7B student (92.8) is already above RL at 32B (91.6). Switch benchmarks to see that the gap is largest on AIME and LiveCodeBench, the hardest sets.

The answer check is not a reasoning check

The most common filter checks only the final answer. A trace with a wrong step can still end at the right answer: by luck, by a second error that cancels the first, or by a guess. Let \(s\) be the share of teacher traces whose reasoning is sound, \(a\) the chance that a sound trace ends with the right answer, and \(g\) the chance that an unsound trace still gives the right answer. Among the traces the answer filter keeps, the share that is sound (the precision of the filter) is, by Bayes’ rule:

\[ \Pr(\text{sound} \mid \text{kept}) = \frac{s\,a}{s\,a + (1 - s)\,g} \tag{120.11} \]

The guess rate \(g\) depends on the answer format: near 0 for a free-form number, \(1/m\) for \(m\)-option multiple choice, 0.5 for yes/no. A step-level judge (a model or a process reward model, session 73) that passes sound traces with probability \(\rho\) and flawed ones with probability \(\phi\) gives:

\[ \Pr(\text{sound} \mid \text{kept}) = \frac{s\,a\,\rho}{s\,a\,\rho + (1 - s)\,g\,\phi} \tag{120.12} \]
Worked example: the same teacher on three answer formats

Let 70% of teacher traces be sound (\(s = 0.7\)) and assume \(a = 1\). For free-form math answers (\(g = 0.02\)) the filter keeps 70.6% of traces and 99.15% of the kept ones are sound. For 4-option multiple choice (\(g = 0.25\)) it keeps 77.5% and only 90.3% are sound: almost one kept trace in ten teaches a wrong reasoning step. For yes/no (\(g = 0.5\)) it is 82.4%.

Add a judge with \(\rho = 0.9\), \(\phi = 0.3\) to the multiple-choice case: precision rises to 96.6%, and 65.3% of traces are kept. The judge throws away 10% of good traces to remove 70% of bad ones that survived the answer check. This is why Fine-tune-CoT found 27.6% flawed traces among correct ones on Date Understanding, a multiple-choice task.

sound, keptflawed, kept (bad training data)dropped
Figure 5. One hundred teacher traces, with \(a = 1\) (every sound trace gets the right answer). The left grid shows the answer check alone; the right grid shows the answer check followed by the step judge (only active when the box is ticked). The readout gives the kept share and the precision, formulas (120.11) and (120.12).
How to read Figure 5

Each square is one trace. The answer filter cannot see the difference between green and orange squares: it keeps every orange trace that guessed right. Pick an answer format with the buttons, then lower \(s\) to model a weaker teacher: precision drops fast when \(g\) is large. Tick the judge and lower \(\phi\) to see how many orange squares a good step check removes, and how many green ones it costs.

A practical filter stack for reasoning traces, from cheap to expensive:

  • Format. The think block is closed, the answer is in the expected place (for example a boxed number), the trace was not cut by the length limit. A cut trace teaches the student never to finish.
  • Outcome. The answer matches the reference, or the code passes unit tests. This is the core filter, with the weakness of (120.11).
  • Language and readability. Drop traces that switch language or are unreadable; DeepSeek-R1 dropped mixed languages, long paragraphs and code blocks [Certain].
  • Length. Very long traces are expensive to train on and can teach the student to ramble; some pipelines keep the shortest correct trace per prompt [Likely].
  • Duplicates and contamination. Hash-based exact dedup, n-gram near-dedup, and removal of anything close to the evaluation sets.
  • Step check. A judge or a process reward model scores steps; slow but it attacks the false positives of (120.11) (session 73).

Two more ideas from the self-training family use the same filter. STaR generates rationales, and when the answer is wrong it tries again with the correct answer given as a hint (rationalization); it fine-tunes on all rationales that led to correct answers and repeats [Certain]. And rejection sampling has a hidden bias: prompts that the teacher rarely solves are kept less often, so the kept set is easier than the prompt set (session 59, formula 59.11). Sample more for hard prompts, or cap the traces per prompt.

Algorithm 4: R1-style reasoning distillation
  1. input: teacher reasoning model, prompts with checkable answers (math, code with tests, science), non-reasoning SFT data, student base model
  2. for each prompt \(x\): sample \(k\) traces \((r, a)\) at temperature about 0.6 to 1 // more samples for hard prompts
  3.   drop traces that fail format, verify, language, length, dedup checks (120.10)
  4.   (optional) score the rest with a step judge (120.12); keep at most one or two per prompt
  5. decontaminate against all evaluation sets; mix in non-reasoning data so that chat skills survive
  6. SFT the student on the result in the same think template it will use at inference; long context, packing with document masks, never cut a trace
  7. (optional) RL with verifiable rewards on top (session 70) to go beyond pure imitation

Why is distillation so much better than RL for the small model (Figure 4)? RL must discover good reasoning by trial and reward; a small model rarely finds a correct long solution, so the reward signal is sparse. Distillation gives it the correct long solution directly. DeepSeek’s conclusion was that distilling strong models into small ones gives excellent results, while small models relying on large-scale RL need huge compute and may not even reach distillation; going beyond the teacher still needs stronger bases and larger-scale RL [Certain]. The capacity gap still applies: very small students can learn better from shorter traces or from a mid-sized teacher (session 59).

Code: samples versus logprobs, and a trace filter

The first script is a tiny, deterministic version of Figure 2 with real training. A hidden teacher has 50 possible answers for each of 32 prompts. The student is a table of logits, trained with Adam on what each kind of API returns. We report the KL from the teacher to the trained student.

import torch

torch.manual_seed(0)
C, V = 32, 50                        # 32 prompts, 50 possible answers each
p = (2.5 * torch.randn(C, V)).softmax(-1)   # the hidden teacher

def api_samples(K, seed):            # black box: K sampled answers per prompt
    g = torch.Generator().manual_seed(seed)
    return torch.multinomial(p, K, replacement=True, generator=g)   # (C, K)

def api_topk(k):                     # black box with top-k logprobs
    return p.topk(k, dim=-1)         # values (C, k), indices (C, k)

def fit(target):                     # target: (C, V) soft labels
    W = torch.zeros(C, V, requires_grad=True)
    opt = torch.optim.Adam([W], lr=0.1)
    for _ in range(400):
        loss = -(target * W.log_softmax(-1)).sum(-1).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    q = W.detach().softmax(-1)
    return (p * (p.log() - q.log())).sum(-1).mean().item()   # KL(p || q)

def from_samples(K, eps=0.0):        # SFT on text = one-hot targets, averaged
    y = api_samples(K, seed=K)
    t = torch.zeros(C, V).scatter_add_(1, y, torch.ones(C, K)) / K
    return (1 - eps) * t + eps / V   # eps > 0: label smoothing

def from_topk(k):                    # exact top-k, rest of the mass spread
    v, i = api_topk(k)
    rest = (1 - v.sum(-1, keepdim=True)) / (V - k)
    return torch.full((C, V), 0.0).add_(rest).scatter_(1, i, v)

print("teacher entropy (nats):", round(-(p * p.log()).sum(-1).mean().item(), 3))
for K in [1, 4, 16, 64, 256]:
    print(f"text, K={K:3d} samples   KL {fit(from_samples(K)):7.3f}"
          f"   with smoothing 0.1  KL {fit(from_samples(K, 0.1)):6.3f}")
for k in [1, 5, 20]:
    print(f"one call, top-{k:2d} logprobs   KL {fit(from_topk(k)):6.3f}")
print(f"full distribution (white-box) KL {fit(p):6.3f}")
teacher entropy (nats): 1.767
text, K=  1 samples   KL   4.205   with smoothing 0.1  KL  2.117
text, K=  4 samples   KL   2.451   with smoothing 0.1  KL  1.114
text, K= 16 samples   KL   0.983   with smoothing 0.1  KL  0.413
text, K= 64 samples   KL   0.360   with smoothing 0.1  KL  0.159
text, K=256 samples   KL   0.098   with smoothing 0.1  KL  0.076
one call, top- 1 logprobs   KL  0.763
one call, top- 5 logprobs   KL  0.154
one call, top-20 logprobs   KL  0.013
full distribution (white-box) KL  0.000

Plain samples overfit: with \(K = 1\), the student puts almost all its mass on one answer and the KL is 4.2 nats, more than the teacher’s entropy. Label smoothing halves it, and the error falls roughly like \(1/K\) as in (120.3). One call with top-5 logprobs (KL 0.154) is about as good as 64 smoothed samples (0.159), and top-20 (0.013) is almost white-box. (Here 50 answers are a toy “vocabulary”; for real LLMs the effective support per token is small, so top-k is even more effective.)

The second script is a minimal filter stack for reasoning traces, formula (120.10): format, answer, language, length and duplicates.

import re, hashlib

def final_answer(trace):
    m = re.findall(r"\\boxed\{([^{}]*)\}", trace)
    return m[-1].strip() if m else None

def keep(trace, gold, max_tokens=16000):
    reasons = []
    if "</think>" not in trace:
        reasons.append("no end of thinking")           # cut or wrong format
    if final_answer(trace) != gold:
        reasons.append("wrong or missing answer")
    letters = [c for c in trace if c.isalpha()]
    if letters and sum(not c.isascii() for c in letters) / len(letters) > 0.05:
        reasons.append("mixed language")                # English task, other script
    if len(trace.split()) > max_tokens:                 # crude token count
        reasons.append("too long")
    return reasons

seen = set()
def dedup(prompt, trace):
    h = hashlib.sha1((prompt + trace).encode()).hexdigest()
    if h in seen: return False
    seen.add(h); return True

gold = "12"
traces = [
    "<think>3 boxes of 4 apples: 3*4 = 12.</think> The answer is \\boxed{12}",
    "<think>3 boxes of 4 apples: 3+4 = 7, hmm, so 12?</think> \\boxed{12}",
    "<think>3 boxes, 4 apples each, 3*4 = 13.</think> \\boxed{13}",
    "<think>每盒4个, 3盒, 3*4 = 12.</think> \\boxed{12}",
    "<think>3 boxes of 4 apples: 3*4 = 12. Let me check again: 3*4",
    "<think>3 boxes of 4 apples: 3*4 = 12.</think> The answer is \\boxed{12}",
]
for t in traces:
    r = keep(t, gold)
    if not r and not dedup("Q1", t): r = ["duplicate"]
    print("KEEP" if not r else "drop", "|", ", ".join(r) or "-", "|", t[:44])
KEEP | - | <think>3 boxes of 4 apples: 3*4 = 12.</think
KEEP | - | <think>3 boxes of 4 apples: 3+4 = 7, hmm, so
drop | wrong or missing answer | <think>3 boxes, 4 apples each, 3*4 = 13.</th
drop | mixed language | <think>每盒4个, 3盒, 3*4 = 12.</think> \boxed{12
drop | no end of thinking, wrong or missing answer | <think>3 boxes of 4 apples: 3*4 = 12. Let me
drop | duplicate | <think>3 boxes of 4 apples: 3*4 = 12.</think

Look at the second line: “3+4 = 7, hmm, so 12?” has a wrong step and a correct boxed answer, and it passes every check. That is the false positive of (120.11). Only a step judge would catch it.

Where this is used

SettingBlack-box methodWhy
A small chat or instruction model from an API teacherSelf-Instruct style data (120.5), explanation system messages (Orca), hard-prompt mining (Lion)No access to logits; text is all you need for SFT
A narrow production task (classification, extraction, routing)Distilling step-by-step (120.9): rationales in training, short labels at test timeSmall model, short outputs, better accuracy per labeled example
A long system prompt or many few-shot examples on every requestContext distillation (120.6), break-even (120.8)Removes prefill compute and KV memory of the prompt
Small reasoning models (1.5B to 70B)R1-style rejection sampling plus SFT (120.10), maybe RL afterBeats RL on the same small base at a fraction of the cost
A data-efficient reasoning models1-style selection of about 1,000 hard, diverse, high-quality tracesQuality of the filter matters more than the count
Video captioning and video QA modelsA strong VLM writes dense captions and QA pairs for clips; a video model is trained on them (LLaVA-Video-178K is synthetic data of this kind)Human video annotation is very expensive; captions from a teacher are text, so black-box works [Likely]
Video retrieval, key-moment detection and Video-RAGTeacher writes captions or timestamps of key moments; a small model or text embedder is trained to match themIndex long archives cheaply with a small model at ingestion time [Likely]
Draft models for speculative decodingTrain the draft on the target model’s own generated textA draft that imitates the target is accepted more often (session 128) [Likely]

Common mistakes

  • Expecting knowledge to transfer through style. Imitation data teaches format and tone fast; facts the student lacks do not appear. Human raters are fooled, targeted benchmarks are not (the false promise).
  • Trusting the answer check. Correct answer does not mean correct reasoning, especially for multiple choice or yes/no (120.11). Add a step judge, or prefer free-form answers when you build the prompt set.
  • Training on cut traces. If the teacher hit its length limit, the trace has no answer and no end of thinking. Keeping it teaches the student to never finish.
  • One sample per prompt with no smoothing. The student becomes over-confident on whatever the teacher said once. Use several samples, label smoothing, or top-k logprobs if available (120.4).
  • Mismatch of chat template or think format. The student must be trained in the template it will be served with (session 9); a teacher’s tags copied into the wrong template break stopping and parsing.
  • Contaminated prompts. Prompts generated by a teacher, or collected from the web, often contain benchmark questions. Decontaminate before training, not after the scores look too good.
  • Ignoring the terms of use. Some API terms forbid training competing models on outputs [Likely]. Check before spending money on data.
  • A student too small for the traces. Very long traces from a huge teacher can be worse for a 1B student than shorter ones from a mid-sized teacher (the capacity gap).

Interview questions

You only have API access to a strong model. How do you build a small model for your task?basic
Collect or generate prompts that cover the task (real logs if possible, otherwise Self-Instruct style generation from seeds), ask the teacher for answers (with explanations or rationales if they help), filter them (format, correctness checks, judge, dedup, decontamination), and fine-tune the student with the SFT loss in its own chat template. This is a Monte Carlo estimate of forward-KL distillation (120.2). If the API returns top-k logprobs, train on soft targets (120.4). Evaluate on held-out real data, not only teacher-style data.
Why is SFT on teacher samples a form of distillation and not just data augmentation?core
The forward KL from teacher to student is \(-H(p) - \E_{y\sim p}[\log q_\theta(y\mid x)]\). The entropy does not depend on the student, so minimizing the KL is minimizing the expected negative log-likelihood of teacher samples, and averaging over \(K\) samples is its Monte Carlo estimate. So it targets the teacher’s distribution, with variance that falls like \(1/K\) (120.3). It is mode-covering like forward KL: the student is pushed to cover everything the teacher may say.
What do top-k logprobs buy you, and why are they often not used?core
They give the teacher’s probabilities of the most likely tokens at each position, so the student can train on soft targets; because LLM distributions are peaked, even \(k = 2\) to 5 holds most of the mass and one call can be worth about 100 text samples (worked example). They are often not used because they are per token of the teacher’s tokenizer, which rarely matches the student’s; many APIs return few or none, especially for hidden reasoning; and the value of reasoning data is mostly in the content, not the distribution.
Explain Distilling step-by-step. Why does the student not output the rationale?core
The teacher writes rationales and labels with few-shot CoT prompting. The student is trained on two tasks with prefixes: [label] x → label and [rationale] x → rationale, loss \(\mathcal{L}_{\text{label}} + \lambda\mathcal{L}_{\text{rationale}}\). At test time only [label] is used, so there is no decode cost for rationale tokens. The rationale task shapes the shared representation. The single-task version (rationale then label as one target) was worse in the paper, for example 43.50 vs 49.58 on ANLI with a 220M T5, and it must decode the rationale.
What is context distillation and when does it pay off?core
The same model with a long context (system prompt, rules, examples) is the teacher; the model without it is the student. You generate inputs, let the teacher answer with the context, and fine-tune the student to give those answers without the context. It saves \(2NC\) FLOPs and the KV cache of \(C\) tokens per request. The break-even is \(R^* = M[(C+u+o) + 3E(u+o)]/C\) requests, independent of model size; with C = 1500, u = 200, o = 300, M = 20k, E = 2 it is about 66.7k requests. It pays off when the prompt is long, stable and used at high volume.
Why did imitation models of ChatGPT look good to raters but fail benchmarks?core
The SFT loss rewards matching the teacher’s tokens, and style (format, tone, confident structure) is in every answer and easy to copy. Knowledge the base model does not have cannot be learned from a limited set of answers; the student learns to sound knowledgeable. Raters judge style and are fooled; targeted benchmarks measure factuality and reasoning. Gudibande et al. found little to no closing of the gap on tasks not heavily covered by the imitation data, and concluded that better base models are the main lever.
Your answer-checked reasoning dataset still teaches wrong steps. Why, and how much?advanced
The check sees only the final answer. Flawed traces that still land on the right answer pass. By Bayes, precision is \(sa/(sa + (1-s)g)\) with \(g\) the chance a flawed trace gets the right answer. With 70% sound traces and 4-option multiple choice (\(g = 0.25\)), precision is 90.3%; for free-form numbers (\(g = 0.02\)) it is 99.15%. Fixes: prefer free-form or executable answers, add a step judge or PRM (with \(\rho = 0.9\), \(\phi = 0.3\) precision rises to 96.6%), sample several traces and keep those that agree, and keep the shortest clean trace.
Why did distillation beat RL for Qwen-32B in DeepSeek-R1?advanced
On the same Qwen 32B base, large-scale RL gave 47.0 on AIME 2024 while SFT on R1 traces gave 72.6, and the distilled model was better on every benchmark reported. RL must discover long correct reasoning by itself and gets a sparse reward when the model rarely succeeds; distillation hands over correct long solutions written by a much stronger model. The cost is that the student is bounded by the teacher: to go beyond it you still need RL on strong bases, and RL after distillation can add more.
How do s1 and DeepSeek-R1 differ in data strategy?advanced
R1 used scale: about 600k rejection-sampled reasoning samples plus 200k non-reasoning samples, filtered by correctness, language mixing, long paragraphs and code blocks. s1 used selection: 1,000 questions chosen for difficulty, diversity and quality, SFT on Qwen2.5-32B-Instruct, plus budget forcing at test time (append “Wait” to extend thinking, 50% to 57% on AIME24). Both say that the base model must already be strong; s1 shows that a small well-chosen set can unlock it, R1 shows how far a broad set can carry many student sizes.
What is the bias of rejection sampling, and how do you fix it?advanced
A prompt is kept only if at least one of \(k\) samples is correct, with probability \(1-(1-s)^k\), and easy prompts give more correct traces (\(ks\) on average). The kept set is easier than the prompt set, and the student sees little of the hard cases. Fixes: adaptive \(k\) (more samples for hard prompts), cap traces per prompt, rationalization (STaR: give the answer as a hint and ask for a rationale, then verify), or a stronger teacher for the hard part.

Exercises

  1. The teacher of Figure 2 returns top-2 logprobs. Write \(\tilde{p}\) from (120.4) and compute \(\KL(p \,\|\, \tilde{p})\).
  2. With \(V_{\text{eff}} = 6\), how many text samples does (120.3) need for an expected KL of 0.01?
  3. Context distillation: \(C = 3000\), \(u = 100\), \(o = 400\), \(M = 50{,}000\), \(E = 1\). Compute \(R^{*}\) from (120.8).
  4. A yes/no task with \(s = 0.5\) and \(a = 1\). What is the precision of the answer filter? And with a judge, \(\rho = 0.9\), \(\phi = 0.3\)?
  5. A teacher solves a hard prompt with probability 0.2 per sample. With \(k = 8\) samples, what is the chance the prompt is kept, and how many correct traces do you expect?
  6. You sample 50,000 prompts × 8 traces × 6,000 output tokens. At a hypothetical price of $1 per million output tokens, what does it cost? If 40% of the traces survive the filters, what is the cost per kept trace?
Answerscheck
1. Missing mass \(1 - 0.65 = 0.35\), spread over 4 replies: \(\tilde{p} = (0.42, 0.23, 0.0875, 0.0875, 0.0875, 0.0875)\); \(\KL = 0.14\ln(0.14/0.0875) + 0.09\ln(0.09/0.0875) + 0.07\ln(0.07/0.0875) + 0.05\ln(0.05/0.0875) = 0.0247\). 2. \(K = 5/(2 \times 0.01) = 250\). 3. \(R^{*} = 50{,}000 \times (3500 + 3 \times 1 \times 500)/3000 = 50{,}000 \times 5000/3000 \approx 83{,}333\) requests. 4. \(0.5/(0.5 + 0.5 \times 0.5) = 66.7\%\); with the judge \(0.45/(0.45 + 0.5 \times 0.5 \times 0.3) = 0.45/0.525 = 85.7\%\). 5. \(1 - 0.8^8 = 0.832\); \(8 \times 0.2 = 1.6\) correct traces on average. 6. \(50{,}000 \times 8 \times 6000 = 2.4 \times 10^9\) tokens, so $2,400; 160,000 kept traces, $0.015 each.

Key takeaways

  • Black-box distillation uses only the teacher’s outputs. SFT on teacher samples is a Monte Carlo estimate of forward-KL distillation (120.1, 120.2); its error falls like \(1/K\) (120.3). Top-k logprobs, if available, are worth many samples (120.4).
  • Instruction-following distillation needs prompts as much as answers: Self-Instruct and Alpaca generate them (120.5), Orca asks for explanations, Lion mines hard prompts. Imitation copies style easily, knowledge hardly.
  • Context distillation moves a long prompt into the weights (120.6); it saves the prompt’s prefill and KV cache on every request, with a break-even (120.8) that does not depend on model size.
  • Distilling step-by-step trains on rationales as a second task (120.9) and answers without them at test time: better accuracy with less data and no decode cost.
  • Reasoning distillation (R1 style) is rejection sampling plus SFT (120.10). It beat RL on the same 32B base (72.6 vs 47.0 on AIME 2024). The filter decides the quality: an answer check passes flawed traces at the rate of (120.11); step judges help (120.12).

Key formulas

\[ \mathcal{L}(\theta) = -\frac{1}{K}\sum_{k} \log q_\theta\big(y^{(k)} \mid x\big), \quad y^{(k)} \sim p, \qquad \E\big[\KL(p \,\|\, \hat{p}_K)\big] \approx \frac{V_{\text{eff}} - 1}{2K} \]
\[ \mathcal{L} = \mathcal{L}_{\text{label}} + \lambda\,\mathcal{L}_{\text{rationale}}, \qquad R^{*} = \frac{M\big[(C+u+o) + 3E(u+o)\big]}{C} \]
\[ \Pr(\text{sound} \mid \text{kept}) = \frac{s\,a\,\rho}{s\,a\,\rho + (1-s)\,g\,\phi} \]

Next, session 121 keeps one model but lets easy tokens and easy inputs use fewer layers: another way to spend less compute where the task is simple.

References for this session

  1. Hsieh et al. (2023). Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
  2. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  3. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  4. Kim & Rush (2016). Sequence-Level Knowledge Distillation
  5. Wang et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions
  6. Taori et al. (2023). Alpaca: A Strong, Replicable Instruction-Following Model
  7. Mukherjee, Mitra, Jawahar et al. (2023). Orca: Progressive Learning from Complex Explanation Traces of GPT-4
  8. Wu, Waheed, Zhang et al. (2023). LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions
  9. Chen et al. (2023). Symbolic Discovery of Optimization Algorithms (Lion)
  10. Gudibande, Wallace, Snell et al. (2023). The False Promise of Imitating Proprietary LLMs
  11. Snell, Klein and Zhong (2022). Learning by Distilling Context
  12. Huang, Chen, Yu and McKeown (2022). In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models
  13. Ho, Schmid and Yun (2022). Large Language Models Are Reasoning Teachers
  14. Zelikman, Wu, Mu and Goodman (2022). STaR: Bootstrapping Reasoning With Reasoning
  15. Muennighoff, Yang, Shi et al. (2025). s1: Simple test-time scaling
  16. Zhou et al. (2023). LIMA: Less Is More for Alignment
  17. Zhang et al. (2024). Video Instruction Tuning With Synthetic Data
  18. Agarwal et al. (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD)