LLM & VLM courseBlog
53Track 4: Post-training and RL · session 1 of 25

From base model to assistant

A pretrained model continues documents. An assistant answers questions, follows instructions, and stops when it is done. This session shows the difference with real outputs from Qwen2.5-0.5B base and instruct, then walks through the post-training stages (SFT, preference tuning, RL), where the instruction data comes from, and why a wrong chat template quietly breaks a good model.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: BeginnerBefore this: Sessions 6, 9 and 40 (next-token prediction, chat templates, evaluating base models)

What you will learn

  • What a base model does and what an assistant does, seen in real greedy outputs from the same 0.5B model before and after post-training
  • The standard pipeline: SFT, then preference tuning (reward model or DPO), then RL, with the real data sizes of InstructGPT and Tülu 3
  • The math of each stage: masked SFT loss, the Bradley-Terry reward loss, the KL-regularized RL objective and its closed-form optimum
  • Where instruction data comes from: human-written (InstructGPT, LIMA), synthetic (Self-Instruct, with its ROUGE-L filter) and distilled (Alpaca, DeepSeek-R1 students)
  • How much post-training really changes: the superficial alignment hypothesis and the token-shift measurement (about 77% of tokens unchanged)
  • Why a wrong or missing chat template breaks a model: what we measured on Qwen2.5-0.5B-Instruct, and how to check it

Why this matters

After pretraining, a language model knows a lot: facts, grammar, code, some reasoning. But it is not an assistant. Ask it a question, and it may write three more questions, or continue your sentence as if it were a web page. This session is about the short, cheap training that turns such a base model into an assistant that answers, follows instructions, refuses some requests, and stops when it is done.

This training is called post-training (everything after pretraining). In 2022, OpenAI showed with InstructGPT that a 1.3B-parameter model after post-training was preferred by human raters over the 175B-parameter GPT-3, a model about 100 times larger. [Certain] Post-training also cost very little: 4.9 petaflop/s-days for their 175B SFT model and 60 for the full RL model, against 3,640 for pretraining GPT-3. [Certain] Every chat model you use today, from Llama and Qwen to GPT and Claude, went through some version of this pipeline.

This session is the map of the Track 4 track. We look at real outputs of one small model before and after post-training, then at each stage (SFT, preference tuning, RL) with its loss and real data sizes, then at where the data comes from, and finally at the chat template: the format that holds it all together, and the most common reason a good model gives bad answers in production. Every model output shown here comes from running Qwen2.5-0.5B (base) and Qwen2.5-0.5B-Instruct ourselves, with greedy decoding on a CPU. [Certain]

What a base model does, and what an assistant does

A base model was trained on one task: predict the next token of a document (session 6). Its loss is the average negative log-likelihood over all tokens of a huge text corpus:

\[ \mathcal{L}_{\text{pre}}(\theta) = -\frac{1}{T} \sum_{t=1}^{T} \log p_\theta\big(x_t \mid x_{\lt t}\big), \qquad \mathbf{x} \sim \text{web, books, code} \tag{53.1} \]
Read the symbols
\(\mathbf{x} = (x_1, \dots, x_T)\)
a document from the pretraining corpus, as tokens
\(p_\theta(x_t \mid x_{\lt t})\)
the model’s probability for the true next token, given all earlier tokens
\(\theta\)
all model weights

Nothing in (53.1) says “be helpful”. The model learns to imitate documents. So when it sees “How do I boil an egg?”, it asks itself: in what kind of document does this line appear, and what comes next? On a forum, more questions may follow. In a quiz, answer options follow. In a dataset of paraphrase tasks, “Do these two questions mean the same?” follows. All of these are good continuations of a document. None of them is what a user wants.

An assistant is a model whose most likely continuation of a user message is a helpful reply in the assistant’s role, which ends with an end-of-turn token. It needs four behaviors that pretraining does not give directly:

  • Answer, do not continue. Treat the input as a request from someone else, not as text to extend.
  • Follow the instruction’s form. “Three tips” means three. “In one sentence” means one.
  • Stop. Emit the end-of-turn token when the answer is complete, so the user can speak again.
  • Keep a persona and rules. Respect the system prompt, refuse clearly harmful requests, admit uncertainty.

Figure 1 shows this with real outputs. The same question goes to the base model and to the instruct model, either as raw text or wrapped in the chat template (session 9).

What the model reads
What it writes (greedy, at most 80 tokens)
Figure 1. Real greedy outputs of Qwen2.5-0.5B (base) and Qwen2.5-0.5B-Instruct for four prompts, with and without the Qwen chat template (which adds a default system prompt). Orange marks a single special token. The bars show the model’s probabilities for the first token of its reply; each token is in quotes, and a leading space inside the quotes is part of the token (⏎ is a newline).
How to read Figure 1

Start with “Boil an egg”, base model, raw text: the model does not answer; it writes a paraphrase-detection exercise, because such exercises were in its training data. Switch to “I feel sad”: the base model continues the diary entry and gets stuck in a loop. Now switch to the instruct model with the chat template: it answers, and on “Capital of France” it ends with <|im_end|>. Look at the bars: with the template, the instruct model is much more sure how to begin (71% for “Bo” on the egg question, against 12% for the most likely raw-text start). Finally, try “Raw chicken” with the instruct model. It says raw chicken is “generally safe”, which is false and dangerous. Post-training taught the format, not the facts.

Three lessons from Figure 1.

First, the base model is a document continuer. On “How do I boil an egg?” it writes “What is the best way to boil an egg? Do those questions have the same meaning? Options are: (1). no. (2). yes.” This is the format of the FLAN-style paraphrase datasets that are part of many pretraining mixtures. On “I feel sad today.” it writes a diary and falls into a repetition loop. [Certain, from our runs]

Second, modern base models are already a little chatty. The Qwen2.5 base model answers “Capital of France” correctly and even writes a numbered recipe when given the chat template. This is not an accident: the Qwen2.5 report says its pretraining data includes synthetic data generated by Qwen2-72B-Instruct. [Certain] Many recent base models have seen instruction-like data before post-training, so the line between “base” and “instruct” is not as sharp as it was for GPT-3. [Likely]

Third, the instruct model needs its template. With raw text, even the instruct model answers and then keeps going (“To verify this, I will use a simple Python code snippet…”) until the token limit. With the template, it answers in one line and stops. We measure this effect carefully in the last part of this session.

The stages: SFT, preference tuning, RL

The classic recipe has three training stages after pretraining. It was set by InstructGPT (Ouyang et al., 2022), building on earlier work on learning from human preferences (Christiano et al., 2017). [Certain] Figure 2 steps through it with the real numbers from InstructGPT and from Tülu 3 (Lambert et al., 2024), a fully open recipe built on Llama 3.1 base models.

Figure 2. The post-training pipeline, one stage per step. The table under each stage gives real data sizes and settings from the InstructGPT paper, the Tülu 3 report, the Llama 2 report and the DeepSeek-R1 report. The last step shows Tülu 3 8B scores after each stage (Table 6 of the Tülu 3 report).
How to read Figure 2

Press Next to move one stage. Each box is a training stage; the arrow means “initialize the next stage from this model”. Under the boxes: what data the stage uses, what the training signal is, and what comes out. In the table, compare the data sizes: pretraining uses trillions of tokens, SFT uses thousands to about a million examples, preference tuning a few hundred thousand pairs, RL tens of thousands of prompts. On the last step, see which stage helps which skill, and note that the safety score goes down as the model becomes more helpful.

Stage 1: supervised fine-tuning (SFT)

SFT trains the base model on examples of the behavior we want: a prompt, and a good answer written by a person or a strong model. The loss is the same next-token loss as (53.1), but only on the answer tokens. The prompt is read, not predicted:

\[ \mathcal{L}_{\text{SFT}}(\theta) = -\frac{1}{\sum_t m_t} \sum_{t} m_t \log \pi_\theta\big(x_t \mid x_{\lt t}\big), \qquad m_t = \begin{cases} 1 & x_t \text{ is an assistant token (incl. end-of-turn)} \\ 0 & \text{otherwise} \end{cases} \tag{53.2} \]
Read the symbols
\(\pi_\theta\)
the model, now called a policy: a rule that picks the next token; same network as \(p_\theta\)
\(\mathbf{x}\)
one chat rendered with the chat template: system, user and assistant turns as one token sequence
\(m_t\)
the loss mask: 1 on what the assistant wrote, 0 on everything else (formula 9.4 in Session 9)

SFT is where the model learns the format: the roles, where answers start, and where they end. Because the mask includes the assistant’s end-of-turn token, SFT is the stage that teaches the model to stop. InstructGPT used about 13k prompts (12,725 training prompts) with answers written by about 40 contractors, and trained for 16 epochs. [Certain] Tülu 3 used 939,344 prompts for 2 epochs. [Certain] The details (masking, packing, epochs, overfitting) are in session 54.

SFT has one limit. It can only show the model good answers; it never shows a bad answer and says “not like this”. And it needs someone to write the ideal answer, which is slow for hard tasks. It is often easier for a person to compare two answers than to write a perfect one. That is what stage 2 uses.

Stage 2: preference data and the reward model

For each prompt \(x\), the SFT model writes several answers, and a person says which one is better. One record is a triple \((x, y_w, y_l)\): the prompt, the preferred (“winning”) answer and the rejected (“losing”) answer. InstructGPT showed labelers between \(K = 4\) and \(K = 9\) answers per prompt and asked for a full ranking, which gives \(\binom{K}{2}\) pairs: up to 36 pairs from one prompt. [Certain]

A reward model \(r_\phi(x, y)\) is a copy of the language model with its output layer replaced by a single number: a score for the whole answer. It is trained with the Bradley-Terry model, which says that the chance a person prefers \(y_w\) depends only on the difference of the two scores:

\[ P(y_w \succ y_l \mid x) = \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big), \qquad \mathcal{L}_{\text{RM}}(\phi) = -\,\E_{(x, y_w, y_l)} \Big[ \log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big) \Big] \tag{53.3} \]
Read the symbols
\(r_\phi(x, y)\)
the reward model’s score for answer \(y\) to prompt \(x\); one number
\(y_w \succ y_l\)
“the person preferred \(y_w\) over \(y_l\)”
\(\sigma(z) = 1/(1 + e^{-z})\)
the logistic function; it turns a score difference into a probability

Only differences matter: adding a constant to every score changes nothing. (InstructGPT therefore shifted its reward model so that human demonstrations score 0 on average. [Certain]) The reward model learns what people like from many comparisons. InstructGPT trained a 6B reward model on 33,207 prompts; Llama 2 collected 1,418,091 of its own comparisons. [Certain] Reward models are the topic of session 67.

Worked example: one reward-model step

The reward model scores the preferred answer 0.5 and the rejected one −0.5. The difference is 1, so the model says the person prefers \(y_w\) with probability \(\sigma(1) = 0.731\). The loss for this pair is \(-\log 0.731 = 0.313\). If the scores were reversed (difference −1), the loss would be \(-\log \sigma(-1) = 1.313\): a large push to fix the order. With \(K = 9\) ranked answers, one prompt gives \(\binom{9}{2} = 36\) such pairs. InstructGPT puts all 36 pairs of one prompt in the same batch, because treating them as separate data points made the reward model overfit after a single pass. [Certain]

Stage 3: reinforcement learning against the reward

Now we have a judge. In the RL stage, the model writes answers to new prompts, the reward model scores them, and the model is updated to make high-scoring answers more likely (session 64, session 66). There is one danger. The reward model is only an estimate of human taste, trained on a limited set of answers. If we push the policy to maximize it without limits, the policy finds answers the reward model overrates: very long text, repeated safety phrases, flattery. This is reward hacking (also called over-optimization). The fix is to stay close to the SFT model, the reference policy \(\pi_{\text{ref}}\), with a KL penalty (session 3):

\[ \max_{\pi_\theta} \; \E_{x \sim \mathcal{D},\; y \sim \pi_\theta(\cdot \mid x)} \big[ r_\phi(x, y) \big] \; - \; \beta \, \E_{x \sim \mathcal{D}} \Big[ \KL\big( \pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x) \big) \Big] \tag{53.4} \]
Read the symbols
\(\mathcal{D}\)
the set of prompts used for RL (only prompts: the answers are sampled from the model itself)
\(\pi_{\text{ref}}\)
the frozen SFT model; the policy starts equal to it
\(\KL(\pi_\theta \| \pi_{\text{ref}})\)
how far the new answer distribution has moved from the SFT one, in nats; zero when they are equal
\(\beta\)
the strength of the leash; InstructGPT used \(\beta = 0.02\), Tülu 3 8B used \(\beta = 0.05\) [Certain]

InstructGPT also added a third term: \(\gamma\) times the pretraining log-likelihood, with \(\gamma = 27.8\), to stop the model from forgetting skills it had before (they call this “PPO-ptx”). [Certain] For a fixed prompt, objective (53.4) has an exact solution. Setting the derivative to zero, with the constraint that probabilities add up to 1, gives:

\[ \pi^*(y \mid x) = \frac{1}{Z(x)} \, \pi_{\text{ref}}(y \mid x) \, \exp\!\Big( \frac{r_\phi(x, y)}{\beta} \Big), \qquad Z(x) = \sum_{y} \pi_{\text{ref}}(y \mid x) \exp\!\Big( \frac{r_\phi(x, y)}{\beta} \Big) \tag{53.5} \]
Read the symbols
\(\pi^*\)
the best possible policy for objective (53.4)
\(Z(x)\)
a normalizer so the probabilities add up to 1; it has a sum over all possible answers, so we can never compute it for real text
\(\exp(r/\beta)\)
the reweighting: every answer keeps its SFT probability, multiplied by a factor that grows with its reward

Read (53.5) as: “start from the SFT model and tilt it toward high reward”. A small \(\beta\) tilts hard; a large \(\beta\) barely moves. Figure 3 lets you feel this; we place it right here because it is the key picture of RLHF.

Figure 3. Formula (53.5) for one prompt (“Is it safe to eat raw chicken?”) and five possible answers. Each answer has an SFT probability (grey), a reward-model score \(r\), and a “true quality” \(q\) that a careful human would give. Answer E is a reward hack: the reward model likes it (\(r = 3\)), people do not (\(q = -1\)). The numbers are a toy example we chose; the formula is exact.
How to read Figure 3

Move \(\beta\) to the left (small \(\beta\), weak leash). The policy puts almost all its mass on E, the answer with the highest reward: expected reward goes up to 3.0, but true quality falls to −1.0. Move \(\beta\) to the right (strong leash): the policy stays close to the grey bars and keeps the bad answer B. Somewhere in the middle, true quality is best: about \(\beta = 1.6\) gives \(\E[q] = 0.63\). This is the whole point of the KL term: the reward is only a proxy, so we must not optimize it all the way.

Worked example: the optimum at \(\beta = 1\)

Take the five answers of Figure 3: \(\pi_{\text{ref}} = (0.25, 0.30, 0.20, 0.15, 0.10)\) and \(r = (1.2, -2.0, 2.0, 1.5, 3.0)\). Multiply each probability by \(e^{r}\): \(0.25 e^{1.2} = 0.830\), \(0.30 e^{-2} = 0.041\), \(0.20 e^{2} = 1.478\), \(0.15 e^{1.5} = 0.672\), \(0.10 e^{3} = 2.009\). The sum is \(Z = 5.029\), so \(\pi^* = (0.165, 0.008, 0.294, 0.134, 0.399)\). The expected reward rises from 0.625 to 2.168, at a cost of \(\KL = 0.553\) nats. The wrong answer B (“Yes, it is safe”) falls from 30% to 0.8%. But the reward hack E rises from 10% to 39.9%.

Two variants of stage 3 matter today.

  • DPO (Rafailov et al., 2023) skips the reward model and the RL loop. It solves (53.5) for \(r\) and puts that into the Bradley-Terry loss (53.3). \(Z(x)\) cancels in the difference, and what remains is a simple classification loss on preference pairs (session 68):
\[ \mathcal{L}_{\text{DPO}}(\theta) = -\,\E_{(x, y_w, y_l)} \left[ \log \sigma\!\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right] \tag{53.6} \]
Read the symbols
\(\beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\)
the “implicit reward” of answer \(y\): how much more likely the policy makes it than the SFT model did
\(\pi_\theta(y \mid x)\)
the product of the token probabilities of the whole answer (in practice, the sum of log-probabilities over the answer tokens)
  • RL with verifiable rewards (RLVR) replaces the learned reward model with a program that checks the answer: a math answer matches the ground truth, the code passes its tests, the text satisfies a format rule. Tülu 3 uses a reward of \(\alpha = 10\) for a verified answer and 0 otherwise, on 29,946 prompts from GSM8K, MATH and instruction-following constraints, plus the KL term of (53.4). [Certain] A program cannot be flattered, so reward hacking is harder (but not impossible). This is the base of reasoning models; DeepSeek-R1-Zero even applied RL directly to a base model with no SFT. [Certain] See session 70.
\[ v(x, y) = \begin{cases} \alpha & \text{if the checker accepts the answer extracted from } y \\ 0 & \text{otherwise} \end{cases}, \qquad \text{objective: } \E\big[v(x, y)\big] - \beta \KL\big(\pi_\theta \,\|\, \pi_{\text{ref}}\big) \tag{53.7} \]
Algorithm 1: the classic three-stage post-training pipeline
  1. input: base model \(\pi_0\), demonstrations \(\mathcal{D}_{\text{SFT}}\), prompt set \(\mathcal{D}_{\text{RL}}\), labelers (or an AI judge, or a checker)
  2. SFT: render each demonstration with the chat template; train \(\pi_0\) with loss (53.2) for 1 to a few epochs \(\rightarrow \pi_{\text{SFT}}\)
  3. collect preferences: for each prompt, sample \(K\) answers from \(\pi_{\text{SFT}}\); ask labelers to rank them; store all \(\binom{K}{2}\) pairs
  4. reward model: init \(r_\phi\) from a language model with a scalar head; train with (53.3) for about one epoch // more epochs overfit
  5. RL: \(\pi_\theta \leftarrow \pi_{\text{SFT}}\), \(\pi_{\text{ref}} \leftarrow \pi_{\text{SFT}}\) (frozen)
    1. repeat: sample prompts \(x\); sample \(y \sim \pi_\theta(\cdot \mid x)\); reward \(r_\phi(x, y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\)
    2. update \(\pi_\theta\) with PPO to increase this reward // objective (53.4)
  6. alternative to steps 4–5: train \(\pi_\theta\) directly on the pairs with DPO (53.6); or use a checker \(v\) instead of \(r_\phi\) (53.7)
  7. evaluate on chat, instruction following, math, code and safety; return \(\pi_\theta\)

Real recipes repeat and mix these steps. Llama 2 ran five rounds of RLHF (RLHF-V1 to V5), collecting fresh preferences on each new model. [Certain] Qwen2.5-Instruct used over 1 million SFT examples, then offline DPO on about 150,000 pairs, then online RL with GRPO. [Certain] Tülu 3 used SFT, then length-normalized DPO, then RLVR. [Certain] The order “SFT first” is common because the preference and RL stages need a model that already writes in the right format: otherwise most sampled answers are not answers at all, and comparing them teaches little. [Likely]

Instruction data: human-written, synthetic, distilled

All three stages need prompts, and SFT also needs answers. There are three ways to get them, and real recipes use all three.

Human-written

People write prompts and answers. This gives the highest control over quality and style, but it is slow and expensive. InstructGPT’s SFT data came from about 40 contractors, plus real prompts sent to the OpenAI API. [Certain] Llama 2 found that a smaller set of high-quality vendor data beat millions of third-party examples, and stopped after 27,540 SFT annotations: “Quality Is All You Need” is the title of that section. [Certain]

LIMA (Zhou et al., 2023) pushed this further: a 65B LLaMA model fine-tuned on only 1,000 carefully chosen examples (750 from Stack Exchange, wikiHow and similar sites, 250 written by the authors), with no RL at all. In a human study, its answers were equivalent to or better than GPT-4’s in 43% of cases, Bard’s in 58% and DaVinci003’s in 65%. [Certain] The authors proposed the superficial alignment hypothesis: almost all knowledge is learned in pretraining, and alignment mainly teaches the style and format of answers. [Certain] We test this idea with a measurement below.

Synthetic: Self-Instruct

Self-Instruct (Wang et al., 2022) asks: can a model write its own instruction data? Start with a small pool of human-written tasks, and let the model write new ones in a loop. The authors started from 175 seed tasks. In each step, they showed the model 8 instructions from the pool (6 human, 2 model-written) and asked for a new one; then the model wrote inputs and outputs for it. [Certain]

The model tends to write the same few tasks again and again. So the key step is a diversity filter: a new instruction is added only if its ROUGE-L similarity to every instruction already in the pool is below 0.7. [Certain] ROUGE-L compares two texts by their longest common subsequence (LCS): the longest list of words that appear in both, in the same order, not necessarily next to each other.

\[ P = \frac{\text{LCS}(a, b)}{|a|}, \qquad R = \frac{\text{LCS}(a, b)}{|b|}, \qquad \text{ROUGE-L}(a, b) = \frac{2PR}{P + R} = \frac{2\,\text{LCS}(a, b)}{|a| + |b|} \tag{53.8} \]
Read the symbols
\(a, b\)
the new instruction and an instruction from the pool, as lists of lowercase words
\(|a|\)
the number of words in \(a\)
\(P, R\)
precision and recall of the shared subsequence; ROUGE-L is their harmonic mean (F1)
Worked example: is “Write a short poem about the sea.” new?

The pool has “Write a short poem about the ocean.” Both have 7 words. The LCS is “write a short poem about the”: 6 words. So \(P = R = 6/7\) and \(\text{ROUGE-L} = 2 \cdot 6 / (7 + 7) = 0.857\). That is above 0.7, so the new instruction is dropped as a near-duplicate. “Write a haiku about autumn leaves.” (6 words) shares only “write a … about” (3 words) with the ocean poem: \(2 \cdot 3 / (6 + 7) = 0.462\), so it is kept.

Figure 4. The Self-Instruct filter, computed live. The pool starts with 8 seed instructions (not shown). 16 candidate instructions arrive in order; each is compared with formula (53.8) to every instruction in the pool, including the candidates kept before it. The bar shows its highest similarity; the red line is the threshold.
How to read Figure 4

At the paper’s threshold of 0.7, 6 of 16 candidates survive. Near-copies such as “Give two tips for staying healthy.” (0.833 against “Give three tips…”) are dropped. Move the threshold down to 0.5: only 5 survive, because “Write a Python function that checks if a number is prime.” (0.632) now counts as too similar. Move it up to 0.9: 14 survive, and the pool fills with paraphrases. “Describe the picture in detail.” is always removed by the second filter in the paper: words like “image” or “picture” mark tasks a text-only model cannot do.

Self-Instruct turned GPT-3 into an instruction follower with 52K instructions and 82K instances, and gave a 33% absolute improvement over the original GPT-3 on Super-NaturalInstructions. [Certain] The data is noisy: in the authors’ own check of 200 samples, 92% of instructions were valid tasks but only 54% of examples were fully correct. [Certain] Later methods add better filters, such as asking a strong model to rate quality, or checking code by running it. Tülu 3 also uses persona-driven prompts (a strong model writes, for example, a math problem from the point of view of a given persona) to make synthetic data more diverse. [Certain]

Algorithm 2: Self-Instruct data generation
  1. input: seed pool \(S\) of human-written tasks (175 in the paper), language model \(M\), threshold \(\tau = 0.7\)
  2. repeat until the pool is large enough:
    1. sample 8 instructions from the pool (6 human-written, 2 model-written) as few-shot examples
    2. ask \(M\) for a new instruction \(c\)
    3. if \(c\) contains a banned word (image, picture, graph) or \(\max_{s \in S} \text{ROUGE-L}(c, s) \ge \tau\): discard \(c\) // formula (53.8)
    4. ask \(M\) for input–output instances of \(c\) (output first for classification tasks, to balance labels)
    5. drop duplicate or invalid instances; add \(c\) and its instances to \(S\)
  3. return all model-written (instruction, input, output) triples as SFT data

Distilled

Distillation means a strong teacher model writes the answers, and a smaller student is fine-tuned on them with ordinary SFT. This is sequence-level knowledge distillation (Kim and Rush, 2016) applied to chat. [Certain] Stanford’s Alpaca (2023) is the famous example: it used the Self-Instruct recipe with OpenAI’s text-davinci-003 as the teacher to make 52K demonstrations for under $500, then fine-tuned LLaMA 7B in about 3 hours on 8 A100 GPUs. [Certain] The DeepSeek-R1 report fine-tuned Qwen and Llama students on about 800k samples written by DeepSeek-R1, with SFT only and no RL stage. [Certain]

Distillation is cheap and strong, but it has three costs. The student copies the teacher’s errors and style. The student can learn to sound like the teacher without having the knowledge to back it up, which can increase confident mistakes. [Likely] And the teacher’s license may forbid using its outputs to train competing models; check it. More in session 59.

SourceExamplesStrengthWeakness
Human-writtenInstructGPT SFT (12,725 prompts), Llama 2 (27,540), LIMA (1,000)Quality and style under control; real user needsSlow and costly; hard tasks need experts
Synthetic (model writes prompts and answers)Self-Instruct (52K), Tülu 3 personasCheap, large, easy to target a skillNoisy (54% fully correct in Self-Instruct); low diversity without filters
Distilled (strong teacher answers)Alpaca (52K from text-davinci-003), DeepSeek-R1 students (800k)Transfers a strong model’s skill to a small oneCopies teacher errors; license limits; style without substance
Preference pairsInstructGPT (33,207 prompts), Llama 2 (1.4M), Tülu 3 8B (271,409)Easier to judge than to write; teaches “not like this”Labeler disagreement; reward hacking

How much does post-training change the model?

LIMA suggested that alignment is “superficial”. Lin et al. (2023) measured this directly with the token distribution shift. Let the aligned model write an answer \(y\) with greedy decoding. At each position, give the base model the same context and ask: where does the aligned model’s token rank in the base model’s list?

\[ \eta_t = \operatorname{rank}_{p_{\text{base}}(\cdot \mid x,\, y_{\lt t})}\!\big(y_t\big), \qquad \text{position } t \text{ is } \begin{cases} \text{unshifted} & \eta_t = 1 \\ \text{marginal} & 2 \le \eta_t \le 3 \\ \text{shifted} & \eta_t \gt 3 \end{cases} \tag{53.9} \]
Read the symbols
\(y_t\)
the token the aligned (instruct) model chose at position \(t\)
\(p_{\text{base}}(\cdot \mid x, y_{\lt t})\)
the base model’s next-token distribution after the same prompt and the same answer so far
\(\eta_t\)
the “base rank”: 1 means the base model would have picked the same token

For Llama-2-7b against Llama-2-7b-chat, they found 77.7% unshifted, 14.5% marginal and only 7.8% shifted positions, and the shifted tokens were mostly style: “Thank”, “Hello”, “However”, “apolog”, and the end token. [Certain] We repeated the measurement for Qwen2.5-0.5B against Qwen2.5-0.5B-Instruct on 30 prompts (2,054 answer tokens, both models given the same chat-template context): 76.6% unshifted, 16.3% marginal, 7.2% shifted. [Certain, our run] The token that shifted most often was <|im_end|> (16 of 147 shifted positions): the base model ranks it thousands of places down. Knowing when to stop is one of the clearest things post-training adds.

unshifted: base rank 1marginal: rank 2–3shifted: rank > 3
Figure 5. Token distribution shift (formula 53.9), measured on real models. The text is Qwen2.5-0.5B-Instruct’s greedy answer (at most 96 tokens). Each token is colored by its rank under Qwen2.5-0.5B (base) with the same context.
How to read Figure 5

Most text is uncolored: the base model would have written the same token. Red tokens are where post-training changed the choice. Click them. In “I feel sad”, the first token “As” (of “As an AI language model”) has base rank 25; the base model wanted “Sure”. In “Haiku”, the poem’s first word “Wh(ispers)” has rank 21. In every answer that finishes, the final <|im_end|> is red, with base ranks in the tens of thousands. Facts such as “Paris” are unshifted: that knowledge came from pretraining.

This supports the superficial alignment view, with two cautions. First, a small shift per token can still change the meaning of an answer: one shifted token (“No” instead of “Yes”) flips a safety answer. Second, the result is about chat-style answers. RL on math and code (RLVR, reasoning models) changes behavior much more, for example by making answers many times longer. [Likely] Lin et al. also showed that a base model with a good system prompt and three fixed in-context examples (their method, URIAL) can come close to SFT-tuned models on their test set, with no training at all. [Certain]

Chat templates, and why a wrong template breaks a model

All three stages train on chats rendered with one fixed chat template (Session 9 explains the tokens in detail). For Qwen2.5 a single-turn prompt is:

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant

<|im_start|> (ID 151644) and <|im_end|> (ID 151645) are single special tokens. The last line, the generation prompt, tells the model “now the assistant speaks”. [Certain] Through SFT and RL, the model learned a very specific habit: after this exact sequence, write an answer, then write <|im_end|>. The habit is tied to the exact tokens. Change them, and the model is in a situation it never trained on.

To see how much this matters, we ran 20 simple factual questions (“What is the capital of Japan?”, “How many legs does a spider have?”, …) through Qwen2.5-0.5B-Instruct with six different prompt formats. For each, we measured three things: the average NLL (negative log-likelihood) per token of a short correct reference answer; the probability that the model writes <|im_end|> right after that answer; and how the greedy output ends within 128 tokens. Figure 6 shows the results.

Prompt sent to Qwen2.5-0.5B-Instruct
Greedy output
real special tokenlooks like a marker, but is ordinary text
Figure 6. One model, six prompt formats, 20 questions (measured by us). The chart shows the mean probability of <|im_end|> right after a correct short answer, on a log scale. The readout gives all metrics for the chosen format.
How to read Figure 6

Compare the bars: only the two formats that contain the real role tokens give the end-of-turn token a sensible probability (0.45 and 0.74). All others are below \(10^{-5}\): about 70,000 to 1,800,000 times lower. Now read the readouts. With the correct template, 15 of 20 answers stop with <|im_end|>. With every wrong format, none do. “No generation prompt” looks fine on the chart, but read its output: the model writes <|im_start|>user itself and answers in the user’s role, and on 4 of 20 questions it replies “I’m sorry, but I can’t assist with that.” With “Markers as text”, the model writes a fake text marker such as ||im_end|>: it imitates what it sees.

The pattern is clear and important. The knowledge survives a wrong template; the turn structure does not. The answer NLL is about the same for the correct template (0.549 nats per token), the Llama 3 format (0.532), the Alpaca format (0.451) and the markers-as-text format (0.427). [Certain, our run] So the model still “knows” that Paris is the capital of France. What breaks is everything post-training added: where to start, which role to play, and when to stop. Only two formats are clearly worse on NLL: raw text (1.607, because the model expects a document, not an answer) and the missing generation prompt (4.148, because after <|im_end|> the model expects a new header, not an answer).

Worked example: how much worse is “no template”?

With the correct template, the mean probability of <|im_end|> after the reference answer is 0.45. With raw text, it is \(2.54 \times 10^{-7}\): about \(0.45 / 2.54\times10^{-7} \approx 1.8\) million times smaller. In the greedy runs, raw-text outputs had a mean length of 116.8 tokens against 74.2, and 14 of 20 hit the 128-token limit. In a product, this means longer latency, higher cost, and answers that drift into unrelated text. The “markers as text” prompt (the template string with special-token parsing turned off) also costs more: 57 tokens instead of 36 for the same chat, because each marker becomes 6 ordinary tokens.

Why does a wrong template hurt so much, if the model knows the answer? Because post-training is narrow. SFT and RL saw millions of tokens, all in one format. The end-of-turn token, the role headers and the system prompt were in exactly the same places every time. The model learned “after <|im_start|>assistant\n, answer and close”, not a general concept of a conversation. A different format looks to the model more like a pretraining document, so it falls back toward base-model behavior. [Likely] The fix is cheap: always render prompts with the model’s own template, as the code below does.

Algorithm 3: check that a model and its template agree
  1. input: model, tokenizer, a few test questions
  2. prompt \(\leftarrow\) apply_chat_template(messages, add_generation_prompt = true); print it and its token IDs // look for the role tokens as single IDs
  3. confirm the special tokens are single IDs, not split into pieces such as <, |, im, …
  4. generate greedily; confirm the output ends with the end-of-turn token well before the length limit
  5. score a known short answer: \(P(\text{end-of-turn} \mid \text{prompt}, \text{answer})\) should be large (here 0.45), not tiny (\(10^{-6}\))
  6. repeat with the exact string your server sends; any difference from step 2 is a bug

In code: preference loss, KL optimum, data filter, template check

A small, self-contained PyTorch script. It runs the Self-Instruct filter (53.8), trains a Bradley-Terry reward model (53.3) on synthetic preferences, and checks the closed-form KL optimum (53.5) against direct gradient ascent on objective (53.4). It runs on a CPU in a few seconds and prints the same numbers every time (PyTorch 2.8).

import re, torch
torch.manual_seed(0)

# 1) Self-Instruct style filter: keep a new instruction only if ROUGE-L < 0.7 to all kept ones
def words(s): return re.sub(r"[^a-z0-9]+", " ", s.lower()).split()
def rouge_l(a, b):
    a, b = words(a), words(b)
    d = [[0] * (len(b) + 1) for _ in range(len(a) + 1)]
    for i in range(len(a)):
        for j in range(len(b)):
            d[i + 1][j + 1] = d[i][j] + 1 if a[i] == b[j] else max(d[i][j + 1], d[i + 1][j])
    l = d[-1][-1]
    return 0.0 if l == 0 else 2 * l / (len(a) + len(b))       # F1 of LCS precision and recall
pool = ["Give three tips for staying healthy.", "Write a short poem about the ocean."]
for c in ["Give two tips for staying healthy.", "Write a haiku about autumn leaves.",
          "Write a short poem about the sea.", "List five ways to save money on groceries."]:
    s = max(rouge_l(c, p) for p in pool)
    if s < 0.7: pool.append(c)
    print(f"{s:.3f}", "keep" if s < 0.7 else "drop", c)
print(len(pool))                                              # 4

# 2) a reward model from pairwise preferences (Bradley-Terry), formula (53.3)
d, n = 8, 2000
w_true = torch.randn(d)                                       # hidden "human taste"
fa, fb = torch.randn(n, d), torch.randn(n, d)                 # features of two answers per prompt
pref_a = torch.rand(n) < torch.sigmoid((fa - fb) @ w_true)    # noisy human choice
fw = torch.where(pref_a[:, None], fa, fb); fl = torch.where(pref_a[:, None], fb, fa)
w = torch.zeros(d, requires_grad=True)
opt = torch.optim.Adam([w], lr=0.05)
for step in range(300):
    loss = -torch.nn.functional.logsigmoid((fw - fl) @ w).mean()
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    acc = (((fw - fl) @ w) > 0).float().mean().item()
    cos = torch.nn.functional.cosine_similarity(w, w_true, dim=0).item()
print(round(loss.item(), 3), round(acc, 3), round(cos, 3))   # 0.25 0.888 0.999

# 3) the KL-regularized optimum, formula (53.5), checked by gradient ascent on logits
ref = torch.tensor([0.25, 0.30, 0.20, 0.15, 0.10])            # SFT policy over 5 answers
r = torch.tensor([1.2, -2.0, 2.0, 1.5, 3.0])                  # reward-model scores
beta = 1.0
closed = ref * torch.exp(r / beta); closed = closed / closed.sum()
z = torch.log(ref).clone().requires_grad_(True)               # start at the reference policy
opt = torch.optim.Adam([z], lr=0.05)
for step in range(2000):
    p = torch.softmax(z, 0)
    J = (p * r).sum() - beta * (p * (p / ref).log()).sum()     # objective (53.4)
    opt.zero_grad(); (-J).backward(); opt.step()
p = torch.softmax(z, 0).detach()
print([round(x, 3) for x in closed.tolist()])          # [0.165, 0.008, 0.294, 0.134, 0.399]
print([round(x, 3) for x in p.tolist()])               # [0.165, 0.008, 0.294, 0.134, 0.399]
kl = (closed * (closed / ref).log()).sum()
print(round((closed * r).sum().item(), 3), round(kl.item(), 3))  # 2.168 0.553

The first part prints 0.833 drop, 0.462 keep, 0.857 drop, 0.000 keep, then a pool of 4. The reward model reaches 88.8% training accuracy, which is close to the best possible here, because the simulated “humans” are noisy; and its weight vector points almost exactly in the direction of the hidden taste (cosine 0.999). The gradient ascent lands on the same five probabilities as formula (53.5). To check a real model’s template (Algorithm 3), the script we used for Figure 6 boils down to this (needs transformers and a 1 GB model download; we ran it with transformers 4.57):

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval()
msgs = [{"role": "user", "content": "What is the capital of France?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True)       # the correct prompt
print(len(ids), tok.convert_ids_to_tokens(ids[:3]))                    # 36 ['<|im_start|>', 'system', 'Ċ']
ans = tok("The capital of France is Paris.", add_special_tokens=False).input_ids
x = torch.tensor([ids + ans + [tok.convert_tokens_to_ids("<|im_end|>")]])
with torch.no_grad():
    logp = torch.log_softmax(model(x).logits[0], -1)
p_end = logp[len(ids) + len(ans) - 1, x[0, -1]].exp().item()
print(round(p_end, 3))         # 0.904 here; about 1e-6 with a wrong template
out = model.generate(torch.tensor([ids]), max_new_tokens=64, do_sample=False)
print(repr(tok.decode(out[0, len(ids):])))                           # 'The capital of France is Paris.<|im_end|>'

Where this is used

SettingWhich stagesWhat matters most
General chat assistants (Llama, Qwen, Tülu)SFT, then DPO and/or RLHF, often RLVRDiverse prompts; evaluate helpfulness, instruction following and safety after every stage
Domain assistants (legal, medical, support bots)SFT from an instruct model on domain chats; sometimes DPOStart from the instruct model, keep its template; mix in general data to avoid forgetting (session 61)
Small on-device modelsDistillation SFT from a large teacher, then DPOTeacher quality and license; the student cannot learn facts it has no capacity for
Reasoning models (math, code)SFT on long reasoning traces, then RL with verifiable rewardsReliable checkers; watch answer length and reward hacking (session 72)
Agents and tool useSFT on tool-call traces in the template’s tool format; RL on task successTool-call tokens and stop tokens exactly as in the template (session 154)
Vision-language assistantsAlignment pretraining, visual instruction SFT, then preference tuningImage placeholders in the template; preference data against hallucination (session 186)
Video QA and Video-RAG assistantsVideo instruction SFT (questions with timestamps), then preference tuningAnswers that point to key moments; frames as placeholders in the chat template; refusing when the video does not show the answer
Video retrieval and classification with an LLM judgeUsually none: use an existing instruct model in zero-shot modeThe judge’s own template and a fixed output format, so that labels can be parsed reliably

Common mistakes

  • Expecting a base model to chat. A base model continues documents. Use few-shot prompts or URIAL-style examples, or use the instruct version. And never evaluate a base model with a chat template (session 40).
  • Fine-tuning an instruct model with a different template. If you SFT Qwen-Instruct on Alpaca-style text, you fight its existing habits and may get a model that knows two formats badly. Use the model’s own template.
  • Forgetting the generation prompt. Without <|im_start|>assistant\n at the end, the model writes the header itself; in our test it chose the user role every time (20 of 20).
  • Masking the end-of-turn token in SFT. Then nothing teaches the model to stop, and it rambles. The end token must have \(m_t = 1\) in (53.2).
  • Too small a KL coefficient, or none. The policy finds what the reward model overrates (length, flattery, repeated phrases). Track the KL and an independent evaluation during RL, not only the reward.
  • Training the reward model for many epochs. InstructGPT found it overfits after one pass; ranked pairs from one prompt are highly correlated. [Certain]
  • Trusting synthetic or distilled data blindly. Self-Instruct data was only 54% fully correct. Filter for duplicates (ROUGE-L, embeddings), quality (a judge model), and benchmark contamination.
  • Measuring only the skill you trained. DPO and RL can raise chat scores and lower safety or math scores. Run the full evaluation suite after each stage.

Interview questions

What is the difference between a base model and an instruct (chat) model?basic
A base model is trained only with next-token prediction on documents (53.1). Given a question, it predicts what follows in a document: often more questions, a quiz format, or a continuation of the text. An instruct model was post-trained (SFT, then usually preference tuning and RL) on chats in a fixed template. It treats the input as a user request, answers in the assistant role, follows the requested form, refuses some requests, and ends its turn with an end-of-turn token. Knowledge comes mostly from pretraining; the behavior comes from post-training. In our Qwen2.5-0.5B runs, the base model answered “How do I boil an egg?” with a paraphrase quiz; the instruct model with steps.
Walk me through the classic post-training pipeline.basic
(1) SFT: fine-tune the base model on prompt–answer demonstrations, loss only on assistant tokens (53.2). (2) Preference data: sample several answers per prompt from the SFT model, have people rank them, and train a reward model with the Bradley-Terry loss (53.3). (3) RL: optimize the SFT model with PPO to maximize the reward minus a KL penalty to the SFT model (53.4). InstructGPT used about 13k SFT prompts, 33k reward-model prompts and 31k PPO prompts. Modern variants replace (2)+(3) with DPO (53.6), or the reward model with a checker (RLVR, 53.7), and often repeat rounds.
Why does SFT come before preference tuning?core
Preference tuning and RL learn from answers that the model itself writes. A base model often does not write answers at all (it continues the document), so comparing two of its outputs teaches little, and RL would spend most samples on useless text. SFT first puts the model into the right format, so its samples are real answers that differ in quality. SFT also gives the reference policy \(\pi_{\text{ref}}\) for the KL term. (Exception: DeepSeek-R1-Zero ran RL directly on a base model with a verifiable reward and a simple format, showing it is possible for reasoning tasks.)
Derive the optimal policy of the KL-regularized RL objective. Why does it matter for DPO?advanced
For one prompt, maximize \(\sum_y \pi(y) r(y) - \beta \sum_y \pi(y) \log(\pi(y)/\pi_{\text{ref}}(y))\) subject to \(\sum_y \pi(y) = 1\). The Lagrangian derivative is \(r(y) - \beta(\log(\pi(y)/\pi_{\text{ref}}(y)) + 1) - \lambda = 0\), so \(\pi^*(y) \propto \pi_{\text{ref}}(y) e^{r(y)/\beta}\) (53.5). Inverting: \(r(y) = \beta \log(\pi^*(y)/\pi_{\text{ref}}(y)) + \beta \log Z\). Put this into the Bradley-Terry loss: \(\beta\log Z\) cancels in the difference \(r(y_w) - r(y_l)\), and we get the DPO loss (53.6), which trains the policy directly on preference pairs, with no reward model and no sampling.
What is the KL penalty for in RLHF, and what happens if \(\beta\) is too small or too large?core
The reward model is an imperfect proxy trained on a limited set of answers. Maximizing it without limits leads to reward hacking: the policy finds outputs the reward model overrates but people do not like (verbosity, flattery, odd repeated phrases), and it can drift away from fluent language. The KL term keeps the policy near the SFT model, where the reward model is reliable. Too small \(\beta\): reward goes up, true quality goes down (Figure 3: \(\E[q]\) falls to \(-1.0\)). Too large \(\beta\): the policy barely changes from SFT. InstructGPT used \(\beta = 0.02\) per token; Tülu 3 8B used \(\beta = 0.05\).
How is a reward model trained, and why are only score differences meaningful?core
Take a language model, replace the output layer by a scalar head, and train it on pairs \((x, y_w, y_l)\) with \(-\log\sigma(r(x,y_w) - r(x,y_l))\) (Bradley-Terry). The probability of a preference depends only on the difference of scores, so adding any constant to all scores (even a prompt-dependent one) gives the same loss. So raw reward values are not comparable across prompts; people normalize them (InstructGPT shifted rewards so human demonstrations average 0). InstructGPT trained on all \(\binom{K}{2}\) pairs of one ranking in one batch for one epoch, because more passes overfit.
Compare human-written, synthetic and distilled instruction data.core
Human-written (InstructGPT, Llama 2’s 27,540 annotations, LIMA’s 1,000): best control of quality and real user intent, but slow and expensive. Synthetic (Self-Instruct: a model writes new tasks from 175 seeds, filtered by ROUGE-L < 0.7): cheap and large, but noisy (54% fully correct) and needs diversity and quality filters. Distilled (Alpaca from text-davinci-003, DeepSeek-R1 students on 800k samples): a strong teacher writes the answers, very effective for small models, but copies teacher errors, may teach style without knowledge, and has license limits. Real recipes (Tülu 3) mix all three and decontaminate against benchmarks.
What is the superficial alignment hypothesis, and what evidence supports it?advanced
It says a model learns almost all its knowledge in pretraining, and alignment mostly teaches the format and style of answers. Evidence: LIMA (65B, only 1,000 SFT examples, no RL) was rated equal to or better than GPT-4 in 43% of cases. Token-shift analysis (URIAL): for Llama-2-7b vs Llama-2-7b-chat, 77.7% of answer tokens are the base model’s top choice and only 7.8% are outside its top 3, mostly stylistic tokens; our own Qwen2.5-0.5B measurement gives 76.6% / 16.3% / 7.2%, with the end-of-turn token as the most-shifted token. Limits: a few shifted tokens can flip meaning, and RL for reasoning changes behavior more deeply.
What happens if you send a chat model a prompt without its chat template, or with another model’s template?core
The model still knows the facts, but its turn behavior breaks. In our test of Qwen2.5-0.5B-Instruct on 20 questions, the answer NLL was about the same with the Llama 3 or Alpaca format as with the correct one (0.53 / 0.45 vs 0.55 nats per token), but the probability of <|im_end|> after the answer fell from 0.45 to below \(10^{-5}\), and 0 of 20 answers ended their turn (15 of 20 with the right template). The model rambled, wrote fake markers as text, or ran to the length limit. Without the generation prompt, it wrote a user header itself and answered as the user.
What is RLVR and why is it popular for math and code?advanced
RL with verifiable rewards uses a program instead of a learned reward model: the reward is \(\alpha\) if the extracted answer matches the ground truth (or the code passes tests, or the format constraint holds) and 0 otherwise (53.7), still with a KL term. A checker cannot be flattered and does not drift, so reward hacking is much harder, and no preference labels are needed. Tülu 3 used \(\alpha = 10\) on about 30k prompts (GSM8K, MATH, IFEval-style constraints) and improved those scores. It only works where answers can be checked; open-ended chat still needs preference data.
Your SFT model gives good answers but never stops writing. List the likely causes.core
(1) The end-of-turn token was masked out of the loss (or removed because PAD = EOS and all PAD-ID labels were set to −100), so the model never learned to emit it. (2) The serving code does not stop on the end-of-turn token (it stops only on end-of-text). (3) The prompt at inference does not match the training template (no generation prompt, wrong markers, markers tokenized as text), so the trained stop habit does not fire. (4) The template at training did not put an end token after assistant turns at all. Fix the mask, the stop list and the template, then confirm that \(P(\text{end} \mid \text{answer})\) is large.

Exercises

  1. A labeler ranks \(K = 6\) answers for one prompt. How many preference pairs does this give? If one pair has reward-model scores 1.0 (preferred) and −1.0 (rejected), what is the Bradley-Terry loss for that pair? What if the scores were swapped?
  2. Two answers, \(\pi_{\text{ref}} = (0.5, 0.5)\), rewards \(r = (1, 0)\), \(\beta = 0.5\). Compute \(\pi^*\) with (53.5) and the KL from \(\pi_{\text{ref}}\).
  3. Compute ROUGE-L between “Translate the sentence into German.” and “Translate the sentence into French.” Is the first kept by Self-Instruct’s filter?
  4. InstructGPT’s 175B SFT model used 4.9 petaflop/s-days and the PPO-ptx model 60, against 3,640 for GPT-3 pretraining. What fraction of the pretraining compute is each?
  5. In Figure 5, “Boil an egg” has 96 answer tokens: 72 unshifted, 16 marginal, 8 shifted. Give the three percentages. Which shifted token do you expect in almost every answer that ends before the length limit, and why?
  6. Your company fine-tunes Qwen2.5-Instruct on support chats formatted as ### Customer: … ### Agent: …. Predict two problems, and give the fix.
  7. Use Figure 3. Find the \(\beta\) where the reward-hack answer E has about 50% of the probability. What are the expected reward and the true quality there?
Answerscheck
1. \(\binom{6}{2} = 15\) pairs. Difference 2: \(\sigma(2) = 0.881\), loss \(= -\log 0.881 = 0.127\). Swapped (difference −2): loss \(= -\log \sigma(-2) = -\log 0.119 = 2.127\). 2. Weights \(0.5 e^{2} = 3.695\) and \(0.5 e^{0} = 0.5\), so \(\pi^* = (0.881, 0.119)\). \(\KL = 0.881 \ln(0.881/0.5) + 0.119 \ln(0.119/0.5) = 0.328\) nats. Expected reward 0.881 (was 0.5). 3. Words: 5 and 5, LCS “translate the sentence into” = 4. \(\text{ROUGE-L} = 2 \cdot 4 / 10 = 0.8 \ge 0.7\): dropped. 4. \(4.9 / 3640 = 0.13\%\) and \(60 / 3640 = 1.6\%\). 5. 75.0%, 16.7%, 8.3%. The end-of-turn token <|im_end|>: the base model was never trained to end a reply there (it ranks it thousands of places down), while the instruct model learned it in SFT. 6. The new format conflicts with the ChatML habits: the model may mix both formats, and it will not learn to stop unless an end marker follows each agent turn and gets a loss; at serving, the old template or stop token may be used by mistake. Fix: render the support chats with the model’s own chat template (customer = user, agent = assistant), mask the loss to assistant turns including <|im_end|>, and serve with the same template. 7. About \(\beta = 0.76\) (E has about 50%; at \(\beta = 0.70\) it has 54.1%). There \(\E[r] \approx 2.35\) and \(\E[q] \approx 0.27\): the reward is high, but true quality is less than half of its best value, 0.63 near \(\beta = 1.6\).

Key takeaways

  • A base model continues documents (53.1). An assistant answers in a role, follows the requested form, and stops with an end-of-turn token. Our Qwen2.5-0.5B runs show both behaviors on the same questions.
  • The classic pipeline is SFT (53.2), then preference data with a Bradley-Terry reward model (53.3), then RL with a KL leash (53.4). Its optimum (53.5) tilts the SFT policy by \(e^{r/\beta}\); DPO (53.6) and RLVR (53.7) are the modern shortcuts.
  • Post-training is cheap compared with pretraining (InstructGPT: under 2% of GPT-3’s compute) and very effective (1.3B InstructGPT preferred over 175B GPT-3).
  • Instruction data is human-written, synthetic (Self-Instruct with a ROUGE-L < 0.7 filter, 53.8) or distilled from a teacher. Quality and diversity beat quantity (LIMA: 1,000 examples).
  • Post-training changes few tokens: about 77% of an aligned answer is the base model’s top choice (53.9). What changes is style, persona and, most clearly, when to stop.
  • A wrong template keeps the knowledge but breaks the turn structure: in our test, P(end-of-turn) fell from 0.45 to below \(10^{-5}\) and no answer ended properly. Always use the model’s own template and check the token IDs.

Key formulas

\[ \mathcal{L}_{\text{SFT}} = -\frac{\sum_t m_t \log \pi_\theta(x_t \mid x_{\lt t})}{\sum_t m_t}, \qquad \mathcal{L}_{\text{RM}} = -\E\big[\log \sigma\big(r(x, y_w) - r(x, y_l)\big)\big] \]
\[ \max_\pi \E[r] - \beta \KL(\pi \,\|\, \pi_{\text{ref}}) \;\Rightarrow\; \pi^*(y \mid x) = \frac{\pi_{\text{ref}}(y \mid x)\, e^{r(x, y)/\beta}}{Z(x)} \]
\[ \mathcal{L}_{\text{DPO}} = -\E\left[\log\sigma\!\left(\beta \log\frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta\log\frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right], \qquad \text{ROUGE-L} = \frac{2\,\text{LCS}(a, b)}{|a| + |b|} \]

References for this session

  1. Ouyang et al. (2022). Training language models to follow instructions with human feedback
  2. Wang et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions
  3. Zhou et al. (2023). LIMA: Less Is More for Alignment
  4. Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
  5. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 8: Post-training
  6. Ouyang et al. (2022). Training language models to follow instructions with human feedback
  7. Wang et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions
  8. Zhou et al. (2023). LIMA: Less Is More for Alignment
  9. Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
  10. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 8: Post-training
  11. Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences
  12. Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  13. Schulman et al. (2017). Proximal Policy Optimization Algorithms
  14. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  15. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  16. Kim & Rush (2016). Sequence-Level Knowledge Distillation
  17. Lambert (2025). Reinforcement Learning from Human Feedback (book)
  18. Taori et al. (2023). Alpaca: A Strong, Replicable Instruction-Following Model
  19. Lin et al. (2023). The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
  20. Qwen Team (2024). Qwen2.5 Technical Report
  21. Hugging Face (2024). Chat templates (Transformers documentation)