From base model to assistant
A pretrained model continues documents. An assistant answers questions, follows instructions, and stops when it is done. This session shows the difference with real outputs from Qwen2.5-0.5B base and instruct, then walks through the post-training stages (SFT, preference tuning, RL), where the instruction data comes from, and why a wrong chat template quietly breaks a good model.
What you will learn
- What a base model does and what an assistant does, seen in real greedy outputs from the same 0.5B model before and after post-training
- The standard pipeline: SFT, then preference tuning (reward model or DPO), then RL, with the real data sizes of InstructGPT and Tülu 3
- The math of each stage: masked SFT loss, the Bradley-Terry reward loss, the KL-regularized RL objective and its closed-form optimum
- Where instruction data comes from: human-written (InstructGPT, LIMA), synthetic (Self-Instruct, with its ROUGE-L filter) and distilled (Alpaca, DeepSeek-R1 students)
- How much post-training really changes: the superficial alignment hypothesis and the token-shift measurement (about 77% of tokens unchanged)
- Why a wrong or missing chat template breaks a model: what we measured on Qwen2.5-0.5B-Instruct, and how to check it
Why this matters
After pretraining, a language model knows a lot: facts, grammar, code, some reasoning. But it is not an assistant. Ask it a question, and it may write three more questions, or continue your sentence as if it were a web page. This session is about the short, cheap training that turns such a base model into an assistant that answers, follows instructions, refuses some requests, and stops when it is done.
This training is called post-training (everything after pretraining). In 2022, OpenAI showed with InstructGPT that a 1.3B-parameter model after post-training was preferred by human raters over the 175B-parameter GPT-3, a model about 100 times larger. [Certain] Post-training also cost very little: 4.9 petaflop/s-days for their 175B SFT model and 60 for the full RL model, against 3,640 for pretraining GPT-3. [Certain] Every chat model you use today, from Llama and Qwen to GPT and Claude, went through some version of this pipeline.
This session is the map of the Track 4 track. We look at real outputs of one small model before and after post-training, then at each stage (SFT, preference tuning, RL) with its loss and real data sizes, then at where the data comes from, and finally at the chat template: the format that holds it all together, and the most common reason a good model gives bad answers in production. Every model output shown here comes from running Qwen2.5-0.5B (base) and Qwen2.5-0.5B-Instruct ourselves, with greedy decoding on a CPU. [Certain]
What a base model does, and what an assistant does
A base model was trained on one task: predict the next token of a document (session 6). Its loss is the average negative log-likelihood over all tokens of a huge text corpus:
- Read the symbols
- \(\mathbf{x} = (x_1, \dots, x_T)\)
- a document from the pretraining corpus, as tokens
- \(p_\theta(x_t \mid x_{\lt t})\)
- the model’s probability for the true next token, given all earlier tokens
- \(\theta\)
- all model weights
Nothing in (53.1) says “be helpful”. The model learns to imitate documents. So when it sees “How do I boil an egg?”, it asks itself: in what kind of document does this line appear, and what comes next? On a forum, more questions may follow. In a quiz, answer options follow. In a dataset of paraphrase tasks, “Do these two questions mean the same?” follows. All of these are good continuations of a document. None of them is what a user wants.
An assistant is a model whose most likely continuation of a user message is a helpful reply in the assistant’s role, which ends with an end-of-turn token. It needs four behaviors that pretraining does not give directly:
- Answer, do not continue. Treat the input as a request from someone else, not as text to extend.
- Follow the instruction’s form. “Three tips” means three. “In one sentence” means one.
- Stop. Emit the end-of-turn token when the answer is complete, so the user can speak again.
- Keep a persona and rules. Respect the system prompt, refuse clearly harmful requests, admit uncertainty.
Figure 1 shows this with real outputs. The same question goes to the base model and to the instruct model, either as raw text or wrapped in the chat template (session 9).
How to read Figure 1
Start with “Boil an egg”, base model, raw text: the model does not answer; it writes a paraphrase-detection exercise, because such exercises were in its training data. Switch to “I feel sad”: the base model continues the diary entry and gets stuck in a loop. Now switch to the instruct model with the chat template: it answers, and on “Capital of France” it ends with <|im_end|>. Look at the bars: with the template, the instruct model is much more sure how to begin (71% for “Bo” on the egg question, against 12% for the most likely raw-text start). Finally, try “Raw chicken” with the instruct model. It says raw chicken is “generally safe”, which is false and dangerous. Post-training taught the format, not the facts.
Three lessons from Figure 1.
First, the base model is a document continuer. On “How do I boil an egg?” it writes “What is the best way to boil an egg? Do those questions have the same meaning? Options are: (1). no. (2). yes.” This is the format of the FLAN-style paraphrase datasets that are part of many pretraining mixtures. On “I feel sad today.” it writes a diary and falls into a repetition loop. [Certain, from our runs]
Second, modern base models are already a little chatty. The Qwen2.5 base model answers “Capital of France” correctly and even writes a numbered recipe when given the chat template. This is not an accident: the Qwen2.5 report says its pretraining data includes synthetic data generated by Qwen2-72B-Instruct. [Certain] Many recent base models have seen instruction-like data before post-training, so the line between “base” and “instruct” is not as sharp as it was for GPT-3. [Likely]
Third, the instruct model needs its template. With raw text, even the instruct model answers and then keeps going (“To verify this, I will use a simple Python code snippet…”) until the token limit. With the template, it answers in one line and stops. We measure this effect carefully in the last part of this session.
The stages: SFT, preference tuning, RL
The classic recipe has three training stages after pretraining. It was set by InstructGPT (Ouyang et al., 2022), building on earlier work on learning from human preferences (Christiano et al., 2017). [Certain] Figure 2 steps through it with the real numbers from InstructGPT and from Tülu 3 (Lambert et al., 2024), a fully open recipe built on Llama 3.1 base models.
How to read Figure 2
Press Next to move one stage. Each box is a training stage; the arrow means “initialize the next stage from this model”. Under the boxes: what data the stage uses, what the training signal is, and what comes out. In the table, compare the data sizes: pretraining uses trillions of tokens, SFT uses thousands to about a million examples, preference tuning a few hundred thousand pairs, RL tens of thousands of prompts. On the last step, see which stage helps which skill, and note that the safety score goes down as the model becomes more helpful.
Stage 1: supervised fine-tuning (SFT)
SFT trains the base model on examples of the behavior we want: a prompt, and a good answer written by a person or a strong model. The loss is the same next-token loss as (53.1), but only on the answer tokens. The prompt is read, not predicted:
- Read the symbols
- \(\pi_\theta\)
- the model, now called a policy: a rule that picks the next token; same network as \(p_\theta\)
- \(\mathbf{x}\)
- one chat rendered with the chat template: system, user and assistant turns as one token sequence
- \(m_t\)
- the loss mask: 1 on what the assistant wrote, 0 on everything else (formula 9.4 in Session 9)
SFT is where the model learns the format: the roles, where answers start, and where they end. Because the mask includes the assistant’s end-of-turn token, SFT is the stage that teaches the model to stop. InstructGPT used about 13k prompts (12,725 training prompts) with answers written by about 40 contractors, and trained for 16 epochs. [Certain] Tülu 3 used 939,344 prompts for 2 epochs. [Certain] The details (masking, packing, epochs, overfitting) are in session 54.
SFT has one limit. It can only show the model good answers; it never shows a bad answer and says “not like this”. And it needs someone to write the ideal answer, which is slow for hard tasks. It is often easier for a person to compare two answers than to write a perfect one. That is what stage 2 uses.
Stage 2: preference data and the reward model
For each prompt \(x\), the SFT model writes several answers, and a person says which one is better. One record is a triple \((x, y_w, y_l)\): the prompt, the preferred (“winning”) answer and the rejected (“losing”) answer. InstructGPT showed labelers between \(K = 4\) and \(K = 9\) answers per prompt and asked for a full ranking, which gives \(\binom{K}{2}\) pairs: up to 36 pairs from one prompt. [Certain]
A reward model \(r_\phi(x, y)\) is a copy of the language model with its output layer replaced by a single number: a score for the whole answer. It is trained with the Bradley-Terry model, which says that the chance a person prefers \(y_w\) depends only on the difference of the two scores:
- Read the symbols
- \(r_\phi(x, y)\)
- the reward model’s score for answer \(y\) to prompt \(x\); one number
- \(y_w \succ y_l\)
- “the person preferred \(y_w\) over \(y_l\)”
- \(\sigma(z) = 1/(1 + e^{-z})\)
- the logistic function; it turns a score difference into a probability
Only differences matter: adding a constant to every score changes nothing. (InstructGPT therefore shifted its reward model so that human demonstrations score 0 on average. [Certain]) The reward model learns what people like from many comparisons. InstructGPT trained a 6B reward model on 33,207 prompts; Llama 2 collected 1,418,091 of its own comparisons. [Certain] Reward models are the topic of session 67.
The reward model scores the preferred answer 0.5 and the rejected one −0.5. The difference is 1, so the model says the person prefers \(y_w\) with probability \(\sigma(1) = 0.731\). The loss for this pair is \(-\log 0.731 = 0.313\). If the scores were reversed (difference −1), the loss would be \(-\log \sigma(-1) = 1.313\): a large push to fix the order. With \(K = 9\) ranked answers, one prompt gives \(\binom{9}{2} = 36\) such pairs. InstructGPT puts all 36 pairs of one prompt in the same batch, because treating them as separate data points made the reward model overfit after a single pass. [Certain]
Stage 3: reinforcement learning against the reward
Now we have a judge. In the RL stage, the model writes answers to new prompts, the reward model scores them, and the model is updated to make high-scoring answers more likely (session 64, session 66). There is one danger. The reward model is only an estimate of human taste, trained on a limited set of answers. If we push the policy to maximize it without limits, the policy finds answers the reward model overrates: very long text, repeated safety phrases, flattery. This is reward hacking (also called over-optimization). The fix is to stay close to the SFT model, the reference policy \(\pi_{\text{ref}}\), with a KL penalty (session 3):
- Read the symbols
- \(\mathcal{D}\)
- the set of prompts used for RL (only prompts: the answers are sampled from the model itself)
- \(\pi_{\text{ref}}\)
- the frozen SFT model; the policy starts equal to it
- \(\KL(\pi_\theta \| \pi_{\text{ref}})\)
- how far the new answer distribution has moved from the SFT one, in nats; zero when they are equal
- \(\beta\)
- the strength of the leash; InstructGPT used \(\beta = 0.02\), Tülu 3 8B used \(\beta = 0.05\) [Certain]
InstructGPT also added a third term: \(\gamma\) times the pretraining log-likelihood, with \(\gamma = 27.8\), to stop the model from forgetting skills it had before (they call this “PPO-ptx”). [Certain] For a fixed prompt, objective (53.4) has an exact solution. Setting the derivative to zero, with the constraint that probabilities add up to 1, gives:
- Read the symbols
- \(\pi^*\)
- the best possible policy for objective (53.4)
- \(Z(x)\)
- a normalizer so the probabilities add up to 1; it has a sum over all possible answers, so we can never compute it for real text
- \(\exp(r/\beta)\)
- the reweighting: every answer keeps its SFT probability, multiplied by a factor that grows with its reward
Read (53.5) as: “start from the SFT model and tilt it toward high reward”. A small \(\beta\) tilts hard; a large \(\beta\) barely moves. Figure 3 lets you feel this; we place it right here because it is the key picture of RLHF.
How to read Figure 3
Move \(\beta\) to the left (small \(\beta\), weak leash). The policy puts almost all its mass on E, the answer with the highest reward: expected reward goes up to 3.0, but true quality falls to −1.0. Move \(\beta\) to the right (strong leash): the policy stays close to the grey bars and keeps the bad answer B. Somewhere in the middle, true quality is best: about \(\beta = 1.6\) gives \(\E[q] = 0.63\). This is the whole point of the KL term: the reward is only a proxy, so we must not optimize it all the way.
Take the five answers of Figure 3: \(\pi_{\text{ref}} = (0.25, 0.30, 0.20, 0.15, 0.10)\) and \(r = (1.2, -2.0, 2.0, 1.5, 3.0)\). Multiply each probability by \(e^{r}\): \(0.25 e^{1.2} = 0.830\), \(0.30 e^{-2} = 0.041\), \(0.20 e^{2} = 1.478\), \(0.15 e^{1.5} = 0.672\), \(0.10 e^{3} = 2.009\). The sum is \(Z = 5.029\), so \(\pi^* = (0.165, 0.008, 0.294, 0.134, 0.399)\). The expected reward rises from 0.625 to 2.168, at a cost of \(\KL = 0.553\) nats. The wrong answer B (“Yes, it is safe”) falls from 30% to 0.8%. But the reward hack E rises from 10% to 39.9%.
Two variants of stage 3 matter today.
- DPO (Rafailov et al., 2023) skips the reward model and the RL loop. It solves (53.5) for \(r\) and puts that into the Bradley-Terry loss (53.3). \(Z(x)\) cancels in the difference, and what remains is a simple classification loss on preference pairs (session 68):
- Read the symbols
- \(\beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\)
- the “implicit reward” of answer \(y\): how much more likely the policy makes it than the SFT model did
- \(\pi_\theta(y \mid x)\)
- the product of the token probabilities of the whole answer (in practice, the sum of log-probabilities over the answer tokens)
- RL with verifiable rewards (RLVR) replaces the learned reward model with a program that checks the answer: a math answer matches the ground truth, the code passes its tests, the text satisfies a format rule. Tülu 3 uses a reward of \(\alpha = 10\) for a verified answer and 0 otherwise, on 29,946 prompts from GSM8K, MATH and instruction-following constraints, plus the KL term of (53.4). [Certain] A program cannot be flattered, so reward hacking is harder (but not impossible). This is the base of reasoning models; DeepSeek-R1-Zero even applied RL directly to a base model with no SFT. [Certain] See session 70.
- input: base model \(\pi_0\), demonstrations \(\mathcal{D}_{\text{SFT}}\), prompt set \(\mathcal{D}_{\text{RL}}\), labelers (or an AI judge, or a checker)
- SFT: render each demonstration with the chat template; train \(\pi_0\) with loss (53.2) for 1 to a few epochs \(\rightarrow \pi_{\text{SFT}}\)
- collect preferences: for each prompt, sample \(K\) answers from \(\pi_{\text{SFT}}\); ask labelers to rank them; store all \(\binom{K}{2}\) pairs
- reward model: init \(r_\phi\) from a language model with a scalar head; train with (53.3) for about one epoch // more epochs overfit
- RL: \(\pi_\theta \leftarrow \pi_{\text{SFT}}\), \(\pi_{\text{ref}} \leftarrow \pi_{\text{SFT}}\) (frozen)
- repeat: sample prompts \(x\); sample \(y \sim \pi_\theta(\cdot \mid x)\); reward \(r_\phi(x, y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\)
- update \(\pi_\theta\) with PPO to increase this reward // objective (53.4)
- alternative to steps 4–5: train \(\pi_\theta\) directly on the pairs with DPO (53.6); or use a checker \(v\) instead of \(r_\phi\) (53.7)
- evaluate on chat, instruction following, math, code and safety; return \(\pi_\theta\)
Real recipes repeat and mix these steps. Llama 2 ran five rounds of RLHF (RLHF-V1 to V5), collecting fresh preferences on each new model. [Certain] Qwen2.5-Instruct used over 1 million SFT examples, then offline DPO on about 150,000 pairs, then online RL with GRPO. [Certain] Tülu 3 used SFT, then length-normalized DPO, then RLVR. [Certain] The order “SFT first” is common because the preference and RL stages need a model that already writes in the right format: otherwise most sampled answers are not answers at all, and comparing them teaches little. [Likely]
Instruction data: human-written, synthetic, distilled
All three stages need prompts, and SFT also needs answers. There are three ways to get them, and real recipes use all three.
Human-written
People write prompts and answers. This gives the highest control over quality and style, but it is slow and expensive. InstructGPT’s SFT data came from about 40 contractors, plus real prompts sent to the OpenAI API. [Certain] Llama 2 found that a smaller set of high-quality vendor data beat millions of third-party examples, and stopped after 27,540 SFT annotations: “Quality Is All You Need” is the title of that section. [Certain]
LIMA (Zhou et al., 2023) pushed this further: a 65B LLaMA model fine-tuned on only 1,000 carefully chosen examples (750 from Stack Exchange, wikiHow and similar sites, 250 written by the authors), with no RL at all. In a human study, its answers were equivalent to or better than GPT-4’s in 43% of cases, Bard’s in 58% and DaVinci003’s in 65%. [Certain] The authors proposed the superficial alignment hypothesis: almost all knowledge is learned in pretraining, and alignment mainly teaches the style and format of answers. [Certain] We test this idea with a measurement below.
Synthetic: Self-Instruct
Self-Instruct (Wang et al., 2022) asks: can a model write its own instruction data? Start with a small pool of human-written tasks, and let the model write new ones in a loop. The authors started from 175 seed tasks. In each step, they showed the model 8 instructions from the pool (6 human, 2 model-written) and asked for a new one; then the model wrote inputs and outputs for it. [Certain]
The model tends to write the same few tasks again and again. So the key step is a diversity filter: a new instruction is added only if its ROUGE-L similarity to every instruction already in the pool is below 0.7. [Certain] ROUGE-L compares two texts by their longest common subsequence (LCS): the longest list of words that appear in both, in the same order, not necessarily next to each other.
- Read the symbols
- \(a, b\)
- the new instruction and an instruction from the pool, as lists of lowercase words
- \(|a|\)
- the number of words in \(a\)
- \(P, R\)
- precision and recall of the shared subsequence; ROUGE-L is their harmonic mean (F1)
The pool has “Write a short poem about the ocean.” Both have 7 words. The LCS is “write a short poem about the”: 6 words. So \(P = R = 6/7\) and \(\text{ROUGE-L} = 2 \cdot 6 / (7 + 7) = 0.857\). That is above 0.7, so the new instruction is dropped as a near-duplicate. “Write a haiku about autumn leaves.” (6 words) shares only “write a … about” (3 words) with the ocean poem: \(2 \cdot 3 / (6 + 7) = 0.462\), so it is kept.
How to read Figure 4
At the paper’s threshold of 0.7, 6 of 16 candidates survive. Near-copies such as “Give two tips for staying healthy.” (0.833 against “Give three tips…”) are dropped. Move the threshold down to 0.5: only 5 survive, because “Write a Python function that checks if a number is prime.” (0.632) now counts as too similar. Move it up to 0.9: 14 survive, and the pool fills with paraphrases. “Describe the picture in detail.” is always removed by the second filter in the paper: words like “image” or “picture” mark tasks a text-only model cannot do.
Self-Instruct turned GPT-3 into an instruction follower with 52K instructions and 82K instances, and gave a 33% absolute improvement over the original GPT-3 on Super-NaturalInstructions. [Certain] The data is noisy: in the authors’ own check of 200 samples, 92% of instructions were valid tasks but only 54% of examples were fully correct. [Certain] Later methods add better filters, such as asking a strong model to rate quality, or checking code by running it. Tülu 3 also uses persona-driven prompts (a strong model writes, for example, a math problem from the point of view of a given persona) to make synthetic data more diverse. [Certain]
- input: seed pool \(S\) of human-written tasks (175 in the paper), language model \(M\), threshold \(\tau = 0.7\)
- repeat until the pool is large enough:
- sample 8 instructions from the pool (6 human-written, 2 model-written) as few-shot examples
- ask \(M\) for a new instruction \(c\)
- if \(c\) contains a banned word (image, picture, graph) or \(\max_{s \in S} \text{ROUGE-L}(c, s) \ge \tau\): discard \(c\) // formula (53.8)
- ask \(M\) for input–output instances of \(c\) (output first for classification tasks, to balance labels)
- drop duplicate or invalid instances; add \(c\) and its instances to \(S\)
- return all model-written (instruction, input, output) triples as SFT data
Distilled
Distillation means a strong teacher model writes the answers, and a smaller student is fine-tuned on them with ordinary SFT. This is sequence-level knowledge distillation (Kim and Rush, 2016) applied to chat. [Certain] Stanford’s Alpaca (2023) is the famous example: it used the Self-Instruct recipe with OpenAI’s text-davinci-003 as the teacher to make 52K demonstrations for under $500, then fine-tuned LLaMA 7B in about 3 hours on 8 A100 GPUs. [Certain] The DeepSeek-R1 report fine-tuned Qwen and Llama students on about 800k samples written by DeepSeek-R1, with SFT only and no RL stage. [Certain]
Distillation is cheap and strong, but it has three costs. The student copies the teacher’s errors and style. The student can learn to sound like the teacher without having the knowledge to back it up, which can increase confident mistakes. [Likely] And the teacher’s license may forbid using its outputs to train competing models; check it. More in session 59.
| Source | Examples | Strength | Weakness |
|---|---|---|---|
| Human-written | InstructGPT SFT (12,725 prompts), Llama 2 (27,540), LIMA (1,000) | Quality and style under control; real user needs | Slow and costly; hard tasks need experts |
| Synthetic (model writes prompts and answers) | Self-Instruct (52K), Tülu 3 personas | Cheap, large, easy to target a skill | Noisy (54% fully correct in Self-Instruct); low diversity without filters |
| Distilled (strong teacher answers) | Alpaca (52K from text-davinci-003), DeepSeek-R1 students (800k) | Transfers a strong model’s skill to a small one | Copies teacher errors; license limits; style without substance |
| Preference pairs | InstructGPT (33,207 prompts), Llama 2 (1.4M), Tülu 3 8B (271,409) | Easier to judge than to write; teaches “not like this” | Labeler disagreement; reward hacking |
How much does post-training change the model?
LIMA suggested that alignment is “superficial”. Lin et al. (2023) measured this directly with the token distribution shift. Let the aligned model write an answer \(y\) with greedy decoding. At each position, give the base model the same context and ask: where does the aligned model’s token rank in the base model’s list?
- Read the symbols
- \(y_t\)
- the token the aligned (instruct) model chose at position \(t\)
- \(p_{\text{base}}(\cdot \mid x, y_{\lt t})\)
- the base model’s next-token distribution after the same prompt and the same answer so far
- \(\eta_t\)
- the “base rank”: 1 means the base model would have picked the same token
For Llama-2-7b against Llama-2-7b-chat, they found 77.7% unshifted, 14.5% marginal and only 7.8% shifted positions, and the shifted tokens were mostly style: “Thank”, “Hello”, “However”, “apolog”, and the end token. [Certain] We repeated the measurement for Qwen2.5-0.5B against Qwen2.5-0.5B-Instruct on 30 prompts (2,054 answer tokens, both models given the same chat-template context): 76.6% unshifted, 16.3% marginal, 7.2% shifted. [Certain, our run] The token that shifted most often was <|im_end|> (16 of 147 shifted positions): the base model ranks it thousands of places down. Knowing when to stop is one of the clearest things post-training adds.
How to read Figure 5
Most text is uncolored: the base model would have written the same token. Red tokens are where post-training changed the choice. Click them. In “I feel sad”, the first token “As” (of “As an AI language model”) has base rank 25; the base model wanted “Sure”. In “Haiku”, the poem’s first word “Wh(ispers)” has rank 21. In every answer that finishes, the final <|im_end|> is red, with base ranks in the tens of thousands. Facts such as “Paris” are unshifted: that knowledge came from pretraining.
This supports the superficial alignment view, with two cautions. First, a small shift per token can still change the meaning of an answer: one shifted token (“No” instead of “Yes”) flips a safety answer. Second, the result is about chat-style answers. RL on math and code (RLVR, reasoning models) changes behavior much more, for example by making answers many times longer. [Likely] Lin et al. also showed that a base model with a good system prompt and three fixed in-context examples (their method, URIAL) can come close to SFT-tuned models on their test set, with no training at all. [Certain]
Chat templates, and why a wrong template breaks a model
All three stages train on chats rendered with one fixed chat template (Session 9 explains the tokens in detail). For Qwen2.5 a single-turn prompt is:
<|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|> <|im_start|>user What is the capital of France?<|im_end|> <|im_start|>assistant
<|im_start|> (ID 151644) and <|im_end|> (ID 151645) are single special tokens. The last line, the generation prompt, tells the model “now the assistant speaks”. [Certain] Through SFT and RL, the model learned a very specific habit: after this exact sequence, write an answer, then write <|im_end|>. The habit is tied to the exact tokens. Change them, and the model is in a situation it never trained on.
To see how much this matters, we ran 20 simple factual questions (“What is the capital of Japan?”, “How many legs does a spider have?”, …) through Qwen2.5-0.5B-Instruct with six different prompt formats. For each, we measured three things: the average NLL (negative log-likelihood) per token of a short correct reference answer; the probability that the model writes <|im_end|> right after that answer; and how the greedy output ends within 128 tokens. Figure 6 shows the results.
<|im_end|> right after a correct short answer, on a log scale. The readout gives all metrics for the chosen format.How to read Figure 6
Compare the bars: only the two formats that contain the real role tokens give the end-of-turn token a sensible probability (0.45 and 0.74). All others are below \(10^{-5}\): about 70,000 to 1,800,000 times lower. Now read the readouts. With the correct template, 15 of 20 answers stop with <|im_end|>. With every wrong format, none do. “No generation prompt” looks fine on the chart, but read its output: the model writes <|im_start|>user itself and answers in the user’s role, and on 4 of 20 questions it replies “I’m sorry, but I can’t assist with that.” With “Markers as text”, the model writes a fake text marker such as ||im_end|>: it imitates what it sees.
The pattern is clear and important. The knowledge survives a wrong template; the turn structure does not. The answer NLL is about the same for the correct template (0.549 nats per token), the Llama 3 format (0.532), the Alpaca format (0.451) and the markers-as-text format (0.427). [Certain, our run] So the model still “knows” that Paris is the capital of France. What breaks is everything post-training added: where to start, which role to play, and when to stop. Only two formats are clearly worse on NLL: raw text (1.607, because the model expects a document, not an answer) and the missing generation prompt (4.148, because after <|im_end|> the model expects a new header, not an answer).
With the correct template, the mean probability of <|im_end|> after the reference answer is 0.45. With raw text, it is \(2.54 \times 10^{-7}\): about \(0.45 / 2.54\times10^{-7} \approx 1.8\) million times smaller. In the greedy runs, raw-text outputs had a mean length of 116.8 tokens against 74.2, and 14 of 20 hit the 128-token limit. In a product, this means longer latency, higher cost, and answers that drift into unrelated text. The “markers as text” prompt (the template string with special-token parsing turned off) also costs more: 57 tokens instead of 36 for the same chat, because each marker becomes 6 ordinary tokens.
Why does a wrong template hurt so much, if the model knows the answer? Because post-training is narrow. SFT and RL saw millions of tokens, all in one format. The end-of-turn token, the role headers and the system prompt were in exactly the same places every time. The model learned “after <|im_start|>assistant\n, answer and close”, not a general concept of a conversation. A different format looks to the model more like a pretraining document, so it falls back toward base-model behavior. [Likely] The fix is cheap: always render prompts with the model’s own template, as the code below does.
- input: model, tokenizer, a few test questions
- prompt \(\leftarrow\) apply_chat_template(messages, add_generation_prompt = true); print it and its token IDs // look for the role tokens as single IDs
- confirm the special tokens are single IDs, not split into pieces such as
<,|,im, … - generate greedily; confirm the output ends with the end-of-turn token well before the length limit
- score a known short answer: \(P(\text{end-of-turn} \mid \text{prompt}, \text{answer})\) should be large (here 0.45), not tiny (\(10^{-6}\))
- repeat with the exact string your server sends; any difference from step 2 is a bug
In code: preference loss, KL optimum, data filter, template check
A small, self-contained PyTorch script. It runs the Self-Instruct filter (53.8), trains a Bradley-Terry reward model (53.3) on synthetic preferences, and checks the closed-form KL optimum (53.5) against direct gradient ascent on objective (53.4). It runs on a CPU in a few seconds and prints the same numbers every time (PyTorch 2.8).
import re, torch torch.manual_seed(0) # 1) Self-Instruct style filter: keep a new instruction only if ROUGE-L < 0.7 to all kept ones def words(s): return re.sub(r"[^a-z0-9]+", " ", s.lower()).split() def rouge_l(a, b): a, b = words(a), words(b) d = [[0] * (len(b) + 1) for _ in range(len(a) + 1)] for i in range(len(a)): for j in range(len(b)): d[i + 1][j + 1] = d[i][j] + 1 if a[i] == b[j] else max(d[i][j + 1], d[i + 1][j]) l = d[-1][-1] return 0.0 if l == 0 else 2 * l / (len(a) + len(b)) # F1 of LCS precision and recall pool = ["Give three tips for staying healthy.", "Write a short poem about the ocean."] for c in ["Give two tips for staying healthy.", "Write a haiku about autumn leaves.", "Write a short poem about the sea.", "List five ways to save money on groceries."]: s = max(rouge_l(c, p) for p in pool) if s < 0.7: pool.append(c) print(f"{s:.3f}", "keep" if s < 0.7 else "drop", c) print(len(pool)) # 4 # 2) a reward model from pairwise preferences (Bradley-Terry), formula (53.3) d, n = 8, 2000 w_true = torch.randn(d) # hidden "human taste" fa, fb = torch.randn(n, d), torch.randn(n, d) # features of two answers per prompt pref_a = torch.rand(n) < torch.sigmoid((fa - fb) @ w_true) # noisy human choice fw = torch.where(pref_a[:, None], fa, fb); fl = torch.where(pref_a[:, None], fb, fa) w = torch.zeros(d, requires_grad=True) opt = torch.optim.Adam([w], lr=0.05) for step in range(300): loss = -torch.nn.functional.logsigmoid((fw - fl) @ w).mean() opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): acc = (((fw - fl) @ w) > 0).float().mean().item() cos = torch.nn.functional.cosine_similarity(w, w_true, dim=0).item() print(round(loss.item(), 3), round(acc, 3), round(cos, 3)) # 0.25 0.888 0.999 # 3) the KL-regularized optimum, formula (53.5), checked by gradient ascent on logits ref = torch.tensor([0.25, 0.30, 0.20, 0.15, 0.10]) # SFT policy over 5 answers r = torch.tensor([1.2, -2.0, 2.0, 1.5, 3.0]) # reward-model scores beta = 1.0 closed = ref * torch.exp(r / beta); closed = closed / closed.sum() z = torch.log(ref).clone().requires_grad_(True) # start at the reference policy opt = torch.optim.Adam([z], lr=0.05) for step in range(2000): p = torch.softmax(z, 0) J = (p * r).sum() - beta * (p * (p / ref).log()).sum() # objective (53.4) opt.zero_grad(); (-J).backward(); opt.step() p = torch.softmax(z, 0).detach() print([round(x, 3) for x in closed.tolist()]) # [0.165, 0.008, 0.294, 0.134, 0.399] print([round(x, 3) for x in p.tolist()]) # [0.165, 0.008, 0.294, 0.134, 0.399] kl = (closed * (closed / ref).log()).sum() print(round((closed * r).sum().item(), 3), round(kl.item(), 3)) # 2.168 0.553
The first part prints 0.833 drop, 0.462 keep, 0.857 drop, 0.000 keep, then a pool of 4. The reward model reaches 88.8% training accuracy, which is close to the best possible here, because the simulated “humans” are noisy; and its weight vector points almost exactly in the direction of the hidden taste (cosine 0.999). The gradient ascent lands on the same five probabilities as formula (53.5). To check a real model’s template (Algorithm 3), the script we used for Figure 6 boils down to this (needs transformers and a 1 GB model download; we ran it with transformers 4.57):
import torch from transformers import AutoTokenizer, AutoModelForCausalLM name = "Qwen/Qwen2.5-0.5B-Instruct" tok = AutoTokenizer.from_pretrained(name) model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval() msgs = [{"role": "user", "content": "What is the capital of France?"}] ids = tok.apply_chat_template(msgs, add_generation_prompt=True) # the correct prompt print(len(ids), tok.convert_ids_to_tokens(ids[:3])) # 36 ['<|im_start|>', 'system', 'Ċ'] ans = tok("The capital of France is Paris.", add_special_tokens=False).input_ids x = torch.tensor([ids + ans + [tok.convert_tokens_to_ids("<|im_end|>")]]) with torch.no_grad(): logp = torch.log_softmax(model(x).logits[0], -1) p_end = logp[len(ids) + len(ans) - 1, x[0, -1]].exp().item() print(round(p_end, 3)) # 0.904 here; about 1e-6 with a wrong template out = model.generate(torch.tensor([ids]), max_new_tokens=64, do_sample=False) print(repr(tok.decode(out[0, len(ids):]))) # 'The capital of France is Paris.<|im_end|>'
Where this is used
| Setting | Which stages | What matters most |
|---|---|---|
| General chat assistants (Llama, Qwen, Tülu) | SFT, then DPO and/or RLHF, often RLVR | Diverse prompts; evaluate helpfulness, instruction following and safety after every stage |
| Domain assistants (legal, medical, support bots) | SFT from an instruct model on domain chats; sometimes DPO | Start from the instruct model, keep its template; mix in general data to avoid forgetting (session 61) |
| Small on-device models | Distillation SFT from a large teacher, then DPO | Teacher quality and license; the student cannot learn facts it has no capacity for |
| Reasoning models (math, code) | SFT on long reasoning traces, then RL with verifiable rewards | Reliable checkers; watch answer length and reward hacking (session 72) |
| Agents and tool use | SFT on tool-call traces in the template’s tool format; RL on task success | Tool-call tokens and stop tokens exactly as in the template (session 154) |
| Vision-language assistants | Alignment pretraining, visual instruction SFT, then preference tuning | Image placeholders in the template; preference data against hallucination (session 186) |
| Video QA and Video-RAG assistants | Video instruction SFT (questions with timestamps), then preference tuning | Answers that point to key moments; frames as placeholders in the chat template; refusing when the video does not show the answer |
| Video retrieval and classification with an LLM judge | Usually none: use an existing instruct model in zero-shot mode | The judge’s own template and a fixed output format, so that labels can be parsed reliably |
Common mistakes
- Expecting a base model to chat. A base model continues documents. Use few-shot prompts or URIAL-style examples, or use the instruct version. And never evaluate a base model with a chat template (session 40).
- Fine-tuning an instruct model with a different template. If you SFT Qwen-Instruct on Alpaca-style text, you fight its existing habits and may get a model that knows two formats badly. Use the model’s own template.
- Forgetting the generation prompt. Without
<|im_start|>assistant\nat the end, the model writes the header itself; in our test it chose the user role every time (20 of 20). - Masking the end-of-turn token in SFT. Then nothing teaches the model to stop, and it rambles. The end token must have \(m_t = 1\) in (53.2).
- Too small a KL coefficient, or none. The policy finds what the reward model overrates (length, flattery, repeated phrases). Track the KL and an independent evaluation during RL, not only the reward.
- Training the reward model for many epochs. InstructGPT found it overfits after one pass; ranked pairs from one prompt are highly correlated. [Certain]
- Trusting synthetic or distilled data blindly. Self-Instruct data was only 54% fully correct. Filter for duplicates (ROUGE-L, embeddings), quality (a judge model), and benchmark contamination.
- Measuring only the skill you trained. DPO and RL can raise chat scores and lower safety or math scores. Run the full evaluation suite after each stage.
Interview questions
What is the difference between a base model and an instruct (chat) model?basic
Walk me through the classic post-training pipeline.basic
Why does SFT come before preference tuning?core
Derive the optimal policy of the KL-regularized RL objective. Why does it matter for DPO?advanced
What is the KL penalty for in RLHF, and what happens if \(\beta\) is too small or too large?core
How is a reward model trained, and why are only score differences meaningful?core
Compare human-written, synthetic and distilled instruction data.core
What is the superficial alignment hypothesis, and what evidence supports it?advanced
What happens if you send a chat model a prompt without its chat template, or with another model’s template?core
<|im_end|> after the answer fell from 0.45 to below \(10^{-5}\), and 0 of 20 answers ended their turn (15 of 20 with the right template). The model rambled, wrote fake markers as text, or ran to the length limit. Without the generation prompt, it wrote a user header itself and answered as the user.What is RLVR and why is it popular for math and code?advanced
Your SFT model gives good answers but never stops writing. List the likely causes.core
Exercises
- A labeler ranks \(K = 6\) answers for one prompt. How many preference pairs does this give? If one pair has reward-model scores 1.0 (preferred) and −1.0 (rejected), what is the Bradley-Terry loss for that pair? What if the scores were swapped?
- Two answers, \(\pi_{\text{ref}} = (0.5, 0.5)\), rewards \(r = (1, 0)\), \(\beta = 0.5\). Compute \(\pi^*\) with (53.5) and the KL from \(\pi_{\text{ref}}\).
- Compute ROUGE-L between “Translate the sentence into German.” and “Translate the sentence into French.” Is the first kept by Self-Instruct’s filter?
- InstructGPT’s 175B SFT model used 4.9 petaflop/s-days and the PPO-ptx model 60, against 3,640 for GPT-3 pretraining. What fraction of the pretraining compute is each?
- In Figure 5, “Boil an egg” has 96 answer tokens: 72 unshifted, 16 marginal, 8 shifted. Give the three percentages. Which shifted token do you expect in almost every answer that ends before the length limit, and why?
- Your company fine-tunes Qwen2.5-Instruct on support chats formatted as
### Customer: … ### Agent: …. Predict two problems, and give the fix. - Use Figure 3. Find the \(\beta\) where the reward-hack answer E has about 50% of the probability. What are the expected reward and the true quality there?
Answerscheck
<|im_end|>: the base model was never trained to end a reply there (it ranks it thousands of places down), while the instruct model learned it in SFT. 6. The new format conflicts with the ChatML habits: the model may mix both formats, and it will not learn to stop unless an end marker follows each agent turn and gets a loss; at serving, the old template or stop token may be used by mistake. Fix: render the support chats with the model’s own chat template (customer = user, agent = assistant), mask the loss to assistant turns including <|im_end|>, and serve with the same template. 7. About \(\beta = 0.76\) (E has about 50%; at \(\beta = 0.70\) it has 54.1%). There \(\E[r] \approx 2.35\) and \(\E[q] \approx 0.27\): the reward is high, but true quality is less than half of its best value, 0.63 near \(\beta = 1.6\).Key takeaways
- A base model continues documents (53.1). An assistant answers in a role, follows the requested form, and stops with an end-of-turn token. Our Qwen2.5-0.5B runs show both behaviors on the same questions.
- The classic pipeline is SFT (53.2), then preference data with a Bradley-Terry reward model (53.3), then RL with a KL leash (53.4). Its optimum (53.5) tilts the SFT policy by \(e^{r/\beta}\); DPO (53.6) and RLVR (53.7) are the modern shortcuts.
- Post-training is cheap compared with pretraining (InstructGPT: under 2% of GPT-3’s compute) and very effective (1.3B InstructGPT preferred over 175B GPT-3).
- Instruction data is human-written, synthetic (Self-Instruct with a ROUGE-L < 0.7 filter, 53.8) or distilled from a teacher. Quality and diversity beat quantity (LIMA: 1,000 examples).
- Post-training changes few tokens: about 77% of an aligned answer is the base model’s top choice (53.9). What changes is style, persona and, most clearly, when to stop.
- A wrong template keeps the knowledge but breaks the turn structure: in our test, P(end-of-turn) fell from 0.45 to below \(10^{-5}\) and no answer ended properly. Always use the model’s own template and check the token IDs.
Key formulas
References for this session
- Ouyang et al. (2022). Training language models to follow instructions with human feedback
- Wang et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions
- Zhou et al. (2023). LIMA: Less Is More for Alignment
- Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 8: Post-training
- Ouyang et al. (2022). Training language models to follow instructions with human feedback
- Wang et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions
- Zhou et al. (2023). LIMA: Less Is More for Alignment
- Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 8: Post-training
- Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Schulman et al. (2017). Proximal Policy Optimization Algorithms
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Kim & Rush (2016). Sequence-Level Knowledge Distillation
- Lambert (2025). Reinforcement Learning from Human Feedback (book)
- Taori et al. (2023). Alpaca: A Strong, Replicable Instruction-Following Model
- Lin et al. (2023). The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
- Qwen Team (2024). Qwen2.5 Technical Report
- Hugging Face (2024). Chat templates (Transformers documentation)