SFT in detail
Supervised fine-tuning (SFT) is next-token training on example conversations, with one change: the loss counts only the tokens the assistant should write. This session goes through the details that decide whether SFT works: the loss mask, how the loss is averaged, packing with document masks, epochs and learning rate, the signs of overfitting, and why a small, clean, diverse dataset often beats a big one.
What you will learn
- The SFT loss: next-token cross-entropy with a loss mask, so only assistant tokens (and their end-of-turn token) are trained
- How the loss is averaged (per token, per sequence, sum), and the gradient-accumulation bug that silently changes the weights
- Packing several examples into one row with a document mask and reset position ids, so a packed row equals separate rows
- Epochs, learning rate, batch size and maximum length, with real recipes from LIMA, Llama 2 and Tülu 3
- How to see overfitting: training loss, validation loss and why generation quality must be checked separately
- Data quality versus quantity, and how to choose a diverse subset
Why this matters
In session 53 we saw the plan: a base model only continues text, and post-training turns it into an assistant. The first and most important stage is supervised fine-tuning (SFT): we show the model many example conversations and train it to write the assistant’s replies. The idea fits in one sentence. The details decide whether it works.
SFT uses the same next-token loss as pretraining (formula 54.1). Four details make it different, and each one is a classic source of silent bugs:
- Loss masking. We train only on the tokens the assistant should write, not on the user’s question or the system prompt. Forget this and the model learns to write user questions.
- How the loss is averaged. Per token, per sequence, or a plain sum. The choice changes how much each example counts. Gradient accumulation can change it again without telling you.
- Packing. Chat examples are short and of very different lengths. We pack several into one row to avoid wasted padding, and we need a document mask so they do not read each other.
- Small data, few steps. SFT sets are thousands to about a million examples, not trillions of tokens. Epochs, learning rate and data quality matter a lot, and overfitting is real.
Real systems use exactly these pieces. Llama 2’s SFT “zero-out[s] the loss on tokens from the user prompt”, packs all prompts and answers into rows of 4,096 tokens, and trains for 2 epochs at learning rate \(2 \times 10^{-5}\). [Certain] Tülu 3 trains Llama 3.1 8B on 939,344 SFT prompts for 2 epochs at \(5 \times 10^{-6}\), and reports that using a sum loss instead of a mean loss mattered. [Certain] LIMA fine-tuned a 65B model on only 1,000 carefully chosen examples. [Certain] By the end of this session you will understand every one of these choices.
The SFT loss: next-token prediction with a mask
An SFT example is a conversation: a list of messages, each with a role (system, user or assistant) and some text. The chat template (session 9) turns it into one token sequence \(x_1, \dots, x_T\) with role headers and end-of-turn tokens. For example, in the ChatML style used by Qwen:
<|im_start|>user What is 2+2?<|im_end|> <|im_start|>assistant 4.<|im_end|>
Pretraining would compute the loss on every token. In SFT we attach a loss mask \(m_t \in \{0, 1\}\) to every token: \(m_t = 1\) if the model should learn to produce token \(t\), and \(m_t = 0\) otherwise. The loss is the average negative log-probability of the masked-in tokens only:
- Read the symbols
- \(i\), \(B\)
- the example index, and the number of examples in the batch
- \(t\), \(T_i\)
- the token position, and the length of example \(i\); position 1 has nothing before it, so it is never a target
- \(m_{i,t}\)
- the loss mask: 1 for tokens the assistant writes (including its end-of-turn token), 0 for system, user and role-header tokens
- \(p_\theta(x_{i,t} \mid \dots)\)
- the probability the model gives to the true next token, after the softmax over the vocabulary
- \(Z\)
- the normalizer: how we turn the sum into an average. The next section shows that this choice matters
The model still reads every token: the user’s question is in the context and attention uses it. The mask only decides which positions produce a gradient. In code, the mask is usually stored inside the labels: a label of -100 means “ignore this position”, because PyTorch’s cross_entropy skips that value by default (ignore_index=-100). [Certain]
Why mask the prompt?
Three reasons. First, the goal is to answer, not to write questions. Gradient on user tokens teaches the model to produce user-style text, and at inference it may start writing the next user turn itself. Second, the prompt tokens dominate in many datasets (long documents with short answers). Without a mask, most of the gradient goes to copying prompts. Third, prompts often come from a different source than answers (scraped questions, synthetic templates), so their quality is lower.
The end-of-turn token (<|im_end|> above) must be trained. It is how the model learns to stop. If it is masked out, the model never learns where an answer ends and keeps writing. LIMA added a special end-of-turn token for this reason, separate from the pretraining end-of-text token. [Certain]
Masking the prompt is the default, not a law. Shi et al. (2024) found that training on the instructions too can help when the outputs are short or the dataset is small, by reducing overfitting. [Certain] In practice, start with assistant-only loss and treat prompt loss as an experiment.
Figure 1 runs formula (54.1) one position at a time, on real numbers. The probabilities come from a tiny language model that your browser trains when the page loads: 120 steps of SFT on 32 small chats with assistant-only loss. Figure 6 shows that training run step by step, and formula (54.10) defines the model. At each position \(t\), the model reads the tokens up to \(x_t\), gives a probability to the true next token \(x_{t+1}\), and the loss \(\ell_t = -\log p\) is added to the sum only if the next token has mask 1.
- \(S \leftarrow 0,\ n \leftarrow 0,\ \mathbf{h} \leftarrow \mathbf{0}\)
- for \(t = 1, \dots, T-1\) do
- \(\mathbf{h} \leftarrow \gamma\,\mathbf{h} + \mathbf{e}(x_t)\) // \(\mathbf{h}\) restarts at 0 for a new chat if the document mask is on
- \(\mathbf{p} \leftarrow \softmax(A\,\mathbf{e}(x_t) + C\,\mathbf{h} + \mathbf{b})\) // (54.10)
- \(\ell_t \leftarrow -\log p[x_{t+1}]\)
- if \(m_{t+1} = 1\): \(S \leftarrow S + \ell_t,\ n \leftarrow n + 1\) // else label −100: skip
- \(\mathcal{L} \leftarrow S / n\) // (54.1) with \(Z = n\)
−100 or, once computed, its loss \(\ell\). The square on the left is the attention mask of the row; the highlighted line is the row of the current position.How to read Figure 1
Press Play and watch the cursor. Blue marks the token being read, orange the token to predict. On user and header positions the model gets losses between 3.1 and 9.3 nats, nearly all worse than a uniform guess over its 27 tokens (\(\ln 27 = 3.30\)): it was never trained to write questions, and nothing is added to \(S\). On the answers the loss is small, and it is added. Switch to “All tokens”: the user tokens now count, and the reported loss jumps from 0.315 to 5.622. Untick the end-of-turn box: the <|im_end|> positions drop out, so the model gets no signal to stop. “Last assistant turn only” is used when earlier assistant turns came from another model and you do not trust them. Choose “Untrained model” to see that every position starts at \(\ln 27\). The two-chat row and the document mask are explained in the packing section below.
In Figure 1 the single chat has 32 tokens. With assistant-only loss, the targets are “Paris .” plus <|im_end|> and “small .” plus <|im_end|>: 6 of 32 tokens, or 18.8%. The trained tiny model gives them losses 0.5841, 0.0386, 0.0095, 1.2199, 0.0328 and 0.0061, so \(S = 1.891\) and \(\mathcal{L} = 1.891 / 6 = 0.315\). The other 25 predictions are computed in the same forward pass and then thrown away. The second answer costs more than the first (1.2199 against 0.5841) because the tiny model’s memory still holds the first question when it answers the second. [Likely] The short example in the code section below has 35 tokens and only 3 targets (8.6%). So the gradient per row in SFT comes from a small part of the tokens, and the number of target tokens, not the number of rows, is what you should count.
How to average the loss
Formula (54.1) has a normalizer \(Z\). There are three common choices, and they are not the same when examples have different numbers of target tokens. Let \(n_i = \sum_t m_{i,t}\) be the number of target tokens in example \(i\), and \(\ell_{i,t} = -\log p_\theta(x_{i,t} \mid x_{i,\lt t})\) the loss of one token.
All three have the same per-token losses inside. They differ in the weight each token gets in the gradient:
- Read the symbols
- \(n_i\)
- number of target tokens (mask = 1) in example \(i\)
- \(\ell_{i,t}\)
- cross-entropy of one target token
- \(w_{i,t}\)
- how much token \(t\) of example \(i\) counts in the gradient
- Token mean: every target token counts the same. Long answers get more total weight. This is what
cross_entropy(..., reduction="mean")gives on a flattened batch. - Sequence mean: every example counts the same. A token in a 10-token answer counts 9 times more than a token in a 90-token answer. Short answers (“Yes.”) get a lot of weight.
- Sum: every token counts the same and the scale is not divided by anything. The gradient size then grows with the number of tokens in the batch, so the learning rate must be chosen for it. Tülu 3 used a sum loss because it “effectively weights all tokens equally”, which gave better results for their mixtures. [Certain]
Example A has \(n_A = 10\) target tokens with average loss 2.0. Example B has \(n_B = 90\) with average loss 1.0. Then:
Token mean: \((10 \times 2.0 + 90 \times 1.0)/100 = 1.10\). Sequence mean: \((2.0 + 1.0)/2 = 1.50\). Sum: \(20 + 90 = 110\).
Weights per token: token mean gives every token \(1/100\). Sequence mean gives A’s tokens \(1/(2 \times 10) = 0.05\) and B’s tokens \(1/(2 \times 90) \approx 0.0056\): a ratio of 9. The reported loss is different (1.10 versus 1.50), and so is the direction of the gradient.
The gradient-accumulation trap
Gradient accumulation (session 39) splits a large batch into \(G\) micro-batches, runs backward on each, and adds the gradients. It should give exactly the same gradient as the full batch. With a token-mean loss it does not, if each micro-batch is averaged over its own tokens and the results are averaged over micro-batches. The correct way divides by the token count of the whole batch:
- Read the symbols
- \(g\), \(G\)
- a micro-batch, and the number of micro-batches in one optimizer step
- \(N_g\)
- target tokens in micro-batch \(g\); \(N\) is the total over all micro-batches
The right side, a “mean of means”, gives micro-batches with few target tokens too much weight. The two sides are equal only when every micro-batch has the same \(N_g\). This was a real bug in popular training code: in October 2024 Hugging Face confirmed that its trainer computed the loss this way, and fixed it by summing the loss and dividing by the total number of target tokens. [Certain] The same issue appears across GPUs in data-parallel training: each GPU averages over its own tokens, then gradients are averaged over GPUs. Tülu 3 notes that “changing gradient accumulation can have large effects on performance due to effectively changing sample weightings”. [Certain]
How to read Figure 2
With the token mean, the orange bar equals the grey bar: each example counts in proportion to its tokens. With the sequence mean, every orange bar is 25%, so the short answer A controls as much as the long answer D. The mean of micro-batch means is in between and depends on how examples happen to fall into micro-batches: move answer A to 400 tokens and watch C and D gain weight although nothing about them changed. The reported loss is different in each mode too, so curves from runs with different settings cannot be compared directly.
Packing: many examples in one row
Chat examples have very different lengths: “Hi” and a 3,000-token code review sit in the same dataset. If each example gets its own row padded to the longest in the batch, much of the GPU work is spent on padding. Packing puts several whole examples into one row of length \(T\). Session 18 introduced packing for pretraining; here we look at what SFT needs on top.
For SFT, an example must stay whole: cutting an answer in the middle would train the model on a fragment with no end-of-turn token. So SFT packing is a bin-packing problem: fill each row with whole examples that fit, for example with first-fit decreasing (sort examples from long to short; put each into the first row that still has room). Examples longer than \(T\) are cut (truncated) or dropped.
A packed row needs three things to behave exactly like separate rows:
- A document mask. Token \(i\) may attend to token \(j\) only if both belong to the same example and \(j \le i\):
\[ M_{ij} = \begin{cases} 0 & \text{if } d(i) = d(j) \text{ and } j \le i \\ -\infty & \text{otherwise} \end{cases} \tag{54.5} \]where \(d(i)\) is the index of the example that token \(i\) belongs to. The mask is block-diagonal: one causal triangle per example.
- Position ids that restart at 0 for each example, so each one sees the same positions (and RoPE angles) as when it is alone:
\[ p_i = i - s_{d(i)} \tag{54.6} \]where \(s_{d}\) is the first position of example \(d\) in the row.
- No loss across the boundary. The last token of example 1 must not be trained to predict the first token of example 2. With chat templates this is automatic, because the first token of each example is a role header with mask 0.
With these three, a packed row gives the same logits as running each example alone (the code below checks it to the last bit). Without the document mask, example 2 can attend to example 1. In SFT this is worse than in pretraining: a later answer can read an earlier, similar example, which is a leak, and the model learns to depend on unrelated context. [Likely] Fast attention kernels apply the mask without building it: FlashAttention’s variable-length interface takes the examples back to back plus their cumulative lengths (cu_seqlens). [Certain]
Figure 1 shows this on real numbers. Choose “Two chats packed in one row”: the second chat starts at row position 32, and its position ids restart at 0. In the tiny model, a fading memory of earlier tokens plays the role of attention, so the document mask means: the memory restarts at the first token of a new chat. With the mask on, every loss in the second chat is exactly what it is when that chat runs alone (“blue” costs 0.204), and the row loss over the 9 targets is 0.241. Untick the document mask: the memory now carries the first chat into the second, and every prediction there changes. “blue” now costs 0.164 and the row loss falls to 0.233. The leak happens to lower the loss here, and that is the danger: information from an unrelated example looks like learning.
Packing also interacts with the loss averaging. In a packed row, a token mean over the row gives long examples more weight, exactly as before. If you want a sequence mean, you must compute it per example inside the row, using the example ids. And packing changes the number of rows per step: a “batch of 128 examples” and a “batch of 128 packed rows” are very different amounts of data. We measure how well rows are filled with the utilization:
- Read the symbols
- \(L_i\)
- length of example \(i\) in tokens (prompt plus answer)
- \(T\)
- row length (maximum sequence length)
- \(R\)
- number of rows used; for padding, replace \(R \cdot T\) by the total padded slots
Eight examples have lengths 120, 900, 300, 2000, 450, 80, 1500 and 600 tokens (sum 5,950), and \(T = 2048\). Padding every example to 2,048 uses \(8 \times 2048 = 16{,}384\) slots: \(U = 5950/16384 = 36.3\%\). First-fit decreasing makes three rows: [2000], [1500, 450, 80] and [900, 600, 300, 120], filled with 2,000, 2,030 and 1,920 tokens. Now \(U = 5950/(3 \times 2048) = 96.8\%\), and the step needs 3 rows instead of 8.
The document mask also saves attention work: one causal triangle per example has \(\sum_i L_i(L_i+1)/2 \approx 3.87\) million query-key pairs, against about 6.29 million for three full causal rows of 2,048. Kernels that skip the blocked tiles get this saving. [Likely]
- sort the examples by length \(L_i\), longest first
- for each example \(i\) in that order do
- for each open row \(r\): if \(u_r + L_i \le T\): put \(i\) in \(r\), \(u_r \leftarrow u_r + L_i\), stop // first fit
- if no row had room: open a new row with \(u = L_i\)
- \(U \leftarrow \sum_i \min(L_i, T) / (R \cdot T)\) // (54.7)
How to read Figure 3
Press Play in packing mode to watch first-fit decreasing work: each example tries the open rows from the top, the formula box shows the test \(u_r + L_i \le T\) with the real numbers, and \(U\) is updated after every placement. In padding mode most of the area is empty padding: about half of the compute is wasted at \(T = 2048\). In packing mode, rows are almost full. Now set \(T = 512\): long examples no longer fit, and their answers are cut (red marks). A cut answer has lost its end-of-turn token, so the model is trained on answers that stop in the middle. This is why the maximum length is a data decision, not only a memory decision: check how many examples exceed it, and drop or split them instead of cutting silently.
Epochs, learning rate, batch and length
SFT starts from a model that is already good. Its job is to change the style and format of the answers, plus some skills, without destroying what pretraining learned. This shapes every hyper-parameter.
Learning rate
SFT learning rates are much lower than pretraining. Llama 2 pretrained its 7B model at a peak of \(3 \times 10^{-4}\) and fine-tuned at \(2 \times 10^{-5}\): 15 times lower. [Certain] Tülu 3 used \(5 \times 10^{-6}\) for 8B and \(2 \times 10^{-6}\) for 70B, with linear decay and 3% warmup. [Certain] Larger models usually get smaller learning rates. [Likely] Too high, and the model forgets general knowledge (catastrophic forgetting, see session 61) and becomes unstable; too low, and it does not pick up the format. LoRA (session 57) uses much larger learning rates, around \(10^{-4}\), because only small adapter matrices are trained. [Likely]
Epochs and steps
The number of optimizer steps is
- Read the symbols
- \(N_{\text{ex}}\)
- number of training examples (or packed rows, if the batch counts rows)
- \(B\)
- effective batch size: per-GPU batch × number of GPUs × accumulation steps
- \(E\)
- number of epochs (full passes over the data)
Most recipes use 1 to 3 epochs. Llama 2 and Tülu 3 used 2; Tülu 3 reports that 2 epochs worked best in their sweep. [Certain] Small, high-quality sets are trained longer: LIMA trained 15 epochs on 1,000 examples and picked a checkpoint between epochs 5 and 10. [Certain] More epochs on a small set means the model sees the same answers many times; it starts to memorize them.
Tülu 3, 8B: \(N_{\text{ex}} = 939{,}344\), \(B = 128\), \(E = 2\), so \(S = 939344 \times 2 / 128 \approx 14{,}677\) steps (slightly more with rounding per epoch), and 3% warmup is about 440 steps. LIMA: \(\lceil 1000/32 \rceil = 32\) steps per epoch, 15 epochs, so about 480 steps in total. Compare with pretraining: Llama 2 took about 500,000 steps. SFT is a short run, so a few bad steps (a spike, a wrong learning rate) matter a lot.
Batch size and maximum length
Effective batches of 32 to 128 examples are common for SFT (LIMA 32, Llama 2 64, Tülu 3 128). [Certain] The maximum length should cover almost all examples: Llama 2 and Tülu 3 used 4,096 tokens, LIMA 2,048. [Certain] Long-context or reasoning SFT (long chains of thought) needs much longer rows, and then packing, document masks and memory tricks become essential (session 39).
Small tricks you will meet
NEFTune adds uniform noise to the input embeddings during SFT only:
where \(L\) is the sequence length, \(d\) the embedding size and \(\alpha\) a strength (5 to 15 in the paper). It reported AlpacaEval going from 29.79% to 64.69% for Llama 2 7B fine-tuned on Alpaca, with smaller gains on other datasets. [Certain] It acts as a regularizer against overfitting; gains depend on the dataset and evaluation. [Likely] Other common choices: weight decay 0.1 with AdamW (as in LIMA and Llama 2), and dropout in LIMA (from 0.0 at the bottom layer to 0.3 at the top for 65B). [Certain]
Signs of overfitting
Overfitting means the model gets better on the training examples but worse on new ones. In SFT, the signs are:
- Training loss keeps falling, often with a sharp drop at the start of each new epoch (the model has seen these exact answers before).
- Validation loss on held-out examples stops falling and then rises.
- Generations start copying phrases from the training set, become formulaic (“As an AI language model…” everywhere), or lose diversity.
- Other skills drop: scores on general benchmarks (MMLU-style knowledge, math) go down compared with the base model, which is forgetting.
There is a catch. For SFT, validation loss is not a reliable measure of answer quality. LIMA reports that perplexity on held-out data did not correlate with generation quality, and chose checkpoints by reading the outputs of a 50-example development set. [Certain] A model can become more confident in its own style (higher loss on reference answers written in a different style) while its answers get better. So use validation loss to catch clear overfitting, and use generation-based evaluation (human or model judges, task benchmarks) to pick the checkpoint (session 40).
How to read Figure 4
With 100 examples, training loss falls all the way while validation loss is lowest around epoch 7 and then climbs: the model is learning the noise in its few examples. Raise the number of examples: the gap between the curves shrinks and the best epoch moves later. Raise the learning rate: everything happens sooner, including overfitting. An LLM is much bigger than this model, but the same pattern appears when a small SFT set is trained for many epochs. [Likely]
Data: quality, quantity and diversity
For SFT, the data is the recipe. The model already knows a lot from pretraining; SFT mostly teaches it the format and the behavior of a good assistant. LIMA calls this the superficial alignment hypothesis: almost all knowledge is learned in pretraining, and alignment teaches which style and format to use. [Certain] If that is true, a small number of excellent examples should be enough, and the evidence partly agrees:
- LIMA: 1,000 curated examples on LLaMA 65B, no RLHF. In a human study, its answers were equal to or preferred over GPT-4 in 43% of cases, 58% against Bard and 65% against DaVinci003. [Certain] In its ablations (7B model, 2,000 examples each), more diverse data (Stack Exchange) beat less diverse data (wikiHow), filtered data beat unfiltered, and making the set up to 16 times larger without more diversity did not help. [Certain] Adding just 30 multi-turn dialogues raised the share of excellent responses in dialogue from 45.2% to 76.1%. [Certain]
- Llama 2: “Quality is all you need”. Meta stopped after 27,540 high-quality SFT annotations, finding that tens of thousands were enough. [Certain]
- AlpaGasus: kept 9k of the 52k Alpaca examples using an LLM to score quality, and the filtered model beat the original Alpaca while training 5.7 times faster. [Certain]
- Tülu 3: a large mix (939,344 prompts) built from many sources, each targeting a skill: math, code, instruction following, safety, multilingual and real user chats. Removing a source hurts its skill: without the WildChat real-user data the average score fell from 60.1 to 58.9, and without the safety data to 58.0. [Certain]
So the modern picture is: quality first, then coverage. A few thousand excellent examples can teach format. Real skills (math, code, tool use, many languages) need targeted data for each skill, and more of it. [Likely] Quantity helps only if it adds new kinds of examples, not more copies of the same kind.
What “quality” means in practice
- Correct answers (verify math and code by running them), in the style you want the model to have.
- Consistent format: the same template, the same way of using markdown, the same refusal style.
- Deduplicated (exact and near-duplicates), so a few examples do not get many epochs’ worth of weight.
- Decontaminated: remove examples that overlap with evaluation benchmarks, or your scores will be inflated. Tülu 3 checks its training prompts against its evaluation sets. [Certain]
- Balanced across tasks: choose the mixture on purpose instead of taking whatever is largest.
Choosing a diverse subset
A typical raw pool is dominated by a few easy kinds of request. Random sampling copies that imbalance. A simple fix is to embed every example (with any sentence-embedding model, see session 60) and pick examples that are far from the ones already picked. The greedy k-center method does this: start from one example, then repeatedly add the example whose distance to its nearest picked example is largest. The coverage radius, the largest distance from any pool example to its nearest picked example, measures how well the subset covers the pool. Data-selection methods used in practice combine such diversity scores with quality and difficulty scores. [Likely]
How to read Figure 5
At \(k = 16\), the random sample misses whole task groups (math and safety), because small groups are rarely drawn. Greedy k-center covers all 8 groups and has a much smaller coverage radius. Note the cost: k-center also likes outliers (a strange example far from everything is picked early), so in practice it is combined with a quality filter that removes broken or off-topic examples first.
Putting it together: the SFT pipeline
- input: messages \([(r_1, c_1), \dots, (r_K, c_K)]\), tokenizer, chat template, max length \(T\)
- \(\text{ids} \leftarrow [\,],\ \text{labels} \leftarrow [\,]\)
- for each message \((r, c)\): // render with the model’s own template
- \(h \leftarrow\) tokens of the role header; append \(h\) to \(\text{ids}\) and \(-100\) for each to \(\text{labels}\)
- \(b \leftarrow\) tokens of \(c\) followed by the end-of-turn token
- append \(b\) to \(\text{ids}\); append \(b\) to \(\text{labels}\) if \(r = \text{assistant}\), else \(-100\) for each // formula (54.1)
- if \(\text{length}(\text{ids}) \gt T\): drop the example, or split the chat at a turn boundary // never cut an answer silently
- if no label is kept: drop the example // zero target tokens gives \(0/0\)
- return \(\text{ids}, \text{labels}\)
- input: examples \((\text{ids}_i, \text{labels}_i)\), row length \(T\), micro-batches per step \(G\)
- sort examples by length, long first; place each into the first row with room // first-fit decreasing
- for each row: concatenate \(\text{ids}\) and \(\text{labels}\); \(\text{doc} \leftarrow\) example index per token; \(\text{pos} \leftarrow\) restart at 0 per example // (54.6)
- \(\text{mask} \leftarrow \text{causal} \wedge (\text{doc}_i = \text{doc}_j)\), or pass
cu_seqlensto a varlen kernel // (54.5) - \(N \leftarrow\) number of labels \(\ne -100\) in all \(G\) micro-batches of this step (summed over GPUs too)
- for \(g = 1 \dots G\):
- \(\text{logits} \leftarrow \text{model}(\text{ids}_g, \text{pos}_g, \text{mask}_g)\)
- \(\text{loss} \leftarrow \operatorname{CE}_{\text{sum}}\big(\text{logits}[:, :-1],\ \text{labels}_g[:, 1:]\big) / N\) //
cross_entropy(..., reduction="sum")// (54.4) - loss.backward() // gradients add up
- clip, optimizer step with a low learning rate and its schedule, zero gradients
A tiny SFT run you can watch
To see SFT learn a format, we need a model small enough to train in a browser in a fraction of a second. Ours has a vocabulary of \(V = 27\) word tokens: the five template tokens (<|im_start|>, <|im_end|>, ↵, user, assistant) and the words of 8 question-answer pairs, such as “capital of France ?” → “Paris .” and “opposite of big ?” → “small .”. Instead of attention, it has a fading memory \(\mathbf{h}_t\) of all tokens read so far:
- Read the symbols
- \(\mathbf{e}(x)\)
- the one-hot vector of token \(x\): length 27, a 1 at the token’s index and 0 elsewhere
- \(\mathbf{h}_t\)
- the memory: each earlier token, weighted by \(\gamma\) to the power of how far back it is; with \(\gamma = 0.9\), a token 5 positions back still has weight \(0.9^5 = 0.59\)
- \(A, C, \mathbf{b}\)
- the weights \(\theta\): \(A\) scores the next token from the current token, \(C\) from the memory, \(\mathbf{b}\) is a bias. In total \(2 \cdot 27^2 + 27 = 1{,}485\) numbers, all starting at 0
The training set has 32 seeded chats with 1 or 2 question-answer turns each. Training is plain SGD on batches of 4 chats, with the token-mean loss (54.1) and \(Z = N\), the number of target tokens in the batch. For a softmax, the gradient of \(\ell_t\) with respect to the scores is simply \(\mathbf{p}_t - \mathbf{e}(x_{t+1})\), so one step is
- Read the symbols
- \(\mathbf{z}_t\)
- the 27 scores before the softmax at position \(t\); \(\mathbf{p}_t\) the probabilities after it
- \(\eta\)
- the learning rate; the figure lets you try 0.5, 1, 2 and 4 (large, because the model is tiny and starts from zero)
- \(m_{t+1}\)
- the loss mask: a position with label −100 gives exactly zero gradient
- take the next 4 chats; labels −100 where \(m = 0\)
- forward: \(\mathbf{p}_t\) at every position // (54.10)
- \(\mathcal{L} \leftarrow \frac{1}{N} \sum_t m_{t+1}\, \ell_t\) // (54.1), \(N\) targets
- backward: \(\partial \mathcal{L} / \partial \mathbf{z}_t = m_{t+1} (\mathbf{p}_t - \mathbf{e}(x_{t+1})) / N\)
- \(\theta \leftarrow \theta - \eta\, \nabla_\theta \mathcal{L}\) // (54.11)
- evaluate: loss on all 32 chats; greedy answers to the 8 questions
<|im_start|> user ↵ question <|im_end|> ↵ <|im_start|> assistant ↵”; decoding stops at <|im_end|> or after 8 tokens. ✓ means the right word, then “.”, then <|im_end|>.How to read Figure 6
Press Play. With assistant-only loss, the orange curve (answers) falls from 3.30 to 0.154 while the blue curve (user and template tokens) rises to 6.709: the model reads the questions but is never trained to write them, and its guesses there get worse than uniform. The format comes first: after step 1 every answer is already “.” then <|im_end|>; the right words come later, and all 8 answers are correct from step 100. Now choose “All tokens”: at step 30 every “answer” is a new user turn, “<|im_start|> user ↵ capital of ?”, because the model learned to write the whole chat; after 120 steps only 4 of 8 answers are right. Choose “end-of-turn masked”: the words are learned, but no answer ever stops, because <|im_end|> was never a target. Step and Back move one phase at a time; at the two fastest speeds Play jumps one whole step per frame.
The PyTorch version below builds a ChatML-style example with a character-level tokenizer (so it runs with no downloads), packs two examples with a document mask, and checks two facts: packed logits equal separate logits, and dividing by the global token count makes gradient accumulation equal to the full batch.
import torch, torch.nn.functional as F torch.manual_seed(0) # 1) a tiny character "tokenizer" with chat special tokens SPECIAL = ["<|im_start|>", "<|im_end|>", "<|pad|>"] CHARS = sorted(set("abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.,?!:+=\n")) VOCAB = SPECIAL + CHARS ID = {t: i for i, t in enumerate(VOCAB)} START, END, PAD = ID["<|im_start|>"], ID["<|im_end|>"], ID["<|pad|>"] enc = lambda s: [ID[c] for c in s] def render(messages): """Chat template -> (ids, mask). mask[t] = 1 if token t is trained.""" ids, mask = [], [] for m in messages: head = [START] + enc(m["role"] + "\n") # role header: never trained ids += head; mask += [0] * len(head) body = enc(m["content"]) + [END] # content + end-of-turn train = 1 if m["role"] == "assistant" else 0 # loss only on assistant turns ids += body; mask += [train] * len(body) # (the end token is trained too) nl = enc("\n"); ids += nl; mask += [0] return ids, mask chat = [{"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "4."}] ids, mask = render(chat) labels = [t if m else -100 for t, m in zip(ids, mask)] # -100 = ignored by cross_entropy print(len(ids), sum(mask)) # 35 tokens, 3 trained # 2) the masked next-token loss, formula (54.1): position t predicts token t+1 def sft_loss(logits, labels, reduction="mean"): lg = logits[:, :-1].reshape(-1, logits.size(-1)) # predictions for positions 0..T-2 lb = labels[:, 1:].reshape(-1) # targets are the NEXT tokens return F.cross_entropy(lg, lb, ignore_index=-100, reduction=reduction) # 3) a tiny causal LM that accepts position ids and an explicit attention mask class TinyLM(torch.nn.Module): def __init__(s, V, d=32, T=64): super().__init__() s.tok, s.pos = torch.nn.Embedding(V, d), torch.nn.Embedding(T, d) s.qkv, s.out, s.head = torch.nn.Linear(d, 3 * d), torch.nn.Linear(d, d), torch.nn.Linear(d, V) def forward(s, ids, pos, allowed): # allowed: (B, T, T) bool x = s.tok(ids) + s.pos(pos) q, k, v = s.qkv(x).chunk(3, dim=-1) x = x + s.out(F.scaled_dot_product_attention(q, k, v, attn_mask=allowed)) return s.head(x) model = TinyLM(len(VOCAB)) causal = lambda n: torch.tril(torch.ones(n, n, dtype=torch.bool)) # 4) packing with a document mask gives exactly the same logits as separate rows chat2 = [{"role": "user", "content": "Say hi."}, {"role": "assistant", "content": "Hi there!"}] a, ma = render(chat); b, mb = render(chat2) la = [t if m else -100 for t, m in zip(a, ma)]; lb_ = [t if m else -100 for t, m in zip(b, mb)] packed = torch.tensor([a + b]) pos = torch.tensor([list(range(len(a))) + list(range(len(b)))]) # positions restart at 0, (54.6) doc = torch.tensor([0] * len(a) + [1] * len(b)) allowed = (doc[:, None] == doc[None, :]) & causal(len(doc)) # block-diagonal causal, (54.5) plabels = torch.tensor([la + [-100] + lb_[1:]]) # never predict sample 2's first token from sample 1 with torch.no_grad(): lp = model(packed, pos, allowed[None]) l1 = model(torch.tensor([a]), torch.arange(len(a))[None], causal(len(a))[None]) l2 = model(torch.tensor([b]), torch.arange(len(b))[None], causal(len(b))[None]) print((lp[0, :len(a)] - l1[0]).abs().max().item(), (lp[0, len(a):] - l2[0]).abs().max().item()) # 0.0 0.0 s_packed = sft_loss(lp, plabels, "sum").item() s_sep = sft_loss(l1, torch.tensor([la]), "sum").item() + sft_loss(l2, torch.tensor([lb_]), "sum").item() print(round(s_packed, 4), round(s_sep, 4)) # the same summed loss # 5) gradient accumulation: divide by the GLOBAL number of trained tokens, (54.4) def grads(loss): model.zero_grad(); loss.backward() return torch.cat([p.grad.flatten() for p in model.parameters()]).clone() micro = [(torch.tensor([a]), torch.tensor([la]), len(a)), (torch.tensor([b]), torch.tensor([lb_]), len(b))] n_total = sum((l[:, 1:] != -100).sum().item() for _, l, _ in micro) def fwd(x, n): return model(x, torch.arange(n)[None], causal(n)[None]) g_full = grads(sum(sft_loss(fwd(x, n), l, "sum") for x, l, n in micro) / n_total) g_right = sum(grads(sft_loss(fwd(x, n), l, "sum") / n_total) for x, l, n in micro) g_wrong = sum(grads(sft_loss(fwd(x, n), l, "mean") / len(micro)) for x, l, n in micro) print(n_total, (g_full - g_right).abs().max().item() < 1e-6, (g_full - g_wrong).abs().max().item() > 1e-3) # 13 True True: the right way matches the full batch, the mean of means does not
Libraries do the same work for you: Hugging Face TRL’s SFTTrainer has options for assistant-only loss and for packing, and its collators build the labels with -100. [Likely] Whatever library you use, print one rendered example with its labels before you train.
Where this is used
| Setting | What SFT data looks like | Details that matter |
|---|---|---|
| General chat assistants (Llama 2 chat, Tülu 3) | Tens of thousands to about a million multi-turn chats, many sources | Assistant-only loss, packing, 2 epochs, low learning rate; the starting point for DPO and RL (session 68) |
| Small, curated alignment (LIMA style) | About 1,000 excellent examples | Many epochs, checkpoint chosen by reading outputs, not by validation loss |
| Domain assistants (legal, medical, internal support) | A few thousand expert answers plus general chat data | Mix in general data against forgetting (session 61); often LoRA (session 57) |
| Reasoning models | Long chain-of-thought answers, often distilled from a stronger model (session 120) | Very long rows (tens of thousands of tokens), packing with varlen attention, answers must not be cut |
| Tool use and agents | Chats with tool calls and tool results | Train on the assistant’s tool calls, mask the tool results (the model does not write them) |
| VLM instruction tuning | Image or video plus question, answer in text | Labels of image and video placeholder tokens are −100; only answer text is trained (session 189) |
| Video-LLMs: video QA, key-moment detection, dense captions | Clips with questions, timestamps and captions | Very uneven lengths (hundreds to thousands of visual tokens), so packing and document masks save the most here; answers with timestamps must be exact (session 205) |
| Video retrieval and classification heads on top of an LLM | Labels written as short text answers | Answers are 1 to 3 tokens long: a sequence mean gives them huge weight compared with long captions in the same mix |
Common mistakes
- No loss mask. Training on user and system tokens. The model learns to write questions and may continue with a fake user turn.
- Masking the end-of-turn token. The model never learns to stop. Check that the last label of each assistant turn is the end token.
- Using a different chat template in training than in inference, or a different one from the base model’s own. Print the rendered text and the token ids (session 9).
- Packing without a document mask or without resetting position ids. Examples read each other, and positions are wrong for every example after the first.
- Mean of micro-batch means. Gradient accumulation or data parallelism with per-micro-batch averaging changes the example weights (54.4). Divide the summed loss by the global number of target tokens.
- Silent truncation. Cutting long examples at the maximum length removes the end of the answer and its end token. Count how many examples are too long, and drop or split them.
- A pretraining-size learning rate or too many epochs on a small set: forgetting and memorized answers. Start around \(10^{-5}\) or lower for full fine-tuning and 1 to 3 epochs, then tune. [Likely]
- Choosing the checkpoint by validation loss alone. Use generation-based evaluation; and decontaminate the training set against your benchmarks.
Interview questions
What is SFT, and how is its loss different from pretraining?basic
Why do we mask the prompt tokens? Is it always better?core
Why must the end-of-turn token be in the loss?basic
Per-token mean, per-sequence mean or sum: what is the difference?core
Gradient accumulation gives a different result from one big batch. Why, and how do you fix it?advanced
Explain packing for SFT. What three things must be right?core
What learning rate and how many epochs would you use for full SFT of an 8B model?core
How do you detect overfitting in SFT, and why is validation loss not enough?core
What did LIMA show, and what are its limits?advanced
You have 2 million raw SFT examples and budget for 100k. How do you choose?advanced
In a VLM or Video-LLM, which tokens get a loss during instruction tuning?core
Exercises
- A chat has a 40-token system prompt, a 200-token user turn and a 60-token answer, plus 5 template tokens per turn (headers, end tokens, newlines; 15 in total). What fraction of tokens are targets with assistant-only loss, if the answer’s end-of-turn token is one of its 5 template tokens and is trained?
- Batch: example A has 5 target tokens with loss 3.0 each, example B has 95 target tokens with loss 1.0 each. Compute the token mean, the sequence mean and the sum.
- Two micro-batches: the first has 100 target tokens with total loss 120, the second 20 target tokens with total loss 40. Compute the correct accumulated loss and the mean of micro-batch means.
- Lengths 1500, 1200, 900, 700, 400 and 300 tokens, \(T = 2048\). Pack them with first-fit decreasing. How many rows, and what is the utilization?
- You have 50,000 examples, an effective batch of 64 and 3 epochs. How many optimizer steps? With 3% warmup, how many warmup steps?
- In a packed row, example 2 starts at position 700 of the row. What position id does the token at row position 745 get?
Answerscheck
Key takeaways
- SFT is next-token training on conversations with a loss mask (54.1): only assistant tokens and their end-of-turn token are targets; everything else is context with label −100.
- Token mean, sequence mean and sum (54.2) weight examples differently (54.3). With gradient accumulation or many GPUs, divide the summed loss by the global target-token count (54.4), or the weights change silently.
- Packing fills rows with whole examples; it needs a block-diagonal document mask (54.5), position ids that restart (54.6) and no loss across boundaries. Utilization (54.7) often goes from under 50% to over 90%.
- SFT is a short, gentle run: learning rate around \(10^{-5}\) or lower for full fine-tuning, 1 to 3 epochs, batches of 32 to 128 examples, length long enough that answers are not cut (54.8).
- Watch for overfitting, but pick checkpoints by generation quality, not validation loss alone.
- Quality first, then coverage: LIMA (1,000 examples), Llama 2 (27,540) and AlpaGasus (9k of 52k) show that clean data wins; Tülu 3 shows that skills need targeted, diverse sources.
Key formulas
References for this session
- Zhou et al. (2023). LIMA: Less Is More for Alignment
- Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
- Raschka, Build a Large Language Model (From Scratch), chapter 7: Fine-tuning to follow instructions
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 10: Alignment
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Jain et al. (2023). NEFTune: Noisy Embeddings Improve Instruction Finetuning
- Chen et al. (2023). AlpaGasus: Training A Better Alpaca with Fewer Data
- Shi et al. (2024). Instruction Tuning With Loss Over Instructions
- Hugging Face (2024). Fixing Gradient Accumulation
- Wang et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions