LLM & VLM courseBlog
54Track 4: Post-training and RL · session 2 of 25

SFT in detail

Supervised fine-tuning (SFT) is next-token training on example conversations, with one change: the loss counts only the tokens the assistant should write. This session goes through the details that decide whether SFT works: the loss mask, how the loss is averaged, packing with document masks, epochs and learning rate, the signs of overfitting, and why a small, clean, diverse dataset often beats a big one.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: BeginnerBefore this: Sessions 4, 9, 18 and 53

What you will learn

  • The SFT loss: next-token cross-entropy with a loss mask, so only assistant tokens (and their end-of-turn token) are trained
  • How the loss is averaged (per token, per sequence, sum), and the gradient-accumulation bug that silently changes the weights
  • Packing several examples into one row with a document mask and reset position ids, so a packed row equals separate rows
  • Epochs, learning rate, batch size and maximum length, with real recipes from LIMA, Llama 2 and Tülu 3
  • How to see overfitting: training loss, validation loss and why generation quality must be checked separately
  • Data quality versus quantity, and how to choose a diverse subset

Why this matters

In session 53 we saw the plan: a base model only continues text, and post-training turns it into an assistant. The first and most important stage is supervised fine-tuning (SFT): we show the model many example conversations and train it to write the assistant’s replies. The idea fits in one sentence. The details decide whether it works.

SFT uses the same next-token loss as pretraining (formula 54.1). Four details make it different, and each one is a classic source of silent bugs:

  • Loss masking. We train only on the tokens the assistant should write, not on the user’s question or the system prompt. Forget this and the model learns to write user questions.
  • How the loss is averaged. Per token, per sequence, or a plain sum. The choice changes how much each example counts. Gradient accumulation can change it again without telling you.
  • Packing. Chat examples are short and of very different lengths. We pack several into one row to avoid wasted padding, and we need a document mask so they do not read each other.
  • Small data, few steps. SFT sets are thousands to about a million examples, not trillions of tokens. Epochs, learning rate and data quality matter a lot, and overfitting is real.

Real systems use exactly these pieces. Llama 2’s SFT “zero-out[s] the loss on tokens from the user prompt”, packs all prompts and answers into rows of 4,096 tokens, and trains for 2 epochs at learning rate \(2 \times 10^{-5}\). [Certain] Tülu 3 trains Llama 3.1 8B on 939,344 SFT prompts for 2 epochs at \(5 \times 10^{-6}\), and reports that using a sum loss instead of a mean loss mattered. [Certain] LIMA fine-tuned a 65B model on only 1,000 carefully chosen examples. [Certain] By the end of this session you will understand every one of these choices.

The SFT loss: next-token prediction with a mask

An SFT example is a conversation: a list of messages, each with a role (system, user or assistant) and some text. The chat template (session 9) turns it into one token sequence \(x_1, \dots, x_T\) with role headers and end-of-turn tokens. For example, in the ChatML style used by Qwen:

<|im_start|>user
What is 2+2?<|im_end|>
<|im_start|>assistant
4.<|im_end|>

Pretraining would compute the loss on every token. In SFT we attach a loss mask \(m_t \in \{0, 1\}\) to every token: \(m_t = 1\) if the model should learn to produce token \(t\), and \(m_t = 0\) otherwise. The loss is the average negative log-probability of the masked-in tokens only:

\[ \mathcal{L}(\theta) = -\frac{1}{Z} \sum_{i=1}^{B} \sum_{t=2}^{T_i} m_{i,t} \, \log p_\theta\!\left(x_{i,t} \mid x_{i,1}, \dots, x_{i,t-1}\right) \tag{54.1} \]
Read the symbols
\(i\), \(B\)
the example index, and the number of examples in the batch
\(t\), \(T_i\)
the token position, and the length of example \(i\); position 1 has nothing before it, so it is never a target
\(m_{i,t}\)
the loss mask: 1 for tokens the assistant writes (including its end-of-turn token), 0 for system, user and role-header tokens
\(p_\theta(x_{i,t} \mid \dots)\)
the probability the model gives to the true next token, after the softmax over the vocabulary
\(Z\)
the normalizer: how we turn the sum into an average. The next section shows that this choice matters

The model still reads every token: the user’s question is in the context and attention uses it. The mask only decides which positions produce a gradient. In code, the mask is usually stored inside the labels: a label of -100 means “ignore this position”, because PyTorch’s cross_entropy skips that value by default (ignore_index=-100). [Certain]

Why mask the prompt?

Three reasons. First, the goal is to answer, not to write questions. Gradient on user tokens teaches the model to produce user-style text, and at inference it may start writing the next user turn itself. Second, the prompt tokens dominate in many datasets (long documents with short answers). Without a mask, most of the gradient goes to copying prompts. Third, prompts often come from a different source than answers (scraped questions, synthetic templates), so their quality is lower.

The end-of-turn token (<|im_end|> above) must be trained. It is how the model learns to stop. If it is masked out, the model never learns where an answer ends and keeps writing. LIMA added a special end-of-turn token for this reason, separate from the pretraining end-of-text token. [Certain]

Masking the prompt is the default, not a law. Shi et al. (2024) found that training on the instructions too can help when the outputs are short or the dataset is small, by reducing overfitting. [Certain] In practice, start with assistant-only loss and treat prompt loss as an experiment.

Figure 1 runs formula (54.1) one position at a time, on real numbers. The probabilities come from a tiny language model that your browser trains when the page loads: 120 steps of SFT on 32 small chats with assistant-only loss. Figure 6 shows that training run step by step, and formula (54.10) defines the model. At each position \(t\), the model reads the tokens up to \(x_t\), gives a probability to the true next token \(x_{t+1}\), and the loss \(\ell_t = -\log p\) is added to the sum only if the next token has mask 1.

not a target (label −100)target (gets a gradient)token being read, \(x_t\)next token, \(x_{t+1}\)second chat in the row
  1. \(S \leftarrow 0,\ n \leftarrow 0,\ \mathbf{h} \leftarrow \mathbf{0}\)
  2. for \(t = 1, \dots, T-1\) do
  3. \(\mathbf{h} \leftarrow \gamma\,\mathbf{h} + \mathbf{e}(x_t)\) // \(\mathbf{h}\) restarts at 0 for a new chat if the document mask is on
  4. \(\mathbf{p} \leftarrow \softmax(A\,\mathbf{e}(x_t) + C\,\mathbf{h} + \mathbf{b})\) // (54.10)
  5. \(\ell_t \leftarrow -\log p[x_{t+1}]\)
  6. if \(m_{t+1} = 1\): \(S \leftarrow S + \ell_t,\ n \leftarrow n + 1\) // else label −100: skip
  7. \(\mathcal{L} \leftarrow S / n\) // (54.1) with \(Z = n\)
Figure 1. The masked loss computed live, position by position, in a ChatML-style template. Words stand in for tokens (a real tokenizer would split some of them further), ↵ is a newline token. Under each token: its position id, and either its label −100 or, once computed, its loss \(\ell\). The square on the left is the attention mask of the row; the highlighted line is the row of the current position.
How to read Figure 1

Press Play and watch the cursor. Blue marks the token being read, orange the token to predict. On user and header positions the model gets losses between 3.1 and 9.3 nats, nearly all worse than a uniform guess over its 27 tokens (\(\ln 27 = 3.30\)): it was never trained to write questions, and nothing is added to \(S\). On the answers the loss is small, and it is added. Switch to “All tokens”: the user tokens now count, and the reported loss jumps from 0.315 to 5.622. Untick the end-of-turn box: the <|im_end|> positions drop out, so the model gets no signal to stop. “Last assistant turn only” is used when earlier assistant turns came from another model and you do not trust them. Choose “Untrained model” to see that every position starts at \(\ln 27\). The two-chat row and the document mask are explained in the packing section below.

Worked example: how few tokens are trained

In Figure 1 the single chat has 32 tokens. With assistant-only loss, the targets are “Paris .” plus <|im_end|> and “small .” plus <|im_end|>: 6 of 32 tokens, or 18.8%. The trained tiny model gives them losses 0.5841, 0.0386, 0.0095, 1.2199, 0.0328 and 0.0061, so \(S = 1.891\) and \(\mathcal{L} = 1.891 / 6 = 0.315\). The other 25 predictions are computed in the same forward pass and then thrown away. The second answer costs more than the first (1.2199 against 0.5841) because the tiny model’s memory still holds the first question when it answers the second. [Likely] The short example in the code section below has 35 tokens and only 3 targets (8.6%). So the gradient per row in SFT comes from a small part of the tokens, and the number of target tokens, not the number of rows, is what you should count.

How to average the loss

Formula (54.1) has a normalizer \(Z\). There are three common choices, and they are not the same when examples have different numbers of target tokens. Let \(n_i = \sum_t m_{i,t}\) be the number of target tokens in example \(i\), and \(\ell_{i,t} = -\log p_\theta(x_{i,t} \mid x_{i,\lt t})\) the loss of one token.

\[ \mathcal{L}_{\text{token}} = \frac{\sum_i \sum_t m_{i,t}\,\ell_{i,t}}{\sum_i n_i}, \qquad \mathcal{L}_{\text{seq}} = \frac{1}{B} \sum_i \frac{\sum_t m_{i,t}\,\ell_{i,t}}{n_i}, \qquad \mathcal{L}_{\text{sum}} = \sum_i \sum_t m_{i,t}\,\ell_{i,t} \tag{54.2} \]

All three have the same per-token losses inside. They differ in the weight each token gets in the gradient:

\[ w^{\text{token}}_{i,t} = \frac{1}{\sum_j n_j}, \qquad w^{\text{seq}}_{i,t} = \frac{1}{B\, n_i}, \qquad w^{\text{sum}}_{i,t} = 1 \tag{54.3} \]
Read the symbols
\(n_i\)
number of target tokens (mask = 1) in example \(i\)
\(\ell_{i,t}\)
cross-entropy of one target token
\(w_{i,t}\)
how much token \(t\) of example \(i\) counts in the gradient
  • Token mean: every target token counts the same. Long answers get more total weight. This is what cross_entropy(..., reduction="mean") gives on a flattened batch.
  • Sequence mean: every example counts the same. A token in a 10-token answer counts 9 times more than a token in a 90-token answer. Short answers (“Yes.”) get a lot of weight.
  • Sum: every token counts the same and the scale is not divided by anything. The gradient size then grows with the number of tokens in the batch, so the learning rate must be chosen for it. Tülu 3 used a sum loss because it “effectively weights all tokens equally”, which gave better results for their mixtures. [Certain]
Worked example: two examples, three losses

Example A has \(n_A = 10\) target tokens with average loss 2.0. Example B has \(n_B = 90\) with average loss 1.0. Then:

Token mean: \((10 \times 2.0 + 90 \times 1.0)/100 = 1.10\). Sequence mean: \((2.0 + 1.0)/2 = 1.50\). Sum: \(20 + 90 = 110\).

Weights per token: token mean gives every token \(1/100\). Sequence mean gives A’s tokens \(1/(2 \times 10) = 0.05\) and B’s tokens \(1/(2 \times 90) \approx 0.0056\): a ratio of 9. The reported loss is different (1.10 versus 1.50), and so is the direction of the gradient.

The gradient-accumulation trap

Gradient accumulation (session 39) splits a large batch into \(G\) micro-batches, runs backward on each, and adds the gradients. It should give exactly the same gradient as the full batch. With a token-mean loss it does not, if each micro-batch is averaged over its own tokens and the results are averaged over micro-batches. The correct way divides by the token count of the whole batch:

\[ \mathcal{L}_{\text{token}} = \sum_{g=1}^{G} \frac{\sum_{(i,t) \in g} m_{i,t}\,\ell_{i,t}}{N}, \qquad N = \sum_{g=1}^{G} \sum_{i \in g} n_i \ \ne\ \frac{1}{G} \sum_{g=1}^{G} \frac{\sum_{(i,t) \in g} m_{i,t}\,\ell_{i,t}}{N_g} \tag{54.4} \]
Read the symbols
\(g\), \(G\)
a micro-batch, and the number of micro-batches in one optimizer step
\(N_g\)
target tokens in micro-batch \(g\); \(N\) is the total over all micro-batches

The right side, a “mean of means”, gives micro-batches with few target tokens too much weight. The two sides are equal only when every micro-batch has the same \(N_g\). This was a real bug in popular training code: in October 2024 Hugging Face confirmed that its trainer computed the loss this way, and fixed it by summing the loss and dividing by the total number of target tokens. [Certain] The same issue appears across GPUs in data-parallel training: each GPU averages over its own tokens, then gradients are averaged over GPUs. Tülu 3 notes that “changing gradient accumulation can have large effects on performance due to effectively changing sample weightings”. [Certain]

share of target tokensshare of gradient weight
Figure 2. One optimizer step over four examples. Micro-batch 1 holds A and B, micro-batch 2 holds C and D. Each example has its own number of target tokens and a fixed loss per token. The bars show how much of the gradient each example controls under each way of averaging.
How to read Figure 2

With the token mean, the orange bar equals the grey bar: each example counts in proportion to its tokens. With the sequence mean, every orange bar is 25%, so the short answer A controls as much as the long answer D. The mean of micro-batch means is in between and depends on how examples happen to fall into micro-batches: move answer A to 400 tokens and watch C and D gain weight although nothing about them changed. The reported loss is different in each mode too, so curves from runs with different settings cannot be compared directly.

Packing: many examples in one row

Chat examples have very different lengths: “Hi” and a 3,000-token code review sit in the same dataset. If each example gets its own row padded to the longest in the batch, much of the GPU work is spent on padding. Packing puts several whole examples into one row of length \(T\). Session 18 introduced packing for pretraining; here we look at what SFT needs on top.

For SFT, an example must stay whole: cutting an answer in the middle would train the model on a fragment with no end-of-turn token. So SFT packing is a bin-packing problem: fill each row with whole examples that fit, for example with first-fit decreasing (sort examples from long to short; put each into the first row that still has room). Examples longer than \(T\) are cut (truncated) or dropped.

A packed row needs three things to behave exactly like separate rows:

  1. A document mask. Token \(i\) may attend to token \(j\) only if both belong to the same example and \(j \le i\):
    \[ M_{ij} = \begin{cases} 0 & \text{if } d(i) = d(j) \text{ and } j \le i \\ -\infty & \text{otherwise} \end{cases} \tag{54.5} \]
    where \(d(i)\) is the index of the example that token \(i\) belongs to. The mask is block-diagonal: one causal triangle per example.
  2. Position ids that restart at 0 for each example, so each one sees the same positions (and RoPE angles) as when it is alone:
    \[ p_i = i - s_{d(i)} \tag{54.6} \]
    where \(s_{d}\) is the first position of example \(d\) in the row.
  3. No loss across the boundary. The last token of example 1 must not be trained to predict the first token of example 2. With chat templates this is automatic, because the first token of each example is a role header with mask 0.

With these three, a packed row gives the same logits as running each example alone (the code below checks it to the last bit). Without the document mask, example 2 can attend to example 1. In SFT this is worse than in pretraining: a later answer can read an earlier, similar example, which is a leak, and the model learns to depend on unrelated context. [Likely] Fast attention kernels apply the mask without building it: FlashAttention’s variable-length interface takes the examples back to back plus their cumulative lengths (cu_seqlens). [Certain]

Figure 1 shows this on real numbers. Choose “Two chats packed in one row”: the second chat starts at row position 32, and its position ids restart at 0. In the tiny model, a fading memory of earlier tokens plays the role of attention, so the document mask means: the memory restarts at the first token of a new chat. With the mask on, every loss in the second chat is exactly what it is when that chat runs alone (“blue” costs 0.204), and the row loss over the 9 targets is 0.241. Untick the document mask: the memory now carries the first chat into the second, and every prediction there changes. “blue” now costs 0.164 and the row loss falls to 0.233. The leak happens to lower the loss here, and that is the danger: information from an unrelated example looks like learning.

Packing also interacts with the loss averaging. In a packed row, a token mean over the row gives long examples more weight, exactly as before. If you want a sequence mean, you must compute it per example inside the row, using the example ids. And packing changes the number of rows per step: a “batch of 128 examples” and a “batch of 128 packed rows” are very different amounts of data. We measure how well rows are filled with the utilization:

\[ U = \frac{\sum_i \min(L_i, T)}{R \cdot T} \tag{54.7} \]
Read the symbols
\(L_i\)
length of example \(i\) in tokens (prompt plus answer)
\(T\)
row length (maximum sequence length)
\(R\)
number of rows used; for padding, replace \(R \cdot T\) by the total padded slots
Worked example: padding versus packing

Eight examples have lengths 120, 900, 300, 2000, 450, 80, 1500 and 600 tokens (sum 5,950), and \(T = 2048\). Padding every example to 2,048 uses \(8 \times 2048 = 16{,}384\) slots: \(U = 5950/16384 = 36.3\%\). First-fit decreasing makes three rows: [2000], [1500, 450, 80] and [900, 600, 300, 120], filled with 2,000, 2,030 and 1,920 tokens. Now \(U = 5950/(3 \times 2048) = 96.8\%\), and the step needs 3 rows instead of 8.

The document mask also saves attention work: one causal triangle per example has \(\sum_i L_i(L_i+1)/2 \approx 3.87\) million query-key pairs, against about 6.29 million for three full causal rows of 2,048. Kernels that skip the blocked tiles get this saving. [Likely]

prompt (loss mask 0)answer (trained)paddinganswer cut by the length limit
  1. sort the examples by length \(L_i\), longest first
  2. for each example \(i\) in that order do
  3. for each open row \(r\): if \(u_r + L_i \le T\): put \(i\) in \(r\), \(u_r \leftarrow u_r + L_i\), stop // first fit
  4. if no row had room: open a new row with \(u = L_i\)
  5. \(U \leftarrow \sum_i \min(L_i, T) / (R \cdot T)\) // (54.7)
Figure 3. Sixteen SFT examples with seeded prompt and answer lengths (prompts 20 to 800 tokens, answers 20 to 1,500, log-uniform). Left: the rows of one pass over the data. Right: the attention mask of the selected packed row, drawn at reduced resolution; each colored triangle is one example, and white means blocked. Click a row to see its mask. While the animation runs, the mask follows the row that just received an example; \(u_r\) is the number of tokens already in row \(r\).
How to read Figure 3

Press Play in packing mode to watch first-fit decreasing work: each example tries the open rows from the top, the formula box shows the test \(u_r + L_i \le T\) with the real numbers, and \(U\) is updated after every placement. In padding mode most of the area is empty padding: about half of the compute is wasted at \(T = 2048\). In packing mode, rows are almost full. Now set \(T = 512\): long examples no longer fit, and their answers are cut (red marks). A cut answer has lost its end-of-turn token, so the model is trained on answers that stop in the middle. This is why the maximum length is a data decision, not only a memory decision: check how many examples exceed it, and drop or split them instead of cutting silently.

Epochs, learning rate, batch and length

SFT starts from a model that is already good. Its job is to change the style and format of the answers, plus some skills, without destroying what pretraining learned. This shapes every hyper-parameter.

Learning rate

SFT learning rates are much lower than pretraining. Llama 2 pretrained its 7B model at a peak of \(3 \times 10^{-4}\) and fine-tuned at \(2 \times 10^{-5}\): 15 times lower. [Certain] Tülu 3 used \(5 \times 10^{-6}\) for 8B and \(2 \times 10^{-6}\) for 70B, with linear decay and 3% warmup. [Certain] Larger models usually get smaller learning rates. [Likely] Too high, and the model forgets general knowledge (catastrophic forgetting, see session 61) and becomes unstable; too low, and it does not pick up the format. LoRA (session 57) uses much larger learning rates, around \(10^{-4}\), because only small adapter matrices are trained. [Likely]

Epochs and steps

The number of optimizer steps is

\[ S = \left\lceil \frac{N_{\text{ex}}}{B} \right\rceil \cdot E \tag{54.8} \]
Read the symbols
\(N_{\text{ex}}\)
number of training examples (or packed rows, if the batch counts rows)
\(B\)
effective batch size: per-GPU batch × number of GPUs × accumulation steps
\(E\)
number of epochs (full passes over the data)

Most recipes use 1 to 3 epochs. Llama 2 and Tülu 3 used 2; Tülu 3 reports that 2 epochs worked best in their sweep. [Certain] Small, high-quality sets are trained longer: LIMA trained 15 epochs on 1,000 examples and picked a checkpoint between epochs 5 and 10. [Certain] More epochs on a small set means the model sees the same answers many times; it starts to memorize them.

Worked example: how many steps is an SFT run?

Tülu 3, 8B: \(N_{\text{ex}} = 939{,}344\), \(B = 128\), \(E = 2\), so \(S = 939344 \times 2 / 128 \approx 14{,}677\) steps (slightly more with rounding per epoch), and 3% warmup is about 440 steps. LIMA: \(\lceil 1000/32 \rceil = 32\) steps per epoch, 15 epochs, so about 480 steps in total. Compare with pretraining: Llama 2 took about 500,000 steps. SFT is a short run, so a few bad steps (a spike, a wrong learning rate) matter a lot.

Batch size and maximum length

Effective batches of 32 to 128 examples are common for SFT (LIMA 32, Llama 2 64, Tülu 3 128). [Certain] The maximum length should cover almost all examples: Llama 2 and Tülu 3 used 4,096 tokens, LIMA 2,048. [Certain] Long-context or reasoning SFT (long chains of thought) needs much longer rows, and then packing, document masks and memory tricks become essential (session 39).

Small tricks you will meet

NEFTune adds uniform noise to the input embeddings during SFT only:

\[ X' = X + \frac{\alpha}{\sqrt{L d}}\, \varepsilon, \qquad \varepsilon_{jk} \sim \mathcal{U}(-1, 1) \tag{54.9} \]

where \(L\) is the sequence length, \(d\) the embedding size and \(\alpha\) a strength (5 to 15 in the paper). It reported AlpacaEval going from 29.79% to 64.69% for Llama 2 7B fine-tuned on Alpaca, with smaller gains on other datasets. [Certain] It acts as a regularizer against overfitting; gains depend on the dataset and evaluation. [Likely] Other common choices: weight decay 0.1 with AdamW (as in LIMA and Llama 2), and dropout in LIMA (from 0.0 at the bottom layer to 0.3 at the top for 65B). [Certain]

Signs of overfitting

Overfitting means the model gets better on the training examples but worse on new ones. In SFT, the signs are:

  • Training loss keeps falling, often with a sharp drop at the start of each new epoch (the model has seen these exact answers before).
  • Validation loss on held-out examples stops falling and then rises.
  • Generations start copying phrases from the training set, become formulaic (“As an AI language model…” everywhere), or lose diversity.
  • Other skills drop: scores on general benchmarks (MMLU-style knowledge, math) go down compared with the base model, which is forgetting.

There is a catch. For SFT, validation loss is not a reliable measure of answer quality. LIMA reports that perplexity on held-out data did not correlate with generation quality, and chose checkpoints by reading the outputs of a 50-example development set. [Certain] A model can become more confident in its own style (higher loss on reference answers written in a different style) while its answers get better. So use validation loss to catch clear overfitting, and use generation-based evaluation (human or model judges, task benchmarks) to pick the checkpoint (session 40).

training lossvalidation loss (600 held-out examples)
Figure 4. Real training in your browser, on a small stand-in model: a softmax classifier with 24 inputs and 6 classes, trained with mini-batch SGD (batch 16) for 30 epochs. The labels come from a hidden teacher with 30% label noise, so the best possible model still has a loss well above 0. The data and the shuffling are seeded.
How to read Figure 4

With 100 examples, training loss falls all the way while validation loss is lowest around epoch 7 and then climbs: the model is learning the noise in its few examples. Raise the number of examples: the gap between the curves shrinks and the best epoch moves later. Raise the learning rate: everything happens sooner, including overfitting. An LLM is much bigger than this model, but the same pattern appears when a small SFT set is trained for many epochs. [Likely]

Data: quality, quantity and diversity

For SFT, the data is the recipe. The model already knows a lot from pretraining; SFT mostly teaches it the format and the behavior of a good assistant. LIMA calls this the superficial alignment hypothesis: almost all knowledge is learned in pretraining, and alignment teaches which style and format to use. [Certain] If that is true, a small number of excellent examples should be enough, and the evidence partly agrees:

  • LIMA: 1,000 curated examples on LLaMA 65B, no RLHF. In a human study, its answers were equal to or preferred over GPT-4 in 43% of cases, 58% against Bard and 65% against DaVinci003. [Certain] In its ablations (7B model, 2,000 examples each), more diverse data (Stack Exchange) beat less diverse data (wikiHow), filtered data beat unfiltered, and making the set up to 16 times larger without more diversity did not help. [Certain] Adding just 30 multi-turn dialogues raised the share of excellent responses in dialogue from 45.2% to 76.1%. [Certain]
  • Llama 2: “Quality is all you need”. Meta stopped after 27,540 high-quality SFT annotations, finding that tens of thousands were enough. [Certain]
  • AlpaGasus: kept 9k of the 52k Alpaca examples using an LLM to score quality, and the filtered model beat the original Alpaca while training 5.7 times faster. [Certain]
  • Tülu 3: a large mix (939,344 prompts) built from many sources, each targeting a skill: math, code, instruction following, safety, multilingual and real user chats. Removing a source hurts its skill: without the WildChat real-user data the average score fell from 60.1 to 58.9, and without the safety data to 58.0. [Certain]

So the modern picture is: quality first, then coverage. A few thousand excellent examples can teach format. Real skills (math, code, tool use, many languages) need targeted data for each skill, and more of it. [Likely] Quantity helps only if it adds new kinds of examples, not more copies of the same kind.

What “quality” means in practice

  • Correct answers (verify math and code by running them), in the style you want the model to have.
  • Consistent format: the same template, the same way of using markdown, the same refusal style.
  • Deduplicated (exact and near-duplicates), so a few examples do not get many epochs’ worth of weight.
  • Decontaminated: remove examples that overlap with evaluation benchmarks, or your scores will be inflated. Tülu 3 checks its training prompts against its evaluation sets. [Certain]
  • Balanced across tasks: choose the mixture on purpose instead of taking whatever is largest.

Choosing a diverse subset

A typical raw pool is dominated by a few easy kinds of request. Random sampling copies that imbalance. A simple fix is to embed every example (with any sentence-embedding model, see session 60) and pick examples that are far from the ones already picked. The greedy k-center method does this: start from one example, then repeatedly add the example whose distance to its nearest picked example is largest. The coverage radius, the largest distance from any pool example to its nearest picked example, measures how well the subset covers the pool. Data-selection methods used in practice combine such diversity scores with quality and difficulty scores. [Likely]

Figure 5. A seeded pool of 600 examples drawn as points in a 2-D embedding space. There are 8 task groups with very different sizes: 270 simple chats, 120 coding, 60 math, 48 writing, 36 safety, 30 tool use, 24 multilingual and 12 video questions. Large circles are the examples kept for SFT.
How to read Figure 5

At \(k = 16\), the random sample misses whole task groups (math and safety), because small groups are rarely drawn. Greedy k-center covers all 8 groups and has a much smaller coverage radius. Note the cost: k-center also likes outliers (a strange example far from everything is picked early), so in practice it is combined with a quality filter that removes broken or off-topic examples first.

Putting it together: the SFT pipeline

Algorithm 1: turn one conversation into a training example
  1. input: messages \([(r_1, c_1), \dots, (r_K, c_K)]\), tokenizer, chat template, max length \(T\)
  2. \(\text{ids} \leftarrow [\,],\ \text{labels} \leftarrow [\,]\)
  3. for each message \((r, c)\): // render with the model’s own template
  4.   \(h \leftarrow\) tokens of the role header; append \(h\) to \(\text{ids}\) and \(-100\) for each to \(\text{labels}\)
  5.   \(b \leftarrow\) tokens of \(c\) followed by the end-of-turn token
  6.   append \(b\) to \(\text{ids}\); append \(b\) to \(\text{labels}\) if \(r = \text{assistant}\), else \(-100\) for each // formula (54.1)
  7. if \(\text{length}(\text{ids}) \gt T\): drop the example, or split the chat at a turn boundary // never cut an answer silently
  8. if no label is kept: drop the example // zero target tokens gives \(0/0\)
  9. return \(\text{ids}, \text{labels}\)
Algorithm 2: pack examples and train one step with correct averaging
  1. input: examples \((\text{ids}_i, \text{labels}_i)\), row length \(T\), micro-batches per step \(G\)
  2. sort examples by length, long first; place each into the first row with room // first-fit decreasing
  3. for each row: concatenate \(\text{ids}\) and \(\text{labels}\); \(\text{doc} \leftarrow\) example index per token; \(\text{pos} \leftarrow\) restart at 0 per example // (54.6)
  4. \(\text{mask} \leftarrow \text{causal} \wedge (\text{doc}_i = \text{doc}_j)\), or pass cu_seqlens to a varlen kernel // (54.5)
  5. \(N \leftarrow\) number of labels \(\ne -100\) in all \(G\) micro-batches of this step (summed over GPUs too)
  6. for \(g = 1 \dots G\):
  7.   \(\text{logits} \leftarrow \text{model}(\text{ids}_g, \text{pos}_g, \text{mask}_g)\)
  8.   \(\text{loss} \leftarrow \operatorname{CE}_{\text{sum}}\big(\text{logits}[:, :-1],\ \text{labels}_g[:, 1:]\big) / N\) // cross_entropy(..., reduction="sum") // (54.4)
  9.   loss.backward() // gradients add up
  10. clip, optimizer step with a low learning rate and its schedule, zero gradients

A tiny SFT run you can watch

To see SFT learn a format, we need a model small enough to train in a browser in a fraction of a second. Ours has a vocabulary of \(V = 27\) word tokens: the five template tokens (<|im_start|>, <|im_end|>, ↵, user, assistant) and the words of 8 question-answer pairs, such as “capital of France ?” → “Paris .” and “opposite of big ?” → “small .”. Instead of attention, it has a fading memory \(\mathbf{h}_t\) of all tokens read so far:

\[ \mathbf{h}_t = \gamma\, \mathbf{h}_{t-1} + \mathbf{e}(x_t), \qquad p_\theta(x_{t+1} \mid x_{\le t}) = \softmax\!\big(A\, \mathbf{e}(x_t) + C\, \mathbf{h}_t + \mathbf{b}\big) \tag{54.10} \]
Read the symbols
\(\mathbf{e}(x)\)
the one-hot vector of token \(x\): length 27, a 1 at the token’s index and 0 elsewhere
\(\mathbf{h}_t\)
the memory: each earlier token, weighted by \(\gamma\) to the power of how far back it is; with \(\gamma = 0.9\), a token 5 positions back still has weight \(0.9^5 = 0.59\)
\(A, C, \mathbf{b}\)
the weights \(\theta\): \(A\) scores the next token from the current token, \(C\) from the memory, \(\mathbf{b}\) is a bias. In total \(2 \cdot 27^2 + 27 = 1{,}485\) numbers, all starting at 0

The training set has 32 seeded chats with 1 or 2 question-answer turns each. Training is plain SGD on batches of 4 chats, with the token-mean loss (54.1) and \(Z = N\), the number of target tokens in the batch. For a softmax, the gradient of \(\ell_t\) with respect to the scores is simply \(\mathbf{p}_t - \mathbf{e}(x_{t+1})\), so one step is

\[ \theta \leftarrow \theta - \eta\, \nabla_\theta \mathcal{L}, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{z}_t} = \frac{m_{t+1}}{N} \big(\mathbf{p}_t - \mathbf{e}(x_{t+1})\big) \tag{54.11} \]
Read the symbols
\(\mathbf{z}_t\)
the 27 scores before the softmax at position \(t\); \(\mathbf{p}_t\) the probabilities after it
\(\eta\)
the learning rate; the figure lets you try 0.5, 1, 2 and 4 (large, because the model is tiny and starts from zero)
\(m_{t+1}\)
the loss mask: a position with label −100 gives exactly zero gradient
loss on assistant tokens, all 32 chatsloss on user and template tokensbatch loss \(\mathcal{L}\) (the trained objective)uniform guess, \(\ln 27\)
  1. take the next 4 chats; labels −100 where \(m = 0\)
  2. forward: \(\mathbf{p}_t\) at every position // (54.10)
  3. \(\mathcal{L} \leftarrow \frac{1}{N} \sum_t m_{t+1}\, \ell_t\) // (54.1), \(N\) targets
  4. backward: \(\partial \mathcal{L} / \partial \mathbf{z}_t = m_{t+1} (\mathbf{p}_t - \mathbf{e}(x_{t+1})) / N\)
  5. \(\theta \leftarrow \theta - \eta\, \nabla_\theta \mathcal{L}\) // (54.11)
  6. evaluate: loss on all 32 chats; greedy answers to the 8 questions
Figure 6. A real SFT run in your browser: the model of formula (54.10), 120 SGD steps of 4 chats each (15 passes over the 32 chats). Each step has three phases: forward and loss, backward and update, evaluation. The boxes at the bottom show the model’s greedy answer to each of the 8 questions after the current step, given the prompt “<|im_start|> user ↵ question <|im_end|> ↵ <|im_start|> assistant ↵”; decoding stops at <|im_end|> or after 8 tokens. ✓ means the right word, then “.”, then <|im_end|>.
How to read Figure 6

Press Play. With assistant-only loss, the orange curve (answers) falls from 3.30 to 0.154 while the blue curve (user and template tokens) rises to 6.709: the model reads the questions but is never trained to write them, and its guesses there get worse than uniform. The format comes first: after step 1 every answer is already “.” then <|im_end|>; the right words come later, and all 8 answers are correct from step 100. Now choose “All tokens”: at step 30 every “answer” is a new user turn, “<|im_start|> user ↵ capital of ?”, because the model learned to write the whole chat; after 120 steps only 4 of 8 answers are right. Choose “end-of-turn masked”: the words are learned, but no answer ever stops, because <|im_end|> was never a target. Step and Back move one phase at a time; at the two fastest speeds Play jumps one whole step per frame.

The PyTorch version below builds a ChatML-style example with a character-level tokenizer (so it runs with no downloads), packs two examples with a document mask, and checks two facts: packed logits equal separate logits, and dividing by the global token count makes gradient accumulation equal to the full batch.

import torch, torch.nn.functional as F
torch.manual_seed(0)

# 1) a tiny character "tokenizer" with chat special tokens
SPECIAL = ["<|im_start|>", "<|im_end|>", "<|pad|>"]
CHARS = sorted(set("abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.,?!:+=\n"))
VOCAB = SPECIAL + CHARS
ID = {t: i for i, t in enumerate(VOCAB)}
START, END, PAD = ID["<|im_start|>"], ID["<|im_end|>"], ID["<|pad|>"]
enc = lambda s: [ID[c] for c in s]

def render(messages):
    """Chat template -> (ids, mask). mask[t] = 1 if token t is trained."""
    ids, mask = [], []
    for m in messages:
        head = [START] + enc(m["role"] + "\n")          # role header: never trained
        ids += head; mask += [0] * len(head)
        body = enc(m["content"]) + [END]                 # content + end-of-turn
        train = 1 if m["role"] == "assistant" else 0     # loss only on assistant turns
        ids += body; mask += [train] * len(body)         # (the end token is trained too)
        nl = enc("\n"); ids += nl; mask += [0]
    return ids, mask

chat = [{"role": "user", "content": "What is 2+2?"},
        {"role": "assistant", "content": "4."}]
ids, mask = render(chat)
labels = [t if m else -100 for t, m in zip(ids, mask)]   # -100 = ignored by cross_entropy
print(len(ids), sum(mask))                               # 35 tokens, 3 trained

# 2) the masked next-token loss, formula (54.1): position t predicts token t+1
def sft_loss(logits, labels, reduction="mean"):
    lg = logits[:, :-1].reshape(-1, logits.size(-1))     # predictions for positions 0..T-2
    lb = labels[:, 1:].reshape(-1)                       # targets are the NEXT tokens
    return F.cross_entropy(lg, lb, ignore_index=-100, reduction=reduction)

# 3) a tiny causal LM that accepts position ids and an explicit attention mask
class TinyLM(torch.nn.Module):
    def __init__(s, V, d=32, T=64):
        super().__init__()
        s.tok, s.pos = torch.nn.Embedding(V, d), torch.nn.Embedding(T, d)
        s.qkv, s.out, s.head = torch.nn.Linear(d, 3 * d), torch.nn.Linear(d, d), torch.nn.Linear(d, V)
    def forward(s, ids, pos, allowed):                   # allowed: (B, T, T) bool
        x = s.tok(ids) + s.pos(pos)
        q, k, v = s.qkv(x).chunk(3, dim=-1)
        x = x + s.out(F.scaled_dot_product_attention(q, k, v, attn_mask=allowed))
        return s.head(x)

model = TinyLM(len(VOCAB))
causal = lambda n: torch.tril(torch.ones(n, n, dtype=torch.bool))

# 4) packing with a document mask gives exactly the same logits as separate rows
chat2 = [{"role": "user", "content": "Say hi."}, {"role": "assistant", "content": "Hi there!"}]
a, ma = render(chat); b, mb = render(chat2)
la = [t if m else -100 for t, m in zip(a, ma)]; lb_ = [t if m else -100 for t, m in zip(b, mb)]
packed = torch.tensor([a + b])
pos = torch.tensor([list(range(len(a))) + list(range(len(b)))])   # positions restart at 0, (54.6)
doc = torch.tensor([0] * len(a) + [1] * len(b))
allowed = (doc[:, None] == doc[None, :]) & causal(len(doc))        # block-diagonal causal, (54.5)
plabels = torch.tensor([la + [-100] + lb_[1:]])  # never predict sample 2's first token from sample 1
with torch.no_grad():
    lp = model(packed, pos, allowed[None])
    l1 = model(torch.tensor([a]), torch.arange(len(a))[None], causal(len(a))[None])
    l2 = model(torch.tensor([b]), torch.arange(len(b))[None], causal(len(b))[None])
print((lp[0, :len(a)] - l1[0]).abs().max().item(), (lp[0, len(a):] - l2[0]).abs().max().item())  # 0.0 0.0
s_packed = sft_loss(lp, plabels, "sum").item()
s_sep = sft_loss(l1, torch.tensor([la]), "sum").item() + sft_loss(l2, torch.tensor([lb_]), "sum").item()
print(round(s_packed, 4), round(s_sep, 4))                         # the same summed loss

# 5) gradient accumulation: divide by the GLOBAL number of trained tokens, (54.4)
def grads(loss):
    model.zero_grad(); loss.backward()
    return torch.cat([p.grad.flatten() for p in model.parameters()]).clone()
micro = [(torch.tensor([a]), torch.tensor([la]), len(a)), (torch.tensor([b]), torch.tensor([lb_]), len(b))]
n_total = sum((l[:, 1:] != -100).sum().item() for _, l, _ in micro)
def fwd(x, n): return model(x, torch.arange(n)[None], causal(n)[None])
g_full  = grads(sum(sft_loss(fwd(x, n), l, "sum") for x, l, n in micro) / n_total)
g_right = sum(grads(sft_loss(fwd(x, n), l, "sum") / n_total) for x, l, n in micro)
g_wrong = sum(grads(sft_loss(fwd(x, n), l, "mean") / len(micro)) for x, l, n in micro)
print(n_total, (g_full - g_right).abs().max().item() < 1e-6, (g_full - g_wrong).abs().max().item() > 1e-3)
# 13 True True: the right way matches the full batch, the mean of means does not

Libraries do the same work for you: Hugging Face TRL’s SFTTrainer has options for assistant-only loss and for packing, and its collators build the labels with -100. [Likely] Whatever library you use, print one rendered example with its labels before you train.

Where this is used

SettingWhat SFT data looks likeDetails that matter
General chat assistants (Llama 2 chat, Tülu 3)Tens of thousands to about a million multi-turn chats, many sourcesAssistant-only loss, packing, 2 epochs, low learning rate; the starting point for DPO and RL (session 68)
Small, curated alignment (LIMA style)About 1,000 excellent examplesMany epochs, checkpoint chosen by reading outputs, not by validation loss
Domain assistants (legal, medical, internal support)A few thousand expert answers plus general chat dataMix in general data against forgetting (session 61); often LoRA (session 57)
Reasoning modelsLong chain-of-thought answers, often distilled from a stronger model (session 120)Very long rows (tens of thousands of tokens), packing with varlen attention, answers must not be cut
Tool use and agentsChats with tool calls and tool resultsTrain on the assistant’s tool calls, mask the tool results (the model does not write them)
VLM instruction tuningImage or video plus question, answer in textLabels of image and video placeholder tokens are −100; only answer text is trained (session 189)
Video-LLMs: video QA, key-moment detection, dense captionsClips with questions, timestamps and captionsVery uneven lengths (hundreds to thousands of visual tokens), so packing and document masks save the most here; answers with timestamps must be exact (session 205)
Video retrieval and classification heads on top of an LLMLabels written as short text answersAnswers are 1 to 3 tokens long: a sequence mean gives them huge weight compared with long captions in the same mix

Common mistakes

  • No loss mask. Training on user and system tokens. The model learns to write questions and may continue with a fake user turn.
  • Masking the end-of-turn token. The model never learns to stop. Check that the last label of each assistant turn is the end token.
  • Using a different chat template in training than in inference, or a different one from the base model’s own. Print the rendered text and the token ids (session 9).
  • Packing without a document mask or without resetting position ids. Examples read each other, and positions are wrong for every example after the first.
  • Mean of micro-batch means. Gradient accumulation or data parallelism with per-micro-batch averaging changes the example weights (54.4). Divide the summed loss by the global number of target tokens.
  • Silent truncation. Cutting long examples at the maximum length removes the end of the answer and its end token. Count how many examples are too long, and drop or split them.
  • A pretraining-size learning rate or too many epochs on a small set: forgetting and memorized answers. Start around \(10^{-5}\) or lower for full fine-tuning and 1 to 3 epochs, then tune. [Likely]
  • Choosing the checkpoint by validation loss alone. Use generation-based evaluation; and decontaminate the training set against your benchmarks.

Interview questions

What is SFT, and how is its loss different from pretraining?basic
SFT trains a pretrained model on example conversations (prompt and ideal answer) with the same next-token cross-entropy as pretraining. The difference is the loss mask: only the tokens the assistant should produce, including its end-of-turn token, get a gradient; system, user and role-header tokens are read as context but have label −100. The data is also much smaller and cleaner, the learning rate is much lower, and training lasts a few epochs.
Why do we mask the prompt tokens? Is it always better?core
The goal is to model \(p(\text{answer} \mid \text{prompt})\), not the prompt itself. Training on the prompt teaches the model to write user-style text (it may start writing the next user turn), lets long prompts dominate the gradient, and fits lower-quality prompt text. It is not always better: Shi et al. (2024) found that also training on instructions can help when outputs are short or the dataset is small, because it reduces overfitting. Assistant-only loss is the default; prompt loss is a tunable choice.
Why must the end-of-turn token be in the loss?basic
Generation stops when the model emits the end-of-turn (or end-of-sequence) token. The model only learns to emit it at the right place if it is trained as a target after each answer. If it is masked, or cut by truncation, the model keeps generating past the end of the answer, often inventing a new user turn. Also make sure the inference server stops on that exact token.
Per-token mean, per-sequence mean or sum: what is the difference?core
The per-token losses are the same; the weights differ (54.3). Token mean gives every target token weight \(1/\sum n_j\), so long answers dominate. Sequence mean gives each example equal total weight, so a token of a short answer can count many times more than a token of a long one (9 times for 10 versus 90 tokens). Sum weights every token equally and does not normalize, so the gradient scale grows with the batch's token count and the learning rate must account for it. Tülu 3 found the sum loss better for its mixtures. The choice also changes the reported loss, so loss curves from different settings are not comparable.
Gradient accumulation gives a different result from one big batch. Why, and how do you fix it?advanced
If each micro-batch computes a token mean over its own tokens and the micro-batch losses are then averaged, micro-batches with few target tokens get more weight per token: a mean of means is not the global mean unless all micro-batches have the same token count. The fix: compute the summed loss per micro-batch and divide by the total number of target tokens in the whole optimizer step, across micro-batches and across data-parallel ranks (an all-reduce of the count). Hugging Face fixed exactly this in its trainer in October 2024.
Explain packing for SFT. What three things must be right?core
Packing places several whole examples into one row of length \(T\) (bin-packing, for example first-fit decreasing) to avoid padding; utilization can go from about 36% to about 97% on realistic lengths. To make a packed row equal to separate rows: (1) a block-diagonal causal document mask so examples cannot attend to each other (or a varlen kernel with cu_seqlens); (2) position ids that restart at 0 for each example; (3) no loss from the last token of one example to the first token of the next (automatic when the first token is a masked role header). Also remember that a batch of packed rows contains many more examples than a batch of padded rows.
What learning rate and how many epochs would you use for full SFT of an 8B model?core
Much lower than pretraining: around \(10^{-5}\) or below. Tülu 3 used \(5 \times 10^{-6}\) for 8B and \(2 \times 10^{-6}\) for 70B, Llama 2 used \(2 \times 10^{-5}\) (15 times below its pretraining peak), with short warmup and linear or cosine decay. One to three epochs is typical (both used 2). Small curated sets can use more epochs (LIMA: 15, checkpoints from epochs 5 to 10). Then sweep: the right value depends on data size, batch size and loss reduction, and results also vary noticeably across random seeds (Tülu 3 reports this).
How do you detect overfitting in SFT, and why is validation loss not enough?core
Signs: training loss keeps falling with drops at each epoch boundary, validation loss rises, generations copy training phrases or lose diversity, and general benchmarks fall below the base model (forgetting). Validation loss is not enough because it measures how well the model predicts the reference answers, not how good its own answers are; LIMA found perplexity did not correlate with generation quality and chose checkpoints by reading outputs. Use generation-based evaluation (judges, task benchmarks) to choose checkpoints.
What did LIMA show, and what are its limits?advanced
LIMA fine-tuned LLaMA 65B on 1,000 curated examples, with no RLHF, and its answers were equal or preferred to GPT-4 in 43% of human comparisons. Ablations showed diversity and quality mattered more than quantity. This supports the superficial alignment hypothesis: pretraining provides the knowledge, SFT mainly teaches format and style. Limits: the evaluation was on general prompts with a strong 65B base model; skills such as math, code, tool use and safety need targeted and larger data, which is why recipes like Tülu 3 use about a million examples from many sources.
You have 2 million raw SFT examples and budget for 100k. How do you choose?advanced
First clean: exact and near-duplicate removal, decontamination against evaluation benchmarks, filters for broken format, wrong language, and refusals you do not want. Then score quality (an LLM judge, reward model, or checks such as running code and verifying math answers). Then ensure coverage: embed the examples, cluster them by task, and choose per-cluster quotas (or a diversity method such as k-center) so rare but important skills are kept; add targeted data for missing skills. Finally, run small ablations: train on the subset and on variants, and evaluate on the skills you care about. AlpaGasus (9k of 52k) and LIMA show that a smaller clean set can beat the full set.
In a VLM or Video-LLM, which tokens get a loss during instruction tuning?core
Only the answer text tokens (and the end-of-turn token). The image or video placeholder positions, which are replaced by hundreds or thousands of visual tokens, get label −100, as do the question and template tokens. Because visual token counts vary a lot between images, clips and frame rates, packing with document masks saves a lot of compute, and the loss averaging choice matters: short answers (a class name, a timestamp) mixed with long captions get very different weights under token mean versus sequence mean.

Exercises

  1. A chat has a 40-token system prompt, a 200-token user turn and a 60-token answer, plus 5 template tokens per turn (headers, end tokens, newlines; 15 in total). What fraction of tokens are targets with assistant-only loss, if the answer’s end-of-turn token is one of its 5 template tokens and is trained?
  2. Batch: example A has 5 target tokens with loss 3.0 each, example B has 95 target tokens with loss 1.0 each. Compute the token mean, the sequence mean and the sum.
  3. Two micro-batches: the first has 100 target tokens with total loss 120, the second 20 target tokens with total loss 40. Compute the correct accumulated loss and the mean of micro-batch means.
  4. Lengths 1500, 1200, 900, 700, 400 and 300 tokens, \(T = 2048\). Pack them with first-fit decreasing. How many rows, and what is the utilization?
  5. You have 50,000 examples, an effective batch of 64 and 3 epochs. How many optimizer steps? With 3% warmup, how many warmup steps?
  6. In a packed row, example 2 starts at position 700 of the row. What position id does the token at row position 745 get?
Answerscheck
1. Total \(40 + 200 + 60 + 15 = 315\) tokens. Targets: the 60 answer tokens plus its end token, 61. Fraction \(61/315 = 19.4\%\). 2. Token mean \((15 + 95)/100 = 1.10\). Sequence mean \((3.0 + 1.0)/2 = 2.00\). Sum \(110\). 3. Correct: \((120 + 40)/120 = 1.333\). Mean of means: \((1.2 + 2.0)/2 = 1.60\); the small micro-batch gets far too much weight. 4. Sorted: 1500 opens row 1 (548 left), 1200 opens row 2 (848 left), 900 opens row 3 (1148 left), 700 goes to row 2 (148 left), 400 goes to row 1 (148 left), 300 goes to row 3 (848 left). Three rows with 1900, 1900 and 1200 tokens; \(U = 5000/6144 = 81.4\%\). 5. \(\lceil 50000/64 \rceil = 782\) steps per epoch (781.25 rounded up), so 2,346 steps; 3% is about 70 warmup steps. 6. \(745 - 700 = 45\), by formula (54.6).

Key takeaways

  • SFT is next-token training on conversations with a loss mask (54.1): only assistant tokens and their end-of-turn token are targets; everything else is context with label −100.
  • Token mean, sequence mean and sum (54.2) weight examples differently (54.3). With gradient accumulation or many GPUs, divide the summed loss by the global target-token count (54.4), or the weights change silently.
  • Packing fills rows with whole examples; it needs a block-diagonal document mask (54.5), position ids that restart (54.6) and no loss across boundaries. Utilization (54.7) often goes from under 50% to over 90%.
  • SFT is a short, gentle run: learning rate around \(10^{-5}\) or lower for full fine-tuning, 1 to 3 epochs, batches of 32 to 128 examples, length long enough that answers are not cut (54.8).
  • Watch for overfitting, but pick checkpoints by generation quality, not validation loss alone.
  • Quality first, then coverage: LIMA (1,000 examples), Llama 2 (27,540) and AlpaGasus (9k of 52k) show that clean data wins; Tülu 3 shows that skills need targeted, diverse sources.

Key formulas

\[ \mathcal{L} = -\frac{1}{Z}\sum_{i,t} m_{i,t} \log p_\theta(x_{i,t} \mid x_{i,\lt t}), \qquad \mathcal{L}_{\text{accum}} = \frac{1}{N} \sum_{g} \sum_{(i,t) \in g} m_{i,t}\,\ell_{i,t} \]
\[ M_{ij} = 0 \ \text{iff}\ d(i) = d(j),\ j \le i; \qquad p_i = i - s_{d(i)}; \qquad U = \frac{\sum_i \min(L_i, T)}{R\,T}; \qquad S = \left\lceil \frac{N_{\text{ex}}}{B} \right\rceil E \]

References for this session

  1. Zhou et al. (2023). LIMA: Less Is More for Alignment
  2. Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
  3. Raschka, Build a Large Language Model (From Scratch), chapter 7: Fine-tuning to follow instructions
  4. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 10: Alignment
  5. Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
  6. Jain et al. (2023). NEFTune: Noisy Embeddings Improve Instruction Finetuning
  7. Chen et al. (2023). AlpaGasus: Training A Better Alpaca with Fewer Data
  8. Shi et al. (2024). Instruction Tuning With Loss Over Instructions
  9. Hugging Face (2024). Fixing Gradient Accumulation
  10. Wang et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions