DPO on a small model
In this lab you align a small chat model with DPO and check whether it really got better. You build and clean a preference dataset, train with TRL’s DPOTrainer and with a 64-line DPO loop you can run on a CPU, tune β, judge the outputs with a position-swapped LLM judge, test the result for length bias, and compare it with plain supervised fine-tuning on the chosen answers.
What you will learn
- How a preference dataset is laid out (prompt, chosen, rejected), and how to clean it: drop ties, check score gaps, measure the length difference between chosen and rejected
- How to compute the DPO loss by hand from summed completion log-probabilities, and what TRL’s logs (rewards/chosen, rewards/rejected, margins, accuracies) mean
- Why both the chosen and the rejected log-probabilities usually go down during DPO, and why SFT on the chosen answers does not separate them
- How to train a model with TRL’s DPOTrainer and SFTTrainer, and what the reference model costs in memory and compute
- What β does, how to sweep it, and why the β with the best judge score is not always the β with the best true quality
- How to measure a win rate with a position-swapped LLM judge, give it an error bar, and correct it for length with a simple length-controlled estimate
- How to compare DPO with SFT only and write down which one you would ship
Why this matters
You have a small chat model that already follows instructions. You also have a set of preference pairs: for each prompt, one answer that people (or a strong model) liked more, the chosen answer, and one they liked less, the rejected answer. In this lab you use DPO to make the model prefer answers like the chosen ones. Then you check, with numbers, whether the model really got better, or whether it only learned to write longer answers.
Session 68 derived the DPO loss: it turns the RLHF objective into one classification-style loss on pairs, with no reward model and no sampling during training. Session 75 showed how heavy the online RL loop is. This lab is the practical side of DPO. Most of the work is not the loss, which is ten lines of code. The work is the data, the choice of \(\beta\), and an honest evaluation.
You will:
- Build and clean a preference dataset, and measure how often the chosen answer is simply the longer one (Figure 1).
- Compute the DPO loss and its logs by hand, then run a 64-line DPO loop on a CPU with a 135M model and read real training curves (Figure 2).
- Train
Qwen2.5-0.5B-Instructwith TRL’sDPOTrainer, and withSFTTraineron the chosen answers only, as the baseline. - Tune \(\beta\) and see what it changes, first in a toy you can train in your browser (Figures 3 and 4).
- Measure win rates with a position-swapped LLM judge, and correct them for length (Figure 5).
- Fill one results table and decide: DPO, SFT only, or both.
DPO and its relatives are used in real post-training pipelines. Zephyr-7B was trained with SFT and then DPO on UltraFeedback, the same dataset as in this lab [Certain]. Tülu 3 uses a length-normalized DPO stage after SFT [Likely]. TRL is the most common open-source library for it, and its documentation uses the same dataset, trl-lib/ultrafeedback_binarized, in its examples [Certain].
Setup
Hardware. Steps 1 to 3 and the toy figures need only a CPU. Step 4 trains a 0.5B model; one GPU with 16 to 24 GB is enough with LoRA, and a 24 GB card is enough for full DPO with a short context [Likely]. Step 6 runs a 7B judge, which needs about 15 GB in BF16 [Likely]; you can also use an API model as the judge.
pip install "transformers>=4.56" trl peft datasets accelerate pyarrow
The TRL code on this page follows the current TRL documentation. TRL changes often (argument names, defaults, logged metrics), so check the documentation of your installed version if an argument fails. We ran the from-scratch loop of Step 3 on a CPU (PyTorch 2.8, transformers 4.49) and report its real output. We did not run the GPU scripts of Steps 4 to 6 for you, and this page does not report their results: you will measure them.
Models.
| Role | Model | Why |
|---|---|---|
| CPU run (Step 3) | HuggingFaceTB/SmolLM2-135M-Instruct: 30 layers, hidden size 576, 134,515,008 parameters, tied embeddings | Small enough to train on a laptop CPU: each of our 32-step runs took about 42 minutes, with two runs sharing the machine |
| Main run (Step 4) | Qwen/Qwen2.5-0.5B-Instruct: 494,032,768 parameters | Already instruction-tuned, so it is a natural SFT starting point and reference model |
| Judge (Step 6) | Qwen/Qwen2.5-7B-Instruct or any stronger model | Must not be one of the contestants |
We counted the SmolLM2 parameters from its config.json: \(49{,}152 \times 576\) embedding weights plus 30 blocks of 3,540,096 weights plus the final norm.
Data. trl-lib/ultrafeedback_binarized, a cleaned version of UltraFeedback. UltraFeedback has prompts with four answers from different models, each scored by GPT-4 [Certain]. The “binarized” version keeps two answers per prompt: the best-scored one as chosen and another one as rejected [Likely]. It has 62,135 training rows and 1,000 test rows, with columns chosen, rejected (each a full conversation), score_chosen and score_rejected. We counted these from the dataset files.
The table you will fill. Copy it now. Every empty cell is a number you measure.
| Model | Train data | Pair accuracy (held out) | Mean answer length (words) | Win rate vs start | Length-controlled win rate | KL to start |
|---|---|---|---|---|---|---|
| A: start (Qwen2.5-0.5B-Instruct) | none | 0.5 (by definition) | 0.5 (itself) | 0.5 | 0 | |
| B: SFT on chosen | chosen answers | |||||
| C: DPO from A, best \(\beta\) | pairs | |||||
| D: SFT on chosen, then DPO | both |
Step 1: build a preference dataset
A preference example has three parts: a prompt \(x\), a chosen answer \(y_w\) (w for “winner”) and a rejected answer \(y_l\) (l for “loser”). TRL accepts this as three columns. The recommended form keeps the prompt separate, so the trainer knows exactly which tokens belong to the answers:
{"prompt": [{"role": "user", "content": "What color is the sky?"}],
"chosen": [{"role": "assistant", "content": "It is blue."}],
"rejected": [{"role": "assistant", "content": "It is green."}]}
ultrafeedback_binarized stores the prompt inside both conversations instead (an “implicit prompt”). TRL handles both forms. In our own loop we take the prompt as the shared part of the two conversations.
Clean the pairs before you train
A pair teaches the model one thing: “move probability from \(y_l\) to \(y_w\)”. If the pair is noise, the model learns noise. Three checks catch most problems:
- Ties. If both answers have the same score, the label is arbitrary. In
ultrafeedback_binarized, 7,554 of the 62,135 pairs (12.2%) have equal scores. Drop them. - Small score gaps. A gap of 0.5 between two judge scores is close to the judge’s own noise. A minimum gap keeps cleaner pairs but fewer of them.
- Length. If the chosen answer is usually the longer one, DPO can learn “longer is better” instead of “better is better”. Measure it.
trl-lib/ultrafeedback_binarized, computed by us with Python. Length is counted in words (split on spaces). “Max length ratio” keeps only pairs where the longer answer has at most that many times the words of the shorter one.How to read Figure 1
With no filter, the chosen answer is longer in 53.5% of pairs, and its median length is 158 words against 130 for the rejected one. That is a mild length bias in the data. Raise the minimum gap to 2: 23,832 pairs remain, the chosen answer is still longer in 55.4% of them, and both answers are shorter: the medians drop to 107 and 83 words. Now set the length ratio to 1.2: the chosen answer is longer in only 45.6% of the 13,305 pairs. So most of the length bias sits in pairs where the two answers have very different lengths. Removing them is one simple way to train a model that does not learn length.
A filter is a trade: fewer pairs, but cleaner ones. For this lab, drop ties and keep everything else; then, as an extra run, train on the length-ratio-filtered set and compare the answer lengths (Step 7).
Making your own pairs (on-policy data)
UltraFeedback answers were written by other models, not by yours. Such data is off-policy: your model may never write answers like them (session 71). A common fix is to sample answers from your own model and let a judge or a reward model rank them. Zephyr and many later recipes use answers from other models; on-policy pairs are used, for example, in iterative DPO [Likely]. Algorithm 1 is the simplest version.
- input: prompts \(\{x\}\), your SFT model \(\pi_\text{SFT}\), a judge or reward model \(J\), \(K = 4\) samples, minimum gap \(g\)
- for each prompt \(x\):
- sample \(y_1, \dots, y_K \sim \pi_\text{SFT}(\cdot \mid x)\) at temperature 0.8 // diversity matters: greedy gives K copies
- score each: \(s_k = J(x, y_k)\)
- \(w \leftarrow \argmax_k s_k\), \(l \leftarrow \argmin_k s_k\)
- if \(s_w - s_l \geq g\): keep \((x, y_w, y_l)\) else: skip // near-ties are noise
- report: pairs kept, share where \(y_w\) is longer, median lengths
# make_pairs.py: score 4 own samples per prompt with a judge, keep best vs worst import json, torch from transformers import AutoTokenizer, AutoModelForCausalLM POLICY, JUDGE = "Qwen/Qwen2.5-0.5B-Instruct", "Qwen/Qwen2.5-7B-Instruct" tok = AutoTokenizer.from_pretrained(POLICY); tok.padding_side = "left" pol = AutoModelForCausalLM.from_pretrained(POLICY, dtype=torch.bfloat16, device_map="auto") jt = AutoTokenizer.from_pretrained(JUDGE) jm = AutoModelForCausalLM.from_pretrained(JUDGE, dtype=torch.bfloat16, device_map="auto") def score(prompt, answer): # 1..10, greedy, last number in the reply msg = [{"role": "system", "content": "Rate the answer from 1 to 10 for helpfulness and " "correctness. Length is not a quality. Reply with one number on the last line."}, {"role": "user", "content": f"Question:\n{prompt}\n\nAnswer:\n{answer}"}] enc = jt.apply_chat_template(msg, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(jm.device) out = jm.generate(**enc, max_new_tokens=8, do_sample=False) txt = jt.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True) nums = [int(t) for t in txt.replace(".", " ").split() if t.isdigit()] return nums[-1] if nums else 0 torch.manual_seed(0) prompts = [json.loads(l)["prompt"] for l in open("prompts.jsonl")] pairs = [] for p in prompts: enc = tok.apply_chat_template([{"role": "user", "content": p}], add_generation_prompt=True, return_tensors="pt", return_dict=True).to(pol.device) out = pol.generate(**enc, do_sample=True, temperature=0.8, top_p=0.95, max_new_tokens=400, num_return_sequences=4) ans = tok.batch_decode(out[:, enc["input_ids"].shape[1]:], skip_special_tokens=True) s = [score(p, a) for a in ans] w, l = max(range(4), key=s.__getitem__), min(range(4), key=s.__getitem__) if s[w] - s[l] >= 2: pairs.append({"prompt": [{"role": "user", "content": p}], "chosen": [{"role": "assistant", "content": ans[w]}], "rejected": [{"role": "assistant", "content": ans[l]}]}) longer = sum(len(x["chosen"][0]["content"].split()) > len(x["rejected"][0]["content"].split()) for x in pairs) / max(1, len(pairs)) print(len(pairs), "pairs kept; chosen longer in", round(longer, 3)) with open("own_pairs.jsonl", "w") as f: for x in pairs: f.write(json.dumps(x) + "\n")
Answers that hit max_new_tokens are cut off. A judge usually scores a cut answer low, so it ends up as the rejected answer, and DPO learns to avoid the long, unfinished style. That can be what you want, but check it: count how many rejected answers were cut.
Step 2: the loss and the logs, by hand
Before running a trainer, compute what it will compute. For one pair, DPO needs four numbers: the log-probability of the chosen and of the rejected answer, under the model being trained \(\pi_\theta\) and under a frozen copy of the starting model, the reference \(\pi_\text{ref}\). The loss is the one from session 68:
Here \(\hat r_w\) and \(\hat r_l\) are the implicit rewards: how much more likely the model now makes an answer than the reference did, times \(\beta\):
- Read the symbols
- \(x, y_w, y_l\)
- the prompt, the chosen answer and the rejected answer
- \(\pi_\theta\)
- the model you train (the policy)
- \(\pi_\text{ref}\)
- the frozen starting model; usually the SFT model you start from
- \(\beta\)
- a positive number (often 0.1) that sets how strongly the model is held near the reference
- \(\hat r_w, \hat r_l\)
- the implicit rewards; TRL logs their batch means as
rewards/chosenandrewards/rejected
The log-probability of a whole answer is the sum over its tokens. Only the answer tokens count, not the prompt tokens:
TRL logs three more numbers per batch. The margin is \(m = \hat r_w - \hat r_l\) (rewards/margins). The reward accuracy is the share of pairs with \(m \gt 0\) (rewards/accuracies). And the gradient of one pair is a weighted push up on \(y_w\) and down on \(y_l\):
The weight \(\sigma(-m)\) is large when the model still ranks the pair wrongly (\(m \lt 0\)) and small when it already ranks it right by a wide margin. Pairs the model has “solved” stop pushing.
At the start, \(\pi_\theta = \pi_\text{ref}\), so \(\hat r_w = \hat r_l = 0\), \(m = 0\) and the loss is \(-\log \sigma(0) = \log 2 = 0.693\). Every DPO run starts at 0.693; if yours does not, the reference is not the starting model.
After some training, the chosen answer has gone from \(-150\) to \(-151\) nats and the rejected one from \(-140\) to \(-145\) nats. Then \(\hat r_w = 0.1 \times (-1) = -0.1\) and \(\hat r_l = 0.1 \times (-5) = -0.5\). The margin is \(m = 0.4\), the loss is \(-\log \sigma(0.4) = 0.513\), and the gradient weight is \(\sigma(-0.4) = 0.401\). Note that the chosen answer became less likely, yet the loss went down and the pair counts as “accurate”. DPO only cares about the gap.
Step 3: DPO from scratch on a CPU
Here is a complete DPO loop without TRL. It uses 256 training pairs and 64 held-out pairs from ultrafeedback_binarized (ties dropped, prompt plus answer at most 512 tokens), batches of 8 pairs, AdamW and one pass over the data: 32 steps. The same script also runs “SFT on chosen” with the same data, so you can compare the two. We ran it on a CPU, and the curves in Figure 2 are its real logs. (Our run also generated answers to measure their length; the version below leaves that part out and prints a shorter log line. We checked that this shorter version runs too.)
- input: a batch of pairs \((x, y_w, y_l)\), policy \(\pi_\theta\), frozen reference \(\pi_\text{ref}\), \(\beta\)
- build \(2B\) sequences: \(x \Vert y_w\) and \(x \Vert y_l\); mask = 1 on answer tokens only
- \(\ell^\theta_w, \ell^\theta_l \leftarrow\) summed masked log-probs under \(\pi_\theta\) (formula 76.3)
- \(\ell^\text{ref}_w, \ell^\text{ref}_l \leftarrow\) the same under \(\pi_\text{ref}\), with no gradient // can be precomputed once
- \(\hat r_w \leftarrow \beta(\ell^\theta_w - \ell^\text{ref}_w)\), \(\hat r_l \leftarrow \beta(\ell^\theta_l - \ell^\text{ref}_l)\)
- loss \(\leftarrow\) mean of \(-\log\sigma(\hat r_w - \hat r_l)\); backward; clip the gradient norm to 1; optimizer step
- log: loss, mean \(\hat r_w\), mean \(\hat r_l\), accuracy, mean \(\ell^\theta_w\), mean \(\ell^\theta_l\)
# dpo_loop.py: DPO (or SFT on chosen) from scratch. Usage: python dpo_loop.py dpo 0.1 2e-5 import sys, json, copy, torch, torch.nn.functional as F import pyarrow.parquet as pq from transformers import AutoModelForCausalLM, AutoTokenizer torch.manual_seed(0) MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct" MODE, BETA, LR = sys.argv[1], float(sys.argv[2]), float(sys.argv[3]) # "dpo" or "sft" N_TRAIN, N_EVAL, BS, MAXLEN = 256, 64, 8, 512 tok = AutoTokenizer.from_pretrained(MODEL) policy = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.float32) ref = copy.deepcopy(policy).eval().requires_grad_(False) # frozen reference # the train split of trl-lib/ultrafeedback_binarized, downloaded as a parquet file rows = pq.read_table("data/train.parquet").slice(0, 4000).to_pylist() def encode(r): # prompt = the shared part of the two conversations p = tok.apply_chat_template(r["chosen"][:-1], add_generation_prompt=True) c = tok.apply_chat_template(r["chosen"]); j = tok.apply_chat_template(r["rejected"]) assert c[:len(p)] == p and j[:len(p)] == p return p, c[len(p):], j[len(p):] pairs = [] for r in rows: if r["score_chosen"] == r["score_rejected"]: continue # drop ties p, c, j = encode(r) if len(p) + max(len(c), len(j)) <= MAXLEN: pairs.append((p, c, j)) if len(pairs) == N_TRAIN + N_EVAL: break train, evals = pairs[:N_TRAIN], pairs[N_TRAIN:] print("chosen longer (tokens):", sum(len(c) > len(j) for _, c, j in pairs) / len(pairs)) def seq_logps(model, prompts, resps): # formula (76.3), answer tokens only seqs = [p + r for p, r in zip(prompts, resps)]; L = max(map(len, seqs)) ids = torch.full((len(seqs), L), tok.pad_token_id) mask = torch.zeros(len(seqs), L, dtype=torch.long); lab = torch.zeros(len(seqs), L, dtype=torch.bool) for i, (p, s) in enumerate(zip(prompts, seqs)): ids[i, :len(s)] = torch.tensor(s); mask[i, :len(s)] = 1; lab[i, len(p):len(s)] = True logits = model(input_ids=ids, attention_mask=mask).logits[:, :-1] lp = torch.log_softmax(logits.float(), -1).gather(-1, ids[:, 1:, None]).squeeze(-1) return (lp * lab[:, 1:]).sum(-1) # the label of position t is token t+1 def batch_logps(model, b): # chosen and rejected in one forward pass P = [x[0] for x in b]; lp = seq_logps(model, P + P, [x[1] for x in b] + [x[2] for x in b]) return lp[:len(b)], lp[len(b):] opt = torch.optim.AdamW(policy.parameters(), lr=LR, weight_decay=0.0) for step in range(N_TRAIN // BS): b = train[step * BS:(step + 1) * BS] pc, pr = batch_logps(policy, b) with torch.no_grad(): rc, rr = batch_logps(ref, b) rw_c, rw_r = BETA * (pc - rc), BETA * (pr - rr) # formula (76.2) if MODE == "dpo": loss = -F.logsigmoid(rw_c - rw_r).mean() # formula (76.1) else: # SFT on chosen, formula (76.5) loss = -pc.sum() / sum(len(x[1]) for x in b) opt.zero_grad(); loss.backward() torch.nn.utils.clip_grad_norm_(policy.parameters(), 1.0); opt.step() print(json.dumps({"step": step + 1, "loss": round(loss.item(), 4), "rewards/chosen": round(rw_c.mean().item(), 4), "rewards/rejected": round(rw_r.mean().item(), 4), "acc": (rw_c > rw_r).float().mean().item(), "logps/chosen": round(pc.mean().item(), 2)})) with torch.no_grad(): # held-out pair accuracy acc = 0 for i in range(0, len(evals), BS): b = evals[i:i + BS]; pc, pr = batch_logps(policy, b); rc, rr = batch_logps(ref, b) acc += ((pc - rc) > (pr - rr)).sum().item() print("held-out pair accuracy:", acc / len(evals))
Our script also generated greedy answers (at most 300 new tokens) for 32 other prompts after training, to measure answer length. We ran three settings: DPO with learning rate \(5 \times 10^{-6}\), DPO with \(2 \times 10^{-5}\), and SFT on chosen with \(2 \times 10^{-5}\), all with \(\beta = 0.1\) for the logged rewards.
dpo_loop.py on a CPU: SmolLM2-135M-Instruct, 256 UltraFeedback pairs, 32 steps of 8 pairs. Thin lines are the per-step values, thick lines a moving average over 6 steps. Each step sees new pairs, so these are noisy “first look” numbers, like a real training log.How to read Figure 2
Start with “DPO, LR 2e-5” and “Implicit rewards”. Both curves go below zero: the policy makes the chosen answers and the rejected answers less likely than the reference does. The rejected curve falls faster. The mean of the last 8 steps is \(-0.263\) for chosen and \(-0.420\) for rejected, so the margin is positive. Switch to “Loss and accuracy”: the loss falls only a little, from 0.696 to 0.665 (mean of the first and last 8 steps), and the batch accuracy goes from 0.531 to 0.578. With 32 steps this is the very start of training. Now pick “DPO, LR 5e-6”: the same shape, about five times smaller (rewards \(-0.049\) and \(-0.094\)). Then pick “SFT on chosen”: both rewards go up, and the rejected one goes up more (0.412 and 0.503). The rejected answers are similar text, so imitating the chosen answers also makes them more likely. The “Log-probs” view shows the raw summed log-probabilities. Each step uses a different batch with different answer lengths, so these jump around and you cannot read a trend from them. That is why TRL logs the implicit rewards: they compare each answer with itself under the reference.
The held-out pair accuracies on the 64 unseen pairs: DPO with LR \(2 \times 10^{-5}\) reaches 0.609, DPO with \(5 \times 10^{-6}\) 0.578, and SFT on chosen 0.422, below chance. So in this run, SFT on chosen ranked the rejected answer above the chosen one more often than not. The answer lengths hardly moved. The start model wrote 214.2 tokens on average on the 32 generation prompts, and 17 of its 32 answers hit the 300-token limit. After training the means were 205.6 (DPO, LR \(2 \times 10^{-5}\)), 209.4 (DPO, LR \(5 \times 10^{-6}\)) and 198.6 tokens (SFT). 32 steps are too few to change the style of a model. The length effects of Step 7 come from full runs.
The held-out pair accuracy is measured on only 64 pairs, so its standard error is about \(\sqrt{0.5 \times 0.5 / 64} = 0.063\). With so few pairs, differences of a few points are noise. The 0.609 of DPO is only about 1.7 standard errors above 0.5, and the 0.031 gap between the two DPO learning rates is pure noise. The gap between DPO and SFT (0.609 against 0.422) is about 2 standard errors of a difference, so it is a hint, not proof. This loop is for understanding and debugging. The real run is next.
Step 4: train with TRL
Now the main experiment, on a GPU. You train two models from Qwen2.5-0.5B-Instruct (model A in the table): B, SFT on the chosen answers, and C, DPO on the pairs. As an extra, D is DPO started from B, with B as the reference. All runs use the same 10,000 training pairs (ties dropped) and the same seed. The SFT baseline is what the DPO paper calls “Preferred-FT”: plain fine-tuning on the chosen answers [Certain].
This is next-token cross-entropy on the chosen answers, averaged over their tokens. It never sees the rejected answers. So it raises the probability of the chosen answers, but nothing pushes the rejected ones down. Figure 2 already showed what that means for the pair accuracy.
# data.py: the shared data from datasets import load_dataset def load_pairs(n_train=10_000, seed=0): ds = load_dataset("trl-lib/ultrafeedback_binarized") ds = ds.filter(lambda r: r["score_chosen"] > r["score_rejected"]) # drop ties train = ds["train"].shuffle(seed=seed).select(range(n_train)) return train, ds["test"]
# sft.py: model B, SFT on the chosen answers only from transformers import AutoTokenizer from trl import SFTConfig, SFTTrainer from data import load_pairs MODEL = "Qwen/Qwen2.5-0.5B-Instruct" train, _ = load_pairs() train = train.map(lambda r: {"messages": r["chosen"]}, remove_columns=train.column_names) cfg = SFTConfig(output_dir="out/B-sft", learning_rate=1e-5, num_train_epochs=1, per_device_train_batch_size=8, gradient_accumulation_steps=4, max_length=1024, assistant_only_loss=True, # loss on the answer, not the prompt lr_scheduler_type="cosine", warmup_steps=10, bf16=True, logging_steps=10, save_strategy="no", report_to="none", seed=0) tr = SFTTrainer(model=MODEL, args=cfg, train_dataset=train, processing_class=AutoTokenizer.from_pretrained(MODEL)) tr.train(); tr.save_model("out/B-sft/final")
assistant_only_loss=True needs a chat template that marks the assistant part; if your TRL version refuses it for this template, use the prompt-completion format of session 54 instead [Likely].
# dpo.py: models C and D. Usage: python dpo.py 0.1 Qwen/Qwen2.5-0.5B-Instruct out/C-b0.1 import sys from transformers import AutoTokenizer from trl import DPOConfig, DPOTrainer from data import load_pairs beta, start, out = float(sys.argv[1]), sys.argv[2], sys.argv[3] train, test = load_pairs() tok = AutoTokenizer.from_pretrained(start) cfg = DPOConfig(output_dir=out, beta=beta, loss_type=["sigmoid"], # plain DPO, formula (76.1) learning_rate=1e-6, # TRL default; DPO needs a small LR num_train_epochs=1, per_device_train_batch_size=4, gradient_accumulation_steps=8, max_length=1024, lr_scheduler_type="cosine", warmup_steps=10, bf16=True, logging_steps=10, eval_strategy="steps", eval_steps=50, save_strategy="no", report_to="none", seed=0) tr = DPOTrainer(model=start, # ref_model=None: a frozen copy of the start args=cfg, train_dataset=train, eval_dataset=test.select(range(200)), processing_class=tok) tr.train(); tr.save_model(f"{out}/final")
With ref_model=None, TRL uses the starting model as the reference [Certain]. For model D, pass the SFT checkpoint out/B-sft/final as start; it then becomes both the starting point and the reference. With a LoRA adapter (peft_config=LoraConfig(...)), TRL does not need a second copy: the reference is the base model with the adapter switched off [Likely]. The TRL documentation suggests a higher learning rate for adapters, around \(10^{-5}\) [Certain].
What the reference model costs
DPO runs four sequences’ worth of forward passes per pair: chosen and rejected, under the policy and under the reference. With the FLOP rule of session 25 (about \(2N\) FLOPs per token forward, \(4N\) backward), the cost per pair with answers of \(T_w\) and \(T_l\) tokens, ignoring the shared prompt, is:
Memory adds the frozen reference to the usual budget of session 38: 16 bytes per trained parameter (FP32 weights, gradients and two Adam states) plus 2 bytes per reference parameter in BF16:
- Read the symbols
- \(N\)
- the number of model parameters
- \(P\)
- the number of LoRA parameters (the base is shared by policy and reference)
- \(T_w, T_l\)
- the lengths in tokens of the chosen and the rejected answer
- \(M_\text{act}\)
- activation memory; DPO stores activations for two sequences per pair
\(N = 494{,}032{,}768\). Full DPO: \(16N = 7.90\) GB for the policy and its optimizer, plus \(2N = 0.99\) GB for the reference, so 8.89 GB before activations. LoRA DPO with \(r = 16\) on all linear layers (\(P = 8{,}798{,}208\), as counted in session 62): \(0.99 + 0.14 = 1.13\) GB.
Compute, with \(T_w = T_l = 300\) tokens: DPO costs \(6N \cdot 600 + 2N \cdot 600 = 4{,}800N\) FLOPs per pair, SFT on the chosen answer \(6N \cdot 300 = 1{,}800N\). So one DPO pair costs about 2.7 times one SFT example. With precompute_ref_log_probs=True, TRL computes the reference log-probs once before training and does not keep the reference in memory during training [Certain]; that saves the reference memory, and the compute too if you train for more than one epoch.
Read the logs while it runs. A healthy DPO run on this data usually looks like Figure 2, only smoother: the loss starts at 0.693 and falls slowly, the accuracy rises above 0.5, rewards/rejected falls faster than rewards/chosen, and the margin grows [Likely]. Warning signs: the loss falls to near 0 within a few hundred steps (too high a learning rate, or the data is too easy to separate, for example by length); logps/chosen falls by hundreds of nats (the model is moving probability to answers outside the data); the accuracy stays at 0.5 (a bug, often a wrong mask or a wrong reference).
Step 5: tune β
What does \(\beta\) do? Session 68 showed that DPO fits the same target as KL-regularized RLHF, whose best policy is the reference tilted by the reward:
The data fixes the reward differences (up to noise). So a small \(\beta\) means large log-ratios: the model moves far from the reference. A large \(\beta\) keeps it close. That makes \(\beta\) the main knob for “how much do I trust these pairs”. Noisy or biased pairs call for a larger \(\beta\); clean pairs and a weak starting model allow a smaller one. The DPO paper used \(\beta = 0.1\) for dialogue and \(\beta = 0.5\) for summarization [Certain]; 0.1 is also the TRL default [Certain].
To see this without a GPU, Figure 3 trains DPO on a toy you can inspect completely. There is one prompt and eight possible complete answers, each with a true quality \(q\) and a length. The reference model (the SFT model) picks them with fixed probabilities. A labeler compares two answers sampled from the reference and prefers answer \(i\) over \(j\) with probability \(\sigma(q_i - q_j + a\,(\text{len}_i - \text{len}_j)/100)\). The number \(a\) is the labeler’s length bias. The policy is a softmax over the eight answers, trained with Adam (learning rate 0.1) on the exact DPO loss, all pairs in every step.
We measure the policy in three ways. The KL to the reference. The true win rate against the reference: the chance that an answer from the policy beats an answer from the reference when compared by quality alone, \(\sigma(q_i - q_j)\). And the judge win rate: the same, but judged by a judge with the same length bias \(a\) as the labeler, as happens when the same kind of model labels the data and judges the result.
How to read Figure 3
Start with the default (\(\beta = 0.1\), \(a = 0.6\), 400 pairs). The chosen answer is longer in 73.5% of the pairs. DPO moves 90.1% of the probability to answer 6 (quality 1.4, 200 words), not to the best answer 5 (quality 1.8, 110 words). The mean length goes from 94 to 191 words, the true win rate is 0.669 and the judge win rate 0.752: the judge shares the labeler’s bias and overstates the gain. Set \(a = 0\): DPO now picks answer 5 (99%), with length 110 and a true win rate of 0.732. Switch to SFT on chosen with \(a = 0.6\): the policy moves much less (KL 0.142) and its true win rate is only 0.585. Look at the reward curve: in DPO both implicit rewards go below zero, the rejected one faster, just like the real run in Figure 2. Finally, set 50 pairs and \(\beta = 0.1\): with so few pairs, the 320-word answer 7 wins 8 of its 9 pairs in the data, and DPO puts all probability on it. Its true quality is 0, the true win rate falls to 0.398, but the judge still reports 0.638.
The last case shows a known weakness of DPO with small data: an answer that only appears as “chosen”, or never loses, can get its probability pushed up without limit, because nothing in the loss holds it back except \(\beta\) and the number of steps [Likely]. This is one of the motivations for IPO (session 69).
A sweep over \(\beta\) makes the trade-off visible. Figure 4 trains the same toy for eight values of \(\beta\) and plots the result of each run.
How to read Figure 4
With \(a = 0.6\) and 400 pairs, the judge likes the smallest \(\beta\) values best (0.754 at \(\beta = 0.02\) and 0.05), because those runs moved most toward the long answer. The true win rate is highest at \(\beta = 0.2\) (0.678). If you choose \(\beta\) by the biased judge, you pick a model that is longer and slightly worse. The KL is about 2 for every \(\beta \leq 0.1\) and falls to 0.37 at \(\beta = 1\) and 0.13 at \(\beta = 2\). Now set \(a = 0\): both curves are the same, and \(\beta\) from 0.02 to 0.1 is best (true win rate 0.732 to 0.733). The lesson for the real lab: sweep \(\beta\), and choose it with a length-controlled metric (Step 7), not with the raw judge score.
The same toy in PyTorch, with autograd instead of a hand-written gradient. It uses the same seeded pair sampler as the figures, so the numbers match them (up to float rounding):
import math, torch, torch.nn.functional as F # One prompt, 8 possible complete answers: true quality, length in words, and the # probability that the SFT model (the reference) writes each one. Q = [-1.5, -1.0, 0.8, 1.5, 1.8, 1.4, 0.0, -0.5] L = [12, 20, 30, 60, 110, 200, 320, 260] P_REF = [0.10, 0.15, 0.20, 0.20, 0.12, 0.10, 0.08, 0.05] q, ln, p_ref = torch.tensor(Q), torch.tensor(L, dtype=torch.float), torch.tensor(P_REF) def mulberry32(a): # tiny seeded RNG, identical to the one in Figure 3 def r(): nonlocal a a = (a + 0x6D2B79F5) & 0xFFFFFFFF t = ((a ^ (a >> 15)) * (a | 1)) & 0xFFFFFFFF t ^= (t + (((t ^ (t >> 7)) * (t | 61)) & 0xFFFFFFFF)) & 0xFFFFFFFF return ((t ^ (t >> 14)) & 0xFFFFFFFF) / 4294967296 return r def sample(r, p): u, c = r(), 0.0 for k, pk in enumerate(p): c += pk if u < c: return k return len(p) - 1 def make_pairs(n, a, seed=1): """Two different answers from the reference; a labeler picks one with P(i wins) = sigmoid(q_i - q_j + a * (len_i - len_j) / 100).""" r, out = mulberry32(seed), [] for _ in range(n): i = sample(r, P_REF); j = sample(r, P_REF) while j == i: j = sample(r, P_REF) p_i = 1 / (1 + math.exp(-(Q[i] - Q[j] + a * (L[i] - L[j]) / 100))) out.append((i, j) if r() < p_i else (j, i)) return torch.tensor(out) # column 0 = chosen, column 1 = rejected def train(pairs, beta, mode="dpo", steps=600, lr=0.1): ref = torch.log(p_ref) logits = ref.clone().requires_grad_(True) # the policy starts at the reference opt = torch.optim.Adam([logits], lr=lr) for _ in range(steps): lp = F.log_softmax(logits, -1) if mode == "dpo": r_w = beta * (lp[pairs[:, 0]] - ref[pairs[:, 0]]) # implicit reward of chosen r_l = beta * (lp[pairs[:, 1]] - ref[pairs[:, 1]]) # implicit reward of rejected loss = -F.logsigmoid(r_w - r_l).mean() # formula (76.1) else: loss = -lp[pairs[:, 0]].mean() # SFT on chosen, formula (76.5) opt.zero_grad(); loss.backward(); opt.step() return F.softmax(logits, -1).detach() def report(name, p): kl = (p * (p / p_ref).log()).sum() true_win = (p[:, None] * p_ref[None, :] * torch.sigmoid(q[:, None] - q[None, :])).sum() print(f"{name:15s} KL {kl:.3f} mean length {(p * ln).sum():5.1f} true win rate vs ref {true_win:.3f}") pairs = make_pairs(400, a=0.6) print(f"chosen is longer in {(ln[pairs[:, 0]] > ln[pairs[:, 1]]).float().mean():.3f} of the pairs") report("reference", p_ref) report("SFT on chosen", train(pairs, 0.1, mode="sft")) for beta in [0.05, 0.2, 1.0]: report(f"DPO beta={beta}", train(pairs, beta))
Output (PyTorch 2.8, CPU, about one second):
chosen is longer in 0.735 of the pairs reference KL 0.000 mean length 94.0 true win rate vs ref 0.500 SFT on chosen KL 0.142 mean length 122.0 true win rate vs ref 0.585 DPO beta=0.05 KL 2.202 mean length 198.2 true win rate vs ref 0.664 DPO beta=0.2 KL 1.508 mean length 175.1 true win rate vs ref 0.678 DPO beta=1.0 KL 0.370 mean length 137.5 true win rate vs ref 0.628
For the real run, sweep \(\beta \in \{0.01, 0.05, 0.1, 0.3, 0.5\}\) with everything else fixed. For each run, record the held-out pair accuracy, the mean answer length on 500 test prompts, a KL estimate, and the judge results of Steps 6 and 7. A simple KL estimate uses the answers \(y_1, \dots, y_n\) that the trained model generated:
The answers must be sampled at temperature 1 for this to be an unbiased estimate of the KL; with greedy answers it is only a rough “how far did we move” number [Certain]. It is measured per answer, in nats, so long answers give large values.
Step 6: win rate with an LLM judge
Pair accuracy says how well the model ranks existing answers. What you care about is the quality of the answers it writes. So generate answers from each model on the 1,000 test prompts (or a fixed subset of 500), and let a judge compare them with the answers of the starting model A. The DPO paper used exactly this, with GPT-4 as the judge, and reported that GPT-4’s judgments agreed with humans about as often as humans agreed with each other [Certain]. It also found that GPT-4 preferred longer, more repetitive summaries than humans did, and added a second prompt that asks for conciseness [Certain].
- input: test prompts, answers of model \(m\) and of the start model A (greedy, same
max_new_tokens), a judge that is not \(m\) or A - for each prompt:
- \(v_1 \leftarrow\) judge(prompt, first = answer of \(m\), second = answer of A)
- \(v_2 \leftarrow\) judge(prompt, first = answer of A, second = answer of \(m\)), mapped back
- if \(v_1 = v_2\): count it else: count a tie // the verdict depended on the order
- store the word counts of both answers // needed in Step 7
- return \(W, T, L\), the win rate (76.10) and its standard error
# judge.py: position-swapped pairwise judge. Usage: python judge.py C-b0.1 A import sys, json, torch from transformers import AutoTokenizer, AutoModelForCausalLM a, b = sys.argv[1], sys.argv[2] # files gen-<name>.json: list of {prompt, answer} JUDGE = "Qwen/Qwen2.5-7B-Instruct" jt = AutoTokenizer.from_pretrained(JUDGE) jm = AutoModelForCausalLM.from_pretrained(JUDGE, dtype=torch.bfloat16, device_map="auto").eval() ga, gb = json.load(open(f"gen-{a}.json")), json.load(open(f"gen-{b}.json")) RUBRIC = ("You compare two answers to the same question. Judge helpfulness, correctness and " "clarity. Do not prefer an answer because it is longer. " "Answer with one word on the last line: A, B or TIE.") def ask(q, s1, s2): msg = [{"role": "system", "content": RUBRIC}, {"role": "user", "content": f"Question:\n{q}\n\nAnswer A:\n{s1}\n\nAnswer B:\n{s2}"}] enc = jt.apply_chat_template(msg, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(jm.device) with torch.no_grad(): out = jm.generate(**enc, max_new_tokens=300, do_sample=False) w = jt.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True).split() last = w[-1].upper().strip(".*:") if w else "TIE" return last if last in ("A", "B", "TIE") else "TIE" rows = [] for x, y in zip(ga, gb): assert x["prompt"] == y["prompt"] v1 = ask(x["prompt"], x["answer"], y["answer"]) v2 = {"A": "B", "B": "A", "TIE": "TIE"}[ask(x["prompt"], y["answer"], x["answer"])] v = v1 if v1 == v2 else "TIE" rows.append({"win": {"A": 1.0, "TIE": 0.5, "B": 0.0}[v], "len_a": len(x["answer"].split()), "len_b": len(y["answer"].split())}) n = len(rows); wr = sum(r["win"] for r in rows) / n se = (sum((r["win"] - wr) ** 2 for r in rows) / (n - 1) / n) ** 0.5 print(f"{a} vs {b}: win rate {wr:.3f} +- {1.96 * se:.3f} (95%), n = {n}") json.dump(rows, open(f"judge-{a}-vs-{b}.json", "w"))
Each prompt gives a score of 1 (win), 0.5 (tie) or 0 (loss). The win rate is their mean, and its standard error comes from the spread of these scores:
Without ties this is the familiar \(\sqrt{\text{WR}(1-\text{WR})/n}\); ties make it a little smaller. Example: 230 wins, 60 ties and 210 losses on 500 prompts give \(\text{WR} = 260 / 500 = 0.52\). The tie-free formula gives \(\mathrm{SE} = 0.022\), so the 95% interval is about \(\pm 0.044\): 0.476 to 0.564 (formula 76.10, which uses the ties, gives \(\pm 0.041\)). That includes 0.5, so this result alone does not show that the model improved.
Step 7: check for length bias
Two things can make a DPO model look better without being better. The data can prefer longer answers (Figure 1), so the model learns to write more. And the judge can prefer longer answers, so it rewards that. Zheng et al. called the second effect verbosity bias [Certain]. Singhal et al. found that in several RLHF setups much of the reward gain came from longer outputs [Certain], and Park et al. studied the same problem for DPO and proposed a length-regularized DPO [Certain].
Three checks, from cheap to careful:
- Report the mean length of every model’s answers next to its win rate. If the win rate rose and the length doubled, be suspicious.
- Look at length-matched prompts: only the prompts where the two answers have about the same length (for example within 30 words). This is honest but uses few prompts.
- Fit a length-controlled win rate. Model the chance that the judge prefers model \(m\) on prompt \(i\) as a logistic function of the length difference, and read off the win rate at a length difference of zero.
- Read the symbols
- \(\Delta_i\)
- the length difference on prompt \(i\), in hundreds of words
- \(w_1\)
- how much the judge’s preference depends on length; 0 means no length effect
- \(w_0\)
- the preference that is left when both answers have the same length
- \(\text{WR}_\text{LC}\)
- the length-controlled win rate: the predicted win rate at equal length
This is a simplified version of the idea behind length-controlled AlpacaEval, which fits a similar regression with a few more terms and then sets the length term to zero [Certain]. It has limits: if longer answers are really better (they contain needed detail), the correction removes some real quality too. It answers the question “would the judge still prefer this model if the answers had the same length?”.
How to read Figure 5
Default: model \(m\) is not better (advantage 0), but writes 100 more words on average, and the judge has a length bias of 1 per 100 words. The raw win rate is 0.704 ± 0.040 (95%): a clear “win” that is entirely length. The fitted length-controlled win rate is 0.530, close to the truth of 0.5. Now set the extra length to 0: the raw and the controlled rates agree (0.486 and 0.488), because there is nothing to correct. Set the judge bias to 0 with 100 extra words: the raw rate is 0.496, and the fit finds almost no length effect (\(w_1 = -0.10\)). With only 100 prompts the length-controlled estimate is noisy (0.540 here); length control needs more prompts than a raw win rate.
# lc.py: raw and length-controlled win rate from judge-*.json (formula 76.11) import sys, json, math rows = json.load(open(sys.argv[1])) X = [(r["len_a"] - r["len_b"]) / 100 for r in rows] Y = [r["win"] for r in rows] # 1, 0.5 or 0; a tie counts as half a win sig = lambda u: 1 / (1 + math.exp(-u)) w0 = w1 = 0.0 for _ in range(30): # Newton's method for logistic regression g0 = g1 = h00 = h01 = h11 = 0.0 for x, y in zip(X, Y): p = sig(w0 + w1 * x); e = y - p; s = p * (1 - p) g0 += e; g1 += e * x; h00 += s; h01 += s * x; h11 += s * x * x det = h00 * h11 - h01 * h01 w0 += (h11 * g0 - h01 * g1) / det; w1 += (h00 * g1 - h01 * g0) / det near = [y for x, y in zip(X, Y) if abs(x) < 0.3] print(f"raw {sum(Y)/len(Y):.3f} length-controlled {sig(w0):.3f} w1 {w1:.3f} " f"length-matched {sum(near)/max(1,len(near)):.3f} on {len(near)} prompts")
Two more fixes, if the length check fails: train on the length-filtered pairs from Step 1, or use a length-normalized loss. TRL’s loss_type="sigmoid_norm" divides the log-probabilities by the number of answer tokens, as SimPO does with its reference-free loss [Certain]. Then rerun Steps 6 and 7.
Step 8: compare with SFT only and decide
Now fill the table: pair accuracy, mean length, raw and length-controlled win rate against A, and KL, for B, C (best \(\beta\)) and D. Compare B with C directly too (run judge.py with B as the second model), because both may beat A while one is clearly better than the other.
What earlier work found, so you know what to look for:
- In the DPO paper, Preferred-FT (SFT on the chosen answers) did not improve much over the SFT model on TL;DR summarization, while DPO reached a win rate of about 61% against the reference summaries at temperature 0, above PPO at 57% [Certain].
- Zephyr reported that DPO after SFT gave a clearly better chat model than SFT alone on MT-Bench and AlpacaEval [Likely]. Those are judge-based benchmarks, so length can play a role.
- The toy of Figure 3 shows the same pattern in a small form: SFT on chosen moves little (KL 0.142, true win rate 0.585), while DPO moves further and gains more, but also follows the labeler’s length bias.
Your 0.5B model, data size and judge are different, so do not assume the answer. A short decision memo, half a page, makes the result reviewable:
- Goal. Example: “a better general assistant at the same answer length and the same serving cost.”
- Results. The table, with 95% intervals and the number of test prompts.
- Decision. B, C or D, with its \(\beta\). If the length-controlled gain of C or D over B is inside the noise, prefer B: it is simpler and cheaper to train.
- Checks. Read 20 answer pairs yourself. Check a few capabilities the preference data did not cover (for example math or code) for regressions (session 61).
Where this is used
| Setting | How DPO is used | What to watch |
|---|---|---|
| Open chat models (Zephyr, Tülu 3 and others) | SFT, then DPO or a variant on pairs from UltraFeedback-style data | Length growth; judge-based benchmarks need length control |
| Small on-device models | Short DPO stage after SFT, often with LoRA | Small models forget easily; check capabilities outside the preference data [Likely] |
| Safety and refusal tuning | Pairs of safe versus unsafe answers (session 74) | Over-refusal: count refusals on harmless prompts |
| Summarization style | Pairs that prefer faithful, concise summaries | The DPO paper used \(\beta = 0.5\) here; judges prefer long summaries |
| VLM hallucination reduction | Pairs of image descriptions with and without invented objects; TRL’s DPOTrainer supports VLMs | Truncation can cut image tokens; TRL recommends max_length=None for VLMs [Certain] |
| Video captions and key-moment descriptions for Video-RAG | Pairs that prefer captions grounded in the frames over generic or invented ones | The judge must see the frames or a transcript, and long captions look better to judges [Likely] |
| Video question answering and action recognition with a VLM | DPO on correct versus wrong answers about actions or events in a clip | Short answers: length bias is small, but position bias in the judge remains [Likely] |
Common mistakes
- Training on ties. Pairs with equal scores (12.2% of
ultrafeedback_binarized) have random labels. Drop them. - A reference that is not the starting model. The loss must start at 0.693 and the implicit rewards at 0. If they do not, the reference or the masking is wrong.
- Counting prompt tokens in the log-probabilities. Formula (76.3) sums over answer tokens only. Including the prompt adds the same number to both sides for the policy and for the reference, so it cancels in theory, but padding and truncation errors then hide easily [Likely].
- Too high a learning rate. DPO uses much smaller rates than SFT (the TRL default is \(10^{-6}\) for full fine-tuning). A loss near 0 after a few hundred steps usually means overfitting, not success.
- Reading reward accuracy as quality. It only measures ranking of training-style pairs. Both log-probabilities can fall while it rises (Figure 2).
- Choosing \(\beta\) by the raw judge score. If the data and the judge share a length bias, the raw score favors the model that moved most toward long answers (Figure 4). Use the length-controlled win rate.
- No SFT-only baseline. Without model B you cannot tell how much of the gain comes from the pairs and how much from simply seeing good answers.
- Different decoding settings per model. Different temperatures or token limits change the answers more than a small training change does.
Interview questions
What are the four log-probabilities DPO needs per pair, and why is the reference model needed at all?basic
Your DPO run starts with a loss of 0.52 instead of 0.693. What is wrong?basic
During DPO, both logps/chosen and logps/rejected go down. Is that a bug?core
["sigmoid", "sft"]), or DPO-Positive-style penalties.What does \(\beta\) control? What do you expect when you lower it?core
Why is SFT on the chosen answers a necessary baseline, and how does it differ from DPO?core
How do you compute a win rate with an LLM judge so that it is trustworthy?core
Explain a length-controlled win rate. When can it be wrong?advanced
How much memory and compute does DPO need compared with SFT for the same model?core
With very few pairs, DPO pushed all probability onto one strange answer. Why can that happen?advanced
What changes if you build pairs from your own model’s samples instead of using UltraFeedback?advanced
Exercises
- A pair has \(\log \pi_\theta(y_w) = -80\), \(\log \pi_\text{ref}(y_w) = -82\), \(\log \pi_\theta(y_l) = -95\), \(\log \pi_\text{ref}(y_l) = -93\), \(\beta = 0.1\). Compute the implicit rewards, the margin, the loss and the gradient weight.
- Using formula (76.6), how many times more compute does a DPO pair need than an SFT example when the chosen answer has 200 tokens and the rejected one 400?
- In Figure 1, which filter keeps at least 20,000 pairs while making the chosen answer longer in less than 50% of them?
- In Figure 3, set \(a = 1.0\), 400 pairs, \(\beta = 0.1\). What happens to the true and the judge win rate compared with \(a = 0.6\)? Explain why the gap between them grows.
- Your judge gives 180 wins, 140 ties and 180 losses on 500 prompts. What is the win rate, and what can you conclude?
- In Figure 5, find a setting where the raw win rate is above 0.9 but the true advantage is the smallest non-zero value. What does the length-controlled estimate say?
Answerscheck
Key takeaways
- A DPO pair needs four summed answer log-probabilities (76.3); the loss (76.1) is a log-sigmoid of the difference of two implicit rewards (76.2) and always starts at 0.693.
- Clean the data first: drop ties, consider a minimum score gap, and measure how often the chosen answer is longer (53.5% in
ultrafeedback_binarized, 45.6% after keeping only length-matched pairs). - Rising reward accuracy is not better generation. Both log-probabilities often fall; check with generated answers.
- \(\beta\) sets how far the model moves (76.8). Sweep it and choose it with a length-controlled metric, because a biased judge favors the model that moved most toward long answers.
- Measure win rates with a position-swapped judge and an error bar (76.10), and report a length-controlled win rate (76.11) and mean answer lengths next to it.
- Always train an SFT-on-chosen baseline. DPO costs about 2.7 times more compute per example (76.6) and a reference model in memory (76.7); it must earn that.
Key formulas
References for this session
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Hugging Face (code). TRL library
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Hugging Face (code). TRL library
- Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback
- Allal et al. (2025). SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- Dubois et al. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Singhal et al. (2023). A Long Way to Go: Investigating Length Correlations in RLHF
- Park et al. (2024). Disentangling Length from Quality in Direct Preference Optimization
- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Tunstall et al. (2023). Zephyr: Direct Distillation of LM Alignment
- Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
- Meng et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward
- Azar et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences
- Lambert (2025). Reinforcement Learning from Human Feedback (book)