LLM & VLM courseBlog
76Track 4: Post-training and RL · session 24 of 25

DPO on a small model

In this lab you align a small chat model with DPO and check whether it really got better. You build and clean a preference dataset, train with TRL’s DPOTrainer and with a 64-line DPO loop you can run on a CPU, tune β, judge the outputs with a position-swapped LLM judge, test the result for length bias, and compare it with plain supervised fine-tuning on the chosen answers.

Time: 4 to 6 hours (reading 1.5 hours, runs and analysis the rest)Level: Intermediate · hands-on labBefore this: Sessions 54, 62 and 68, Python, one CUDA GPU with 16 to 24 GB (a CPU is enough for Steps 1 to 4)

What you will learn

  • How a preference dataset is laid out (prompt, chosen, rejected), and how to clean it: drop ties, check score gaps, measure the length difference between chosen and rejected
  • How to compute the DPO loss by hand from summed completion log-probabilities, and what TRL’s logs (rewards/chosen, rewards/rejected, margins, accuracies) mean
  • Why both the chosen and the rejected log-probabilities usually go down during DPO, and why SFT on the chosen answers does not separate them
  • How to train a model with TRL’s DPOTrainer and SFTTrainer, and what the reference model costs in memory and compute
  • What β does, how to sweep it, and why the β with the best judge score is not always the β with the best true quality
  • How to measure a win rate with a position-swapped LLM judge, give it an error bar, and correct it for length with a simple length-controlled estimate
  • How to compare DPO with SFT only and write down which one you would ship

Why this matters

You have a small chat model that already follows instructions. You also have a set of preference pairs: for each prompt, one answer that people (or a strong model) liked more, the chosen answer, and one they liked less, the rejected answer. In this lab you use DPO to make the model prefer answers like the chosen ones. Then you check, with numbers, whether the model really got better, or whether it only learned to write longer answers.

Session 68 derived the DPO loss: it turns the RLHF objective into one classification-style loss on pairs, with no reward model and no sampling during training. Session 75 showed how heavy the online RL loop is. This lab is the practical side of DPO. Most of the work is not the loss, which is ten lines of code. The work is the data, the choice of \(\beta\), and an honest evaluation.

You will:

  1. Build and clean a preference dataset, and measure how often the chosen answer is simply the longer one (Figure 1).
  2. Compute the DPO loss and its logs by hand, then run a 64-line DPO loop on a CPU with a 135M model and read real training curves (Figure 2).
  3. Train Qwen2.5-0.5B-Instruct with TRL’s DPOTrainer, and with SFTTrainer on the chosen answers only, as the baseline.
  4. Tune \(\beta\) and see what it changes, first in a toy you can train in your browser (Figures 3 and 4).
  5. Measure win rates with a position-swapped LLM judge, and correct them for length (Figure 5).
  6. Fill one results table and decide: DPO, SFT only, or both.

DPO and its relatives are used in real post-training pipelines. Zephyr-7B was trained with SFT and then DPO on UltraFeedback, the same dataset as in this lab [Certain]. Tülu 3 uses a length-normalized DPO stage after SFT [Likely]. TRL is the most common open-source library for it, and its documentation uses the same dataset, trl-lib/ultrafeedback_binarized, in its examples [Certain].

Setup

Hardware. Steps 1 to 3 and the toy figures need only a CPU. Step 4 trains a 0.5B model; one GPU with 16 to 24 GB is enough with LoRA, and a 24 GB card is enough for full DPO with a short context [Likely]. Step 6 runs a 7B judge, which needs about 15 GB in BF16 [Likely]; you can also use an API model as the judge.

pip install "transformers>=4.56" trl peft datasets accelerate pyarrow

The TRL code on this page follows the current TRL documentation. TRL changes often (argument names, defaults, logged metrics), so check the documentation of your installed version if an argument fails. We ran the from-scratch loop of Step 3 on a CPU (PyTorch 2.8, transformers 4.49) and report its real output. We did not run the GPU scripts of Steps 4 to 6 for you, and this page does not report their results: you will measure them.

Models.

RoleModelWhy
CPU run (Step 3)HuggingFaceTB/SmolLM2-135M-Instruct: 30 layers, hidden size 576, 134,515,008 parameters, tied embeddingsSmall enough to train on a laptop CPU: each of our 32-step runs took about 42 minutes, with two runs sharing the machine
Main run (Step 4)Qwen/Qwen2.5-0.5B-Instruct: 494,032,768 parametersAlready instruction-tuned, so it is a natural SFT starting point and reference model
Judge (Step 6)Qwen/Qwen2.5-7B-Instruct or any stronger modelMust not be one of the contestants

We counted the SmolLM2 parameters from its config.json: \(49{,}152 \times 576\) embedding weights plus 30 blocks of 3,540,096 weights plus the final norm.

Data. trl-lib/ultrafeedback_binarized, a cleaned version of UltraFeedback. UltraFeedback has prompts with four answers from different models, each scored by GPT-4 [Certain]. The “binarized” version keeps two answers per prompt: the best-scored one as chosen and another one as rejected [Likely]. It has 62,135 training rows and 1,000 test rows, with columns chosen, rejected (each a full conversation), score_chosen and score_rejected. We counted these from the dataset files.

The table you will fill. Copy it now. Every empty cell is a number you measure.

ModelTrain dataPair accuracy (held out)Mean answer length (words)Win rate vs startLength-controlled win rateKL to start
A: start (Qwen2.5-0.5B-Instruct)none0.5 (by definition)0.5 (itself)0.50
B: SFT on chosenchosen answers
C: DPO from A, best \(\beta\)pairs
D: SFT on chosen, then DPOboth

Step 1: build a preference dataset

A preference example has three parts: a prompt \(x\), a chosen answer \(y_w\) (w for “winner”) and a rejected answer \(y_l\) (l for “loser”). TRL accepts this as three columns. The recommended form keeps the prompt separate, so the trainer knows exactly which tokens belong to the answers:

{"prompt":   [{"role": "user", "content": "What color is the sky?"}],
 "chosen":   [{"role": "assistant", "content": "It is blue."}],
 "rejected": [{"role": "assistant", "content": "It is green."}]}

ultrafeedback_binarized stores the prompt inside both conversations instead (an “implicit prompt”). TRL handles both forms. In our own loop we take the prompt as the shared part of the two conversations.

Clean the pairs before you train

A pair teaches the model one thing: “move probability from \(y_l\) to \(y_w\)”. If the pair is noise, the model learns noise. Three checks catch most problems:

  • Ties. If both answers have the same score, the label is arbitrary. In ultrafeedback_binarized, 7,554 of the 62,135 pairs (12.2%) have equal scores. Drop them.
  • Small score gaps. A gap of 0.5 between two judge scores is close to the judge’s own noise. A minimum gap keeps cleaner pairs but fewer of them.
  • Length. If the chosen answer is usually the longer one, DPO can learn “longer is better” instead of “better is better”. Measure it.
pairs keptchosen answer is longermedian words: chosen and rejected
Figure 1. Real statistics of all 62,135 training pairs of trl-lib/ultrafeedback_binarized, computed by us with Python. Length is counted in words (split on spaces). “Max length ratio” keeps only pairs where the longer answer has at most that many times the words of the shorter one.
How to read Figure 1

With no filter, the chosen answer is longer in 53.5% of pairs, and its median length is 158 words against 130 for the rejected one. That is a mild length bias in the data. Raise the minimum gap to 2: 23,832 pairs remain, the chosen answer is still longer in 55.4% of them, and both answers are shorter: the medians drop to 107 and 83 words. Now set the length ratio to 1.2: the chosen answer is longer in only 45.6% of the 13,305 pairs. So most of the length bias sits in pairs where the two answers have very different lengths. Removing them is one simple way to train a model that does not learn length.

A filter is a trade: fewer pairs, but cleaner ones. For this lab, drop ties and keep everything else; then, as an extra run, train on the length-ratio-filtered set and compare the answer lengths (Step 7).

Making your own pairs (on-policy data)

UltraFeedback answers were written by other models, not by yours. Such data is off-policy: your model may never write answers like them (session 71). A common fix is to sample answers from your own model and let a judge or a reward model rank them. Zephyr and many later recipes use answers from other models; on-policy pairs are used, for example, in iterative DPO [Likely]. Algorithm 1 is the simplest version.

Algorithm 1: on-policy preference pairs from your own model
  1. input: prompts \(\{x\}\), your SFT model \(\pi_\text{SFT}\), a judge or reward model \(J\), \(K = 4\) samples, minimum gap \(g\)
  2. for each prompt \(x\):
  3.   sample \(y_1, \dots, y_K \sim \pi_\text{SFT}(\cdot \mid x)\) at temperature 0.8 // diversity matters: greedy gives K copies
  4.   score each: \(s_k = J(x, y_k)\)
  5.   \(w \leftarrow \argmax_k s_k\), \(l \leftarrow \argmin_k s_k\)
  6.   if \(s_w - s_l \geq g\): keep \((x, y_w, y_l)\) else: skip // near-ties are noise
  7. report: pairs kept, share where \(y_w\) is longer, median lengths
# make_pairs.py: score 4 own samples per prompt with a judge, keep best vs worst
import json, torch
from transformers import AutoTokenizer, AutoModelForCausalLM

POLICY, JUDGE = "Qwen/Qwen2.5-0.5B-Instruct", "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(POLICY); tok.padding_side = "left"
pol = AutoModelForCausalLM.from_pretrained(POLICY, dtype=torch.bfloat16, device_map="auto")
jt = AutoTokenizer.from_pretrained(JUDGE)
jm = AutoModelForCausalLM.from_pretrained(JUDGE, dtype=torch.bfloat16, device_map="auto")

def score(prompt, answer):        # 1..10, greedy, last number in the reply
    msg = [{"role": "system", "content": "Rate the answer from 1 to 10 for helpfulness and "
            "correctness. Length is not a quality. Reply with one number on the last line."},
           {"role": "user", "content": f"Question:\n{prompt}\n\nAnswer:\n{answer}"}]
    enc = jt.apply_chat_template(msg, add_generation_prompt=True, return_tensors="pt",
                                 return_dict=True).to(jm.device)
    out = jm.generate(**enc, max_new_tokens=8, do_sample=False)
    txt = jt.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True)
    nums = [int(t) for t in txt.replace(".", " ").split() if t.isdigit()]
    return nums[-1] if nums else 0

torch.manual_seed(0)
prompts = [json.loads(l)["prompt"] for l in open("prompts.jsonl")]
pairs = []
for p in prompts:
    enc = tok.apply_chat_template([{"role": "user", "content": p}], add_generation_prompt=True,
                                  return_tensors="pt", return_dict=True).to(pol.device)
    out = pol.generate(**enc, do_sample=True, temperature=0.8, top_p=0.95,
                       max_new_tokens=400, num_return_sequences=4)
    ans = tok.batch_decode(out[:, enc["input_ids"].shape[1]:], skip_special_tokens=True)
    s = [score(p, a) for a in ans]
    w, l = max(range(4), key=s.__getitem__), min(range(4), key=s.__getitem__)
    if s[w] - s[l] >= 2:
        pairs.append({"prompt": [{"role": "user", "content": p}],
                      "chosen": [{"role": "assistant", "content": ans[w]}],
                      "rejected": [{"role": "assistant", "content": ans[l]}]})
longer = sum(len(x["chosen"][0]["content"].split()) > len(x["rejected"][0]["content"].split())
             for x in pairs) / max(1, len(pairs))
print(len(pairs), "pairs kept; chosen longer in", round(longer, 3))
with open("own_pairs.jsonl", "w") as f:
    for x in pairs: f.write(json.dumps(x) + "\n")

Answers that hit max_new_tokens are cut off. A judge usually scores a cut answer low, so it ends up as the rejected answer, and DPO learns to avoid the long, unfinished style. That can be what you want, but check it: count how many rejected answers were cut.

Step 2: the loss and the logs, by hand

Before running a trainer, compute what it will compute. For one pair, DPO needs four numbers: the log-probability of the chosen and of the rejected answer, under the model being trained \(\pi_\theta\) and under a frozen copy of the starting model, the reference \(\pi_\text{ref}\). The loss is the one from session 68:

\[ \mathcal{L}_\text{DPO} = -\log \sigma\big(\hat r_w - \hat r_l\big), \qquad \sigma(u) = \frac{1}{1 + e^{-u}} \tag{76.1} \]

Here \(\hat r_w\) and \(\hat r_l\) are the implicit rewards: how much more likely the model now makes an answer than the reference did, times \(\beta\):

\[ \hat r_w = \beta \big[\log \pi_\theta(y_w \mid x) - \log \pi_\text{ref}(y_w \mid x)\big], \qquad \hat r_l = \beta \big[\log \pi_\theta(y_l \mid x) - \log \pi_\text{ref}(y_l \mid x)\big] \tag{76.2} \]
Read the symbols
\(x, y_w, y_l\)
the prompt, the chosen answer and the rejected answer
\(\pi_\theta\)
the model you train (the policy)
\(\pi_\text{ref}\)
the frozen starting model; usually the SFT model you start from
\(\beta\)
a positive number (often 0.1) that sets how strongly the model is held near the reference
\(\hat r_w, \hat r_l\)
the implicit rewards; TRL logs their batch means as rewards/chosen and rewards/rejected

The log-probability of a whole answer is the sum over its tokens. Only the answer tokens count, not the prompt tokens:

\[ \log \pi(y \mid x) = \sum_{t=1}^{T} \log \pi\big(y_t \mid x, y_{\lt t}\big) \tag{76.3} \]

TRL logs three more numbers per batch. The margin is \(m = \hat r_w - \hat r_l\) (rewards/margins). The reward accuracy is the share of pairs with \(m \gt 0\) (rewards/accuracies). And the gradient of one pair is a weighted push up on \(y_w\) and down on \(y_l\):

\[ \nabla_\theta \mathcal{L}_\text{DPO} = -\beta\, \sigma(-m)\, \Big[\nabla_\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x)\Big] \tag{76.4} \]

The weight \(\sigma(-m)\) is large when the model still ranks the pair wrongly (\(m \lt 0\)) and small when it already ranks it right by a wide margin. Pairs the model has “solved” stop pushing.

Worked example: one pair, \(\beta = 0.1\)

At the start, \(\pi_\theta = \pi_\text{ref}\), so \(\hat r_w = \hat r_l = 0\), \(m = 0\) and the loss is \(-\log \sigma(0) = \log 2 = 0.693\). Every DPO run starts at 0.693; if yours does not, the reference is not the starting model.

After some training, the chosen answer has gone from \(-150\) to \(-151\) nats and the rejected one from \(-140\) to \(-145\) nats. Then \(\hat r_w = 0.1 \times (-1) = -0.1\) and \(\hat r_l = 0.1 \times (-5) = -0.5\). The margin is \(m = 0.4\), the loss is \(-\log \sigma(0.4) = 0.513\), and the gradient weight is \(\sigma(-0.4) = 0.401\). Note that the chosen answer became less likely, yet the loss went down and the pair counts as “accurate”. DPO only cares about the gap.

Step 3: DPO from scratch on a CPU

Here is a complete DPO loop without TRL. It uses 256 training pairs and 64 held-out pairs from ultrafeedback_binarized (ties dropped, prompt plus answer at most 512 tokens), batches of 8 pairs, AdamW and one pass over the data: 32 steps. The same script also runs “SFT on chosen” with the same data, so you can compare the two. We ran it on a CPU, and the curves in Figure 2 are its real logs. (Our run also generated answers to measure their length; the version below leaves that part out and prints a shorter log line. We checked that this shorter version runs too.)

Algorithm 2: one DPO training step
  1. input: a batch of pairs \((x, y_w, y_l)\), policy \(\pi_\theta\), frozen reference \(\pi_\text{ref}\), \(\beta\)
  2. build \(2B\) sequences: \(x \Vert y_w\) and \(x \Vert y_l\); mask = 1 on answer tokens only
  3. \(\ell^\theta_w, \ell^\theta_l \leftarrow\) summed masked log-probs under \(\pi_\theta\) (formula 76.3)
  4. \(\ell^\text{ref}_w, \ell^\text{ref}_l \leftarrow\) the same under \(\pi_\text{ref}\), with no gradient // can be precomputed once
  5. \(\hat r_w \leftarrow \beta(\ell^\theta_w - \ell^\text{ref}_w)\), \(\hat r_l \leftarrow \beta(\ell^\theta_l - \ell^\text{ref}_l)\)
  6. loss \(\leftarrow\) mean of \(-\log\sigma(\hat r_w - \hat r_l)\); backward; clip the gradient norm to 1; optimizer step
  7. log: loss, mean \(\hat r_w\), mean \(\hat r_l\), accuracy, mean \(\ell^\theta_w\), mean \(\ell^\theta_l\)
# dpo_loop.py: DPO (or SFT on chosen) from scratch.  Usage: python dpo_loop.py dpo 0.1 2e-5
import sys, json, copy, torch, torch.nn.functional as F
import pyarrow.parquet as pq
from transformers import AutoModelForCausalLM, AutoTokenizer
torch.manual_seed(0)
MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct"
MODE, BETA, LR = sys.argv[1], float(sys.argv[2]), float(sys.argv[3])   # "dpo" or "sft"
N_TRAIN, N_EVAL, BS, MAXLEN = 256, 64, 8, 512
tok = AutoTokenizer.from_pretrained(MODEL)
policy = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.float32)
ref = copy.deepcopy(policy).eval().requires_grad_(False)          # frozen reference

# the train split of trl-lib/ultrafeedback_binarized, downloaded as a parquet file
rows = pq.read_table("data/train.parquet").slice(0, 4000).to_pylist()
def encode(r):                    # prompt = the shared part of the two conversations
    p = tok.apply_chat_template(r["chosen"][:-1], add_generation_prompt=True)
    c = tok.apply_chat_template(r["chosen"]); j = tok.apply_chat_template(r["rejected"])
    assert c[:len(p)] == p and j[:len(p)] == p
    return p, c[len(p):], j[len(p):]
pairs = []
for r in rows:
    if r["score_chosen"] == r["score_rejected"]: continue           # drop ties
    p, c, j = encode(r)
    if len(p) + max(len(c), len(j)) <= MAXLEN: pairs.append((p, c, j))
    if len(pairs) == N_TRAIN + N_EVAL: break
train, evals = pairs[:N_TRAIN], pairs[N_TRAIN:]
print("chosen longer (tokens):", sum(len(c) > len(j) for _, c, j in pairs) / len(pairs))

def seq_logps(model, prompts, resps):          # formula (76.3), answer tokens only
    seqs = [p + r for p, r in zip(prompts, resps)]; L = max(map(len, seqs))
    ids = torch.full((len(seqs), L), tok.pad_token_id)
    mask = torch.zeros(len(seqs), L, dtype=torch.long); lab = torch.zeros(len(seqs), L, dtype=torch.bool)
    for i, (p, s) in enumerate(zip(prompts, seqs)):
        ids[i, :len(s)] = torch.tensor(s); mask[i, :len(s)] = 1; lab[i, len(p):len(s)] = True
    logits = model(input_ids=ids, attention_mask=mask).logits[:, :-1]
    lp = torch.log_softmax(logits.float(), -1).gather(-1, ids[:, 1:, None]).squeeze(-1)
    return (lp * lab[:, 1:]).sum(-1)              # the label of position t is token t+1

def batch_logps(model, b):                     # chosen and rejected in one forward pass
    P = [x[0] for x in b]; lp = seq_logps(model, P + P, [x[1] for x in b] + [x[2] for x in b])
    return lp[:len(b)], lp[len(b):]

opt = torch.optim.AdamW(policy.parameters(), lr=LR, weight_decay=0.0)
for step in range(N_TRAIN // BS):
    b = train[step * BS:(step + 1) * BS]
    pc, pr = batch_logps(policy, b)
    with torch.no_grad(): rc, rr = batch_logps(ref, b)
    rw_c, rw_r = BETA * (pc - rc), BETA * (pr - rr)               # formula (76.2)
    if MODE == "dpo":
        loss = -F.logsigmoid(rw_c - rw_r).mean()                  # formula (76.1)
    else:                                                         # SFT on chosen, formula (76.5)
        loss = -pc.sum() / sum(len(x[1]) for x in b)
    opt.zero_grad(); loss.backward()
    torch.nn.utils.clip_grad_norm_(policy.parameters(), 1.0); opt.step()
    print(json.dumps({"step": step + 1, "loss": round(loss.item(), 4),
        "rewards/chosen": round(rw_c.mean().item(), 4), "rewards/rejected": round(rw_r.mean().item(), 4),
        "acc": (rw_c > rw_r).float().mean().item(), "logps/chosen": round(pc.mean().item(), 2)}))

with torch.no_grad():                          # held-out pair accuracy
    acc = 0
    for i in range(0, len(evals), BS):
        b = evals[i:i + BS]; pc, pr = batch_logps(policy, b); rc, rr = batch_logps(ref, b)
        acc += ((pc - rc) > (pr - rr)).sum().item()
print("held-out pair accuracy:", acc / len(evals))

Our script also generated greedy answers (at most 300 new tokens) for 32 other prompts after training, to measure answer length. We ran three settings: DPO with learning rate \(5 \times 10^{-6}\), DPO with \(2 \times 10^{-5}\), and SFT on chosen with \(2 \times 10^{-5}\), all with \(\beta = 0.1\) for the logged rewards.

Figure 2. Real logs of dpo_loop.py on a CPU: SmolLM2-135M-Instruct, 256 UltraFeedback pairs, 32 steps of 8 pairs. Thin lines are the per-step values, thick lines a moving average over 6 steps. Each step sees new pairs, so these are noisy “first look” numbers, like a real training log.
How to read Figure 2

Start with “DPO, LR 2e-5” and “Implicit rewards”. Both curves go below zero: the policy makes the chosen answers and the rejected answers less likely than the reference does. The rejected curve falls faster. The mean of the last 8 steps is \(-0.263\) for chosen and \(-0.420\) for rejected, so the margin is positive. Switch to “Loss and accuracy”: the loss falls only a little, from 0.696 to 0.665 (mean of the first and last 8 steps), and the batch accuracy goes from 0.531 to 0.578. With 32 steps this is the very start of training. Now pick “DPO, LR 5e-6”: the same shape, about five times smaller (rewards \(-0.049\) and \(-0.094\)). Then pick “SFT on chosen”: both rewards go up, and the rejected one goes up more (0.412 and 0.503). The rejected answers are similar text, so imitating the chosen answers also makes them more likely. The “Log-probs” view shows the raw summed log-probabilities. Each step uses a different batch with different answer lengths, so these jump around and you cannot read a trend from them. That is why TRL logs the implicit rewards: they compare each answer with itself under the reference.

The held-out pair accuracies on the 64 unseen pairs: DPO with LR \(2 \times 10^{-5}\) reaches 0.609, DPO with \(5 \times 10^{-6}\) 0.578, and SFT on chosen 0.422, below chance. So in this run, SFT on chosen ranked the rejected answer above the chosen one more often than not. The answer lengths hardly moved. The start model wrote 214.2 tokens on average on the 32 generation prompts, and 17 of its 32 answers hit the 300-token limit. After training the means were 205.6 (DPO, LR \(2 \times 10^{-5}\)), 209.4 (DPO, LR \(5 \times 10^{-6}\)) and 198.6 tokens (SFT). 32 steps are too few to change the style of a model. The length effects of Step 7 come from full runs.

The held-out pair accuracy is measured on only 64 pairs, so its standard error is about \(\sqrt{0.5 \times 0.5 / 64} = 0.063\). With so few pairs, differences of a few points are noise. The 0.609 of DPO is only about 1.7 standard errors above 0.5, and the 0.031 gap between the two DPO learning rates is pure noise. The gap between DPO and SFT (0.609 against 0.422) is about 2 standard errors of a difference, so it is a hint, not proof. This loop is for understanding and debugging. The real run is next.

Step 4: train with TRL

Now the main experiment, on a GPU. You train two models from Qwen2.5-0.5B-Instruct (model A in the table): B, SFT on the chosen answers, and C, DPO on the pairs. As an extra, D is DPO started from B, with B as the reference. All runs use the same 10,000 training pairs (ties dropped) and the same seed. The SFT baseline is what the DPO paper calls “Preferred-FT”: plain fine-tuning on the chosen answers [Certain].

\[ \mathcal{L}_\text{SFT} = -\frac{1}{\sum_i |y_w^{(i)}|} \sum_i \log \pi_\theta\big(y_w^{(i)} \mid x^{(i)}\big) \tag{76.5} \]

This is next-token cross-entropy on the chosen answers, averaged over their tokens. It never sees the rejected answers. So it raises the probability of the chosen answers, but nothing pushes the rejected ones down. Figure 2 already showed what that means for the pair accuracy.

# data.py: the shared data
from datasets import load_dataset
def load_pairs(n_train=10_000, seed=0):
    ds = load_dataset("trl-lib/ultrafeedback_binarized")
    ds = ds.filter(lambda r: r["score_chosen"] > r["score_rejected"])     # drop ties
    train = ds["train"].shuffle(seed=seed).select(range(n_train))
    return train, ds["test"]
# sft.py: model B, SFT on the chosen answers only
from transformers import AutoTokenizer
from trl import SFTConfig, SFTTrainer
from data import load_pairs
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
train, _ = load_pairs()
train = train.map(lambda r: {"messages": r["chosen"]}, remove_columns=train.column_names)
cfg = SFTConfig(output_dir="out/B-sft", learning_rate=1e-5, num_train_epochs=1,
                per_device_train_batch_size=8, gradient_accumulation_steps=4, max_length=1024,
                assistant_only_loss=True,          # loss on the answer, not the prompt
                lr_scheduler_type="cosine", warmup_steps=10, bf16=True,
                logging_steps=10, save_strategy="no", report_to="none", seed=0)
tr = SFTTrainer(model=MODEL, args=cfg, train_dataset=train,
                processing_class=AutoTokenizer.from_pretrained(MODEL))
tr.train(); tr.save_model("out/B-sft/final")

assistant_only_loss=True needs a chat template that marks the assistant part; if your TRL version refuses it for this template, use the prompt-completion format of session 54 instead [Likely].

# dpo.py: models C and D.  Usage: python dpo.py 0.1 Qwen/Qwen2.5-0.5B-Instruct out/C-b0.1
import sys
from transformers import AutoTokenizer
from trl import DPOConfig, DPOTrainer
from data import load_pairs
beta, start, out = float(sys.argv[1]), sys.argv[2], sys.argv[3]
train, test = load_pairs()
tok = AutoTokenizer.from_pretrained(start)
cfg = DPOConfig(output_dir=out, beta=beta, loss_type=["sigmoid"],   # plain DPO, formula (76.1)
                learning_rate=1e-6,               # TRL default; DPO needs a small LR
                num_train_epochs=1, per_device_train_batch_size=4, gradient_accumulation_steps=8,
                max_length=1024, lr_scheduler_type="cosine", warmup_steps=10, bf16=True,
                logging_steps=10, eval_strategy="steps", eval_steps=50,
                save_strategy="no", report_to="none", seed=0)
tr = DPOTrainer(model=start,                       # ref_model=None: a frozen copy of the start
                args=cfg, train_dataset=train, eval_dataset=test.select(range(200)),
                processing_class=tok)
tr.train(); tr.save_model(f"{out}/final")

With ref_model=None, TRL uses the starting model as the reference [Certain]. For model D, pass the SFT checkpoint out/B-sft/final as start; it then becomes both the starting point and the reference. With a LoRA adapter (peft_config=LoraConfig(...)), TRL does not need a second copy: the reference is the base model with the adapter switched off [Likely]. The TRL documentation suggests a higher learning rate for adapters, around \(10^{-5}\) [Certain].

What the reference model costs

DPO runs four sequences’ worth of forward passes per pair: chosen and rejected, under the policy and under the reference. With the FLOP rule of session 25 (about \(2N\) FLOPs per token forward, \(4N\) backward), the cost per pair with answers of \(T_w\) and \(T_l\) tokens, ignoring the shared prompt, is:

\[ C_\text{DPO} \approx \underbrace{6N (T_w + T_l)}_{\text{policy, forward + backward}} + \underbrace{2N (T_w + T_l)}_{\text{reference, forward}}, \qquad C_\text{SFT} \approx 6N\, T_w \tag{76.6} \]

Memory adds the frozen reference to the usual budget of session 38: 16 bytes per trained parameter (FP32 weights, gradients and two Adam states) plus 2 bytes per reference parameter in BF16:

\[ M_\text{DPO, full} \approx 16N + 2N + M_\text{act}, \qquad M_\text{DPO, LoRA} \approx 2N + 16P + M_\text{act} \tag{76.7} \]
Read the symbols
\(N\)
the number of model parameters
\(P\)
the number of LoRA parameters (the base is shared by policy and reference)
\(T_w, T_l\)
the lengths in tokens of the chosen and the rejected answer
\(M_\text{act}\)
activation memory; DPO stores activations for two sequences per pair
Worked example: Qwen2.5-0.5B-Instruct

\(N = 494{,}032{,}768\). Full DPO: \(16N = 7.90\) GB for the policy and its optimizer, plus \(2N = 0.99\) GB for the reference, so 8.89 GB before activations. LoRA DPO with \(r = 16\) on all linear layers (\(P = 8{,}798{,}208\), as counted in session 62): \(0.99 + 0.14 = 1.13\) GB.

Compute, with \(T_w = T_l = 300\) tokens: DPO costs \(6N \cdot 600 + 2N \cdot 600 = 4{,}800N\) FLOPs per pair, SFT on the chosen answer \(6N \cdot 300 = 1{,}800N\). So one DPO pair costs about 2.7 times one SFT example. With precompute_ref_log_probs=True, TRL computes the reference log-probs once before training and does not keep the reference in memory during training [Certain]; that saves the reference memory, and the compute too if you train for more than one epoch.

Read the logs while it runs. A healthy DPO run on this data usually looks like Figure 2, only smoother: the loss starts at 0.693 and falls slowly, the accuracy rises above 0.5, rewards/rejected falls faster than rewards/chosen, and the margin grows [Likely]. Warning signs: the loss falls to near 0 within a few hundred steps (too high a learning rate, or the data is too easy to separate, for example by length); logps/chosen falls by hundreds of nats (the model is moving probability to answers outside the data); the accuracy stays at 0.5 (a bug, often a wrong mask or a wrong reference).

Step 5: tune β

What does \(\beta\) do? Session 68 showed that DPO fits the same target as KL-regularized RLHF, whose best policy is the reference tilted by the reward:

\[ \pi^*(y \mid x) \propto \pi_\text{ref}(y \mid x)\, \exp\!\big(r(x, y) / \beta\big) \quad\Longleftrightarrow\quad \log \frac{\pi^*(y \mid x)}{\pi_\text{ref}(y \mid x)} = \frac{r(x, y)}{\beta} - \log Z(x) \tag{76.8} \]

The data fixes the reward differences (up to noise). So a small \(\beta\) means large log-ratios: the model moves far from the reference. A large \(\beta\) keeps it close. That makes \(\beta\) the main knob for “how much do I trust these pairs”. Noisy or biased pairs call for a larger \(\beta\); clean pairs and a weak starting model allow a smaller one. The DPO paper used \(\beta = 0.1\) for dialogue and \(\beta = 0.5\) for summarization [Certain]; 0.1 is also the TRL default [Certain].

To see this without a GPU, Figure 3 trains DPO on a toy you can inspect completely. There is one prompt and eight possible complete answers, each with a true quality \(q\) and a length. The reference model (the SFT model) picks them with fixed probabilities. A labeler compares two answers sampled from the reference and prefers answer \(i\) over \(j\) with probability \(\sigma(q_i - q_j + a\,(\text{len}_i - \text{len}_j)/100)\). The number \(a\) is the labeler’s length bias. The policy is a softmax over the eight answers, trained with Adam (learning rate 0.1) on the exact DPO loss, all pairs in every step.

We measure the policy in three ways. The KL to the reference. The true win rate against the reference: the chance that an answer from the policy beats an answer from the reference when compared by quality alone, \(\sigma(q_i - q_j)\). And the judge win rate: the same, but judged by a judge with the same length bias \(a\) as the labeler, as happens when the same kind of model labels the data and judges the result.

reference (SFT model)policy after trainingimplicit reward, chosen (curve)implicit reward, rejected (curve)
Figure 3. Live DPO on a toy with eight answers. Each bar is labeled with the answer’s true quality \(q\) and length in words. The pairs are sampled with a seeded random generator, so the same settings always give the same result. The small chart on the right shows the mean implicit rewards (76.2) of the chosen and rejected answers over the 600 training steps.
How to read Figure 3

Start with the default (\(\beta = 0.1\), \(a = 0.6\), 400 pairs). The chosen answer is longer in 73.5% of the pairs. DPO moves 90.1% of the probability to answer 6 (quality 1.4, 200 words), not to the best answer 5 (quality 1.8, 110 words). The mean length goes from 94 to 191 words, the true win rate is 0.669 and the judge win rate 0.752: the judge shares the labeler’s bias and overstates the gain. Set \(a = 0\): DPO now picks answer 5 (99%), with length 110 and a true win rate of 0.732. Switch to SFT on chosen with \(a = 0.6\): the policy moves much less (KL 0.142) and its true win rate is only 0.585. Look at the reward curve: in DPO both implicit rewards go below zero, the rejected one faster, just like the real run in Figure 2. Finally, set 50 pairs and \(\beta = 0.1\): with so few pairs, the 320-word answer 7 wins 8 of its 9 pairs in the data, and DPO puts all probability on it. Its true quality is 0, the true win rate falls to 0.398, but the judge still reports 0.638.

The last case shows a known weakness of DPO with small data: an answer that only appears as “chosen”, or never loses, can get its probability pushed up without limit, because nothing in the loss holds it back except \(\beta\) and the number of steps [Likely]. This is one of the motivations for IPO (session 69).

A sweep over \(\beta\) makes the trade-off visible. Figure 4 trains the same toy for eight values of \(\beta\) and plots the result of each run.

true win rate vs referencejudge win rate (judge has bias \(a\))KL to reference (right axis)
Figure 4. A \(\beta\) sweep on the toy of Figure 3: \(\beta \in\) {0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2}, 600 Adam steps each. The marked points are the \(\beta\) with the highest true win rate and the \(\beta\) with the highest judge win rate.
How to read Figure 4

With \(a = 0.6\) and 400 pairs, the judge likes the smallest \(\beta\) values best (0.754 at \(\beta = 0.02\) and 0.05), because those runs moved most toward the long answer. The true win rate is highest at \(\beta = 0.2\) (0.678). If you choose \(\beta\) by the biased judge, you pick a model that is longer and slightly worse. The KL is about 2 for every \(\beta \leq 0.1\) and falls to 0.37 at \(\beta = 1\) and 0.13 at \(\beta = 2\). Now set \(a = 0\): both curves are the same, and \(\beta\) from 0.02 to 0.1 is best (true win rate 0.732 to 0.733). The lesson for the real lab: sweep \(\beta\), and choose it with a length-controlled metric (Step 7), not with the raw judge score.

The same toy in PyTorch, with autograd instead of a hand-written gradient. It uses the same seeded pair sampler as the figures, so the numbers match them (up to float rounding):

import math, torch, torch.nn.functional as F

# One prompt, 8 possible complete answers: true quality, length in words, and the
# probability that the SFT model (the reference) writes each one.
Q     = [-1.5, -1.0, 0.8, 1.5, 1.8, 1.4, 0.0, -0.5]
L     = [12, 20, 30, 60, 110, 200, 320, 260]
P_REF = [0.10, 0.15, 0.20, 0.20, 0.12, 0.10, 0.08, 0.05]
q, ln, p_ref = torch.tensor(Q), torch.tensor(L, dtype=torch.float), torch.tensor(P_REF)

def mulberry32(a):                       # tiny seeded RNG, identical to the one in Figure 3
    def r():
        nonlocal a
        a = (a + 0x6D2B79F5) & 0xFFFFFFFF
        t = ((a ^ (a >> 15)) * (a | 1)) & 0xFFFFFFFF
        t ^= (t + (((t ^ (t >> 7)) * (t | 61)) & 0xFFFFFFFF)) & 0xFFFFFFFF
        return ((t ^ (t >> 14)) & 0xFFFFFFFF) / 4294967296
    return r

def sample(r, p):
    u, c = r(), 0.0
    for k, pk in enumerate(p):
        c += pk
        if u < c: return k
    return len(p) - 1

def make_pairs(n, a, seed=1):
    """Two different answers from the reference; a labeler picks one with
    P(i wins) = sigmoid(q_i - q_j + a * (len_i - len_j) / 100)."""
    r, out = mulberry32(seed), []
    for _ in range(n):
        i = sample(r, P_REF); j = sample(r, P_REF)
        while j == i: j = sample(r, P_REF)
        p_i = 1 / (1 + math.exp(-(Q[i] - Q[j] + a * (L[i] - L[j]) / 100)))
        out.append((i, j) if r() < p_i else (j, i))
    return torch.tensor(out)             # column 0 = chosen, column 1 = rejected

def train(pairs, beta, mode="dpo", steps=600, lr=0.1):
    ref = torch.log(p_ref)
    logits = ref.clone().requires_grad_(True)       # the policy starts at the reference
    opt = torch.optim.Adam([logits], lr=lr)
    for _ in range(steps):
        lp = F.log_softmax(logits, -1)
        if mode == "dpo":
            r_w = beta * (lp[pairs[:, 0]] - ref[pairs[:, 0]])   # implicit reward of chosen
            r_l = beta * (lp[pairs[:, 1]] - ref[pairs[:, 1]])   # implicit reward of rejected
            loss = -F.logsigmoid(r_w - r_l).mean()              # formula (76.1)
        else:
            loss = -lp[pairs[:, 0]].mean()                      # SFT on chosen, formula (76.5)
        opt.zero_grad(); loss.backward(); opt.step()
    return F.softmax(logits, -1).detach()

def report(name, p):
    kl = (p * (p / p_ref).log()).sum()
    true_win = (p[:, None] * p_ref[None, :] * torch.sigmoid(q[:, None] - q[None, :])).sum()
    print(f"{name:15s} KL {kl:.3f}   mean length {(p * ln).sum():5.1f}   true win rate vs ref {true_win:.3f}")

pairs = make_pairs(400, a=0.6)
print(f"chosen is longer in {(ln[pairs[:, 0]] > ln[pairs[:, 1]]).float().mean():.3f} of the pairs")
report("reference", p_ref)
report("SFT on chosen", train(pairs, 0.1, mode="sft"))
for beta in [0.05, 0.2, 1.0]:
    report(f"DPO beta={beta}", train(pairs, beta))

Output (PyTorch 2.8, CPU, about one second):

chosen is longer in 0.735 of the pairs
reference       KL 0.000   mean length  94.0   true win rate vs ref 0.500
SFT on chosen   KL 0.142   mean length 122.0   true win rate vs ref 0.585
DPO beta=0.05   KL 2.202   mean length 198.2   true win rate vs ref 0.664
DPO beta=0.2    KL 1.508   mean length 175.1   true win rate vs ref 0.678
DPO beta=1.0    KL 0.370   mean length 137.5   true win rate vs ref 0.628

For the real run, sweep \(\beta \in \{0.01, 0.05, 0.1, 0.3, 0.5\}\) with everything else fixed. For each run, record the held-out pair accuracy, the mean answer length on 500 test prompts, a KL estimate, and the judge results of Steps 6 and 7. A simple KL estimate uses the answers \(y_1, \dots, y_n\) that the trained model generated:

\[ \widehat{\KL}(\pi_\theta \,\|\, \pi_\text{ref}) = \frac{1}{n} \sum_{i=1}^{n} \Big[\log \pi_\theta(y_i \mid x_i) - \log \pi_\text{ref}(y_i \mid x_i)\Big], \qquad y_i \sim \pi_\theta(\cdot \mid x_i) \tag{76.9} \]

The answers must be sampled at temperature 1 for this to be an unbiased estimate of the KL; with greedy answers it is only a rough “how far did we move” number [Certain]. It is measured per answer, in nats, so long answers give large values.

Step 6: win rate with an LLM judge

Pair accuracy says how well the model ranks existing answers. What you care about is the quality of the answers it writes. So generate answers from each model on the 1,000 test prompts (or a fixed subset of 500), and let a judge compare them with the answers of the starting model A. The DPO paper used exactly this, with GPT-4 as the judge, and reported that GPT-4’s judgments agreed with humans about as often as humans agreed with each other [Certain]. It also found that GPT-4 preferred longer, more repetitive summaries than humans did, and added a second prompt that asks for conciseness [Certain].

Algorithm 3: pairwise win rate with a position swap
  1. input: test prompts, answers of model \(m\) and of the start model A (greedy, same max_new_tokens), a judge that is not \(m\) or A
  2. for each prompt:
  3.   \(v_1 \leftarrow\) judge(prompt, first = answer of \(m\), second = answer of A)
  4.   \(v_2 \leftarrow\) judge(prompt, first = answer of A, second = answer of \(m\)), mapped back
  5.   if \(v_1 = v_2\): count it else: count a tie // the verdict depended on the order
  6.   store the word counts of both answers // needed in Step 7
  7. return \(W, T, L\), the win rate (76.10) and its standard error
# judge.py: position-swapped pairwise judge.  Usage: python judge.py C-b0.1 A
import sys, json, torch
from transformers import AutoTokenizer, AutoModelForCausalLM
a, b = sys.argv[1], sys.argv[2]                  # files gen-<name>.json: list of {prompt, answer}
JUDGE = "Qwen/Qwen2.5-7B-Instruct"
jt = AutoTokenizer.from_pretrained(JUDGE)
jm = AutoModelForCausalLM.from_pretrained(JUDGE, dtype=torch.bfloat16, device_map="auto").eval()
ga, gb = json.load(open(f"gen-{a}.json")), json.load(open(f"gen-{b}.json"))
RUBRIC = ("You compare two answers to the same question. Judge helpfulness, correctness and "
          "clarity. Do not prefer an answer because it is longer. "
          "Answer with one word on the last line: A, B or TIE.")
def ask(q, s1, s2):
    msg = [{"role": "system", "content": RUBRIC},
           {"role": "user", "content": f"Question:\n{q}\n\nAnswer A:\n{s1}\n\nAnswer B:\n{s2}"}]
    enc = jt.apply_chat_template(msg, add_generation_prompt=True, return_tensors="pt",
                                 return_dict=True).to(jm.device)
    with torch.no_grad():
        out = jm.generate(**enc, max_new_tokens=300, do_sample=False)
    w = jt.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True).split()
    last = w[-1].upper().strip(".*:") if w else "TIE"
    return last if last in ("A", "B", "TIE") else "TIE"
rows = []
for x, y in zip(ga, gb):
    assert x["prompt"] == y["prompt"]
    v1 = ask(x["prompt"], x["answer"], y["answer"])
    v2 = {"A": "B", "B": "A", "TIE": "TIE"}[ask(x["prompt"], y["answer"], x["answer"])]
    v = v1 if v1 == v2 else "TIE"
    rows.append({"win": {"A": 1.0, "TIE": 0.5, "B": 0.0}[v],
                 "len_a": len(x["answer"].split()), "len_b": len(y["answer"].split())})
n = len(rows); wr = sum(r["win"] for r in rows) / n
se = (sum((r["win"] - wr) ** 2 for r in rows) / (n - 1) / n) ** 0.5
print(f"{a} vs {b}: win rate {wr:.3f} +- {1.96 * se:.3f} (95%), n = {n}")
json.dump(rows, open(f"judge-{a}-vs-{b}.json", "w"))

Each prompt gives a score of 1 (win), 0.5 (tie) or 0 (loss). The win rate is their mean, and its standard error comes from the spread of these scores:

\[ \text{WR} = \frac{W + \tfrac12 T}{n}, \qquad \mathrm{SE} = \sqrt{\frac{1}{n(n-1)} \sum_{i=1}^{n} \big(s_i - \text{WR}\big)^2}, \quad s_i \in \{0, \tfrac12, 1\} \tag{76.10} \]

Without ties this is the familiar \(\sqrt{\text{WR}(1-\text{WR})/n}\); ties make it a little smaller. Example: 230 wins, 60 ties and 210 losses on 500 prompts give \(\text{WR} = 260 / 500 = 0.52\). The tie-free formula gives \(\mathrm{SE} = 0.022\), so the 95% interval is about \(\pm 0.044\): 0.476 to 0.564 (formula 76.10, which uses the ties, gives \(\pm 0.041\)). That includes 0.5, so this result alone does not show that the model improved.

Step 7: check for length bias

Two things can make a DPO model look better without being better. The data can prefer longer answers (Figure 1), so the model learns to write more. And the judge can prefer longer answers, so it rewards that. Zheng et al. called the second effect verbosity bias [Certain]. Singhal et al. found that in several RLHF setups much of the reward gain came from longer outputs [Certain], and Park et al. studied the same problem for DPO and proposed a length-regularized DPO [Certain].

Three checks, from cheap to careful:

  1. Report the mean length of every model’s answers next to its win rate. If the win rate rose and the length doubled, be suspicious.
  2. Look at length-matched prompts: only the prompts where the two answers have about the same length (for example within 30 words). This is honest but uses few prompts.
  3. Fit a length-controlled win rate. Model the chance that the judge prefers model \(m\) on prompt \(i\) as a logistic function of the length difference, and read off the win rate at a length difference of zero.
\[ P(m \text{ wins on prompt } i) = \sigma\big(w_0 + w_1\, \Delta_i\big), \quad \Delta_i = \frac{\text{len}_m(i) - \text{len}_A(i)}{100}, \qquad \text{WR}_\text{LC} = \sigma(w_0) \tag{76.11} \]
Read the symbols
\(\Delta_i\)
the length difference on prompt \(i\), in hundreds of words
\(w_1\)
how much the judge’s preference depends on length; 0 means no length effect
\(w_0\)
the preference that is left when both answers have the same length
\(\text{WR}_\text{LC}\)
the length-controlled win rate: the predicted win rate at equal length

This is a simplified version of the idea behind length-controlled AlpacaEval, which fits a similar regression with a few more terms and then sets the length term to zero [Certain]. It has limits: if longer answers are really better (they contain needed detail), the correction removes some real quality too. It answers the question “would the judge still prefer this model if the answers had the same length?”.

win rate per length bin (dot size = prompts)fitted \(\sigma(w_0 + w_1 \Delta)\)true win rate at equal length
Figure 5. Simulated judge verdicts (seeded). On each prompt, the length difference is drawn from a normal distribution with the chosen mean and a standard deviation of 60 words; the judge prefers \(m\) with probability \(\sigma(\text{advantage} + \text{bias} \cdot \Delta)\). The fit is formula (76.11), solved with Newton’s method.
How to read Figure 5

Default: model \(m\) is not better (advantage 0), but writes 100 more words on average, and the judge has a length bias of 1 per 100 words. The raw win rate is 0.704 ± 0.040 (95%): a clear “win” that is entirely length. The fitted length-controlled win rate is 0.530, close to the truth of 0.5. Now set the extra length to 0: the raw and the controlled rates agree (0.486 and 0.488), because there is nothing to correct. Set the judge bias to 0 with 100 extra words: the raw rate is 0.496, and the fit finds almost no length effect (\(w_1 = -0.10\)). With only 100 prompts the length-controlled estimate is noisy (0.540 here); length control needs more prompts than a raw win rate.

# lc.py: raw and length-controlled win rate from judge-*.json (formula 76.11)
import sys, json, math
rows = json.load(open(sys.argv[1]))
X = [(r["len_a"] - r["len_b"]) / 100 for r in rows]
Y = [r["win"] for r in rows]                     # 1, 0.5 or 0; a tie counts as half a win
sig = lambda u: 1 / (1 + math.exp(-u))
w0 = w1 = 0.0
for _ in range(30):                              # Newton's method for logistic regression
    g0 = g1 = h00 = h01 = h11 = 0.0
    for x, y in zip(X, Y):
        p = sig(w0 + w1 * x); e = y - p; s = p * (1 - p)
        g0 += e; g1 += e * x; h00 += s; h01 += s * x; h11 += s * x * x
    det = h00 * h11 - h01 * h01
    w0 += (h11 * g0 - h01 * g1) / det; w1 += (h00 * g1 - h01 * g0) / det
near = [y for x, y in zip(X, Y) if abs(x) < 0.3]
print(f"raw {sum(Y)/len(Y):.3f}  length-controlled {sig(w0):.3f}  w1 {w1:.3f}  "
      f"length-matched {sum(near)/max(1,len(near)):.3f} on {len(near)} prompts")

Two more fixes, if the length check fails: train on the length-filtered pairs from Step 1, or use a length-normalized loss. TRL’s loss_type="sigmoid_norm" divides the log-probabilities by the number of answer tokens, as SimPO does with its reference-free loss [Certain]. Then rerun Steps 6 and 7.

Step 8: compare with SFT only and decide

Now fill the table: pair accuracy, mean length, raw and length-controlled win rate against A, and KL, for B, C (best \(\beta\)) and D. Compare B with C directly too (run judge.py with B as the second model), because both may beat A while one is clearly better than the other.

What earlier work found, so you know what to look for:

  • In the DPO paper, Preferred-FT (SFT on the chosen answers) did not improve much over the SFT model on TL;DR summarization, while DPO reached a win rate of about 61% against the reference summaries at temperature 0, above PPO at 57% [Certain].
  • Zephyr reported that DPO after SFT gave a clearly better chat model than SFT alone on MT-Bench and AlpacaEval [Likely]. Those are judge-based benchmarks, so length can play a role.
  • The toy of Figure 3 shows the same pattern in a small form: SFT on chosen moves little (KL 0.142, true win rate 0.585), while DPO moves further and gains more, but also follows the labeler’s length bias.

Your 0.5B model, data size and judge are different, so do not assume the answer. A short decision memo, half a page, makes the result reviewable:

  1. Goal. Example: “a better general assistant at the same answer length and the same serving cost.”
  2. Results. The table, with 95% intervals and the number of test prompts.
  3. Decision. B, C or D, with its \(\beta\). If the length-controlled gain of C or D over B is inside the noise, prefer B: it is simpler and cheaper to train.
  4. Checks. Read 20 answer pairs yourself. Check a few capabilities the preference data did not cover (for example math or code) for regressions (session 61).

Where this is used

SettingHow DPO is usedWhat to watch
Open chat models (Zephyr, Tülu 3 and others)SFT, then DPO or a variant on pairs from UltraFeedback-style dataLength growth; judge-based benchmarks need length control
Small on-device modelsShort DPO stage after SFT, often with LoRASmall models forget easily; check capabilities outside the preference data [Likely]
Safety and refusal tuningPairs of safe versus unsafe answers (session 74)Over-refusal: count refusals on harmless prompts
Summarization stylePairs that prefer faithful, concise summariesThe DPO paper used \(\beta = 0.5\) here; judges prefer long summaries
VLM hallucination reductionPairs of image descriptions with and without invented objects; TRL’s DPOTrainer supports VLMsTruncation can cut image tokens; TRL recommends max_length=None for VLMs [Certain]
Video captions and key-moment descriptions for Video-RAGPairs that prefer captions grounded in the frames over generic or invented onesThe judge must see the frames or a transcript, and long captions look better to judges [Likely]
Video question answering and action recognition with a VLMDPO on correct versus wrong answers about actions or events in a clipShort answers: length bias is small, but position bias in the judge remains [Likely]

Common mistakes

  1. Training on ties. Pairs with equal scores (12.2% of ultrafeedback_binarized) have random labels. Drop them.
  2. A reference that is not the starting model. The loss must start at 0.693 and the implicit rewards at 0. If they do not, the reference or the masking is wrong.
  3. Counting prompt tokens in the log-probabilities. Formula (76.3) sums over answer tokens only. Including the prompt adds the same number to both sides for the policy and for the reference, so it cancels in theory, but padding and truncation errors then hide easily [Likely].
  4. Too high a learning rate. DPO uses much smaller rates than SFT (the TRL default is \(10^{-6}\) for full fine-tuning). A loss near 0 after a few hundred steps usually means overfitting, not success.
  5. Reading reward accuracy as quality. It only measures ranking of training-style pairs. Both log-probabilities can fall while it rises (Figure 2).
  6. Choosing \(\beta\) by the raw judge score. If the data and the judge share a length bias, the raw score favors the model that moved most toward long answers (Figure 4). Use the length-controlled win rate.
  7. No SFT-only baseline. Without model B you cannot tell how much of the gain comes from the pairs and how much from simply seeing good answers.
  8. Different decoding settings per model. Different temperatures or token limits change the answers more than a small training change does.

Interview questions

What are the four log-probabilities DPO needs per pair, and why is the reference model needed at all?basic
The summed answer log-probabilities of the chosen and of the rejected answer, under the trained policy and under the frozen reference. The reference turns raw log-probabilities into log-ratios (implicit rewards). Without it, the loss would only compare the two answers’ raw likelihoods, which depend strongly on length and on how common the words are, and there would be nothing that keeps the model near its starting point. The reference plays the role of the KL anchor of RLHF; \(\beta\) sets its strength.
Your DPO run starts with a loss of 0.52 instead of 0.693. What is wrong?basic
At step 0 the policy equals the reference, so both implicit rewards are 0, the margin is 0 and the loss is \(\log 2 = 0.693\). A different value means the policy and the reference differ at the start: a wrong reference checkpoint, dropout switched on in one of them, different dtypes or attention implementations, different masking or padding for the two models, or precomputed reference log-probs from different data. TRL switches dropout off by default for this reason.
During DPO, both logps/chosen and logps/rejected go down. Is that a bug?core
No, it is the usual behaviour. The loss only depends on the difference of the two log-ratios, so the cheapest way to raise the margin is often to push the rejected answer down more than the chosen one. The probability that leaves both goes to other answers, outside the data. In our CPU run with learning rate \(2 \times 10^{-5}\), both implicit rewards ended below zero while the accuracy rose. It becomes a problem when the chosen log-probs fall a lot: then the model may also lose good behaviour. Fixes include a larger \(\beta\), a lower learning rate, adding an SFT term on the chosen answers (TRL can combine loss types, for example ["sigmoid", "sft"]), or DPO-Positive-style penalties.
What does \(\beta\) control? What do you expect when you lower it?core
\(\beta\) is the weight of the KL anchor in the RLHF objective that DPO solves. The best policy is \(\pi_\text{ref}\exp(r/\beta)\) up to normalization, so for fixed reward differences a smaller \(\beta\) gives larger log-ratios: the model moves further from the reference. Lower \(\beta\) means a larger KL, more aggressive use of the pairs, and more risk of following noise or biases in them (for example length). In our toy with a length-biased labeler, the KL rose from 0.13 at \(\beta = 2\) to about 2.2 at \(\beta = 0.05\), and the true win rate peaked at \(\beta = 0.2\) while the biased judge preferred the smallest values.
Why is SFT on the chosen answers a necessary baseline, and how does it differ from DPO?core
It tells you how much of the improvement comes from simply imitating good answers, which is cheaper and simpler. SFT on chosen raises the likelihood of the chosen answers but never uses the rejected ones, so it does not learn what to avoid; it often raises the rejected answers’ likelihood too, because they are similar. DPO uses the contrast, with weights that focus on pairs the model still ranks wrongly. The DPO paper found that this “Preferred-FT” baseline improved little over the SFT model on summarization, while DPO improved clearly. You still need the baseline in your own setting, because the result depends on the data.
How do you compute a win rate with an LLM judge so that it is trustworthy?core
Use a judge that is not one of the contestants, a fixed rubric, greedy judge decoding, and the same decoding settings for all models. Judge each prompt twice with the answer order swapped, and count disagreements as ties (this removes position bias). Report the win rate \((W + T/2)/n\) with a standard error and the number of prompts. Report answer lengths and a length-controlled win rate. Read a sample of verdicts yourself. If the judge model also produced the preference labels, its biases are in the data and in the evaluation, so use a different judge if you can.
Explain a length-controlled win rate. When can it be wrong?advanced
Fit a logistic regression of the per-prompt judge outcome on the length difference of the two answers, \(\sigma(w_0 + w_1\Delta)\), and report \(\sigma(w_0)\), the predicted win rate when the lengths are equal. It removes the part of the preference that the judge gives for length alone. It can be wrong in two ways. If the extra length is real value (a needed example, a missing step), the correction removes real quality. And if the two models almost never have similar lengths, the estimate at \(\Delta = 0\) is an extrapolation and noisy. Length-controlled AlpacaEval uses a richer version of the same idea.
How much memory and compute does DPO need compared with SFT for the same model?core
Memory: the policy as in SFT (about 16 bytes per parameter with mixed-precision Adam), plus a frozen reference in BF16 (2 bytes per parameter), plus activations for two sequences per pair. For Qwen2.5-0.5B that is about 8.9 GB before activations. With LoRA the reference is the base with the adapter off, so there is no extra copy. Compute: policy forward and backward on both answers plus a reference forward on both, about \(8N(T_w + T_l)\) FLOPs per pair against \(6NT_w\) for an SFT example; 2.7 times as much at equal lengths. Precomputing the reference log-probs removes the reference from memory during training.
With very few pairs, DPO pushed all probability onto one strange answer. Why can that happen?advanced
The DPO loss of a pair keeps falling as the margin grows, and only a margin of infinity gives zero loss. If an answer is chosen in its pairs and never rejected, its log-ratio can grow without limit; only \(\beta\), the learning rate and the number of steps slow it down. With little data this happens more often, and the answer may be one that a biased labeler liked (in our toy, the longest answer, with true quality 0). IPO replaces the log-sigmoid with a squared loss toward a fixed target margin, which bounds this effect; more data, a larger \(\beta\), early stopping, and on-policy pairs also help.
What changes if you build pairs from your own model’s samples instead of using UltraFeedback?advanced
The pairs become on-policy: both answers are things your model actually writes, so the gradient moves probability among answers it really produces, and the reference log-probs are not tiny. The ranking now comes from your judge or reward model, so its biases (length, style, position) go straight into the data. You need enough diversity (temperature sampling, several samples), a minimum score gap to avoid noise, and a check for truncated answers. Repeating this loop (sample, judge, DPO, sample again) is iterative or online DPO, which moves DPO closer to RLHF.

Exercises

  1. A pair has \(\log \pi_\theta(y_w) = -80\), \(\log \pi_\text{ref}(y_w) = -82\), \(\log \pi_\theta(y_l) = -95\), \(\log \pi_\text{ref}(y_l) = -93\), \(\beta = 0.1\). Compute the implicit rewards, the margin, the loss and the gradient weight.
  2. Using formula (76.6), how many times more compute does a DPO pair need than an SFT example when the chosen answer has 200 tokens and the rejected one 400?
  3. In Figure 1, which filter keeps at least 20,000 pairs while making the chosen answer longer in less than 50% of them?
  4. In Figure 3, set \(a = 1.0\), 400 pairs, \(\beta = 0.1\). What happens to the true and the judge win rate compared with \(a = 0.6\)? Explain why the gap between them grows.
  5. Your judge gives 180 wins, 140 ties and 180 losses on 500 prompts. What is the win rate, and what can you conclude?
  6. In Figure 5, find a setting where the raw win rate is above 0.9 but the true advantage is the smallest non-zero value. What does the length-controlled estimate say?
Answerscheck
1. \(\hat r_w = 0.1 \times 2 = 0.2\), \(\hat r_l = 0.1 \times (-2) = -0.2\), margin 0.4, loss \(-\log\sigma(0.4) = 0.513\), weight \(\sigma(-0.4) = 0.401\). 2. DPO: \(8N \times 600 = 4{,}800N\); SFT: \(6N \times 200 = 1{,}200N\); ratio 4.0. 3. Minimum gap 0 or “more than 0” with a length ratio of at most 1.5: 25,038 pairs at 49.5%, or 21,185 pairs at 49.8%. 4. At \(a = 1.0\) the true win rate is 0.662 and the judge win rate 0.794 (at \(a = 0.6\): 0.669 and 0.752). The labeler is more length-biased, so DPO again settles on the long answer 6 (99.7%), and the judge, with the same stronger bias, rewards that length more. 5. \((180 + 70)/500 = 0.5\): no measured difference. Here the 95% interval from formula (76.10) is about \(\pm 0.037\), so a true change larger than about 4 points would probably have been seen. 6. Advantage 0.2, extra length 150 words, judge bias 1.5, 500 prompts: the raw win rate is 0.904 against a true 0.55 at equal length. The length-controlled estimate is 0.664: much closer, but not exact, because with a mean difference of 150 words only 15 prompts have nearly equal lengths, so \(\Delta = 0\) is at the edge of the data.

Key takeaways

  • A DPO pair needs four summed answer log-probabilities (76.3); the loss (76.1) is a log-sigmoid of the difference of two implicit rewards (76.2) and always starts at 0.693.
  • Clean the data first: drop ties, consider a minimum score gap, and measure how often the chosen answer is longer (53.5% in ultrafeedback_binarized, 45.6% after keeping only length-matched pairs).
  • Rising reward accuracy is not better generation. Both log-probabilities often fall; check with generated answers.
  • \(\beta\) sets how far the model moves (76.8). Sweep it and choose it with a length-controlled metric, because a biased judge favors the model that moved most toward long answers.
  • Measure win rates with a position-swapped judge and an error bar (76.10), and report a length-controlled win rate (76.11) and mean answer lengths next to it.
  • Always train an SFT-on-chosen baseline. DPO costs about 2.7 times more compute per example (76.6) and a reference model in memory (76.7); it must earn that.

Key formulas

\[ \mathcal{L}_\text{DPO} = -\log\sigma(\hat r_w - \hat r_l), \quad \hat r = \beta\big[\log\pi_\theta(y \mid x) - \log\pi_\text{ref}(y \mid x)\big], \quad \nabla \propto -\beta\,\sigma(-m)\,(\nabla\log\pi_\theta(y_w) - \nabla\log\pi_\theta(y_l)) \]
\[ \pi^* \propto \pi_\text{ref}\, e^{r/\beta}, \quad \text{WR} = \frac{W + T/2}{n}, \quad \text{WR}_\text{LC} = \sigma(w_0) \text{ from } \sigma(w_0 + w_1\Delta), \quad C_\text{DPO} \approx 8N(T_w + T_l) \]

References for this session

  1. Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  2. Hugging Face (code). TRL library
  3. Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  4. Hugging Face (code). TRL library
  5. Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback
  6. Allal et al. (2025). SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
  7. Dubois et al. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
  8. Singhal et al. (2023). A Long Way to Go: Investigating Length Correlations in RLHF
  9. Park et al. (2024). Disentangling Length from Quality in Direct Preference Optimization
  10. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
  11. Tunstall et al. (2023). Zephyr: Direct Distillation of LM Alignment
  12. Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
  13. Meng et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward
  14. Azar et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences
  15. Lambert (2025). Reinforcement Learning from Human Feedback (book)