DPO: alignment without a separate reward model
Classic RLHF trains a reward model and then runs PPO against it, with four models in memory and a sampling loop. DPO (Direct Preference Optimization) shows that the best policy of the KL-regularized RLHF objective has a closed form, so the reward can be written as a log-ratio of two language models. Put that into the Bradley–Terry preference model and RLHF becomes one classification-style loss on preference pairs.
What you will learn
- The KL-regularized RL objective of RLHF, and why its best policy is the reference policy tilted by exp(r / β)
- How to turn the closed-form policy around: any reward is β times a log-ratio of policies plus a term that depends only on the prompt
- Why that prompt-only term (the partition function) cancels in the Bradley–Terry model, and how this gives the DPO loss
- The DPO gradient: which answers it pushes up and down, and why the weight σ(r̂_l − r̂_w) matters
- What β and the reference model do, and how to read the training logs (implicit rewards, margins, accuracy, log-probabilities)
- What DPO cannot do: it learns only from a fixed (offline) set of pairs, so it cannot discover good answers that are not in the data
- A full DPO training step in pseudo-code and in runnable PyTorch, with memory numbers compared to PPO-RLHF
Why this matters
In session 67 we trained a reward model on pairs of answers (“A is better than B”), and in session 66 we used PPO to push a language model toward high reward while a KL penalty kept it close to a frozen reference. That pipeline works, but it is heavy: two training stages, four models in GPU memory, and a slow loop that must generate new answers at every step. DPO (Direct Preference Optimization, Rafailov et al., 2023) gets the same objective with one loss on the preference pairs, no reward model, no sampling and no critic.
The trick is a piece of algebra. The RLHF objective has a best policy that we can write in closed form. Turn that formula around, and the reward becomes a log-ratio between the policy and the reference model. Put this reward into the Bradley–Terry model of preferences, and an unknown normalizing term cancels. What is left is a logistic loss that we can minimize with plain gradient descent, like supervised fine-tuning. The paper’s subtitle says it: “Your language model is secretly a reward model”. [Certain]
- Simple. A DPO training step is two forward passes (chosen and rejected answer) through the policy, plus log-probabilities from a frozen reference model. No generation during training.
- Widely used. Zephyr-7B was trained with DPO on AI-ranked answers (UltraFeedback, \(\beta = 0.1\)). [Certain] Every round of Llama 3 post-training was SFT followed by DPO; Meta reports that DPO “required less compute for large-scale models and performed better” than PPO in their tests, especially on instruction following. [Certain] Tülu 3 uses a length-normalized DPO in its pipeline. [Likely]
- Limits. DPO learns only from a fixed set of pairs (offline data). It cannot explore, and careful comparisons find that well-tuned PPO can still beat it, for example on code generation. [Certain] Knowing exactly why is a common interview topic.
The starting point: the KL-regularized objective
Let us recall the setting from session 67 and session 66. A prompt \(x\) comes from a dataset. The policy \(\pi_\theta(y \mid x)\) is our language model; it gives a probability to every full answer \(y\). The reference policy \(\pi_\text{ref}\) is a frozen copy of the model before this stage, usually the SFT model from session 54. A reward function \(r(x, y)\) says how good an answer is. RLHF maximizes
- Read the symbols
- \(r(x, y)\)
- the reward of answer \(y\) to prompt \(x\); in classic RLHF this is the learned reward model
- \(\KL(\pi \,\|\, \pi_\text{ref})\)
- \(\sum_y \pi(y \mid x) \log \frac{\pi(y \mid x)}{\pi_\text{ref}(y \mid x)}\), how far the policy has moved from the reference (see session 3)
- \(\beta\)
- a positive number, the price of moving away from the reference; large \(\beta\) means a short leash
The KL term matters for two reasons. The reward model was trained on answers that look like the SFT model’s answers, so it is only accurate near \(\pi_\text{ref}\); far away, the policy finds answers that fool it (reward hacking). And without the KL term, the best policy would put all its probability on one answer, losing all diversity. [Certain]
In PPO-RLHF we never solve (68.1) exactly. We sample answers, score them, and take small policy-gradient steps. DPO starts from a different question: what is the exact best policy for (68.1), if we could search over all possible distributions?
The best policy has a closed form
Fix one prompt \(x\) and drop it from the notation. Write the objective as one expectation over \(y \sim \pi\), and use \(\log a - \log b = \log(a/b)\):
In the last step we divided \(\pi_\text{ref}(y)\, e^{r(y)/\beta}\) by its total \(Z\), so that it becomes a probability distribution, and called it \(\pi^*\):
- Read the symbols
- \(\pi^*\)
- the best possible policy for objective (68.1) with this reward and this \(\beta\)
- \(Z(x)\)
- the partition function: the number that makes the probabilities sum to 1. It depends on the prompt only, but it is a sum over every possible answer, so for a language model we can never compute it
- \(e^{r/\beta}\)
- a multiplier for each answer: answers with high reward get their reference probability multiplied by a large number
Now read (68.2). The term \(\beta \log Z\) does not depend on \(\pi\). A KL divergence is never negative and is zero only when the two distributions are equal. So the objective is largest exactly when \(\pi = \pi^*\), and its best value is \(\beta \log Z\). We have solved RLHF in one line, at least on paper. [Certain]
So the best policy is a re-weighting of the reference: it never gives probability to an answer that \(\pi_\text{ref}\) gives zero probability, and \(\beta\) sets how strongly the reward re-weights. As \(\beta \to \infty\), \(\pi^* \to \pi_\text{ref}\). As \(\beta \to 0\), \(\pi^*\) puts all its probability on the answers with the highest reward.
One prompt with three answers. Reference probabilities \(\pi_\text{ref} = (0.5, 0.3, 0.2)\), rewards \(r = (0, 1, 2)\).
Multipliers \(e^{r/\beta} = (1, 2.718, 7.389)\). Unnormalized weights: \(0.5 \times 1 = 0.5\), \(0.3 \times 2.718 = 0.815\), \(0.2 \times 7.389 = 1.478\). Their sum is \(Z = 2.793\).
So \(\pi^* = (0.179, 0.292, 0.529)\). The best answer went from 20% to 53%, but the worst one still keeps 18%. The expected reward is \(1.350\), the KL to the reference is \(0.323\) nats, and the objective is \(1.350 - 0.323 = 1.027 = \log Z\), as (68.2) promised.
With \(\beta = 0.5\) the leash is looser: \(\pi^* = (0.037, 0.163, 0.801)\), expected reward \(1.764\), KL \(0.916\) nats. With \(\beta = 2\): \(\pi^* = (0.325, 0.322, 0.353)\), KL only \(0.084\).
How to read Figure 2
Start at \(\beta = 0.5\). Answer D has the highest reward but a small reference probability (0.10); the best policy raises it to about 0.65. Move \(\beta\) to 5: the blue bars almost match the grey ones, because moving away is expensive. Move \(\beta\) to 0.1: almost all probability goes to D. Now lower D’s reward below C’s: the winner changes. Notice that no blue bar is ever zero when its grey bar is not zero: the best policy only re-weights the reference.
Turning it around: the reward is a log-ratio
Formula (68.3) maps a reward to a policy. Take the logarithm of both sides and solve for the reward:
This is the key idea of DPO. If we know the best policy and the reference, we know the reward, up to one number per prompt, \(\beta \log Z(x)\). Check it on the worked example with \(\beta = 1\): \(\log(0.179/0.5) + \log 2.793 = -1.027 + 1.027 = 0\) and \(\log(0.529/0.2) + 1.027 = 0.973 + 1.027 = 2.000\). These are the rewards we started from.
That unknown \(\beta \log Z(x)\) looks like a problem: we cannot compute it. But we do not need to. The next step only ever uses differences of rewards for the same prompt, and in a difference it cancels.
Bradley–Terry, and the DPO loss
From session 67: the Bradley–Terry model says that when a person compares two answers \(y_1\) and \(y_2\) to the same prompt, the probability that they prefer \(y_1\) is a sigmoid of the reward difference:
A reward model is trained by maximizing the log of this probability on human-labeled pairs \((x, y_w, y_l)\), where \(y_w\) is the preferred (“winner”, chosen) answer and \(y_l\) the other one (“loser”, rejected). Now put (68.4) into (68.5). The two \(\beta \log Z(x)\) terms are equal (same prompt) and cancel:
The human preference probability is now written with policies only. So instead of fitting a reward model and then finding its best policy, we fit the policy itself by maximum likelihood on the pairs. Replace \(\pi^*\) by our model \(\pi_\theta\) and take the negative log-likelihood. This is the DPO loss:
- Read the symbols
- \(\mathcal{D}\)
- a fixed dataset of preference triples: a prompt, a chosen answer, a rejected answer
- \(\pi_\theta(y \mid x)\)
- the probability our model gives to the whole answer; in practice we use its log, a sum over tokens (formula 68.10)
- \(\pi_\text{ref}\)
- the frozen reference, usually the SFT model; also the starting point of \(\pi_\theta\)
- \(\beta\)
- the same KL price as in (68.1); typical values are 0.1 to 0.5 [Likely]
- \(\log \sigma(\cdot)\)
- the log-likelihood of the observed preference; the loss is a logistic regression loss
Each term inside the sigmoid has a name. The implicit reward of an answer is
so the DPO loss is just the reward-model loss of session 67, \(-\log \sigma(\hat r_\theta(x, y_w) - \hat r_\theta(x, y_l))\), with a special choice of reward model: one whose reward is \(\beta\) times a log-ratio of two language models. If this reward model fits the data perfectly, its best policy, by (68.3), is \(\pi_\theta\) itself. We train a reward model and get its optimal policy for free. [Certain]
- start: \(\max_\pi \E_\pi[r] - \beta\,\KL(\pi \,\|\, \pi_\text{ref})\) // RLHF objective (68.1)
- rewrite it as \(-\beta\,\KL(\pi \,\|\, \pi^*) + \beta \log Z\) // (68.2)
- so the best policy is \(\pi^* \propto \pi_\text{ref}\, e^{r/\beta}\) // (68.3)
- invert: \(r = \beta \log(\pi^*/\pi_\text{ref}) + \beta \log Z(x)\) // (68.4)
- plug into Bradley–Terry; \(\beta \log Z(x)\) cancels; replace \(\pi^*\) by \(\pi_\theta\); minimize the negative log-likelihood // (68.6), (68.7)
What the gradient does
Write \(m = \hat r_\theta(x, y_w) - \hat r_\theta(x, y_l)\) for the margin of one pair. The loss of the pair is \(-\log \sigma(m)\), and its derivative with respect to \(m\) is \(-\sigma(-m)\). By the chain rule,
Read it in three parts. Gradient descent moves against this gradient, so it raises \(\log \pi_\theta(y_w)\) and lowers \(\log \pi_\theta(y_l)\). The size of the push is the weight \(\sigma(\hat r_l - \hat r_w)\): close to 1 when the implicit reward orders the pair wrongly, close to 0 when it already orders it right by a wide margin. Pairs that are already “solved” stop contributing. And the factor \(\beta\) scales everything.
The weight is important. The DPO paper reports that a naive version, which only raises \(y_w\) and lowers \(y_l\) without this weight (an “unlikelihood”-style loss), makes the language model degenerate. [Certain] Without the weight, the model keeps pushing \(\log \pi_\theta(y_l)\) toward minus infinity, and since the two answers share many tokens, it damages the chosen answers too.
Note also what the gradient does not say. It controls only the difference of the two log-ratios. Nothing stops both log-probabilities from going down, as long as the rejected one goes down faster. This happens in practice and is a known failure mode, especially when the chosen and rejected answers differ in only a few tokens (Pal et al., 2024). [Certain] Figure 3 shows it for one pair, and Figure 4 shows it during training.
The reference gives the chosen answer \(\log \pi_\text{ref}(y_w) = -40.0\) and the rejected answer \(\log \pi_\text{ref}(y_l) = -38.0\) (sums of token log-probabilities). At the start, \(\pi_\theta = \pi_\text{ref}\): both implicit rewards are 0, the margin is 0, the loss is \(\log 2 = 0.693\), and the weight is \(\sigma(0) = 0.5\).
After some training, \(\log \pi_\theta(y_w) = -39.0\) and \(\log \pi_\theta(y_l) = -39.5\). Implicit rewards: \(\hat r_w = 0.1 \times (-39.0 + 40.0) = 0.10\), \(\hat r_l = 0.1 \times (-39.5 + 38.0) = -0.15\). Margin \(0.25\), loss \(-\log \sigma(0.25) = 0.576\), the model’s preference probability \(\sigma(0.25) = 0.562\), and the weight drops to \(0.438\).
A different outcome: \(\log \pi_\theta(y_w) = -42\) and \(\log \pi_\theta(y_l) = -46\). Now \(\hat r_w = -0.2\), \(\hat r_l = -0.8\), margin \(0.6\), loss \(0.438\): lower than before. Yet the chosen answer became \(e^{2} \approx 7.4\) times less likely. The loss is happy; you may not be.
How to read Figure 3
The default is the worked example: margin 0.25, loss 0.576. Push the chosen slider up: the dots move right, the loss falls and the weight shrinks, so this pair will soon stop mattering. Now set the chosen change to −2 and the rejected change to −8 (the third case of the worked example): the margin is 0.6 and the loss 0.438, lower than at the start, but the readout warns that the chosen answer is less likely than under the reference. Then raise \(\beta\) to 0.5 with the same sliders: the same change in log-probabilities now gives a five times larger margin, so the loss saturates after much smaller moves. That is how a large \(\beta\) keeps the policy close to the reference.
From pairs to tokens: computing the loss on a real model
A language model gives probabilities to tokens, not to whole answers. The log-probability of an answer \(y = (y_1, \dots, y_T)\) is the sum over its tokens:
So every implicit reward costs one forward pass of the policy and one of the reference over prompt plus answer. Only the answer tokens are summed; the prompt tokens are masked out, exactly like the loss mask in session 54. Since the reference is frozen, its log-probabilities for the whole dataset can be computed once before training and stored, so only the policy needs to be in GPU memory during training.
- input: a batch of triples \((x, y_w, y_l)\); policy \(\pi_\theta\); frozen \(\pi_\text{ref}\); \(\beta\)
- build two sequences per triple: \([x; y_w]\) and \([x; y_l]\), with a mask that is 1 on answer tokens only
- \(\ell_w, \ell_l \leftarrow\) masked sums of token log-probs under \(\pi_\theta\) // (68.10), with gradient
- \(\ell^\text{ref}_w, \ell^\text{ref}_l \leftarrow\) the same under \(\pi_\text{ref}\) // no gradient; can be precomputed
- \(\hat r_w \leftarrow \beta(\ell_w - \ell^\text{ref}_w)\), \(\;\hat r_l \leftarrow \beta(\ell_l - \ell^\text{ref}_l)\) // implicit rewards (68.8)
- \(\mathcal{L} \leftarrow \text{mean}\big(-\log\sigma(\hat r_w - \hat r_l)\big)\) // (68.7)
- backward, clip, AdamW step // as in Session 4
- log: mean \(\hat r_w\), mean \(\hat r_l\), margin, accuracy \(= \text{mean}[\hat r_w \gt \hat r_l]\), and \(\ell_w, \ell_l\)
Figure 4 runs Algorithm 2 live on a toy problem where we know everything: one prompt and the five answers A to E of Figure 5 below, with reference probabilities \((0.30, 0.30, 0.25, 0.10, 0.05)\) and hidden true rewards \((0, 0.5, 1, 2, -1)\). The simulation samples a dataset of pairs: two random answers, labeled with the Bradley–Terry probability of the true rewards (68.5), and then flipped with probability “label noise”. The policy is a softmax over five logits \(\theta\), so an “answer” is a single token and \(\log \pi_\theta(y) = \theta_y - \log \sum_k e^{\theta_k}\). For this policy \(\nabla_\theta \log \pi_\theta(y_w) - \nabla_\theta \log \pi_\theta(y_l) = e_{y_w} - e_{y_l}\) exactly (the normalizer cancels), so the gradient step of (68.9) adds the same amount to the chosen logit and subtracts it from the rejected one. We use one pair per step and plain SGD instead of AdamW, so every number in the panel can be checked by hand.
- take the next pair \((y_w, y_l)\) from the shuffled dataset
- \(\rho_w \leftarrow \log \dfrac{\pi_\theta(y_w)}{\pi_\text{ref}(y_w)}\), \(\;\rho_l \leftarrow \log \dfrac{\pi_\theta(y_l)}{\pi_\text{ref}(y_l)}\)
- \(\hat r_w \leftarrow \beta \rho_w\), \(\;\hat r_l \leftarrow \beta \rho_l\) // implicit rewards (68.8)
- \(\ell \leftarrow -\log \sigma(\hat r_w - \hat r_l)\) // (68.7)
- \(w \leftarrow \sigma(\hat r_l - \hat r_w)\) // gradient weight (68.9)
- \(\theta_{y_w} \mathrel{+}= \eta \beta w\); \(\;\theta_{y_l} \mathrel{-}= \eta \beta w\) // SGD step
How to read Figure 4
Step through one pair with Next line: at the start both log-ratios are 0, the loss is \(\ln 2 = 0.693\) and the weight is 0.5, as in the worked example. Then press Play. With the defaults (\(\beta = 0.1\), \(\eta = 1\), noise 0.10, 100 pairs, seed 1) the run ends with loss 0.570 and reward accuracy 77.0%, and the third chart shows the effect from the worked example: the mean log-probability of the chosen answers rises by only 0.015 nats in the first few steps, then falls to −3.01 nats, while the rejected ones fall to −8.27. Why, in this toy? Most answers win some pairs and lose others. DPO keeps raising the answers that win most often (B and D end with 0.52 and 0.46 of the probability), so A and C lose probability even in the pairs where they were chosen. The loss and the accuracy are happy anyway: only the gap matters. Raise \(\beta\) to 0.5: the same data now give a much smaller drop (chosen −0.28, rejected −1.42) and a KL of 0.26 nats instead of 0.93, because a large \(\beta\) saturates the loss after smaller moves. Set the noise to 0 (with \(\beta = 0.5\)): the policy moves close to the dashed \(\pi^\ast\) (D gets 0.80, \(\pi^\ast\) says 0.65; finite data pushes too far, as in Figure 5). (Speeds 1 and 2 run one line of the algorithm at a time, 0.65 s or 0.2 s per line; speeds 3 to 5 run whole iterations, more of them per frame. “Next line” and the next-step button work at any speed.)
The PyTorch version below trains a tiny causal model on 8 pairs, then checks the theory: on exact, noise-free preference data, the DPO minimum is the closed-form \(\pi^*\) of (68.3).
import copy, torch import torch.nn.functional as F torch.manual_seed(0) def seq_logp(model, ids, resp_mask): """Sum of log pi(token) over the response tokens only. ids: (B, T). Formula (68.10).""" logits = model(ids[:, :-1]) # predict token t+1 from tokens up to t logp = torch.log_softmax(logits.float(), dim=-1) tok = torch.gather(logp, 2, ids[:, 1:].unsqueeze(-1)).squeeze(-1) return (tok * resp_mask[:, 1:]).sum(-1) # (B,) def dpo_loss(pol_w, pol_l, ref_w, ref_l, beta=0.1): r_w = beta * (pol_w - ref_w) # implicit reward of the chosen answer (68.8) r_l = beta * (pol_l - ref_l) # implicit reward of the rejected answer loss = -F.logsigmoid(r_w - r_l).mean() # formula (68.7); logsigmoid is numerically stable stats = {"margin": (r_w - r_l).mean().item(), "acc": (r_w > r_l).float().mean().item()} return loss, stats # a tiny causal "language model": embedding -> linear (a bigram model) V, T = 16, 8 class Tiny(torch.nn.Module): def __init__(s): super().__init__(); s.emb = torch.nn.Embedding(V, 32); s.out = torch.nn.Linear(32, V) def forward(s, x): return s.out(s.emb(x)) policy = Tiny() ref = copy.deepcopy(policy).eval() # the reference starts equal to the policy for p in ref.parameters(): p.requires_grad_(False) # 8 preference pairs: same 4-token prompt, two different 4-token answers B = 8 prompt = torch.randint(0, V, (B, 4)) chosen = torch.cat([prompt, torch.randint(0, V // 2, (B, 4))], 1) # chosen answers use tokens 0..7 rejected = torch.cat([prompt, torch.randint(V // 2, V, (B, 4))], 1) # rejected answers use tokens 8..15 mask = torch.cat([torch.zeros(B, 4), torch.ones(B, 4)], 1) # 1 on response tokens with torch.no_grad(): # reference log-probs: computed once ref_w, ref_l = seq_logp(ref, chosen, mask), seq_logp(ref, rejected, mask) opt = torch.optim.AdamW(policy.parameters(), lr=1e-2, weight_decay=0.0) for step in range(101): pol_w, pol_l = seq_logp(policy, chosen, mask), seq_logp(policy, rejected, mask) loss, st = dpo_loss(pol_w, pol_l, ref_w, ref_l, beta=0.1) opt.zero_grad(); loss.backward(); opt.step() if step % 50 == 0: print(f"step {step:3d} loss {loss.item():.4f} margin {st['margin']:.3f} acc {st['acc']:.2f} " f"d_logp chosen {(pol_w - ref_w).mean().item():+.2f} rejected {(pol_l - ref_l).mean().item():+.2f}") # theory check: on exact preference data, DPO's optimum is the closed-form pi* (formula 68.3) ref_p = torch.tensor([0.30, 0.30, 0.25, 0.10, 0.05]); r = torch.tensor([0.0, 0.5, 1.0, 2.0, -1.0]); beta = 0.5 pi_star = ref_p * torch.exp(r / beta); pi_star /= pi_star.sum() z = ref_p.log().clone().requires_grad_(True) # a "policy" that is just 5 logits opt = torch.optim.SGD([z], lr=8.0) i, j = torch.meshgrid(torch.arange(5), torch.arange(5), indexing="ij") wpair = ref_p[i] * ref_p[j] * torch.sigmoid(r[i] - r[j]) * (i != j) # P(pair) * P(i preferred to j) wpair /= wpair.sum() for _ in range(300): h = beta * (torch.log_softmax(z, 0) - ref_p.log()) # implicit rewards loss = -(wpair * F.logsigmoid(h[i] - h[j])).sum() opt.zero_grad(); loss.backward(); opt.step() print("DPO ", torch.softmax(z, 0).detach().numpy().round(3)) print("pi* ", pi_star.numpy().round(3))
Output (PyTorch 2.8, CPU):
step 0 loss 0.6931 margin 0.000 acc 0.00 d_logp chosen +0.00 rejected +0.00 step 50 loss 0.0056 margin 5.220 acc 1.00 d_logp chosen +5.21 rejected -46.99 step 100 loss 0.0018 margin 6.348 acc 1.00 d_logp chosen +5.10 rejected -58.38 DPO [0.036 0.097 0.219 0.648 0.001] pi* [0.036 0.097 0.219 0.648 0.001]
Three lessons hide in this output. First, the loss starts at \(\log 2\) because policy and reference are equal; if your first DPO loss is not 0.693, your reference is wrong. Second, accuracy at step 0 is 0, not 0.5: with \(\hat r_w = \hat r_l\) exactly, the strict test \(\hat r_w \gt \hat r_l\) fails. Third, almost all of the margin comes from pushing the rejected answers down (−58 nats) rather than the chosen ones up (+5): making a sequence very unlikely is easy, and the loss does not care which side moves. Between steps 50 and 100 the chosen log-probability even falls a little (from +5.21 to +5.10).
Does DPO find the RLHF solution? Coverage and offline data
The derivation is exact. If the preference data were infinite, labeled by a Bradley–Terry model with a true reward \(r^*\), and covered every pair of answers, then the DPO minimum would be exactly the \(\pi^*\) of (68.3), the same policy that a perfect RLHF run would find. The code above shows this numerically. [Certain]
Real data break the assumptions in two ways.
- Finite data. With a few labels per pair, the empirical preference is often 100% one way. Then the loss keeps falling as the margin grows, and the implicit reward difference goes to infinity, whatever \(\beta\) says. The KL leash of (68.1) is not really enforced. This is the overfitting problem that IPO fixes (session 69). [Certain]
- Coverage. DPO only sees answers that are in the dataset. The pairs were usually generated by some other model (often not your \(\pi_\text{ref}\)), and they are never refreshed. An excellent answer that nobody wrote down gets no direct signal; its probability changes only as a side effect of the others. PPO, in contrast, samples from the current policy, so it can find and reward new good answers. That is the meaning of offline versus online: the data are fixed before training versus produced during training (session 71).
Figure 5 trains DPO on a toy problem where we know everything: one prompt, five answers A to E, a reference policy, and true rewards. Answer D is the best but rare under the reference.
How to read Figure 5
With all pairs and exact labels (infinite data), the blue bars land on the outlines: DPO recovers the RLHF optimum, here D goes from 0.10 to 0.648 at \(\beta = 0.5\), and the KL to \(\pi^*\) is about 0. With 60 sampled pairs (seeded, so the same every time), the labels are noisy and some rare pairs were seen only once, with a one-sided label. After 300 steps D ends at 0.865 instead of 0.648, and the KL to \(\pi^*\) is 0.128 nats: finite data pushes margins further than the true preferences justify. Choose “D never in the data”: the best answer is never compared, so DPO cannot learn that it is good. Its probability falls to about 0.04 while C takes over. An online method that sampled D and scored it could fix this; DPO on this dataset never will.
Closed form: \(e^{r/\beta} = (1, 2.718, 7.389, 54.6, 0.135)\); weights \(\pi_\text{ref}\, e^{r/\beta} = (0.300, 0.815, 1.847, 5.460, 0.007)\); \(Z = 8.43\); \(\pi^* = (0.036, 0.097, 0.219, 0.648, 0.001)\).
DPO with all pairs reaches \((0.036, 0.097, 0.219, 0.648, 0.001)\), the same to three digits. DPO without D reaches \((0.097, 0.263, 0.596, 0.041, 0.002)\): the relative order of A, B, C, E is right, but D, the best answer, lost 60% of its reference probability. The KL from DPO to \(\pi^*\) is 0.85 nats.
The two routes side by side
Figure 6 runs both routes from Figure 1 on the same pairs, from the same reference. The RLHF route has two stages, as in session 67: first it fits a reward model \(r_\phi\) (one score per answer) with the Bradley–Terry loss for 150 iterations; then it trains a policy with the sampled policy gradient of reward minus \(\beta\) times the log-ratio, 16 samples per iteration, with a leave-one-out baseline. The DPO route trains its policy with (68.7) from the first iteration. Both see the same minibatches of 10 pairs and use the same learning rate.
- draw a minibatch of 10 pairs \((y_w, y_l)\)
- RLHF stage 1 (iterations 1–150): \(r_\phi \mathrel{+}= \eta\,\overline{(1 - \sigma(r_w - r_l))(e_{y_w} - e_{y_l})}\) // (67.4)
- RLHF stage 2: sample 16 answers \(y \sim \pi_\text{RL}\); \(R \leftarrow r_\phi(y) - \beta \log \frac{\pi_\text{RL}(y)}{\pi_\text{ref}(y)}\)
- RLHF stage 2: \(\theta_\text{RL} \mathrel{+}= \eta\,\overline{(R - b)(e_y - \pi_\text{RL})}\) // policy gradient
- DPO: \(\theta_\text{DPO} \mathrel{+}= \eta\beta\,\overline{\sigma(\hat r_l - \hat r_w)(e_{y_w} - e_{y_l})}\) // (68.9)
- record the true reward and the KL of both policies
How to read Figure 6
Press Play at speed 4. During the first 150 iterations only DPO’s policy moves; the RLHF policy waits for its reward model (its curve stays flat at the reference value 0.55). Then the RLHF policy catches up quickly. With the defaults (\(\beta = 0.5\), noise 0.10, 100 pairs, seed 1) both end almost at the same place: expected true reward 0.877 (RLHF) and 0.867 (DPO), KL 0.24 and 0.25 nats, and nearly the same bars. Both are well below the \(\pi^\ast\) line (1.562), because they learned from 100 noisy pairs, not from the true reward: on this data D wins only a little more often than the others, so its small reference probability (0.10) keeps it near 0.20. The bottom-right chart shows why the two agree: the reward model and DPO’s implicit reward end up ordering the answers the same way. Set the noise to 0: both reach about 1.72 and put almost 0.79 on D. With 400 noisy pairs they differ more (1.40 against 1.15): the routes are equivalent at their optimum, not step by step, and the RLHF reward model here only gets 150 iterations of training. Lower \(\beta\) and both policies move further from the reference.
β and the reference model
What β does. In theory, \(\beta\) is the KL price in (68.1): small \(\beta\) allows the policy to move far from the reference. In the loss, \(\beta\) multiplies the log-ratios. With a small \(\beta\), the model must change its log-probabilities a lot before the margin is large and the loss saturates; with a large \(\beta\), small changes already saturate the loss, so training stops early and the policy stays close to \(\pi_\text{ref}\). The original paper used \(\beta = 0.1\) for most experiments and \(0.5\) for TL;DR summarization, with RMSprop, learning rate \(10^{-6}\), batch size 64 and 150 warmup steps. [Certain] Zephyr used \(\beta = 0.1\) with a peak learning rate of \(5 \times 10^{-7}\); Llama 3 used \(\beta = 0.1\). [Certain] DPO learning rates are 10 to 100 times smaller than SFT learning rates. [Likely]
What the reference model does. It appears in three places: it is the anchor of the KL in (68.1), it is the starting point of \(\pi_\theta\), and it is the baseline in every implicit reward. Its quality matters because the theory assumes the preference answers look like samples from \(\pi_\text{ref}\). If you take a public preference dataset written by other models, first run SFT on the chosen answers, so that the reference is not far from the data. The DPO paper does exactly this when no SFT model is available. [Certain]
Can we drop it? If we remove \(\pi_\text{ref}\) from (68.7), the loss becomes \(-\log\sigma(\beta[\log\pi_\theta(y_w) - \log\pi_\theta(y_l)])\). Nothing then anchors the model: it can drift arbitrarily, and the unnormalized log-probability prefers short answers. Methods such as ORPO and SimPO remove the reference on purpose and add other fixes (an SFT term, length normalization, a margin). That is the topic of session 69.
What DPO can and cannot do
| PPO-RLHF | DPO | |
|---|---|---|
| Objective | (68.1) with a learned reward model | the same (68.1), solved in closed form |
| Training stages after SFT | 2: reward model, then RL | 1 |
| Models in memory | 4 (policy, critic, reference, reward model) | 2 (policy, reference), or 1 with precomputed log-probs |
| Generation during training | yes, every iteration (usually the slowest part) | no |
| Data | online: fresh samples from the current policy | offline: a fixed set of pairs |
| Hyper-parameters | many (clip, KL coefficient, GAE, critic lr, batch sizes) | few (\(\beta\), lr, epochs) |
| Can explore new answers | yes | no |
| Reusable reward model | yes (also for best-of-N, data filtering) | only implicit, tied to \(\pi_\text{ref}\) |
| Known weak spots | reward hacking, instability, cost | overfitting on one-sided pairs, falling chosen log-probs, length growth, out-of-distribution answers |
A common middle ground is iterative (or online) DPO: train with DPO, sample new answers from the new policy, label them with a reward model or a judge LLM, and run DPO again on the fresh pairs. This brings back part of the exploration of online RL while keeping the simple loss (Xiong et al., 2023). [Certain] Llama 3’s several rounds of SFT and DPO, each on newly collected preference data, follow this pattern. [Likely]
Length. Human and AI judges often prefer longer answers, and the chosen answer in many datasets is longer on average. DPO then learns to write longer answers, even when quality does not improve (Park et al., 2024). [Likely] Length-normalized variants and length-controlled evaluation are the usual fixes.
Where this is used
| Setting | What DPO does there | Notes |
|---|---|---|
| Chat assistants (Zephyr, Llama 3, Tülu 3) | Preference stage after SFT: helpfulness, instruction following, tone | Pairs from humans or from a judge model; \(\beta \approx 0.1\) [Certain for Zephyr and Llama 3] |
| Summarization (TL;DR in the DPO paper) | Prefer summaries that people rate higher | Win rate against reference summaries about 61% for DPO vs 57% for PPO, judged by GPT-4 [Certain] |
| Safety and refusals | Prefer safe answers over harmful ones, and helpful answers over needless refusals | Needs pairs on both sides, or the model over-refuses (session 74) |
| Vision-language models | Pairs of image descriptions: faithful vs hallucinated objects | Same loss; the image is part of \(x\). Used to reduce hallucination in VLMs [Likely] |
| Video captioning and video QA | Prefer captions that match the actual events and their order over captions with invented actions or wrong timing | Rejected answers can be made by editing a correct caption (swap two actions, change a time); keep edits small but watch the falling-likelihood problem [Speculation] |
| Key-moment detection with an LLM head | Prefer answers that cite the right time span | Pairs from a temporal-IoU score; an online method with a verifiable reward is often a better fit (session 70) [Speculation] |
| Code and math | Pairs of correct and incorrect solutions | Works, but online RL with a verifier is now usually preferred for reasoning [Likely] |
Common mistakes
- Summing the prompt tokens into the log-probability. The prompt is the same for both answers and the reference, so it only adds noise and padding bugs. Mask everything except the answer, including padding.
- Averaging token log-probs when you meant to sum. Plain DPO uses the sum (68.10). Averaging changes the method (it is a length-normalized variant) and the meaning of \(\beta\).
- A reference that is not the starting policy. At step 0 the loss must be exactly 0.693 and every implicit reward 0. If not, the reference or the masking is wrong (for example, a different chat template or dropout left on).
- Different templates or tokenization for policy and reference. The log-ratio then measures the template, not the answer. Use the same tokenizer and template (session 9). Llama 3 also masks special formatting tokens out of the DPO loss for stability. [Certain]
- Watching only the loss and accuracy. Both can look great while the chosen log-probability falls and the model gets worse. Log the chosen and rejected log-probs and the KL to the reference; consider adding a small SFT (negative log-likelihood) term on the chosen answers, as Llama 3 did with coefficient 0.2. [Certain]
- Too many epochs or too high a learning rate. DPO overfits fast on one-sided pairs: margins keep growing and generation quality drops. One to three epochs with a learning rate around \(10^{-7}\) to \(10^{-6}\) are common. [Likely]
- Using pairs where both answers are bad, or both good. The loss still pushes them apart. Filter pairs with a small quality gap, or label noise dominates.
- Comparing implicit rewards across prompts. They are defined only up to a per-prompt constant \(\beta\log Z(x)\).
Interview questions
Derive the DPO loss starting from the RLHF objective.core
Why is the partition function \(Z(x)\) not a problem?basic
What does \(\beta\) control, and what happens at the extremes?basic
Explain the DPO gradient. Why is the weight \(\sigma(\hat r_l - \hat r_w)\) important?core
During DPO training, the log-probability of the chosen answers goes down. Is that a bug?advanced
Why is DPO called “offline”, and what does that cost?core
Compare the compute and memory of DPO and PPO-RLHF for a 7B model.core
Why do we need a reference model at all? What if you remove it?core
In what sense is “your language model secretly a reward model”? How would you use it as one?advanced
Your DPO model writes longer and longer answers. Why, and what do you do?core
Under what conditions does DPO give exactly the RLHF solution?advanced
You have only single thumbs-up or thumbs-down labels, not pairs. Can you use DPO?basic
Exercises
- For the three-answer example (\(\pi_\text{ref} = (0.5, 0.3, 0.2)\), \(r = (0, 1, 2)\)), compute \(\pi^*\) and the best objective value for \(\beta = 0.5\).
- One pair with \(\beta = 0.1\): the policy raised the chosen log-probability by 3 nats and lowered the rejected one by 2 nats, relative to the reference. Compute the margin, the loss and the gradient weight.
- Show that adding a prompt-only term \(f(x)\) to the reward changes neither \(\pi^*\) in (68.3) nor the Bradley–Terry probability (68.5).
- What does \(\pi^*\) become as \(\beta \to \infty\) and as \(\beta \to 0\)?
- Training logs with \(\beta = 0.1\) show mean rewards/chosen \(= -0.5\) and rewards/rejected \(= -2.0\). What happened to the log-probabilities? What is \(\sigma\) of the mean margin, and why should this worry you?
- Estimate the memory, before activations, of DPO and of PPO-RLHF for a 70B model (AdamW mixed precision, frozen models in BF16, reference not precomputed).
Answerscheck
Key takeaways
- RLHF maximizes reward minus \(\beta\) times the KL to a reference (68.1). Its exact best policy is the reference re-weighted by \(e^{r/\beta}\) and normalized by \(Z(x)\) (68.3).
- Inverting it, any reward equals \(\beta\log(\pi^*/\pi_\text{ref})\) plus a prompt-only term (68.4). In Bradley–Terry that term cancels (68.6).
- So the reward-model loss, written for this special reward, is the DPO loss (68.7): logistic regression on the difference of implicit rewards (68.8), computed from summed token log-probs (68.10).
- The gradient (68.9) raises the chosen answer, lowers the rejected one, and weights each pair by how wrong it still is. It only controls the difference, so both log-probs can fall (Figure 4).
- DPO needs one stage, two models and no generation, but it is offline: it cannot explore, it overfits one-sided pairs, and it can lose good answers that are not in the data. Iterative DPO and online RL address this. On the same pairs, the two-stage RLHF route and DPO reach nearly the same policy (Figure 6).
Key formulas
References for this session
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Lambert (2025). Reinforcement Learning from Human Feedback (book)
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 10: Alignment
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Lambert (2025). Reinforcement Learning from Human Feedback (book)
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 10: Alignment
- Ouyang et al. (2022). Training language models to follow instructions with human feedback
- Schulman et al. (2017). Proximal Policy Optimization Algorithms
- Tunstall et al. (2023). Zephyr: Direct Distillation of LM Alignment
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Pal et al. (2024). Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
- Xu et al. (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
- Azar et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences
- Xiong et al. (2023). Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
- Park et al. (2024). Disentangling Length from Quality in Direct Preference Optimization
- Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
- Hugging Face (code). TRL library