LLM & VLM courseBlog
68Track 4: Post-training and RL · session 16 of 25

DPO: alignment without a separate reward model

Classic RLHF trains a reward model and then runs PPO against it, with four models in memory and a sampling loop. DPO (Direct Preference Optimization) shows that the best policy of the KL-regularized RLHF objective has a closed form, so the reward can be written as a log-ratio of two language models. Put that into the Bradley–Terry preference model and RLHF becomes one classification-style loss on preference pairs.

Time: 3 to 4 hours (reading 2 to 2.5 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 3, 54, 66 and 67

What you will learn

  • The KL-regularized RL objective of RLHF, and why its best policy is the reference policy tilted by exp(r / β)
  • How to turn the closed-form policy around: any reward is β times a log-ratio of policies plus a term that depends only on the prompt
  • Why that prompt-only term (the partition function) cancels in the Bradley–Terry model, and how this gives the DPO loss
  • The DPO gradient: which answers it pushes up and down, and why the weight σ(r̂_l − r̂_w) matters
  • What β and the reference model do, and how to read the training logs (implicit rewards, margins, accuracy, log-probabilities)
  • What DPO cannot do: it learns only from a fixed (offline) set of pairs, so it cannot discover good answers that are not in the data
  • A full DPO training step in pseudo-code and in runnable PyTorch, with memory numbers compared to PPO-RLHF

Why this matters

In session 67 we trained a reward model on pairs of answers (“A is better than B”), and in session 66 we used PPO to push a language model toward high reward while a KL penalty kept it close to a frozen reference. That pipeline works, but it is heavy: two training stages, four models in GPU memory, and a slow loop that must generate new answers at every step. DPO (Direct Preference Optimization, Rafailov et al., 2023) gets the same objective with one loss on the preference pairs, no reward model, no sampling and no critic.

The trick is a piece of algebra. The RLHF objective has a best policy that we can write in closed form. Turn that formula around, and the reward becomes a log-ratio between the policy and the reference model. Put this reward into the Bradley–Terry model of preferences, and an unknown normalizing term cancels. What is left is a logistic loss that we can minimize with plain gradient descent, like supervised fine-tuning. The paper’s subtitle says it: “Your language model is secretly a reward model”. [Certain]

  • Simple. A DPO training step is two forward passes (chosen and rejected answer) through the policy, plus log-probabilities from a frozen reference model. No generation during training.
  • Widely used. Zephyr-7B was trained with DPO on AI-ranked answers (UltraFeedback, \(\beta = 0.1\)). [Certain] Every round of Llama 3 post-training was SFT followed by DPO; Meta reports that DPO “required less compute for large-scale models and performed better” than PPO in their tests, especially on instruction following. [Certain] Tülu 3 uses a length-normalized DPO in its pipeline. [Likely]
  • Limits. DPO learns only from a fixed set of pairs (offline data). It cannot explore, and careful comparisons find that well-tuned PPO can still beat it, for example on code generation. [Certain] Knowing exactly why is a common interview topic.
Figure 1. Two ways to use the same preference data. Top: classic RLHF trains a reward model, then runs PPO, which samples new answers, scores them and trains a policy and a critic. Bottom: DPO trains the policy directly on the pairs, with a frozen reference model. The memory numbers are for a 7B model trained with AdamW in mixed precision (16 bytes per trained parameter, 2 bytes per frozen one).

The starting point: the KL-regularized objective

Let us recall the setting from session 67 and session 66. A prompt \(x\) comes from a dataset. The policy \(\pi_\theta(y \mid x)\) is our language model; it gives a probability to every full answer \(y\). The reference policy \(\pi_\text{ref}\) is a frozen copy of the model before this stage, usually the SFT model from session 54. A reward function \(r(x, y)\) says how good an answer is. RLHF maximizes

\[ \max_{\pi}\ \E_{x \sim \mathcal{D}}\Big[\ \E_{y \sim \pi(\cdot \mid x)}\big[r(x, y)\big] \;-\; \beta\, \KL\big(\pi(\cdot \mid x)\,\big\|\,\pi_\text{ref}(\cdot \mid x)\big) \Big] \tag{68.1} \]
Read the symbols
\(r(x, y)\)
the reward of answer \(y\) to prompt \(x\); in classic RLHF this is the learned reward model
\(\KL(\pi \,\|\, \pi_\text{ref})\)
\(\sum_y \pi(y \mid x) \log \frac{\pi(y \mid x)}{\pi_\text{ref}(y \mid x)}\), how far the policy has moved from the reference (see session 3)
\(\beta\)
a positive number, the price of moving away from the reference; large \(\beta\) means a short leash

The KL term matters for two reasons. The reward model was trained on answers that look like the SFT model’s answers, so it is only accurate near \(\pi_\text{ref}\); far away, the policy finds answers that fool it (reward hacking). And without the KL term, the best policy would put all its probability on one answer, losing all diversity. [Certain]

In PPO-RLHF we never solve (68.1) exactly. We sample answers, score them, and take small policy-gradient steps. DPO starts from a different question: what is the exact best policy for (68.1), if we could search over all possible distributions?

The best policy has a closed form

Fix one prompt \(x\) and drop it from the notation. Write the objective as one expectation over \(y \sim \pi\), and use \(\log a - \log b = \log(a/b)\):

\[ \E_{\pi}\big[r(y)\big] - \beta\,\E_{\pi}\Big[\log \frac{\pi(y)}{\pi_\text{ref}(y)}\Big] = -\beta\, \E_{\pi}\Big[\log \frac{\pi(y)}{\pi_\text{ref}(y)\, e^{r(y)/\beta}}\Big] = -\beta\, \KL\big(\pi \,\|\, \pi^*\big) + \beta \log Z \tag{68.2} \]

In the last step we divided \(\pi_\text{ref}(y)\, e^{r(y)/\beta}\) by its total \(Z\), so that it becomes a probability distribution, and called it \(\pi^*\):

\[ \pi^*(y \mid x) = \frac{1}{Z(x)}\, \pi_\text{ref}(y \mid x)\, \exp\!\Big(\frac{r(x, y)}{\beta}\Big), \qquad Z(x) = \sum_{y} \pi_\text{ref}(y \mid x)\, \exp\!\Big(\frac{r(x, y)}{\beta}\Big) \tag{68.3} \]
Read the symbols
\(\pi^*\)
the best possible policy for objective (68.1) with this reward and this \(\beta\)
\(Z(x)\)
the partition function: the number that makes the probabilities sum to 1. It depends on the prompt only, but it is a sum over every possible answer, so for a language model we can never compute it
\(e^{r/\beta}\)
a multiplier for each answer: answers with high reward get their reference probability multiplied by a large number

Now read (68.2). The term \(\beta \log Z\) does not depend on \(\pi\). A KL divergence is never negative and is zero only when the two distributions are equal. So the objective is largest exactly when \(\pi = \pi^*\), and its best value is \(\beta \log Z\). We have solved RLHF in one line, at least on paper. [Certain]

So the best policy is a re-weighting of the reference: it never gives probability to an answer that \(\pi_\text{ref}\) gives zero probability, and \(\beta\) sets how strongly the reward re-weights. As \(\beta \to \infty\), \(\pi^* \to \pi_\text{ref}\). As \(\beta \to 0\), \(\pi^*\) puts all its probability on the answers with the highest reward.

Worked example: three answers, \(\beta = 1\)

One prompt with three answers. Reference probabilities \(\pi_\text{ref} = (0.5, 0.3, 0.2)\), rewards \(r = (0, 1, 2)\).

Multipliers \(e^{r/\beta} = (1, 2.718, 7.389)\). Unnormalized weights: \(0.5 \times 1 = 0.5\), \(0.3 \times 2.718 = 0.815\), \(0.2 \times 7.389 = 1.478\). Their sum is \(Z = 2.793\).

So \(\pi^* = (0.179, 0.292, 0.529)\). The best answer went from 20% to 53%, but the worst one still keeps 18%. The expected reward is \(1.350\), the KL to the reference is \(0.323\) nats, and the objective is \(1.350 - 0.323 = 1.027 = \log Z\), as (68.2) promised.

With \(\beta = 0.5\) the leash is looser: \(\pi^* = (0.037, 0.163, 0.801)\), expected reward \(1.764\), KL \(0.916\) nats. With \(\beta = 2\): \(\pi^* = (0.325, 0.322, 0.353)\), KL only \(0.084\).

reference \(\pi_\text{ref}\)best policy \(\pi^*\), formula (68.3)
Figure 2. The closed-form best policy for five answers A to E with fixed reference probabilities and rewards. The \(\beta\) slider is on a log scale (from 0.1 to 5). The readout checks formula (68.2): the expected reward minus \(\beta\) times the KL equals \(\beta \log Z\).
How to read Figure 2

Start at \(\beta = 0.5\). Answer D has the highest reward but a small reference probability (0.10); the best policy raises it to about 0.65. Move \(\beta\) to 5: the blue bars almost match the grey ones, because moving away is expensive. Move \(\beta\) to 0.1: almost all probability goes to D. Now lower D’s reward below C’s: the winner changes. Notice that no blue bar is ever zero when its grey bar is not zero: the best policy only re-weights the reference.

Turning it around: the reward is a log-ratio

Formula (68.3) maps a reward to a policy. Take the logarithm of both sides and solve for the reward:

\[ r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_\text{ref}(y \mid x)} + \beta \log Z(x) \tag{68.4} \]

This is the key idea of DPO. If we know the best policy and the reference, we know the reward, up to one number per prompt, \(\beta \log Z(x)\). Check it on the worked example with \(\beta = 1\): \(\log(0.179/0.5) + \log 2.793 = -1.027 + 1.027 = 0\) and \(\log(0.529/0.2) + 1.027 = 0.973 + 1.027 = 2.000\). These are the rewards we started from.

That unknown \(\beta \log Z(x)\) looks like a problem: we cannot compute it. But we do not need to. The next step only ever uses differences of rewards for the same prompt, and in a difference it cancels.

Bradley–Terry, and the DPO loss

From session 67: the Bradley–Terry model says that when a person compares two answers \(y_1\) and \(y_2\) to the same prompt, the probability that they prefer \(y_1\) is a sigmoid of the reward difference:

\[ p(y_1 \succ y_2 \mid x) = \sigma\big(r(x, y_1) - r(x, y_2)\big), \qquad \sigma(u) = \frac{1}{1 + e^{-u}} \tag{68.5} \]

A reward model is trained by maximizing the log of this probability on human-labeled pairs \((x, y_w, y_l)\), where \(y_w\) is the preferred (“winner”, chosen) answer and \(y_l\) the other one (“loser”, rejected). Now put (68.4) into (68.5). The two \(\beta \log Z(x)\) terms are equal (same prompt) and cancel:

\[ p(y_w \succ y_l \mid x) = \sigma\!\Big( \beta \log \frac{\pi^*(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \beta \log \frac{\pi^*(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)} \Big) \tag{68.6} \]

The human preference probability is now written with policies only. So instead of fitting a reward model and then finding its best policy, we fit the policy itself by maximum likelihood on the pairs. Replace \(\pi^*\) by our model \(\pi_\theta\) and take the negative log-likelihood. This is the DPO loss:

\[ \mathcal{L}_\text{DPO}(\theta) = -\,\E_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\!\Big( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)} \Big) \right] \tag{68.7} \]
Read the symbols
\(\mathcal{D}\)
a fixed dataset of preference triples: a prompt, a chosen answer, a rejected answer
\(\pi_\theta(y \mid x)\)
the probability our model gives to the whole answer; in practice we use its log, a sum over tokens (formula 68.10)
\(\pi_\text{ref}\)
the frozen reference, usually the SFT model; also the starting point of \(\pi_\theta\)
\(\beta\)
the same KL price as in (68.1); typical values are 0.1 to 0.5 [Likely]
\(\log \sigma(\cdot)\)
the log-likelihood of the observed preference; the loss is a logistic regression loss

Each term inside the sigmoid has a name. The implicit reward of an answer is

\[ \hat r_\theta(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_\text{ref}(y \mid x)} \tag{68.8} \]

so the DPO loss is just the reward-model loss of session 67, \(-\log \sigma(\hat r_\theta(x, y_w) - \hat r_\theta(x, y_l))\), with a special choice of reward model: one whose reward is \(\beta\) times a log-ratio of two language models. If this reward model fits the data perfectly, its best policy, by (68.3), is \(\pi_\theta\) itself. We train a reward model and get its optimal policy for free. [Certain]

Algorithm 1: the DPO derivation in five steps
  1. start: \(\max_\pi \E_\pi[r] - \beta\,\KL(\pi \,\|\, \pi_\text{ref})\) // RLHF objective (68.1)
  2. rewrite it as \(-\beta\,\KL(\pi \,\|\, \pi^*) + \beta \log Z\) // (68.2)
  3. so the best policy is \(\pi^* \propto \pi_\text{ref}\, e^{r/\beta}\) // (68.3)
  4. invert: \(r = \beta \log(\pi^*/\pi_\text{ref}) + \beta \log Z(x)\) // (68.4)
  5. plug into Bradley–Terry; \(\beta \log Z(x)\) cancels; replace \(\pi^*\) by \(\pi_\theta\); minimize the negative log-likelihood // (68.6), (68.7)

What the gradient does

Write \(m = \hat r_\theta(x, y_w) - \hat r_\theta(x, y_l)\) for the margin of one pair. The loss of the pair is \(-\log \sigma(m)\), and its derivative with respect to \(m\) is \(-\sigma(-m)\). By the chain rule,

\[ \nabla_\theta \mathcal{L}_\text{DPO} = -\beta\, \E_{\mathcal{D}} \Big[ \underbrace{\sigma\big(\hat r_\theta(x, y_l) - \hat r_\theta(x, y_w)\big)}_{\text{weight}} \Big( \nabla_\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x) \Big) \Big] \tag{68.9} \]

Read it in three parts. Gradient descent moves against this gradient, so it raises \(\log \pi_\theta(y_w)\) and lowers \(\log \pi_\theta(y_l)\). The size of the push is the weight \(\sigma(\hat r_l - \hat r_w)\): close to 1 when the implicit reward orders the pair wrongly, close to 0 when it already orders it right by a wide margin. Pairs that are already “solved” stop contributing. And the factor \(\beta\) scales everything.

The weight is important. The DPO paper reports that a naive version, which only raises \(y_w\) and lowers \(y_l\) without this weight (an “unlikelihood”-style loss), makes the language model degenerate. [Certain] Without the weight, the model keeps pushing \(\log \pi_\theta(y_l)\) toward minus infinity, and since the two answers share many tokens, it damages the chosen answers too.

Note also what the gradient does not say. It controls only the difference of the two log-ratios. Nothing stops both log-probabilities from going down, as long as the rejected one goes down faster. This happens in practice and is a known failure mode, especially when the chosen and rejected answers differ in only a few tokens (Pal et al., 2024). [Certain] Figure 3 shows it for one pair, and Figure 4 shows it during training.

Worked example: one pair, \(\beta = 0.1\)

The reference gives the chosen answer \(\log \pi_\text{ref}(y_w) = -40.0\) and the rejected answer \(\log \pi_\text{ref}(y_l) = -38.0\) (sums of token log-probabilities). At the start, \(\pi_\theta = \pi_\text{ref}\): both implicit rewards are 0, the margin is 0, the loss is \(\log 2 = 0.693\), and the weight is \(\sigma(0) = 0.5\).

After some training, \(\log \pi_\theta(y_w) = -39.0\) and \(\log \pi_\theta(y_l) = -39.5\). Implicit rewards: \(\hat r_w = 0.1 \times (-39.0 + 40.0) = 0.10\), \(\hat r_l = 0.1 \times (-39.5 + 38.0) = -0.15\). Margin \(0.25\), loss \(-\log \sigma(0.25) = 0.576\), the model’s preference probability \(\sigma(0.25) = 0.562\), and the weight drops to \(0.438\).

A different outcome: \(\log \pi_\theta(y_w) = -42\) and \(\log \pi_\theta(y_l) = -46\). Now \(\hat r_w = -0.2\), \(\hat r_l = -0.8\), margin \(0.6\), loss \(0.438\): lower than before. Yet the chosen answer became \(e^{2} \approx 7.4\) times less likely. The loss is happy; you may not be.

loss \(-\log\sigma(m)\)weight \(\sigma(-m)\)
Figure 3. One preference pair. The two top sliders set how much the policy’s log-probability of the chosen and of the rejected answer moved away from the reference (in nats). The plot shows the loss (68.7) and the gradient weight from (68.9) against the margin \(m = \hat r_w - \hat r_l\); the dots mark the current pair.
How to read Figure 3

The default is the worked example: margin 0.25, loss 0.576. Push the chosen slider up: the dots move right, the loss falls and the weight shrinks, so this pair will soon stop mattering. Now set the chosen change to −2 and the rejected change to −8 (the third case of the worked example): the margin is 0.6 and the loss 0.438, lower than at the start, but the readout warns that the chosen answer is less likely than under the reference. Then raise \(\beta\) to 0.5 with the same sliders: the same change in log-probabilities now gives a five times larger margin, so the loss saturates after much smaller moves. That is how a large \(\beta\) keeps the policy close to the reference.

From pairs to tokens: computing the loss on a real model

A language model gives probabilities to tokens, not to whole answers. The log-probability of an answer \(y = (y_1, \dots, y_T)\) is the sum over its tokens:

\[ \log \pi_\theta(y \mid x) = \sum_{t=1}^{T} \log \pi_\theta\big(y_t \mid x, y_{\lt t}\big) \tag{68.10} \]

So every implicit reward costs one forward pass of the policy and one of the reference over prompt plus answer. Only the answer tokens are summed; the prompt tokens are masked out, exactly like the loss mask in session 54. Since the reference is frozen, its log-probabilities for the whole dataset can be computed once before training and stored, so only the policy needs to be in GPU memory during training.

Algorithm 2: one DPO training step
  1. input: a batch of triples \((x, y_w, y_l)\); policy \(\pi_\theta\); frozen \(\pi_\text{ref}\); \(\beta\)
  2. build two sequences per triple: \([x; y_w]\) and \([x; y_l]\), with a mask that is 1 on answer tokens only
  3. \(\ell_w, \ell_l \leftarrow\) masked sums of token log-probs under \(\pi_\theta\) // (68.10), with gradient
  4. \(\ell^\text{ref}_w, \ell^\text{ref}_l \leftarrow\) the same under \(\pi_\text{ref}\) // no gradient; can be precomputed
  5. \(\hat r_w \leftarrow \beta(\ell_w - \ell^\text{ref}_w)\), \(\;\hat r_l \leftarrow \beta(\ell_l - \ell^\text{ref}_l)\) // implicit rewards (68.8)
  6. \(\mathcal{L} \leftarrow \text{mean}\big(-\log\sigma(\hat r_w - \hat r_l)\big)\) // (68.7)
  7. backward, clip, AdamW step // as in Session 4
  8. log: mean \(\hat r_w\), mean \(\hat r_l\), margin, accuracy \(= \text{mean}[\hat r_w \gt \hat r_l]\), and \(\ell_w, \ell_l\)

Figure 4 runs Algorithm 2 live on a toy problem where we know everything: one prompt and the five answers A to E of Figure 5 below, with reference probabilities \((0.30, 0.30, 0.25, 0.10, 0.05)\) and hidden true rewards \((0, 0.5, 1, 2, -1)\). The simulation samples a dataset of pairs: two random answers, labeled with the Bradley–Terry probability of the true rewards (68.5), and then flipped with probability “label noise”. The policy is a softmax over five logits \(\theta\), so an “answer” is a single token and \(\log \pi_\theta(y) = \theta_y - \log \sum_k e^{\theta_k}\). For this policy \(\nabla_\theta \log \pi_\theta(y_w) - \nabla_\theta \log \pi_\theta(y_l) = e_{y_w} - e_{y_l}\) exactly (the normalizer cancels), so the gradient step of (68.9) adds the same amount to the chosen logit and subtracts it from the rejected one. We use one pair per step and plain SGD instead of AdamW, so every number in the panel can be checked by hand.

Running: DPO with SGD, one pair per step
  1. take the next pair \((y_w, y_l)\) from the shuffled dataset
  2. \(\rho_w \leftarrow \log \dfrac{\pi_\theta(y_w)}{\pi_\text{ref}(y_w)}\), \(\;\rho_l \leftarrow \log \dfrac{\pi_\theta(y_l)}{\pi_\text{ref}(y_l)}\)
  3. \(\hat r_w \leftarrow \beta \rho_w\), \(\;\hat r_l \leftarrow \beta \rho_l\) // implicit rewards (68.8)
  4. \(\ell \leftarrow -\log \sigma(\hat r_w - \hat r_l)\) // (68.7)
  5. \(w \leftarrow \sigma(\hat r_l - \hat r_w)\) // gradient weight (68.9)
  6. \(\theta_{y_w} \mathrel{+}= \eta \beta w\); \(\;\theta_{y_l} \mathrel{-}= \eta \beta w\) // SGD step
DPO policy\(\pi^*\) of the true reward (68.3)lossreward accuracychosenrejectedimplicit reward \(\beta \log \pi_\theta/\pi_\text{ref}\)true reward
Figure 4. DPO as a live simulation. Each step runs the six lines on one labeled pair; the highlighted line just ran, and the panel shows its numbers. The charts are computed on the whole dataset after every step: the loss (68.7), the reward accuracy (fraction of pairs with \(\hat r_w \gt \hat r_l\)), and the mean change of \(\log \pi_\theta\) from the reference over all chosen and over all rejected answers. Training stops after 40 epochs. Changing \(\beta\), the noise, the number of pairs or the seed restarts the run.
How to read Figure 4

Step through one pair with Next line: at the start both log-ratios are 0, the loss is \(\ln 2 = 0.693\) and the weight is 0.5, as in the worked example. Then press Play. With the defaults (\(\beta = 0.1\), \(\eta = 1\), noise 0.10, 100 pairs, seed 1) the run ends with loss 0.570 and reward accuracy 77.0%, and the third chart shows the effect from the worked example: the mean log-probability of the chosen answers rises by only 0.015 nats in the first few steps, then falls to −3.01 nats, while the rejected ones fall to −8.27. Why, in this toy? Most answers win some pairs and lose others. DPO keeps raising the answers that win most often (B and D end with 0.52 and 0.46 of the probability), so A and C lose probability even in the pairs where they were chosen. The loss and the accuracy are happy anyway: only the gap matters. Raise \(\beta\) to 0.5: the same data now give a much smaller drop (chosen −0.28, rejected −1.42) and a KL of 0.26 nats instead of 0.93, because a large \(\beta\) saturates the loss after smaller moves. Set the noise to 0 (with \(\beta = 0.5\)): the policy moves close to the dashed \(\pi^\ast\) (D gets 0.80, \(\pi^\ast\) says 0.65; finite data pushes too far, as in Figure 5). (Speeds 1 and 2 run one line of the algorithm at a time, 0.65 s or 0.2 s per line; speeds 3 to 5 run whole iterations, more of them per frame. “Next line” and the next-step button work at any speed.)

The PyTorch version below trains a tiny causal model on 8 pairs, then checks the theory: on exact, noise-free preference data, the DPO minimum is the closed-form \(\pi^*\) of (68.3).

import copy, torch
import torch.nn.functional as F
torch.manual_seed(0)

def seq_logp(model, ids, resp_mask):
    """Sum of log pi(token) over the response tokens only. ids: (B, T). Formula (68.10)."""
    logits = model(ids[:, :-1])                       # predict token t+1 from tokens up to t
    logp = torch.log_softmax(logits.float(), dim=-1)
    tok = torch.gather(logp, 2, ids[:, 1:].unsqueeze(-1)).squeeze(-1)
    return (tok * resp_mask[:, 1:]).sum(-1)           # (B,)

def dpo_loss(pol_w, pol_l, ref_w, ref_l, beta=0.1):
    r_w = beta * (pol_w - ref_w)                      # implicit reward of the chosen answer (68.8)
    r_l = beta * (pol_l - ref_l)                      # implicit reward of the rejected answer
    loss = -F.logsigmoid(r_w - r_l).mean()            # formula (68.7); logsigmoid is numerically stable
    stats = {"margin": (r_w - r_l).mean().item(),
             "acc": (r_w > r_l).float().mean().item()}
    return loss, stats

# a tiny causal "language model": embedding -> linear (a bigram model)
V, T = 16, 8
class Tiny(torch.nn.Module):
    def __init__(s):
        super().__init__(); s.emb = torch.nn.Embedding(V, 32); s.out = torch.nn.Linear(32, V)
    def forward(s, x): return s.out(s.emb(x))
policy = Tiny()
ref = copy.deepcopy(policy).eval()                    # the reference starts equal to the policy
for p in ref.parameters(): p.requires_grad_(False)

# 8 preference pairs: same 4-token prompt, two different 4-token answers
B = 8
prompt = torch.randint(0, V, (B, 4))
chosen = torch.cat([prompt, torch.randint(0, V // 2, (B, 4))], 1)      # chosen answers use tokens 0..7
rejected = torch.cat([prompt, torch.randint(V // 2, V, (B, 4))], 1)   # rejected answers use tokens 8..15
mask = torch.cat([torch.zeros(B, 4), torch.ones(B, 4)], 1)             # 1 on response tokens

with torch.no_grad():                                 # reference log-probs: computed once
    ref_w, ref_l = seq_logp(ref, chosen, mask), seq_logp(ref, rejected, mask)

opt = torch.optim.AdamW(policy.parameters(), lr=1e-2, weight_decay=0.0)
for step in range(101):
    pol_w, pol_l = seq_logp(policy, chosen, mask), seq_logp(policy, rejected, mask)
    loss, st = dpo_loss(pol_w, pol_l, ref_w, ref_l, beta=0.1)
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 50 == 0:
        print(f"step {step:3d}  loss {loss.item():.4f}  margin {st['margin']:.3f}  acc {st['acc']:.2f}  "
              f"d_logp chosen {(pol_w - ref_w).mean().item():+.2f}  rejected {(pol_l - ref_l).mean().item():+.2f}")

# theory check: on exact preference data, DPO's optimum is the closed-form pi* (formula 68.3)
ref_p = torch.tensor([0.30, 0.30, 0.25, 0.10, 0.05]); r = torch.tensor([0.0, 0.5, 1.0, 2.0, -1.0]); beta = 0.5
pi_star = ref_p * torch.exp(r / beta); pi_star /= pi_star.sum()
z = ref_p.log().clone().requires_grad_(True)          # a "policy" that is just 5 logits
opt = torch.optim.SGD([z], lr=8.0)
i, j = torch.meshgrid(torch.arange(5), torch.arange(5), indexing="ij")
wpair = ref_p[i] * ref_p[j] * torch.sigmoid(r[i] - r[j]) * (i != j)   # P(pair) * P(i preferred to j)
wpair /= wpair.sum()
for _ in range(300):
    h = beta * (torch.log_softmax(z, 0) - ref_p.log())           # implicit rewards
    loss = -(wpair * F.logsigmoid(h[i] - h[j])).sum()
    opt.zero_grad(); loss.backward(); opt.step()
print("DPO  ", torch.softmax(z, 0).detach().numpy().round(3))
print("pi*  ", pi_star.numpy().round(3))

Output (PyTorch 2.8, CPU):

step   0  loss 0.6931  margin 0.000  acc 0.00  d_logp chosen +0.00  rejected +0.00
step  50  loss 0.0056  margin 5.220  acc 1.00  d_logp chosen +5.21  rejected -46.99
step 100  loss 0.0018  margin 6.348  acc 1.00  d_logp chosen +5.10  rejected -58.38
DPO   [0.036 0.097 0.219 0.648 0.001]
pi*   [0.036 0.097 0.219 0.648 0.001]

Three lessons hide in this output. First, the loss starts at \(\log 2\) because policy and reference are equal; if your first DPO loss is not 0.693, your reference is wrong. Second, accuracy at step 0 is 0, not 0.5: with \(\hat r_w = \hat r_l\) exactly, the strict test \(\hat r_w \gt \hat r_l\) fails. Third, almost all of the margin comes from pushing the rejected answers down (−58 nats) rather than the chosen ones up (+5): making a sequence very unlikely is easy, and the loss does not care which side moves. Between steps 50 and 100 the chosen log-probability even falls a little (from +5.21 to +5.10).

Does DPO find the RLHF solution? Coverage and offline data

The derivation is exact. If the preference data were infinite, labeled by a Bradley–Terry model with a true reward \(r^*\), and covered every pair of answers, then the DPO minimum would be exactly the \(\pi^*\) of (68.3), the same policy that a perfect RLHF run would find. The code above shows this numerically. [Certain]

Real data break the assumptions in two ways.

  • Finite data. With a few labels per pair, the empirical preference is often 100% one way. Then the loss keeps falling as the margin grows, and the implicit reward difference goes to infinity, whatever \(\beta\) says. The KL leash of (68.1) is not really enforced. This is the overfitting problem that IPO fixes (session 69). [Certain]
  • Coverage. DPO only sees answers that are in the dataset. The pairs were usually generated by some other model (often not your \(\pi_\text{ref}\)), and they are never refreshed. An excellent answer that nobody wrote down gets no direct signal; its probability changes only as a side effect of the others. PPO, in contrast, samples from the current policy, so it can find and reward new good answers. That is the meaning of offline versus online: the data are fixed before training versus produced during training (session 71).

Figure 5 trains DPO on a toy problem where we know everything: one prompt, five answers A to E, a reference policy, and true rewards. Answer D is the best but rare under the reference.

Figure 5. DPO on a toy problem with five answers. Reference probabilities \((0.30, 0.30, 0.25, 0.10, 0.05)\) and true rewards \((0, 0.5, 1, 2, -1)\). Left: reference (grey), the DPO policy after the chosen number of gradient steps (blue), and the closed-form \(\pi^*\) of (68.3) (outline). Right: the DPO probability of each answer during training; dashed lines are \(\pi^*\). Labels follow Bradley–Terry with the true rewards.
How to read Figure 5

With all pairs and exact labels (infinite data), the blue bars land on the outlines: DPO recovers the RLHF optimum, here D goes from 0.10 to 0.648 at \(\beta = 0.5\), and the KL to \(\pi^*\) is about 0. With 60 sampled pairs (seeded, so the same every time), the labels are noisy and some rare pairs were seen only once, with a one-sided label. After 300 steps D ends at 0.865 instead of 0.648, and the KL to \(\pi^*\) is 0.128 nats: finite data pushes margins further than the true preferences justify. Choose “D never in the data”: the best answer is never compared, so DPO cannot learn that it is good. Its probability falls to about 0.04 while C takes over. An online method that sampled D and scored it could fix this; DPO on this dataset never will.

The numbers behind Figure 5 (\(\beta = 0.5\))

Closed form: \(e^{r/\beta} = (1, 2.718, 7.389, 54.6, 0.135)\); weights \(\pi_\text{ref}\, e^{r/\beta} = (0.300, 0.815, 1.847, 5.460, 0.007)\); \(Z = 8.43\); \(\pi^* = (0.036, 0.097, 0.219, 0.648, 0.001)\).

DPO with all pairs reaches \((0.036, 0.097, 0.219, 0.648, 0.001)\), the same to three digits. DPO without D reaches \((0.097, 0.263, 0.596, 0.041, 0.002)\): the relative order of A, B, C, E is right, but D, the best answer, lost 60% of its reference probability. The KL from DPO to \(\pi^*\) is 0.85 nats.

The two routes side by side

Figure 6 runs both routes from Figure 1 on the same pairs, from the same reference. The RLHF route has two stages, as in session 67: first it fits a reward model \(r_\phi\) (one score per answer) with the Bradley–Terry loss for 150 iterations; then it trains a policy with the sampled policy gradient of reward minus \(\beta\) times the log-ratio, 16 samples per iteration, with a leave-one-out baseline. The DPO route trains its policy with (68.7) from the first iteration. Both see the same minibatches of 10 pairs and use the same learning rate.

Running: one iteration of each route
  1. draw a minibatch of 10 pairs \((y_w, y_l)\)
  2. RLHF stage 1 (iterations 1–150): \(r_\phi \mathrel{+}= \eta\,\overline{(1 - \sigma(r_w - r_l))(e_{y_w} - e_{y_l})}\) // (67.4)
  3. RLHF stage 2: sample 16 answers \(y \sim \pi_\text{RL}\); \(R \leftarrow r_\phi(y) - \beta \log \frac{\pi_\text{RL}(y)}{\pi_\text{ref}(y)}\)
  4. RLHF stage 2: \(\theta_\text{RL} \mathrel{+}= \eta\,\overline{(R - b)(e_y - \pi_\text{RL})}\) // policy gradient
  5. DPO: \(\theta_\text{DPO} \mathrel{+}= \eta\beta\,\overline{\sigma(\hat r_l - \hat r_w)(e_{y_w} - e_{y_l})}\) // (68.9)
  6. record the true reward and the KL of both policies
RLHF (reward model + policy gradient)DPO (and its implicit reward)true reward / \(\pi^*\) values\(\pi^*\) of the true reward
Figure 6. RLHF against DPO on one dataset of preference pairs (answers and reference as in Figure 5). The bar over a sum, \(\overline{\,\cdot\,}\), means the average over the minibatch (or over the 16 samples). Lines 3 and 4 do nothing during stage 1. Bottom right: the true reward, the reward model \(r_\phi\) and DPO’s implicit reward \(\beta \log \pi_\text{DPO}/\pi_\text{ref}\), all shifted to mean 0, because only differences matter (formula 68.4).
How to read Figure 6

Press Play at speed 4. During the first 150 iterations only DPO’s policy moves; the RLHF policy waits for its reward model (its curve stays flat at the reference value 0.55). Then the RLHF policy catches up quickly. With the defaults (\(\beta = 0.5\), noise 0.10, 100 pairs, seed 1) both end almost at the same place: expected true reward 0.877 (RLHF) and 0.867 (DPO), KL 0.24 and 0.25 nats, and nearly the same bars. Both are well below the \(\pi^\ast\) line (1.562), because they learned from 100 noisy pairs, not from the true reward: on this data D wins only a little more often than the others, so its small reference probability (0.10) keeps it near 0.20. The bottom-right chart shows why the two agree: the reward model and DPO’s implicit reward end up ordering the answers the same way. Set the noise to 0: both reach about 1.72 and put almost 0.79 on D. With 400 noisy pairs they differ more (1.40 against 1.15): the routes are equivalent at their optimum, not step by step, and the RLHF reward model here only gets 150 iterations of training. Lower \(\beta\) and both policies move further from the reference.

β and the reference model

What β does. In theory, \(\beta\) is the KL price in (68.1): small \(\beta\) allows the policy to move far from the reference. In the loss, \(\beta\) multiplies the log-ratios. With a small \(\beta\), the model must change its log-probabilities a lot before the margin is large and the loss saturates; with a large \(\beta\), small changes already saturate the loss, so training stops early and the policy stays close to \(\pi_\text{ref}\). The original paper used \(\beta = 0.1\) for most experiments and \(0.5\) for TL;DR summarization, with RMSprop, learning rate \(10^{-6}\), batch size 64 and 150 warmup steps. [Certain] Zephyr used \(\beta = 0.1\) with a peak learning rate of \(5 \times 10^{-7}\); Llama 3 used \(\beta = 0.1\). [Certain] DPO learning rates are 10 to 100 times smaller than SFT learning rates. [Likely]

What the reference model does. It appears in three places: it is the anchor of the KL in (68.1), it is the starting point of \(\pi_\theta\), and it is the baseline in every implicit reward. Its quality matters because the theory assumes the preference answers look like samples from \(\pi_\text{ref}\). If you take a public preference dataset written by other models, first run SFT on the chosen answers, so that the reference is not far from the data. The DPO paper does exactly this when no SFT model is available. [Certain]

Can we drop it? If we remove \(\pi_\text{ref}\) from (68.7), the loss becomes \(-\log\sigma(\beta[\log\pi_\theta(y_w) - \log\pi_\theta(y_l)])\). Nothing then anchors the model: it can drift arbitrarily, and the unnormalized log-probability prefers short answers. Methods such as ORPO and SimPO remove the reference on purpose and add other fixes (an SFT term, length normalization, a margin). That is the topic of session 69.

What DPO can and cannot do

PPO-RLHFDPO
Objective(68.1) with a learned reward modelthe same (68.1), solved in closed form
Training stages after SFT2: reward model, then RL1
Models in memory4 (policy, critic, reference, reward model)2 (policy, reference), or 1 with precomputed log-probs
Generation during trainingyes, every iteration (usually the slowest part)no
Dataonline: fresh samples from the current policyoffline: a fixed set of pairs
Hyper-parametersmany (clip, KL coefficient, GAE, critic lr, batch sizes)few (\(\beta\), lr, epochs)
Can explore new answersyesno
Reusable reward modelyes (also for best-of-N, data filtering)only implicit, tied to \(\pi_\text{ref}\)
Known weak spotsreward hacking, instability, costoverfitting on one-sided pairs, falling chosen log-probs, length growth, out-of-distribution answers

A common middle ground is iterative (or online) DPO: train with DPO, sample new answers from the new policy, label them with a reward model or a judge LLM, and run DPO again on the fresh pairs. This brings back part of the exploration of online RL while keeping the simple loss (Xiong et al., 2023). [Certain] Llama 3’s several rounds of SFT and DPO, each on newly collected preference data, follow this pattern. [Likely]

Length. Human and AI judges often prefer longer answers, and the chosen answer in many datasets is longer on average. DPO then learns to write longer answers, even when quality does not improve (Park et al., 2024). [Likely] Length-normalized variants and length-controlled evaluation are the usual fixes.

Where this is used

SettingWhat DPO does thereNotes
Chat assistants (Zephyr, Llama 3, Tülu 3)Preference stage after SFT: helpfulness, instruction following, tonePairs from humans or from a judge model; \(\beta \approx 0.1\) [Certain for Zephyr and Llama 3]
Summarization (TL;DR in the DPO paper)Prefer summaries that people rate higherWin rate against reference summaries about 61% for DPO vs 57% for PPO, judged by GPT-4 [Certain]
Safety and refusalsPrefer safe answers over harmful ones, and helpful answers over needless refusalsNeeds pairs on both sides, or the model over-refuses (session 74)
Vision-language modelsPairs of image descriptions: faithful vs hallucinated objectsSame loss; the image is part of \(x\). Used to reduce hallucination in VLMs [Likely]
Video captioning and video QAPrefer captions that match the actual events and their order over captions with invented actions or wrong timingRejected answers can be made by editing a correct caption (swap two actions, change a time); keep edits small but watch the falling-likelihood problem [Speculation]
Key-moment detection with an LLM headPrefer answers that cite the right time spanPairs from a temporal-IoU score; an online method with a verifiable reward is often a better fit (session 70) [Speculation]
Code and mathPairs of correct and incorrect solutionsWorks, but online RL with a verifier is now usually preferred for reasoning [Likely]

Common mistakes

  • Summing the prompt tokens into the log-probability. The prompt is the same for both answers and the reference, so it only adds noise and padding bugs. Mask everything except the answer, including padding.
  • Averaging token log-probs when you meant to sum. Plain DPO uses the sum (68.10). Averaging changes the method (it is a length-normalized variant) and the meaning of \(\beta\).
  • A reference that is not the starting policy. At step 0 the loss must be exactly 0.693 and every implicit reward 0. If not, the reference or the masking is wrong (for example, a different chat template or dropout left on).
  • Different templates or tokenization for policy and reference. The log-ratio then measures the template, not the answer. Use the same tokenizer and template (session 9). Llama 3 also masks special formatting tokens out of the DPO loss for stability. [Certain]
  • Watching only the loss and accuracy. Both can look great while the chosen log-probability falls and the model gets worse. Log the chosen and rejected log-probs and the KL to the reference; consider adding a small SFT (negative log-likelihood) term on the chosen answers, as Llama 3 did with coefficient 0.2. [Certain]
  • Too many epochs or too high a learning rate. DPO overfits fast on one-sided pairs: margins keep growing and generation quality drops. One to three epochs with a learning rate around \(10^{-7}\) to \(10^{-6}\) are common. [Likely]
  • Using pairs where both answers are bad, or both good. The loss still pushes them apart. Filter pairs with a small quality gap, or label noise dominates.
  • Comparing implicit rewards across prompts. They are defined only up to a per-prompt constant \(\beta\log Z(x)\).

Interview questions

Derive the DPO loss starting from the RLHF objective.core
Start from \(\max_\pi \E_{y\sim\pi}[r] - \beta\KL(\pi\|\pi_\text{ref})\). Combine the terms: \(-\beta\E_\pi[\log(\pi / (\pi_\text{ref}e^{r/\beta}))] = -\beta\KL(\pi\|\pi^*) + \beta\log Z\), with \(\pi^* = \pi_\text{ref}e^{r/\beta}/Z\). The KL is minimized at \(\pi = \pi^*\). Taking logs gives \(r = \beta\log(\pi^*/\pi_\text{ref}) + \beta\log Z(x)\). Insert into Bradley–Terry \(\sigma(r_w - r_l)\): the \(\beta\log Z(x)\) terms cancel. Replace \(\pi^*\) by \(\pi_\theta\) and minimize the negative log-likelihood of the observed preferences: \(-\log\sigma(\beta\log\frac{\pi_\theta(y_w)}{\pi_\text{ref}(y_w)} - \beta\log\frac{\pi_\theta(y_l)}{\pi_\text{ref}(y_l)})\).
Why is the partition function \(Z(x)\) not a problem?basic
\(Z(x)\) is a sum over all possible answers, so it cannot be computed for a language model. But it depends only on the prompt. Bradley–Terry uses only the difference of rewards of two answers to the same prompt, so \(\beta\log Z(x)\) appears in both terms and cancels. This is also why implicit rewards can only be compared within one prompt.
What does \(\beta\) control, and what happens at the extremes?basic
\(\beta\) is the price of the KL to the reference. Large \(\beta\): the best policy stays close to \(\pi_\text{ref}\), and in the loss a small change in log-probabilities already gives a big margin, so training saturates early. Small \(\beta\): the policy may move far, toward putting all mass on the top answers, and the loss needs large log-probability changes before it saturates. Typical values are 0.1 to 0.5. In practice, with finite one-sided data the margin can still grow without bound, so \(\beta\) is a weaker leash than the theory suggests.
Explain the DPO gradient. Why is the weight \(\sigma(\hat r_l - \hat r_w)\) important?core
\(\nabla\mathcal{L} = -\beta\E[\sigma(\hat r_l - \hat r_w)(\nabla\log\pi(y_w) - \nabla\log\pi(y_l))]\). It raises the chosen answer and lowers the rejected one, weighted by how wrong the implicit reward currently is. Pairs already ordered correctly by a wide margin get almost no weight, which prevents the model from pushing log-probabilities to minus infinity. The DPO paper reports that the unweighted version (pure unlikelihood-style) makes the model degenerate.
During DPO training, the log-probability of the chosen answers goes down. Is that a bug?advanced
Not necessarily a code bug: the loss depends only on the difference of the two log-ratios, so both can fall as long as the rejected one falls faster. It is common when chosen and rejected answers share most tokens (small edit distance), because lowering shared tokens lowers both. It can hurt quality, since probability mass moves to answers that are in neither set. Fixes: an SFT/NLL term on chosen answers (Llama 3 used 0.2), DPO-Positive (Smaug), a lower learning rate, fewer epochs, or a larger \(\beta\). Log both log-probs to catch it.
Why is DPO called “offline”, and what does that cost?core
Its data are a fixed set of pairs collected before training, usually from other models, and never refreshed. So DPO cannot try new answers and learn from them; answers not in the data get no direct signal and their probability changes only through side effects. If the data come from a policy very different from \(\pi_\text{ref}\), the theory’s assumptions fail. PPO samples on-policy and can explore and correct its own mistakes, at higher cost. Iterative/online DPO (resample, relabel, retrain) is the usual compromise.
Compare the compute and memory of DPO and PPO-RLHF for a 7B model.core
PPO trains a policy and a critic (16 bytes per parameter each with AdamW: 112 GB + 112 GB) and keeps a frozen reference and reward model (14 GB each in BF16): 252 GB before activations, plus generation at every iteration. DPO trains only the policy (112 GB) plus a frozen reference (14 GB), 126 GB, or 112 GB with precomputed reference log-probs, and needs no generation: just forward and backward on chosen and rejected sequences.
Why do we need a reference model at all? What if you remove it?core
The reference is the anchor of the KL term, the baseline of each implicit reward, and the initial policy. Without it, the loss only compares \(\log\pi_\theta(y_w)\) and \(\log\pi_\theta(y_l)\); nothing keeps the model close to its starting point, and raw sequence log-probabilities favor short answers. Reference-free methods (ORPO, SimPO) add other terms: an SFT loss, length normalization, a target margin.
In what sense is “your language model secretly a reward model”? How would you use it as one?advanced
After DPO, \(\hat r(x, y) = \beta\log(\pi_\theta(y|x)/\pi_\text{ref}(y|x))\) is a reward model in the Bradley–Terry sense, and the paper shows every reward class can be represented this way. You can rank candidate answers to the same prompt with it (for best-of-N or to label new pairs) using two forward passes. It is defined only up to a per-prompt constant, so it cannot be compared across prompts, and it is reliable only near the training data distribution.
Your DPO model writes longer and longer answers. Why, and what do you do?core
In many preference datasets the chosen answer is longer on average, because judges reward detail. DPO learns whatever separates chosen from rejected, including length. Fixes: balance lengths in the data, use length-normalized or length-regularized variants, add a margin or SFT term, evaluate with length-controlled win rates, and keep \(\beta\) and epochs moderate.
Under what conditions does DPO give exactly the RLHF solution?advanced
If preferences follow Bradley–Terry with some true reward \(r^*\), the data cover all relevant pairs with enough labels to estimate the preference probabilities, the policy class can represent \(\pi^*\), and optimization finds the minimum, then the DPO minimum is \(\pi^* \propto \pi_\text{ref}e^{r^*/\beta}\), the same as the optimum of the KL-regularized objective. With finite, one-sided labels the margins diverge (the IPO critique); with missing coverage the result depends on what is in the data.
You have only single thumbs-up or thumbs-down labels, not pairs. Can you use DPO?basic
Not directly: DPO needs a chosen and a rejected answer for the same prompt. You can build pairs where the same prompt has one good and one bad answer, or use a method designed for unpaired labels such as KTO (next session).

Exercises

  1. For the three-answer example (\(\pi_\text{ref} = (0.5, 0.3, 0.2)\), \(r = (0, 1, 2)\)), compute \(\pi^*\) and the best objective value for \(\beta = 0.5\).
  2. One pair with \(\beta = 0.1\): the policy raised the chosen log-probability by 3 nats and lowered the rejected one by 2 nats, relative to the reference. Compute the margin, the loss and the gradient weight.
  3. Show that adding a prompt-only term \(f(x)\) to the reward changes neither \(\pi^*\) in (68.3) nor the Bradley–Terry probability (68.5).
  4. What does \(\pi^*\) become as \(\beta \to \infty\) and as \(\beta \to 0\)?
  5. Training logs with \(\beta = 0.1\) show mean rewards/chosen \(= -0.5\) and rewards/rejected \(= -2.0\). What happened to the log-probabilities? What is \(\sigma\) of the mean margin, and why should this worry you?
  6. Estimate the memory, before activations, of DPO and of PPO-RLHF for a 70B model (AdamW mixed precision, frozen models in BF16, reference not precomputed).
Answerscheck
1. Multipliers \(e^{2r} = (1, 7.389, 54.60)\), weights \((0.5, 2.217, 10.92)\), \(Z = 13.636\); \(\pi^* = (0.037, 0.163, 0.801)\). Best objective \(\beta\log Z = 0.5 \times \log 13.636 = 1.306\) (expected reward 1.764 minus \(0.5 \times 0.916\)). 2. \(\hat r_w = 0.3\), \(\hat r_l = -0.2\), margin 0.5; loss \(-\log\sigma(0.5) = 0.474\); weight \(\sigma(-0.5) = 0.378\). 3. \(e^{(r + f)/\beta} = e^{f/\beta}e^{r/\beta}\): the factor \(e^{f(x)/\beta}\) is the same for all answers and divides out with \(Z\). In Bradley–Terry, \(f(x)\) appears in both rewards and cancels in the difference. 4. \(\beta\to\infty\): \(e^{r/\beta}\to 1\), so \(\pi^*\to\pi_\text{ref}\). \(\beta\to 0\): all probability goes to the answer(s) with the highest reward among those with \(\pi_\text{ref} \gt 0\). 5. \(\log\pi_\theta - \log\pi_\text{ref} = \hat r/\beta\): chosen fell by 5 nats (probability \(\times\, 0.0067\)), rejected by 20 nats. Margin 1.5, \(\sigma(1.5) = 0.818\). Accuracy and margin look good, but the chosen answers are now about 150 times less likely than under the reference, so the mass went to answers in neither set: check generations and consider an SFT term. 6. Policy \(70\times10^9\times16 = 1{,}120\) GB; each frozen BF16 model 140 GB. DPO: 1,120 + 140 = 1,260 GB. PPO: 2 × 1,120 + 2 × 140 = 2,520 GB, so about twice as much, plus the generation engine.

Key takeaways

  • RLHF maximizes reward minus \(\beta\) times the KL to a reference (68.1). Its exact best policy is the reference re-weighted by \(e^{r/\beta}\) and normalized by \(Z(x)\) (68.3).
  • Inverting it, any reward equals \(\beta\log(\pi^*/\pi_\text{ref})\) plus a prompt-only term (68.4). In Bradley–Terry that term cancels (68.6).
  • So the reward-model loss, written for this special reward, is the DPO loss (68.7): logistic regression on the difference of implicit rewards (68.8), computed from summed token log-probs (68.10).
  • The gradient (68.9) raises the chosen answer, lowers the rejected one, and weights each pair by how wrong it still is. It only controls the difference, so both log-probs can fall (Figure 4).
  • DPO needs one stage, two models and no generation, but it is offline: it cannot explore, it overfits one-sided pairs, and it can lose good answers that are not in the data. Iterative DPO and online RL address this. On the same pairs, the two-stage RLHF route and DPO reach nearly the same policy (Figure 6).

Key formulas

\[ \pi^*(y|x) = \frac{\pi_\text{ref}(y|x)\,e^{r(x,y)/\beta}}{Z(x)}, \qquad r = \beta\log\frac{\pi^*}{\pi_\text{ref}} + \beta\log Z(x) \]
\[ \mathcal{L}_\text{DPO} = -\E\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_\text{ref}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_\text{ref}(y_l|x)}\Big)\Big] \]
\[ \nabla_\theta\mathcal{L}_\text{DPO} = -\beta\,\E\big[\sigma(\hat r_l - \hat r_w)\,(\nabla\log\pi_\theta(y_w|x) - \nabla\log\pi_\theta(y_l|x))\big] \]

References for this session

  1. Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  2. Lambert (2025). Reinforcement Learning from Human Feedback (book)
  3. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 10: Alignment
  4. Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  5. Lambert (2025). Reinforcement Learning from Human Feedback (book)
  6. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 10: Alignment
  7. Ouyang et al. (2022). Training language models to follow instructions with human feedback
  8. Schulman et al. (2017). Proximal Policy Optimization Algorithms
  9. Tunstall et al. (2023). Zephyr: Direct Distillation of LM Alignment
  10. Llama Team, Meta (2024). The Llama 3 Herd of Models
  11. Pal et al. (2024). Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
  12. Xu et al. (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
  13. Azar et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences
  14. Xiong et al. (2023). Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  15. Park et al. (2024). Disentangling Length from Quality in Direct Preference Optimization
  16. Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training
  17. Hugging Face (code). TRL library