LLM & VLM courseBlog
69Track 4: Post-training and RL · session 17 of 25

The DPO family: IPO, KTO, ORPO, SimPO

DPO turned preference learning into one simple classification loss. Then people found its weak spots: it overfits when preferences look certain, it needs pairs, it needs an SFT stage and a reference model, and it can favor long answers. This session shows four well-known fixes (IPO, KTO, ORPO, SimPO), what each one changes in the loss, and when to pick which.

Time: 2.5 to 3 hours (reading 2 hours, exercises 45 minutes)Level: BeginnerBefore this: Sessions 67 and 68

What you will learn

  • How every DPO-style method is a loss on an implicit reward margin, and why the shape of that loss matters
  • Why DPO pushes the margin to infinity when the data says one answer always wins, and how IPO's squared loss gives a finite target 1/(2τ)
  • How KTO learns from single thumbs-up or thumbs-down labels with a reference point z0 and separate weights for good and bad examples
  • How ORPO adds an odds-ratio term to the normal SFT loss, so one stage and no reference model are needed
  • How SimPO uses the average log-probability as its reward, adds a target margin γ, and drops the reference model
  • How to compute all five losses from per-token log-probabilities in PyTorch, and how to choose between them

Why this matters

In session 68 we replaced the reward model and the RL loop of RLHF with one loss on pairs of answers. DPO is simple, stable and cheap, so it spread fast. Then teams found four practical problems with it, and each problem got its own fix.

  1. Overfitting on "certain" pairs. If the data says answer A always beats answer B, DPO keeps pushing B's probability toward zero, no matter how large \(\beta\) is. IPO fixes this with a squared loss that has a finite target.
  2. It needs pairs. Real products collect single thumbs-up or thumbs-down clicks, not side-by-side comparisons. KTO learns from these single labels.
  3. It needs an SFT stage and a reference model. Standard practice is SFT first, then DPO with a frozen copy of the SFT model. ORPO does both in one stage with no reference model.
  4. It can favor long answers, and the reference model costs memory. SimPO uses the average log-probability per token as its reward, adds a margin, and drops the reference model.

All five methods are offline: they learn from a fixed dataset, without sampling new answers during training. They are all available in the Hugging Face TRL library, so in practice you can switch between them by changing a config. [Likely] Llama 3 used DPO in its post-training rounds [Certain], Zephyr-7B was trained with DPO [Certain], and the ORPO and SimPO papers released 7B to 9B chat models trained with their methods [Certain].

The key idea of this session is that all of these methods are one recipe with different parts: pick an implicit reward for an answer, compare rewards of good and bad answers, and pass the difference through a loss function. Once you see the parts, each new paper in this family is easy to read.

The common template: a reward margin and a loss shape

Recall DPO. For a prompt \(x\), a chosen answer \(y_w\) ("winner") and a rejected answer \(y_l\) ("loser"), the loss is:

\[ \mathcal{L}_{\text{DPO}} = -\log \sigma\big(\beta\, h_\theta\big), \qquad h_\theta = \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \tag{69.1} \]
Read the symbols
\(\pi_\theta(y \mid x)\)
probability of the whole answer under the model we train: the product of its token probabilities, so \(\log \pi_\theta\) is the sum of token log-probabilities
\(\pi_{\text{ref}}\)
the reference model, a frozen copy of the starting (usually SFT) model
\(h_\theta\)
the margin: how much more the model has raised the winner than the loser, measured against the reference
\(\beta\)
a positive number, often 0.1; it plays the role of the KL penalty strength in RLHF
\(\sigma\)
the sigmoid, \(\sigma(z) = 1/(1 + e^{-z})\)

Read it as logistic regression. The "implicit reward" of an answer is \(r(y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\). DPO says: the reward gap \(\beta h_\theta\) should be large and positive. So every method in this family is defined by three choices:

ChoiceDPO's answerWho changes it
What is the reward of an answer?\(\beta \times\) log-ratio to the reference, summed over tokensSimPO (average log-prob, no reference), ORPO (log-odds, no reference)
What data do we compare?a chosen and a rejected answer to the same promptKTO (single answers, each labeled good or bad)
Which loss shape on the margin?\(-\log \sigma\) (logistic)IPO (squared), SimPO (logistic with a margin \(\gamma\)), KTO (sigmoid "value")

The loss shape decides how hard the method pushes at each margin. The gradient of DPO with respect to the margin is:

\[ \frac{\partial \mathcal{L}_{\text{DPO}}}{\partial h_\theta} = -\beta\, \sigma\big(-\beta\, h_\theta\big) \tag{69.2} \]

This number is always negative: DPO always wants a larger margin. The push gets weaker as the margin grows, but it never becomes zero. For a pair that the model already ranks right by a wide margin, DPO still pushes, a little. That fact is the root of the first problem.

Worked example: one pair, used for every method below

The chosen answer has 20 tokens; the rejected one has 40. Summed log-probabilities: policy \(\log \pi_\theta(y_w) = -30\), \(\log \pi_\theta(y_l) = -50\); reference \(\log \pi_{\text{ref}}(y_w) = -32\), \(\log \pi_{\text{ref}}(y_l) = -49\).

Log-ratios: winner \(-30 - (-32) = +2\), loser \(-50 - (-49) = -1\). Margin \(h_\theta = 2 - (-1) = 3\).

With \(\beta = 0.1\): \(\beta h = 0.3\), loss \(= -\log \sigma(0.3) = 0.554\). The gradient (69.2) is \(-0.1 \times \sigma(-0.3) = -0.1 \times 0.426 = -0.0426\). The model is already right about this pair, yet the loss is still well above zero and still pushes.

IPO: a finite target instead of "more is always better"

The problem: DPO's best margin can be infinite

Suppose the true probability that \(y_w\) beats \(y_l\) is \(p\). If we see the pair many times, labeled each way in proportion \(p\), the expected DPO loss is \(-p \log \sigma(\beta h) - (1-p) \log \sigma(-\beta h)\). Setting its derivative to zero gives \(\sigma(\beta h) = p\), so the best margin is:

\[ h^*_{\text{DPO}} = \frac{1}{\beta} \log \frac{p}{1 - p} \tag{69.3} \]

As \(p \to 1\), this goes to infinity. And in a real dataset, \(p\) looks like 1 almost always: each pair usually appears once, labeled one way, so the empirical win rate is \(\hat p = 1\) even if the true \(p\) is 0.8. Azar et al. point out that in this case DPO drives \(\pi_\theta(y_l)\) to 0 for any value of \(\beta\): the KL regularization, which \(\beta\) was supposed to provide, stops working. [Certain] The model becomes overconfident on the training pairs and can drift far from the reference.

The IPO loss

IPO (Identity Preference Optimization, from the paper's general \(\Psi\)PO objective with \(\Psi\) set to the identity) keeps the same margin \(h_\theta\) but replaces the logistic loss with a squared error to a fixed target:

\[ \mathcal{L}_{\text{IPO}} = \Big( h_\theta - \frac{1}{2\tau} \Big)^2 \tag{69.4} \]
Read the symbols
\(h_\theta\)
the same log-ratio margin as in DPO (69.1); no \(\beta\) inside
\(\tau\)
the regularization strength, the counterpart of DPO's \(\beta\) (TRL reuses the name beta for it [Likely]); larger \(\tau\) means a smaller target and a model closer to the reference
\(1/(2\tau)\)
the target margin; with \(\tau = 0.1\) it is 5

Repeat the same calculation as for DPO. With a pair labeled in proportion \(p\), the expected loss is \(p\,(h - c)^2 + (1-p)(-h - c)^2\) with \(c = 1/(2\tau)\). Its minimum is at

\[ h^*_{\text{IPO}} = \frac{2p - 1}{2\tau} \tag{69.5} \]

This is always between \(-1/(2\tau)\) and \(+1/(2\tau)\). Even when \(p = 1\), the best margin is finite. So \(\tau\) really controls how far the model may move. (This is the two-answer case of the paper's general result \(h^* = \tau^{-1}\big(p(y \succ \mu) - p(y' \succ \mu)\big)\).)

The gradient is \(\partial \mathcal{L}_{\text{IPO}} / \partial h = 2\,(h - 1/(2\tau))\). Below the target it pushes the margin up; above the target it pushes it down. Figure 1 compares the push of the two losses.

DPO pushIPO pushIPO target \(1/(2\tau)\)
Figure 1. How hard each loss pushes the margin up, as a function of the margin. The push is \(-\partial \mathcal{L}/\partial h\), divided by its value at \(h = 0\) so both curves start at 1. Positive means "make the margin larger"; negative means "make it smaller". The IPO line is cut off at −1.5 to keep the plot readable.
How to read Figure 1

The blue DPO curve stays above zero for every margin: it decays but never crosses zero, so DPO never says "this is enough". The orange IPO line crosses zero exactly at the target \(1/(2\tau)\) and becomes negative after it. Move the margin slider past the target and read the signs in the text below. Make \(\tau\) larger and the target moves left: a stronger leash. The readout shows the real (not rescaled) losses and gradients.

Figure 2 shows what this means during training. We take a toy "model" with only two possible answers and a uniform reference, so the margin is simply the difference of two logits and \(\pi_\theta(y_l) = \sigma(-h)\). We train it with Adam (learning rate 0.05) on data where \(y_w\) wins a fraction \(\hat p\) of the time.

DPO marginIPO marginIPO optimum (69.5)
Figure 2. Live training of the margin \(h\) with DPO and with IPO, Adam with learning rate 0.05, starting from \(h = 0\) (the reference). The y-axis grows to fit the DPO curve.
How to read Figure 2

With \(\hat p = 1\) (every pair seen once, which is the normal case) the DPO margin keeps growing as long as you train: about 28 after 1,000 steps and about 48 after 3,000, so \(\pi_\theta(y_l)\) falls to around \(10^{-12}\) and below. IPO stops at 5. Lower \(\hat p\) to 0.8: now DPO also stops, at \(\log 4 / \beta = 13.86\) (69.3), while IPO stops at 3 (69.5). Adam makes the DPO problem worse: it rescales the tiny gradient back to a normal step size, so a small push still moves the weights.

Worked example: the same pair under IPO

Margin \(h = 3\) from the first example, \(\tau = 0.1\), target \(1/(2\tau) = 5\). Loss \((3 - 5)^2 = 4\). Gradient \(2(3 - 5) = -4\): push up. If training later reaches \(h = 7\), the gradient becomes \(2(7 - 5) = +4\): push down. DPO at \(h = 7\) would still push up with \(-0.1\,\sigma(-0.7) = -0.033\).

Best margins for different \(p\), with \(\beta = \tau = 0.1\): at \(p = 0.8\), DPO 13.86 and IPO 3; at \(p = 0.99\), DPO 45.95 and IPO 4.9; at \(p = 1\), DPO infinite and IPO 5.

KTO: learning from thumbs up and thumbs down

Why single labels

Pairwise data is expensive: a person must read two answers to the same prompt and compare. Many products instead log one answer and one click: thumbs up (desirable) or thumbs down (undesirable). The two kinds of label are also usually unbalanced, for example many more good examples than bad ones. KTO (Kahneman-Tversky Optimization, named after the prospect theory of human choices) is built for exactly this data. The paper reports that it matches or beats DPO at model sizes from 1B to 30B. [Certain]

The math

KTO keeps DPO's implicit reward, one answer at a time:

\[ r_\theta(x, y) = \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} \tag{69.6} \]

Without a second answer to compare with, it needs a reference point. KTO uses the KL divergence between the policy and the reference, which is the average reward of a "typical" answer:

\[ z_0 = \KL\big(\pi_\theta(y' \mid x) \,\|\, \pi_{\text{ref}}(y' \mid x)\big) \approx \max\Big(0,\ \frac{1}{m} \sum_{i=1}^{m} \log \frac{\pi_\theta(y_j \mid x_i)}{\pi_{\text{ref}}(y_j \mid x_i)}\Big) \tag{69.7} \]

In practice \(z_0\) is estimated in each micro-batch of size \(m\) from mismatched pairs: answer \(y_j\) belongs to a different prompt than \(x_i\) (the batch is shifted by one). Mismatched answers are used because the real answers were chosen to be very good or very bad, so their rewards are not typical. The estimate is clipped at 0, and no gradient flows through it. [Certain]

Each answer then gets a "value", a sigmoid of how far its reward is above (good answers) or below (bad answers) the reference point:

\[ v(x, y) = \begin{cases} \lambda_D\, \sigma\big(\beta\,(r_\theta(x, y) - z_0)\big) & \text{if } y \text{ is desirable} \\ \lambda_U\, \sigma\big(\beta\,(z_0 - r_\theta(x, y))\big) & \text{if } y \text{ is undesirable} \end{cases} \tag{69.8} \]
\[ \mathcal{L}_{\text{KTO}} = \E_{(x, y)}\big[\lambda_y - v(x, y)\big] \tag{69.9} \]
Read the symbols
\(\lambda_D, \lambda_U\)
weights for desirable and undesirable examples; \(\lambda_y\) is the one that matches the label of \(y\)
\(\beta\)
how sharp the sigmoid is; the paper says \(\beta = 0.1\) works well in most cases
\(z_0\)
the reference point from (69.7), treated as a constant
\(\lambda_y - v\)
between 0 and \(\lambda_y\); it is 0 when the answer is far on the right side of \(z_0\)

The sigmoid is the important shape. It is steep near \(z_0\) and flat far away. So answers that are already clearly good (or clearly bad) get almost no gradient, which limits overfitting in the same spirit as IPO. Prospect theory adds one more idea: people feel losses more strongly than gains (loss aversion). The two weights let you set this balance. The paper recommends choosing them so that

\[ \frac{\lambda_D\, n_D}{\lambda_U\, n_U} \in \Big[1, \frac{3}{2}\Big] \tag{69.10} \]

where \(n_D\) and \(n_U\) are the numbers of desirable and undesirable examples. This interval was found by experiment. [Certain] In one Llama-7B experiment the authors discarded up to 90% of the desirable data, adjusted the weights to keep (69.10), and still beat DPO. [Certain] Treat that as a result for one setup, not a guarantee.

desirable: loss \(1 - \sigma(\beta(r - z_0))\)undesirable: loss \(1 - \sigma(\beta(z_0 - r))\)\(z_0\)
Figure 3. The KTO loss for one answer (with \(\lambda_D = \lambda_U = 1\)) as a function of its reward \(r_\theta\). The dots are the two answers set by the sliders. The last line of the readout applies rule (69.10) to the counts you choose, with \(\lambda_U = 1\).
How to read Figure 3

The green curve falls as the reward grows: a good answer is pushed to the right of \(z_0\). The orange curve rises: a bad answer is pushed to the left. Both are steepest at \(z_0\) and flat far from it, so far-away answers stop contributing. Slide \(z_0\) to the right: it means the policy has already moved away from the reference on typical answers, so a good answer must now score higher to count as good. Set \(n_D = 1000\), \(n_U = 100\): the rule gives \(\lambda_D\) between 0.10 and 0.15, so each good example counts about ten times less.

Worked example: the pair as two separate KTO examples

Use the same answers, but now as unpaired data: \(y_w\) is desirable with reward \(r = +2\), \(y_l\) is undesirable with \(r = -1\). Take \(z_0 = 0.4\), \(\beta = 0.1\), \(\lambda_D = \lambda_U = 1\).

Desirable: \(1 - \sigma(0.1 \times (2 - 0.4)) = 1 - \sigma(0.16) = 0.460\). Undesirable: \(1 - \sigma(0.1 \times (0.4 + 1)) = 1 - \sigma(0.14) = 0.465\). Gradients with respect to \(r\): \(-0.0248\) and \(+0.0249\): raise the good answer, lower the bad one.

Class weights: with 1,000 thumbs-up and 100 thumbs-down and \(\lambda_U = 1\), rule (69.10) gives \(1 \le 1000\,\lambda_D / 100 \le 1.5\), so \(\lambda_D \in [0.10, 0.15]\).

ORPO: SFT and preference in one loss

Why merge the stages

The usual pipeline is two stages: SFT on good answers, then DPO with the SFT model as a frozen reference. That means two training runs and, during DPO, a second model in memory. Hong et al. observed that SFT alone raises the probability of the rejected style too, because rejected answers are often similar in form to chosen ones. A small extra penalty on the rejected style is enough. [Certain] ORPO (Odds Ratio Preference Optimization) adds that penalty to the SFT loss and trains from the base model in one stage, without a reference model.

The math

ORPO works with the length-normalized probability of an answer (the geometric mean of its token probabilities) and its odds:

\[ P_\theta(y \mid x) = \exp\Big(\frac{1}{|y|} \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{\lt t})\Big), \qquad \text{odds}_\theta(y \mid x) = \frac{P_\theta(y \mid x)}{1 - P_\theta(y \mid x)} \tag{69.11} \]

The odds ratio compares the two answers, and the preference term is a logistic loss on its logarithm:

\[ \mathcal{L}_{\text{OR}} = -\log \sigma\Big(\log \frac{\text{odds}_\theta(y_w \mid x)}{\text{odds}_\theta(y_l \mid x)}\Big) \tag{69.12} \]
\[ \mathcal{L}_{\text{ORPO}} = \E_{(x, y_w, y_l)}\big[\mathcal{L}_{\text{SFT}} + \lambda\, \mathcal{L}_{\text{OR}}\big] \tag{69.13} \]
Read the symbols
\(|y|\)
the number of tokens in the answer
\(P_\theta\)
a number between 0 and 1, the "per-token" probability of the answer
\(\mathcal{L}_{\text{SFT}}\)
the usual next-token cross-entropy on the chosen answer \(y_w\) only
\(\lambda\)
weight of the preference term; the paper used 0.1 for Mistral-7B, 0.2 for Llama-2 7B, 0.25 for Phi-2

Why odds instead of plain probability ratios? The log-odds is \(\log P - \log(1 - P)\). Its gradient is

\[ \nabla_\theta \mathcal{L}_{\text{OR}} = -\,\delta \cdot \Big( \frac{\nabla_\theta \log P_\theta(y_w)}{1 - P_\theta(y_w)} - \frac{\nabla_\theta \log P_\theta(y_l)}{1 - P_\theta(y_l)} \Big), \qquad \delta = \Big[1 + \frac{\text{odds}_\theta(y_w)}{\text{odds}_\theta(y_l)}\Big]^{-1} \tag{69.14} \]

(The paper writes it as \(\delta \cdot h(d)\) with \(h(d)\) the bracket; the minus sign here means that a gradient descent step raises \(y_w\) and lowers \(y_l\).) Two things to notice. First, \(\delta\) shrinks to 0 when the chosen answer already has much larger odds: like DPO, a solved pair stops mattering. Second, the factor \(1/(1 - P)\) grows when \(P\) is close to 1, so the loss pushes extra hard against a rejected answer that the model finds very likely. The authors argue that the plain probability ratio pushes too hard and makes the model forget the SFT signal, while the odds ratio gives a milder contrast that works together with SFT. [Likely]

log-odds \(\log(P/(1-P))\)\(\log P\) (for comparison)chosenrejected
Figure 4. ORPO turns the mean log-probability of each answer into log-odds. The vertical gap between the two dots on the solid curve is the log odds ratio that goes into (69.12).
How to read Figure 4

For small probabilities (left side) log-odds and log-probability are almost the same line. Near \(P = 1\) (right side) the log-odds shoots up. Start from the default: the model gives the rejected answer the higher per-token probability, so the log odds ratio is negative and \(\mathcal{L}_{\text{OR}}\) is above \(\log 2 = 0.693\). Move the chosen slider to the right of the rejected one and watch \(\mathcal{L}_{\text{OR}}\) fall. Note that the SFT part (\(-\)mean log-prob of the chosen answer) is usually much larger than \(\lambda \mathcal{L}_{\text{OR}}\).

Worked example: the pair under ORPO

Mean log-probs: chosen \(-30/20 = -1.5\), rejected \(-50/40 = -1.25\). So \(P_w = e^{-1.5} = 0.223\), \(P_l = e^{-1.25} = 0.287\). Odds: \(0.223/0.777 = 0.287\) and \(0.287/0.713 = 0.402\). Log odds ratio \(= \log(0.287/0.402) = -0.335\).

\(\mathcal{L}_{\text{OR}} = -\log \sigma(-0.335) = 0.875\). SFT loss (mean token NLL of the chosen answer) \(= 1.5\). With \(\lambda = 0.1\): total \(= 1.5 + 0.0875 = 1.587\). Gradient factors: \(\delta = 0.583\), \(1/(1 - P_w) = 1.287\), \(1/(1 - P_l) = 1.402\).

Note that DPO said this pair was already right (margin +3), while ORPO says it is wrong. They measure different things: DPO measures change against the reference, ORPO measures the current per-token probabilities.

The ORPO paper fine-tuned Phi-2 (2.7B), Llama-2 (7B) and Mistral (7B) on UltraFeedback only. Its Mistral-ORPO models reached 12.20% on AlpacaEval 2.0, 66.19% on IFEval (instruction-level, loose) and 7.32 on MT-Bench, as reported by the authors. [Certain]

SimPO: average log-probability as the reward

Two problems with DPO's reward

Meng et al. point at two issues. First, DPO's reward is a sum over tokens, but decoding cares about something closer to the average log-probability per token. The two can disagree: the paper reports that for only about half of the training pairs does a correct DPO reward ranking also mean a correct ranking by average likelihood. [Certain] Second, the reference model costs memory and compute at every step. SimPO (Simple Preference Optimization) changes the reward to the average log-probability and drops the reference model:

\[ r_{\text{SimPO}}(x, y) = \frac{\beta}{|y|} \log \pi_\theta(y \mid x) = \frac{\beta}{|y|} \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{\lt t}) \tag{69.15} \]

and adds a target margin \(\gamma\) inside the Bradley-Terry loss, so the winner must beat the loser by at least a gap, not just by any amount:

\[ \mathcal{L}_{\text{SimPO}} = -\log \sigma\Big( \frac{\beta}{|y_w|} \log \pi_\theta(y_w \mid x) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l \mid x) - \gamma \Big) \tag{69.16} \]
Read the symbols
\(\beta\)
a scale for the reward; much larger than DPO's: the paper recommends 2.0 to 2.5
\(\gamma\)
the target reward margin; the paper recommends 0.5 to 1.5
\(\gamma / \beta\)
the required gap in mean log-probability per token; with \(\gamma = 1\), \(\beta = 2.5\) it is 0.4 nats per token

Why length normalization is needed

A summed log-probability is more negative for every extra token. Without a reference model to cancel this, a long answer starts with a big handicap compared to a short one, whatever its quality. The SimPO paper's ablation shows that removing length normalization was the most harmful change: outputs became long and repetitive, and the likelihood of answers became strongly tied to their length (Spearman correlation 0.82 without normalization versus 0.34 with it). [Certain] The average removes the direct effect of length. Figure 5 lets you see when the sum and the average disagree.

chosenrejected
Figure 5. Left: summed log-probability (what a reference-free reward without normalization would use). Right: mean log-probability per token (SimPO's reward before multiplying by \(\beta\)). Bars grow downward because log-probabilities are negative; shorter bars are better.
How to read Figure 5

In the default, the chosen answer is better per token (\(-0.9\) versus \(-1.1\)) but 2.5 times longer, so its sum is much worse (\(-180\) versus \(-88\)). A summed reward would call the rejected answer "better" and push the model toward short answers on this pair, and toward long ones on pairs where the long answer is chosen. The mean has no such length effect. The SimPO margin is \(2.5 \times 0.2 - 1 = -0.5\): still negative, because \(\gamma\) asks for a per-token gap of 0.4 and the gap is only 0.2. Set \(\gamma = 0\) and the loss drops below \(\log 2\).

Worked example: the pair under SimPO

Mean log-probs: chosen \(-1.5\), rejected \(-1.25\). With \(\beta = 2.5\) and \(\gamma = 1\): margin \(= 2.5 \times (-1.5 + 1.25) - 1 = -0.625 - 1 = -1.625\). Loss \(= -\log \sigma(-1.625) = 1.805\).

So SimPO strongly wants to change this pair, DPO (margin +3) does not, and a summed reference-free reward (\(-30\) versus \(-50\)) would call it solved. Three methods, three opinions about the same numbers, because each uses a different reward.

The SimPO paper reports gains over DPO of up to 6.4 points on AlpacaEval 2 and up to 7.5 points on Arena-Hard, and its best model (from Gemma-2-9B-it) reached a 72.4% length-controlled win rate on AlpacaEval 2 and 59.1% on Arena-Hard. [Certain] These are the authors' own numbers on their setups; independent comparisons often find the methods closer once each is tuned well. [Likely]

Side by side, and how to choose

All the pairwise members can be written as one form: a loss function \(\ell\) applied to a reward gap, minus an optional margin:

\[ \mathcal{L} = \ell\big( r(x, y_w) - r(x, y_l) - \gamma \big) \tag{69.17} \]

DPO: \(\ell(z) = -\log \sigma(z)\), \(r = \beta \log(\pi_\theta/\pi_{\text{ref}})\), \(\gamma = 0\). IPO: \(\ell(z) = (z - 1/(2\tau))^2\) with \(r = \log(\pi_\theta/\pi_{\text{ref}})\). SimPO: \(\ell = -\log\sigma\), \(r\) from (69.15), \(\gamma \gt 0\). ORPO: \(\ell = -\log\sigma\), \(r\) = log-odds, plus the SFT term. KTO replaces the pair with a comparison against \(z_0\).

The five losses at work: a toy you can train

Figures 6 and 7 train all five methods on one toy problem, so you can watch formula (69.17) in the loop. There is one prompt and six possible answers A to F: short and correct, detailed and correct, long and padded, short and vague, long and wrong, off-topic. They have lengths \((3, 8, 12, 2, 10, 5)\) tokens, reference probabilities \((0.25, 0.15, 0.10, 0.20, 0.10, 0.20)\), and hidden true qualities \((1.0, 1.5, 0.3, -0.5, -1.0, -1.5)\). The policy is a softmax over the six answers. To make the length matter as it does in a real model, each answer has one parameter per token: its logit is \(|y| \times \phi_y\), so one optimizer step changes a long answer’s total log-probability more than a short one’s. For ORPO’s odds we also need a per-token probability; we use \(\log \pi_\theta(y)/|y| - 1\), as if each token also paid 1 nat for the words that every answer shares (this constant cancels in all the other margins). Pairs are labeled with \(\sigma(q_i - q_j + b \cdot (|y_i| - |y_j|)/4)\), where \(b\) is a labeler length bias, and then flipped with probability “label noise”. All methods use Adam. We use \(\beta\) for DPO and KTO, \(\tau = \beta\) for IPO, \(\beta = 2.5\) and a margin \(\gamma\) for SimPO, and \(\lambda = 0.1\) for ORPO. KTO treats \(y_w\) as a desirable and \(y_l\) as an undesirable example, with \(\lambda_D = \lambda_U = 1\), and its \(z_0\) is the exact KL of the toy policy (69.7).

This toy shows how each loss pushes. It does not rank the methods: a real model shares parameters across answers in far more complex ways, and its results depend on tuning.

Running: one step of the chosen method
  1. draw a minibatch of 10 pairs \((y_w, y_l)\)
  2. compute \(\log \pi_\theta(y)\), \(\log \pi_\text{ref}(y)\) and \(|y|\) for both answers
  3. reward of each answer: log-ratio (DPO, IPO, KTO), log-odds (ORPO), or \(\beta \log \pi_\theta / |y|\) (SimPO)
  4. margin \(z = r(y_w) - r(y_l) - \gamma\) // (69.17); KTO compares each answer with \(z_0\)
  5. loss \(\ell(z)\) and its push \(-\ell'(z)\) for every pair
  6. Adam step on the mean loss
a pair (color of the method)KTO: an undesirable answer (green: desirable)hollow dot: the label was flipped by noisemean lossfraction of pairs with a positive margin
Figure 6. One method at a time. Left: the method’s loss as a function of its own margin, with every pair of the dataset as a dot at its current margin; the dots move as the policy trains. For ORPO the curve is the odds-ratio term \(-\log\sigma(\cdot)\) only (the SFT term does not depend on the margin). For KTO there are two curves, for desirable (green) and undesirable (red) answers, against \(r - z_0\). The dashed vertical line is IPO’s target \(1/(2\tau)\) or SimPO’s \(\gamma/\beta\). Training stops after 300 iterations.
How to read Figure 6

Start with DPO and press Play. The dots run to the right along the curve and never stop: the push \(\beta\sigma(-\beta h)\) is small but never zero. Some dots stay on the left, high-loss side. Many of them are hollow (9 of the 60 labels were flipped by noise), but not all: a pair’s margin depends only on its two answers, and when other pairs of the same two answers carry the opposite label, they cannot all be satisfied. With the defaults, after 300 iterations DPO has a mean loss of 0.468, 80.0% of pairs with a positive margin, and it has put 0.999 of the probability on B: KL 1.89 nats, close to the largest possible value here, \(\log(1/0.15) = 1.90\). Switch to IPO: the dots stop near and below the dashed target \(1/(2\tau) = 5\) (the largest IPO margin is 5.5, while DPO’s reach 28) and the KL stops at 0.83. Pairs whose labels disagree pull in opposite directions, so the squared loss stays large (15.2) even at the end. KTO ends at KL 0.91, SimPO at 0.79. ORPO moves least (KL 0.26): most of its loss is the SFT term on the chosen answers, so it learns roughly “how often each answer was chosen”. Raise \(\beta\) for DPO to 0.5 and the curve bends five times faster, so the dots stop pushing much sooner. (Speeds 1 and 2 run one line of the algorithm at a time, 0.65 s or 0.2 s per line; speeds 3 to 5 run whole iterations, more of them per frame. “Next line” and the next-step button work at any speed.)

Running: the five losses on one minibatch
  1. draw a minibatch of 10 pairs (KTO: \(y_w\) is desirable, \(y_l\) undesirable)
  2. DPO: \(-\log \sigma(\beta h)\), then an Adam step // (69.1)
  3. IPO: \((h - 1/(2\tau))^2\), then an Adam step // (69.4)
  4. KTO: \(\lambda_D - v(y_w) + \lambda_U - v(y_l)\), then an Adam step // (69.8), (69.9)
  5. ORPO: \(\mathcal{L}_\text{SFT} + \lambda\,\mathcal{L}_\text{OR}\), then an Adam step // (69.13)
  6. SimPO: \(-\log \sigma\big(\beta(\bar\ell_w - \bar\ell_l) - \gamma\big)\) with \(\bar\ell = \log\pi_\theta(y)/|y|\), then an Adam step // (69.16)
  7. record quality, KL and length of all five policies
DPOIPOKTOORPOSimPOreference
Figure 7. A race: five policies start from the same reference and see the same minibatches of the same pairs, each trained with its own loss. The table at the bottom right shows each policy’s probability for answers A to F (the number after the letter is the length in tokens). Training stops after 400 iterations.
How to read Figure 7

With speed 1, Next line walks through one minibatch: each line computes one method’s loss and the panel shows its numbers on the first pair. Then Play at speed 4. With the defaults (\(\beta = \tau = 0.1\), \(\gamma = 0.5\), noise 0.10, length bias 0.5, 100 pairs, seed 1), after 400 iterations: DPO puts all its probability on B (quality 1.49, KL 1.87 nats); IPO stays closer (quality 1.02, KL 0.78); KTO splits between B and the long padded C (KL 1.40, length 9.9 tokens); SimPO reaches quality 1.02 at KL 0.77 with length 6.3; ORPO barely leaves the reference (KL 0.16, quality 0.28). Now raise the length bias to 1.0: the labelers often prefer padding, and DPO follows them to C (0.77 of its probability, length 11.1, quality 0.57). SimPO’s per-token reward keeps it shorter (8.3 tokens, quality 0.77). Set the length bias to 0: every method ends at 8 tokens or fewer.

MethodDataReference model?Separate SFT first?RewardLoss shapeMain fix
DPOpairsyesyes\(\beta \log \pi_\theta/\pi_{\text{ref}}\), summedlogisticremoves the reward model and RL
IPOpairsyesyeslog-ratio, summedsquared, target \(1/(2\tau)\)finite target, no overfitting on certain pairs
KTOsingle answers, good or badyesusually [Likely]log-ratio vs \(z_0\)sigmoid value, weights \(\lambda_D, \lambda_U\)no pairs needed; handles imbalance
ORPOpairsnonolog-odds of mean token probabilitySFT + \(\lambda\) logisticone stage, one model in memory
SimPOpairsnoyes\(\beta \times\) mean log-problogistic with margin \(\gamma\)length-normalized reward, no reference

A practical way to choose, as a starting point and not a rule:

  • You have clean pairs and memory for two models: start with DPO, it is the best-understood baseline.
  • Your pairs are noisy or each appears once and the model drifts or becomes overconfident: try IPO, or DPO with label smoothing (a small chance that each label is flipped). [Likely]
  • You only have thumbs-up/down logs, or very unbalanced good and bad sets: KTO.
  • You start from a base model and want one short run with one model in memory: ORPO.
  • Answers get longer and longer during DPO, or memory is tight: SimPO (tune the learning rate carefully).

What none of them can do: they all learn from a fixed set of answers. They never see the model's own new mistakes during training, and their implicit rewards are only trained on the answers in the data. For that you need online methods that sample from the current model, such as iterative or online DPO and the RL methods of session 70. [Likely]

Algorithms and code

Algorithm 1: one training step for a pairwise method (DPO, IPO, SimPO, ORPO)
  1. input: batch of \((x, y_w, y_l)\), policy \(\pi_\theta\), reference \(\pi_{\text{ref}}\) (only for DPO and IPO), method and its hyper-parameters
  2. build two sequences per example: prompt + chosen, prompt + rejected; mask the prompt tokens and the padding
  3. run the policy once on the concatenated batch; take per-token log-probs of the answer tokens
  4. \(s_w, s_l \leftarrow\) sums of those log-probs; \(a_w = s_w / |y_w|\), \(a_l = s_l / |y_l|\) // averages for SimPO and ORPO
  5. if DPO or IPO: run \(\pi_{\text{ref}}\) without gradients to get \(s^{\text{ref}}_w, s^{\text{ref}}_l\); \(h \leftarrow (s_w - s^{\text{ref}}_w) - (s_l - s^{\text{ref}}_l)\)
  6. DPO: \(L = -\log\sigma(\beta h)\) // (69.1); IPO: \(L = (h - 1/(2\tau))^2\) // (69.4)
  7. SimPO: \(L = -\log\sigma(\beta(a_w - a_l) - \gamma)\) // (69.16)
  8. ORPO: \(L = -a_w + \lambda \cdot \big(-\log\sigma(\text{logodds}(a_w) - \text{logodds}(a_l))\big)\), \(\text{logodds}(a) = a - \log(1 - e^{a})\) // (69.11)–(69.13)
  9. average \(L\) over the batch; backward; clip; optimizer step
  10. log the margin, the chosen and rejected rewards, and the mean answer length
Algorithm 2: one KTO step on unpaired labels
  1. input: batch of \((x_i, y_i, \text{label}_i)\) with label desirable or undesirable, \(\beta\), \(\lambda_D\), \(\lambda_U\)
  2. \(r_i \leftarrow \log \pi_\theta(y_i \mid x_i) - \log \pi_{\text{ref}}(y_i \mid x_i)\) // (69.6), summed over answer tokens
  3. make mismatched pairs: \(j = (i \bmod m) + 1\), pair prompt \(x_i\) with answer \(y_j\)
  4. without gradients: \(z_0 \leftarrow \max\big(0, \operatorname{mean}_i [\log \pi_\theta(y_j \mid x_i) - \log \pi_{\text{ref}}(y_j \mid x_i)]\big)\) // (69.7)
  5. desirable: \(\ell_i = \lambda_D (1 - \sigma(\beta (r_i - z_0)))\); undesirable: \(\ell_i = \lambda_U (1 - \sigma(\beta (z_0 - r_i)))\) // (69.8), (69.9)
  6. \(L = \operatorname{mean}_i\, \ell_i\); backward; optimizer step
  7. check rule (69.10) for the whole dataset when you set \(\lambda_D, \lambda_U\)

The PyTorch code below computes all five losses from a tiny causal language model, so it runs on a CPU in seconds. Then it trains the same start with DPO and with IPO and prints the final margins. It is deterministic.

import copy, torch
import torch.nn.functional as F
torch.manual_seed(0)

V, D = 32, 16                                   # tiny vocabulary and width
class TinyLM(torch.nn.Module):                  # stands in for a real causal LM
    def __init__(self):
        super().__init__()
        self.emb = torch.nn.Embedding(V, D); self.out = torch.nn.Linear(D, V)
    def forward(self, ids):                     # (B, T) to logits (B, T, V)
        return self.out(torch.tanh(self.emb(ids).cumsum(1)))

def seq_logps(model, ids, resp_mask):
    """Sum and mean of log p(token) over response tokens only."""
    logits = model(ids[:, :-1])
    lp = torch.gather(F.log_softmax(logits, -1), 2, ids[:, 1:, None]).squeeze(2)
    m = resp_mask[:, 1:].float()                # predict token t from tokens before t
    s = (lp * m).sum(1)
    return s, s / m.sum(1)                      # (summed, length-normalized)

# losses; every input is a per-example tensor of shape (B,)
def dpo(pw, pl, rw, rl, beta=0.1):              # (69.1)
    return -F.logsigmoid(beta * ((pw - rw) - (pl - rl))).mean()
def ipo(pw, pl, rw, rl, tau=0.1):               # (69.4)
    return (((pw - rw) - (pl - rl)) - 1 / (2 * tau)).pow(2).mean()
def simpo(aw, al, beta=2.5, gamma=1.0):         # (69.16), aw/al are mean log-probs
    return -F.logsigmoid(beta * (aw - al) - gamma).mean()
def orpo(aw, al, lam=0.1):                      # (69.11)-(69.13)
    log_odds = lambda a: a - torch.log1p(-torch.exp(a))   # log(P/(1-P)), P = exp(a)
    l_or = -F.logsigmoid(log_odds(aw) - log_odds(al))
    return (-aw + lam * l_or).mean()            # SFT loss on the chosen answer + odds-ratio term
def kto(r, desirable, z0, beta=0.1, lam_d=1.0, lam_u=1.0):  # (69.8)-(69.9)
    v = torch.where(desirable, lam_d * torch.sigmoid(beta * (r - z0)),
                               lam_u * torch.sigmoid(beta * (z0 - r)))
    lam = torch.where(desirable, torch.full_like(r, lam_d), torch.full_like(r, lam_u))
    return (lam - v).mean()

# a fake batch: 4 prompts of 4 tokens, chosen and rejected answers of different lengths
B, P = 4, 4
def make(n):                                    # prompt + n answer tokens, right-padded to 12
    ids = torch.zeros(B, 12, dtype=torch.long); mask = torch.zeros(B, 12, dtype=torch.bool)
    ids[:, :P + n] = torch.randint(1, V, (B, P + n)); mask[:, P:P + n] = True
    return ids, mask
prompts = torch.randint(1, V, (B, P))
cw, mw = make(6); cl, ml = make(8)
cw[:, :P] = prompts; cl[:, :P] = prompts         # same prompt for chosen and rejected

policy = TinyLM(); ref = copy.deepcopy(policy).eval()
for p_ in ref.parameters(): p_.requires_grad_(False)

with torch.no_grad():
    rw, _ = seq_logps(ref, cw, mw); rl, _ = seq_logps(ref, cl, ml)
pw, aw = seq_logps(policy, cw, mw); pl, al = seq_logps(policy, cl, ml)
print("at start: DPO %.4f  IPO %.4f  SimPO %.4f  ORPO %.4f" % (
    dpo(pw, pl, rw, rl).item(), ipo(pw, pl, rw, rl).item(),
    simpo(aw, al).item(), orpo(aw, al).item()))     # DPO = log 2 and IPO = 25 when policy == ref

# KTO: unpaired labels. Reference point z0 from mismatched pairs (shift prompts by one), no gradient
with torch.no_grad():
    mis = cw.clone(); mis[:, :P] = prompts.roll(1, 0)
    z0 = (seq_logps(policy, mis, mw)[0] - seq_logps(ref, mis, mw)[0]).mean().clamp(min=0)
r = torch.cat([pw - rw, pl - rl]); lab = torch.tensor([True] * B + [False] * B)
print("KTO %.4f  z0 %.4f" % (kto(r, lab, z0).item(), z0.item()))

# train with DPO and with IPO from the same start; watch the margin
for name, lossf in [("DPO", dpo), ("IPO", ipo)]:
    pol = copy.deepcopy(ref).train()
    for p_ in pol.parameters(): p_.requires_grad_(True)
    opt = torch.optim.Adam(pol.parameters(), lr=1e-2)
    for step in range(300):
        pw, _ = seq_logps(pol, cw, mw); pl, _ = seq_logps(pol, cl, ml)
        loss = lossf(pw, pl, rw, rl)
        opt.zero_grad(); loss.backward(); opt.step()
    margin = ((pw - rw) - (pl - rl)).mean().item()
    print("%s after 300 steps: mean margin %.2f" % (name, margin))   # IPO stops near 1/(2*tau) = 5

On PyTorch 2.x (CPU) it prints, at start, DPO 0.6931 (\(= \log 2\), because policy and reference are the same), IPO 25.0000 (\(= 5^2\)), SimPO 1.1930, ORPO 3.6044, then KTO 0.5000 with \(z_0 = 0\). After 300 steps the mean DPO margin is about 85.7 and still growing, while the IPO margin is 5.00. Exact digits may differ slightly across PyTorch versions. [Likely]

Where this is used

SettingTypical methodWhy
Chat model post-training (Llama 3, Zephyr)DPO after SFT, often in several rounds with fresh pairsstable, cheap, well understood [Certain for the two named models]
Product feedback logs (thumbs up/down in a chat app)KTOsingle labels, unbalanced classes
Small team, one GPU, base model to chat modelORPOone stage, no reference model in memory
Verbose models (answers keep growing in DPO)SimPO, or DPO with a length penaltylength-normalized reward [Likely]
Noisy crowd-sourced pairsIPO, or DPO with label smoothingbounded target, less overconfidence [Likely]
Vision-language models: fewer hallucinated objects in image captionsDPO-style losses on pairs of faithful vs hallucinated captionsthe same losses work for any model that gives token log-probs (session 190) [Likely]
Video captioning and video question answeringDPO or KTO on answers judged against the video (by people or a judge model)a judge can label single answers as faithful or not, which fits KTO [Speculation]
Video key-moment detection and Video-RAG answersKTO on user clicks ("this clip answered my question" or not)clicks are single, unpaired signals [Speculation]

Common mistakes

  • Mixing sums and means. DPO, IPO and KTO use summed token log-probs in the log-ratio; SimPO and ORPO use means. Using means in DPO silently changes the useful range of \(\beta\).
  • Counting prompt or padding tokens. Log-probs must cover answer tokens only. Including the prompt adds the same constant to both answers in DPO (so it cancels there), but it does not cancel in SimPO, ORPO or KTO.
  • Reusing DPO's \(\beta\) everywhere. DPO \(\beta \approx 0.1\), SimPO \(\beta \approx 2\)–2.5, ORPO \(\lambda \approx 0.1\)–0.25, IPO \(\tau\) on its own scale. Each method needs its own search.
  • Letting gradients flow through KTO's \(z_0\). The paper stops the gradient; without that, the model can lower \(z_0\) instead of improving answers.
  • Ignoring class imbalance in KTO. With 10 times more thumbs-up than thumbs-down and equal weights, the bad examples barely matter. Set \(\lambda_D, \lambda_U\) with rule (69.10).
  • Reading a falling loss as success. DPO's loss can keep falling because the loser's probability goes to zero while the winner's probability also falls. Log the chosen and rejected log-probs separately, the margin, and the answer length.
  • Training reference-free methods too long or too fast. SimPO and ORPO have no anchor to the start; too high a learning rate or too many epochs causes drift and broken outputs. [Likely]

Interview questions

What is the implicit reward in DPO, and what is the "margin"?basic
The implicit reward of an answer is \(\beta \log(\pi_\theta(y \mid x)/\pi_{\text{ref}}(y \mid x))\): how much the model has raised that answer compared with the reference, times \(\beta\). The margin \(h\) is the chosen answer's log-ratio minus the rejected one's. DPO is logistic regression on \(\beta h\).
Why does DPO overfit when preferences are deterministic, and how does IPO fix it?core
For a pair labeled with win rate \(p\), DPO's best margin is \(\log(p/(1-p))/\beta\), which is infinite at \(p = 1\). Datasets usually show each pair once, so \(\hat p = 1\) for every pair, and DPO pushes the loser's probability to 0 for any \(\beta\): the KL control is lost. IPO regresses the same margin to a fixed target \(1/(2\tau)\) with a squared loss, so its best margin is \((2p-1)/(2\tau)\), finite even at \(p = 1\), and the gradient reverses past the target.
Write the gradient of the DPO loss with respect to the margin and explain its sign.core
\(\partial \mathcal{L}/\partial h = -\beta\sigma(-\beta h)\). It is negative for every \(h\), so gradient descent always increases the margin; it shrinks for large margins but never reaches zero. The factor \(\sigma(-\beta h)\) is a per-pair weight: pairs the model gets wrong get weight near 1, pairs it gets very right get weight near 0.
How does KTO train without pairs? What is \(z_0\) and how is it computed?core
Each answer is compared to a reference point instead of another answer. The reward is the log-ratio \(r = \log(\pi_\theta/\pi_{\text{ref}})\); good answers should have \(r \gt z_0\), bad answers \(r \lt z_0\). The loss is \(\lambda_y - v\) with \(v = \lambda_D\sigma(\beta(r - z_0))\) or \(\lambda_U\sigma(\beta(z_0 - r))\). \(z_0\) estimates the KL between policy and reference: the mean log-ratio of mismatched prompt-answer pairs in the micro-batch, clipped at 0, with no gradient.
Your feedback data has 20,000 thumbs-up and 2,000 thumbs-down. How do you set KTO's weights?core
Use \(\lambda_D n_D / (\lambda_U n_U) \in [1, 1.5]\). With \(\lambda_U = 1\): \(1 \le 20000\lambda_D/2000 \le 1.5\), so \(\lambda_D \in [0.10, 0.15]\). Alternatively keep \(\lambda_D = 1\) and set \(\lambda_U\) between 6.67 and 10.
What does ORPO change compared with SFT followed by DPO?core
ORPO adds \(\lambda\) times an odds-ratio term to the SFT loss and trains in one stage from the base model, with no reference model. The term is \(-\log\sigma(\log(\text{odds}(y_w)/\text{odds}(y_l)))\), with odds computed from the length-normalized answer probability. It saves one training run and one model in memory. The SFT part keeps the model learning the chosen answers, so no anchor model is needed.
Why does ORPO use odds instead of a plain probability ratio?advanced
The log-odds \(\log P - \log(1-P)\) has gradient \(\nabla \log P/(1-P)\). Its contrast is milder than pushing the probability ratio to extremes, and the authors argue a probability-ratio term pushes the rejected likelihood down too hard and fights the SFT term. The \(1/(1-P)\) factor still pushes hard against rejected answers that are already very likely. The ORPO weight \(\delta = 1/(1 + \text{OR})\) fades for solved pairs.
Why does SimPO normalize by length, and what is \(\gamma\) for?core
Without a reference model, summed log-probabilities penalize every extra token, so the reward mostly measures length, not quality; the paper's ablation without normalization gave long, repetitive outputs. The average log-prob per token removes this and matches what decoding ranks. \(\gamma\) is a target margin inside the sigmoid: the winner must beat the loser by at least \(\gamma/\beta\) nats per token before the loss gets small, which improves separation.
Which of DPO, IPO, KTO, ORPO and SimPO need a reference model? What does that cost?basic
DPO, IPO and KTO do; ORPO and SimPO do not. The reference is a frozen copy of the starting model: for a 7B model in BF16, about 14 GB more weights plus its forward passes. A common trick is to precompute the reference log-probs once for the whole dataset.
Name a limitation that all these methods share.advanced
They are offline: they learn only from answers in a fixed dataset, usually produced by other models. The implicit reward is never checked on the model's own new answers, so the model can exploit regions the data does not cover, and it cannot learn from its own mistakes. Online or iterative variants (generate new answers, label them, train again) and RL methods such as PPO and GRPO address this. [Likely]
During DPO training the loss falls nicely but the model's answers get worse. What do you check?advanced
Log the chosen and rejected log-probs separately: often both fall, the rejected one faster, so the margin grows while the chosen answer becomes less likely. Check answer length growth, the KL to the reference, and whether many pairs have near-certain labels. Fixes: larger \(\beta\), fewer epochs, lower learning rate, IPO or label smoothing, an SFT term on the chosen answer, or SimPO-style length normalization.

Exercises

  1. With \(\beta = 0.2\), what margin \(h\) does DPO aim for when \(p = 0.9\)? What does IPO aim for with \(\tau = 0.2\)?
  2. Compute the DPO loss and its gradient with respect to \(h\) at \(h = 0\) for any \(\beta\).
  3. IPO, \(\tau = 0.25\), current margin \(h = 3\). Compute the loss and the gradient. Which way does it push?
  4. KTO: a desirable answer has reward \(r = 0.4\) and \(z_0 = 0.4\). What is its loss with \(\lambda_D = 1\)? What is the gradient with respect to \(r\) if \(\beta = 0.1\)?
  5. SimPO with \(\beta = 2\), \(\gamma = 1\): the chosen answer has mean log-prob \(-0.8\), the rejected \(-1.4\). Compute the margin and the loss.
  6. ORPO: both answers have the same mean log-prob. What is \(\mathcal{L}_{\text{OR}}\)? What does that tell you about the starting point of ORPO training?
Answerscheck
1. DPO: \(\log(0.9/0.1)/0.2 = 2.197/0.2 = 10.99\). IPO: \((2 \times 0.9 - 1)/(2 \times 0.2) = 0.8/0.4 = 2\). 2. Loss \(-\log\sigma(0) = \log 2 = 0.693\); gradient \(-\beta\sigma(0) = -\beta/2\). 3. Target \(1/(2 \times 0.25) = 2\). Loss \((3 - 2)^2 = 1\); gradient \(2(3 - 2) = +2\): it pushes the margin down, toward 2. 4. \(1 - \sigma(0) = 0.5\); gradient \(-\beta\sigma(0)(1 - \sigma(0)) = -0.1 \times 0.25 = -0.025\). 5. Margin \(2 \times (-0.8 + 1.4) - 1 = 1.2 - 1 = 0.2\); loss \(-\log\sigma(0.2) = 0.598\). 6. Equal odds, log odds ratio 0, \(\mathcal{L}_{\text{OR}} = \log 2 = 0.693\). At the start the two answers are usually close, so the odds-ratio term starts near \(\lambda \log 2\), small next to the SFT loss.

Key takeaways

  • Every DPO-family method is: an implicit reward per answer, a comparison (pair or reference point), and a loss shape on the gap (69.17).
  • DPO's logistic loss always pushes the margin up (69.2); with near-certain labels its best margin is infinite (69.3), so \(\beta\) stops acting as a leash.
  • IPO regresses the margin to \(1/(2\tau)\) (69.4), so its best margin \((2p-1)/(2\tau)\) is finite (69.5).
  • KTO learns from single good or bad labels by comparing each reward with a reference point \(z_0\) (69.6)–(69.9); set class weights with (69.10).
  • ORPO adds \(\lambda\) times a log-odds-ratio loss to SFT (69.11)–(69.13): one stage, no reference model.
  • SimPO uses \(\beta \times\) mean log-prob as the reward with a margin \(\gamma\) (69.15)–(69.16): length-normalized, no reference model, but needs a careful learning rate.
  • The loss shape decides how hard each pair pushes. In the toy race of Figures 6 and 7, DPO pushes margins without limit and follows a labeler length bias furthest, while IPO stops near its target.
  • All are offline: they never learn from the model's own new answers. That is the job of online and RL methods.

Key formulas

\[ \mathcal{L}_{\text{DPO}} = -\log\sigma(\beta h), \quad \mathcal{L}_{\text{IPO}} = \Big(h - \frac{1}{2\tau}\Big)^2, \quad h = \log\frac{\pi_\theta(y_w)}{\pi_{\text{ref}}(y_w)} - \log\frac{\pi_\theta(y_l)}{\pi_{\text{ref}}(y_l)} \]
\[ \mathcal{L}_{\text{KTO}} = \lambda_y - \lambda_y\,\sigma\big(\pm\beta(r_\theta - z_0)\big), \qquad \mathcal{L}_{\text{ORPO}} = \mathcal{L}_{\text{SFT}} - \lambda\log\sigma\Big(\log\frac{\text{odds}(y_w)}{\text{odds}(y_l)}\Big) \]
\[ \mathcal{L}_{\text{SimPO}} = -\log\sigma\Big(\frac{\beta}{|y_w|}\log\pi_\theta(y_w) - \frac{\beta}{|y_l|}\log\pi_\theta(y_l) - \gamma\Big) \]

References for this session

  1. Azar et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences
  2. Ethayarajh et al. (2024). KTO: Model Alignment as Prospect Theoretic Optimization
  3. Hong et al. (2024). ORPO: Monolithic Preference Optimization without Reference Model
  4. Meng et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward
  5. Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  6. Azar et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences
  7. Ethayarajh et al. (2024). KTO: Model Alignment as Prospect Theoretic Optimization
  8. Hong et al. (2024). ORPO: Monolithic Preference Optimization without Reference Model
  9. Meng et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward
  10. Lambert (2025). Reinforcement Learning from Human Feedback (book)
  11. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 10: Alignment
  12. Tunstall et al. (2023). Zephyr: Direct Distillation of LM Alignment
  13. Llama Team, Meta (2024). The Llama 3 Herd of Models
  14. Hugging Face (code). TRL library