VLM preference optimization and RL
After SFT, a vision-language model still describes things that are not in the image and still makes mistakes on counts, charts and geometry. This session explains the last training stage that fixes this: preference tuning (DPO, mDPO, MPO), fine-grained human corrections (RLHF-V), and RL with answers a program can check (GRPO). It ends with how to design rewards for vision tasks, such as exact match, IoU and format rewards, and how models hack them.
What you will learn
- Why SFT is not enough for a VLM: it never pushes down the model's own hallucinations, which come from the language prior
- DPO for VLMs, its gradient, and the 'unconditional preference' trap: pairs that can be judged without the image; mDPO's image-conditioned pair and anchor
- Where preference pairs come from: human rankings, human corrections, AI feedback, and rules such as correctness sampling and Dropout NTP
- RLHF-V: fine-grained corrections with dense DPO, where edited tokens count gamma = 5 times
- MPO: the 0.8 / 0.2 / 1.0 mix of DPO, BCO and SFT losses, and how it raised InternVL2-8B on MathVista from 58.3 to 67.0
- GRPO for visual reasoning with checkable answers: group advantages, the KL estimator, and why all-right or all-wrong groups give no signal
- Reward design for vision tasks: normalized exact match, numeric tolerance, IoU and temporal IoU, format rewards, and common reward hacks
Why this matters
After supervised fine-tuning, a vision-language model (VLM) follows instructions. But it still says things that are not in the image, and it still makes mistakes when it counts, reads a chart or solves a geometry problem. This session is about the last training stage that fixes these two problems: preference optimization and reinforcement learning (RL).
In session 189 the model learned from good answers only. Supervised fine-tuning (SFT) says “write this”. It never says “do not write that”. So the model never learns which of its own habits are bad. A VLM has one very strong bad habit: its language model has read billions of sentences, so it knows that “a dog on a sofa” often comes with “a cat” or “a lamp”. When the image is unclear, it writes what is likely in text, not what is in the picture. This is called hallucination: fluent text that the image does not support (see session 156).
The fix is feedback on the model’s own answers. There are three common forms, shown in Figure 1:
- Preference pairs (DPO, MPO): for one image and question, a better answer and a worse answer. The model learns to prefer the better one.
- Fine-grained corrections (RLHF-V): a person fixes only the wrong words in the model’s answer. The training then focuses on exactly those words.
- RL with a checker (GRPO): when the answer can be checked by a program (a count, a chart value, a box position), the model samples several answers, a rule scores them, and the model moves toward the ones that scored well.
Real systems use all three. LLaVA-RLHF trained a reward model on human comparisons and then optimized the VLM against it with RL [Certain]. RLHF-V used about 1.4k human-corrected answers and reported a 34.8% drop in hallucination rate on its base model [Certain]. MPO built about 3 million preference samples with almost no humans and raised InternVL2-8B on MathVista from 58.3 to 67.0 [Certain]. And many 2025 reasoning VLMs, such as Vision-R1, finish with GRPO on problems that have checkable answers [Certain]. By the end of this session you will know how each one works, why it works, and how to pick one.
From SFT to preferences: what changes for a VLM
A preference pair for a VLM has four parts: an image \(I\), a question \(x\), a chosen answer \(y_c\) and a rejected answer \(y_r\). The pair says only “\(y_c\) is better than \(y_r\) for this image”. It does not say how much better, and the chosen answer does not have to be perfect.
The usual model of such a choice is the Bradley–Terry model: each answer has a hidden score (a reward) \(r\), and the chance that people prefer \(y_c\) depends only on the difference of the two scores:
Classic RLHF first trains a reward model with this formula and then runs PPO against it (see session 67 and session 66). LLaVA-RLHF did this for a VLM; it also gave the reward model extra facts, such as the image caption, so that it was harder to fool (“Factually Augmented RLHF”) [Certain]. That pipeline needs four models in memory and is hard to tune.
DPO in one formula
Direct Preference Optimization (DPO) skips the reward model. It uses the policy itself as the reward: the reward of an answer is how much more likely the trained model \(\pi_\theta\) makes it, compared with a frozen copy \(\pi_0\) of the SFT model (the reference model). For a VLM, the image is simply part of the condition (see session 68):
- Read the symbols
- \(I, x\)
- the image (as visual tokens) and the text question
- \(\pi_\theta(y \mid I, x)\)
- probability of the whole answer under the model we train: the product of its token probabilities
- \(\pi_0\)
- the frozen SFT model; it never changes during this stage
- \(\Delta_c, \Delta_r\)
- log-ratios: how much the model has raised (positive) or lowered (negative) each answer since SFT
- \(\beta\)
- how strongly we stay close to \(\pi_0\); common values are 0.1 (MPO) to 0.5 (RLHF-V)
- \(\beta\Delta_c - \beta\Delta_r\)
- the margin \(z\): the difference of the two implicit rewards
The gradient explains the behaviour. With \(z = \beta(\Delta_c - \Delta_r)\):
Each step pushes the chosen answer up and the rejected answer down. The weight \(\sigma(-z)\) is large when the model still ranks the pair wrongly (\(z \lt 0\)) and goes to 0 when the pair is already well separated. So DPO spends its effort on the pairs it gets wrong. Explore this in Figure 2.
The chosen caption has log-probability \(-42.0\) under the model and \(-43.0\) under the reference, so \(\Delta_c = 1.0\). The rejected caption has \(-40.5\) and \(-40.0\), so \(\Delta_r = -0.5\). Note that the rejected caption is still more likely than the chosen one in absolute terms; DPO only looks at the changes.
With \(\beta = 0.1\): the margin is \(z = 0.1 \times (1.0 - (-0.5)) = 0.15\). The loss is \(-\log \sigma(0.15) = 0.621\), a little below the starting value \(\log 2 = 0.693\) (at the start \(\pi_\theta = \pi_0\), so \(z = 0\)). The gradient weight is \(\sigma(-0.15) = 0.463\): the pair still gets almost half of the full push.
How to read Figure 2
Start at the defaults: the loss is 0.621 and the weight is 0.463, as in the worked example. Move \(\Delta_r\) up to 1.0: now the rejected answer has gained as much as the chosen one, the margin is 0, and the loss is back to \(\log 2\). Now lower both sliders by the same amount: the loss does not change. DPO cares only about the gap, so it is happy to make both answers less likely. This is one reason MPO adds an SFT term and mDPO adds an “anchor” (below). Finally raise \(\beta\): the same log-ratios give a larger margin, so the loss saturates sooner and the model is held closer to \(\pi_0\).
The VLM trap: preferences that ignore the image
Here is the problem that is special to VLMs. In most pairs, the chosen and the rejected answer differ in words (“one dog” vs “two dogs”). The model can often tell which one is better from the text alone: the rejected one is longer, more detailed, or uses typical hallucination phrases. Then DPO can lower its loss without looking at the image. Wang et al. (mDPO) tested this: they trained with DPO when the image was removed from the pairs and found that the model performed about as well as with the image on some hallucination benchmarks [Likely]. They call this the unconditional preference problem: the model overlooks the image condition [Certain].
The fix in mDPO is a second pair in which the answer is the same and the image changes. Take the chosen answer \(y_c\). Under the real image \(I\) it should be likely; under a corrupted image \(I'\) (the paper makes it by randomly cropping the image; its default setting is called “Crop 0–20%”) it should be less likely:
The only way to lower this loss is to make the answer depend on the image. mDPO also adds an anchor term \(-\log\sigma(\beta\Delta_c - \delta)\) with \(\delta = 0\), which pushes the chosen answer’s reward to be positive, so that its likelihood cannot fall (the problem you saw in Figure 2). The full loss is the sum of DPO, CoPO and the anchor [Certain]. On Bunny-v1.0-3B with 10K pairs from the Silkie dataset, Object HalBench CHAIRs (share of captions that contain a hallucinated object) went from 44.3 with DPO to 27.0 with mDPO [Certain].
Where preference data comes from
Pairs are the expensive part. There are four main sources. Each trades cost against how well the pair targets the model’s real errors.
| Source | How the pair is made | Example | Strength / weakness |
|---|---|---|---|
| Human ranking | People see two model answers and pick the better (or less hallucinated) one | LLaVA-RLHF | Real human judgment; slow, costly, and coarse: one bit per pair |
| Human correction | People edit only the wrong parts of the model’s answer; edited = chosen, original = rejected | RLHF-V: about 1.4k samples, 2.65 corrected segments per answer on average | Very precise signal, small data; still needs trained annotators |
| AI feedback | A strong VLM scores or ranks sampled answers | Silkie (GPT-4V feedback [Likely], 80K samples), RLAIF-V (open-source models as judges) | Cheap and large; inherits the judge’s own hallucinations |
| Rules and tricks | Known answer: correct samples = chosen, wrong = rejected. No known answer: make a bad answer on purpose | MPO / MMPR: about 3M samples; Dropout NTP | Scales without people; needs ground truth or a good “corruption” |
RLAIF-V reports that its 7B model cut object hallucination by 80.7% and overall hallucination by 33.7% with open-source AI feedback only [Certain]. Its key idea is to ask the judge simple yes/no questions about small claims inside an answer instead of asking “which answer is better?” as a whole [Likely]. Other common tricks make the rejected answer by editing the chosen one: swap an object, change a count, or change a color [Likely]. These “injected” negatives are cheap, but they are not the model’s own errors, so they teach less than pairs built from the model’s samples.
A rule that holds for all sources: the pairs should come from (or be close to) the model you are training. Pairs built from another model’s answers are off-policy: the model may never write such text, so pushing it down changes little (see session 71).
Fine-grained corrections: RLHF-V
A long caption with one wrong phrase is a weak training signal if it is treated as one unit. The pair “whole caption A is better than whole caption B” does not say which words were wrong. RLHF-V changes how the data is collected and how the loss reads it.
Data. The base model (Muffin, a 13B VLM) writes a description. An annotator rewrites only the hallucinated segments. In their data, objects were 41.2% of the errors, positions 20.3%, numbers 16.5% and attributes 10.0% [Certain]. The corrected answer is \(y_c\); the original is \(y_r\). The two answers share most of their words.
Loss. RLHF-V calls its method Dense Direct Preference Optimization (DDPO). It replaces the plain sum of token log-probabilities by a weighted average that counts the changed tokens \(\gamma\) times:
- Read the symbols
- \(y_u\)
- the unchanged tokens of the answer
- \(y_e\)
- the edited tokens: the corrected words in \(y_c\), or the wrong words in \(y_r\)
- \(\gamma\)
- how much more an edited token counts; RLHF-V uses \(\gamma = 5\)
- \(N\)
- the normalizer, so that a longer answer does not get a larger score just for being long
This score is used in place of \(\log\pi\) inside the DPO loss (190.2), for both the policy and the reference. RLHF-V trained with \(\beta = 0.5\) and learning rate \(5\times10^{-7}\) for 7 epochs, in under one hour on 8 A100 GPUs [Certain]. On Object HalBench, the share of responses with a hallucination went from 50.5% to 12.2% [Certain].
One more detail makes corrections even more useful. In a corrected pair, the words before the first edit are identical and have identical contexts, so their log-ratios are the same in \(y_c\) and \(y_r\) and cancel exactly in \(\Delta_c - \Delta_r\). Words after an edit have a slightly different context, so they cancel only roughly. In effect, the pair already points at the edit; \(\gamma\) makes it point harder. Try it in Figure 3.
How to read Figure 3
By default 3 of the 19 words are edits (“a lamp”, “closed”). With plain DPO (\(\gamma = 1\)) they are 15.8% of the score: the 16 unchanged words dominate the average. With \(\gamma = 5\) they become \(15/31 = 48.4\%\). Click words to mark more or fewer edits. Notice that one edit in a long caption gets a tiny share at \(\gamma = 1\); that is exactly the case where a sequence-level pair is weakest and DDPO helps most.
MPO: three losses for reasoning and hallucination
Mixed Preference Optimization (MPO), from the InternVL team, was built for a different weakness: chain-of-thought (CoT) reasoning. The authors found that a VLM often scores lower with CoT than with a direct answer (InternVL2-8B: 58.3 direct vs 56.8 CoT on MathVista) [Certain]. SFT on CoT data had not taught it to reason well. Their answer combines three losses:
The three parts teach three different things:
- Preference loss (DPO, 190.2): which of two answers is better. This is a relative signal.
- Quality loss (BCO, binary classifier optimization): is this answer good or bad on its own? Each answer is judged alone, against a moving average \(\delta\) of past rewards:
- Generation loss (SFT): the plain next-token loss on the chosen answer, divided by its length: \(\mathcal{L}_{\text{SFT}} = -\log \pi_\theta(y_c \mid I, x) / |y_c|\). It keeps the chosen answer likely, which DPO alone does not (Figure 2).
MPO trained with \(\beta = 0.1\), learning rate \(5\times10^{-6}\), one epoch, and global batch 256 [Certain].
Use the pair from the DPO example: \(\Delta_c = 1.0\), \(\Delta_r = -0.5\), \(\beta = 0.1\). Let the chosen answer have 40 tokens with total log-probability \(-42.0\), and take \(\delta = 0\).
DPO: \(-\log\sigma(0.15) = 0.621\). BCO: \(-\log\sigma(0.1) - \log\sigma(0.05) = 0.644 + 0.669 = 1.313\). SFT: \(42.0 / 40 = 1.050\).
Total: \(0.8 \times 0.621 + 0.2 \times 1.313 + 1.0 \times 1.050 = 0.497 + 0.263 + 1.050 = 1.809\). The SFT term is the largest part, as the weights intend: the preference terms steer, the SFT term keeps the model fluent.
Building pairs without people
MPO’s dataset, MMPR, has about 3 million samples: about 2.5M from questions with a clear correct answer and about 750K without one [Certain]. It uses two pipelines.
- input: SFT model \(\pi_0\), set of (image \(I\), question \(x\), optional gold answer \(a\))
- for each \((I, x, a)\) with a gold answer:
- sample up to 32 reasoning answers from \(\pi_0\) at temperature 1.0
- \(Y^+ \leftarrow\) answers whose final answer equals \(a\); \(Y^- \leftarrow\) the rest, including answers with no clear final answer
- make up to 15 pairs \((y_c \in Y^+, y_r \in Y^-)\) // skip the question if \(Y^+\) or \(Y^-\) is empty
- for each \((I, x)\) without a gold answer: // Dropout NTP
- \(y \leftarrow\) one answer from \(\pi_0(\cdot \mid I, x)\); cut it in half: \(y = [y_{1/2},\, \cdot\,]\)
- \(\tilde y \leftarrow\) let \(\pi_0\) finish \(y_{1/2}\) without the image // what the language prior writes
- add the pair \((y_c = y,\; y_r = \tilde y)\)
- return all pairs // train with (190.6)
Dropout NTP (“next-token prediction with the image dropped”) is a neat way to produce the exact error you want to remove. The second half, written without the image, contains whatever the language model thinks is likely: that is the hallucination. The paper tried cutting at 25% and 75% instead of 50%, and both were worse [Certain]. With all this, InternVL2-8B-MPO reached 67.0 on MathVista, 8.7 points above InternVL2-8B, and also improved slightly on hallucination benchmarks (POPE 86.9 → 88.1, MMHal-Bench score 3.3 → 3.5) [Certain].
GRPO for visual reasoning with checkable answers
Preference pairs are collected once. As the model improves, they become stale: the model no longer makes those mistakes, and it makes new ones nobody labeled. Online RL avoids this. The model writes new answers at every step, and a reward scores them. For many vision tasks the reward does not need a person or a reward model: a program can check the answer. This is RL with verifiable rewards (RLVR):
- Counting: “How many apples?” The gold answer is 7. Reward 1 if the answer is 7.
- Charts and documents: “What was revenue in 2021?” Reward 1 if the number is within a small tolerance of the table value.
- Geometry and visual math: multiple choice or a number; the same check as text math.
- Grounding and detection: “Draw a box around the red cup.” Reward = overlap with the true box (IoU, next section).
GRPO (Group Relative Policy Optimization), introduced in DeepSeekMath, is the standard algorithm here (see session 70). For each question it samples a group of \(G\) answers \(o_1, \dots, o_G\) from the current model, scores them, and uses the group itself as the baseline:
Every token of answer \(o_i\) gets the same advantage \(\hat A_i\). Answers better than the group average are pushed up; worse ones are pushed down. PPO needs a value model, a second network of about the same size as the policy, to estimate the baseline; GRPO replaces it with the group mean, which saves a lot of memory [Certain]. For a VLM this matters: the policy already carries a vision encoder and many visual tokens per sample.
The objective is PPO’s clipped objective with this advantage, plus a KL penalty to the reference model:
- Read the symbols
- \(\rho_{i,t}\)
- how much more (or less) likely token \(t\) of answer \(i\) is now than under the model that sampled it
- \(\varepsilon\)
- the clip range (often 0.2): the update cannot gain from moving a token’s probability ratio outside \([1-\varepsilon, 1+\varepsilon]\)
- \(\mathbb{D}_{i,t}\)
- a per-token estimate of the KL divergence to the reference model; always \(\ge 0\), and 0 when the two models agree
- \(\beta\)
- the KL weight; DeepSeekMath used 0.04 (with \(G = 64\) and learning rate \(10^{-6}\))
Example for (190.10): if the reference gives a token 0.8 times the probability the policy gives it, \(\mathbb{D} = 0.8 - \log 0.8 - 1 = 0.023\). DeepSeekMath-RL 7B improved over its SFT model from 82.9% to 88.2% on GSM8K and from 46.8% to 51.7% on MATH [Certain]. Vision-R1 used GRPO on 10K multimodal math problems, after a cold-start SFT on about 200K CoT samples, and reports 73.5% on MathVista for its 7B model [Certain].
\(G = 8\) answers to “How many apples?”; 3 say 7 (reward 1), 5 are wrong (reward 0). Mean \(= 3/8 = 0.375\). Standard deviation (population) \(= \sqrt{0.375 \times 0.625} = 0.484\).
Correct answers: \(\hat A = (1 - 0.375)/0.484 = +1.29\). Wrong answers: \(\hat A = (0 - 0.375)/0.484 = -0.77\). The rare correct answers get the bigger push. If you use the sample standard deviation (divide by \(G - 1\)) instead, the values are \(+1.21\) and \(-0.72\); libraries differ here.
When the group gives no signal
Look at (190.8) again. If all \(G\) answers are correct, or all are wrong, every \(r_i\) equals the mean and every advantage is 0. That question teaches nothing, but it still costs \(G\) full generations. If the model answers a question correctly with probability \(p\), the chance that a group is useless is
With \(G = 8\): \(p = 0.5\) gives 0.8%, but \(p = 0.9\) or \(p = 0.1\) gives 43%, and \(p = 0.95\) gives 66%. Questions that are too easy or too hard waste most of the compute. This is why VLM RL recipes filter the data to questions of medium difficulty for the current model, and why some sample more answers per question [Likely]. Figure 4 lets you see this happen.
How to read Figure 4
At \(p = 0.4\), most groups mix right and wrong answers, and the bars point up for correct and down for wrong ones. Push \(p\) to 0.95 and press “Sample a new group” a few times: most groups are all correct and every bar is 0. The readout gives the exact probability of a useless group from (190.11). Then raise \(G\) to 16: the probability drops, at twice the generation cost.
Two more details matter in practice. First, dividing by the answer length \(|o_i|\) in (190.9) and by the standard deviation in (190.8) both add biases; Dr. GRPO argues for removing them (for example, long wrong answers are punished less per token) [Likely]. Second, the KL term and a small learning rate keep the model from drifting away from the SFT model, which protects its general skills such as captioning and OCR [Likely].
Reward design for vision tasks
With GRPO the reward is the task definition. The model will find any way to raise it, including ways you did not intend. This is called reward hacking. A good vision reward is cheap to compute, hard to fool, and gives partial credit where partial answers are meaningful.
Exact match, with normalization
For counting, multiple choice and short answers, the reward is 1 if the extracted answer equals the gold answer, else 0. The hard part is extraction and normalization: “7”, “ 7 ”, “7.” and “seven” are the same count; “(B)” and “B” are the same choice. Ask for a fixed answer slot (for example inside <answer>…</answer> tags) and normalize before comparing. For chart values, compare numbers with a relative tolerance instead of strings, since “12.3” and “12.30” are equal and the chart may only show two digits.
IoU for boxes, and temporal IoU for video
For grounding (“where is the red cup?”) and detection, the answer is a box. A 0/1 reward would be almost always 0 at the start. Intersection over union (IoU) gives partial credit:
IoU is 1 for a perfect box and 0 for a box that does not touch the target. The same formula on time intervals gives temporal IoU, the natural reward for key-moment detection in video (“when does the goal happen?”). Visual-RFT used a detection reward that adds the average IoU of the predicted boxes, a confidence reward and a format reward, on Qwen2-VL-2B and 7B [Certain]. In its few-shot COCO setting, the 2B model reached 41.5 mAP with 2 examples per class, against 21.0 for SFT on the same data [Certain].
IoU can be hacked too. A box that covers the whole image always overlaps the target. For a 4×4 target in a 10×10 image, the whole-image box gets IoU \(16/100 = 0.16\) for free, every time, with no looking. Two fixes: give 0 below a threshold (for example IoU \(\lt 0.5\)), or subtract a penalty for boxes much larger than needed. Try both in Figure 5.
Format reward, and how to combine
A format reward gives a small bonus if the answer follows the required layout, for example a reasoning part in <think>…</think> and a final answer in <answer>…</answer>. It makes extraction reliable. Combine the pieces as
Keep \(\lambda_f\) small (for example 0.1 against a task reward in \([0, 1]\)). If the format bonus is as large as the task reward, the model can collect half the reward with a perfect layout and a wrong answer. And remember that in GRPO only differences inside a group matter (190.8): a format bonus that every answer already earns adds nothing.
How to read Figure 5
The default box is shifted by 1 in each direction: intersection 9, union 23, IoU 0.391. Press “Whole image (hack)”: raw IoU still pays 0.16 for a useless answer. Switch to “IoU, 0 below 0.5”: the hack pays 0, but so does the honest near miss, so early in training the model gets less signal. The size penalty used here, \(\max(0, |\hat b|/|b| - 2) \times 0.05\) with a floor at 0, is one simple choice among many: it only punishes boxes more than twice too big.
Putting it together: one GRPO step for a VLM
- input: policy \(\pi_\theta\), frozen reference \(\pi_{\text{ref}}\), batch of (image, question, gold) with known type: count, choice, number, or box
- for each question: encode the image once; sample \(G\) answers from \(\pi_{\theta_{\text{old}}}\) // the visual tokens can be shared by all \(G\) samples
- \(r_i \leftarrow R_{\text{task}}(o_i, \text{gold}) + \lambda_f R_{\text{format}}(o_i)\) // (190.12), (190.13)
- if all \(r_i\) are equal: drop the question // no signal, (190.11)
- \(\hat A_i \leftarrow (r_i - \operatorname{mean} r)/(\operatorname{std} r + 10^{-6})\) // (190.8)
- compute log-probs of every answer token under \(\pi_\theta\), \(\pi_{\theta_{\text{old}}}\), \(\pi_{\text{ref}}\) // with the image in the context
- loss \(\leftarrow -\mathcal{J}(\theta)\) from (190.9), (190.10); only answer tokens count, never image or prompt tokens
- clip the gradient norm, take an AdamW step with a small learning rate (around \(10^{-6}\))
- log: mean reward, share of dropped groups, answer length, KL to reference // a rising KL or exploding length is an early warning
The PyTorch code below implements the pieces of this session on a tiny model. Part 1 is a toy VLM (an image vector plus the previous token). Part 2 trains it with the MPO loss on corrected pairs, using the RLHF-V token weights. Part 3 has the verifiable rewards and the GRPO advantages. It runs on a CPU in a few seconds and prints the same numbers every time.
import re, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # ---- 1) a tiny "VLM": image feature + previous token -> next-token logits ---- V, D = 12, 16 class TinyVLM(nn.Module): def __init__(s): super().__init__(); s.img = nn.Linear(8, D); s.tok = nn.Embedding(V, D); s.out = nn.Linear(D, V) def forward(s, img, ids): # img (B,8), ids (B,T) h = torch.tanh(s.img(img)[:, None, :] + s.tok(ids)) return s.out(h) # (B,T,V): logits for the NEXT token def seq_logp(model, img, ids, w): """Weighted sum of log p(token) over answer tokens, divided by sum of weights. w = 0 on prompt/padding, 1 on normal answer tokens, gamma on corrected tokens (DDPO).""" logits = model(img, ids[:, :-1]) lp = torch.gather(F.log_softmax(logits, -1), 2, ids[:, 1:, None]).squeeze(-1) w = w[:, 1:] return (lp * w).sum(-1), (lp * w).sum(-1) / w.sum(-1) # ---- 2) the three preference losses: DPO (190.2), BCO (190.7), MPO (190.8) ---- def dpo_loss(lc, lr, lc0, lr0, beta=0.1): return -F.logsigmoid(beta * ((lc - lc0) - (lr - lr0))).mean() def bco_loss(lc, lr, lc0, lr0, delta, beta=0.1): rc, rr = beta * (lc - lc0), beta * (lr - lr0) return (-F.logsigmoid(rc - delta) - F.logsigmoid(-(rr - delta))).mean() # one pair: same image, chosen = human-corrected caption, rejected = original B, T = 4, 10 img = torch.randn(B, 8) yc = torch.randint(0, V, (B, T)); yr = yc.clone() yr[:, 6:8] = (yc[:, 6:8] + 3) % V # tokens 6-7 are the hallucinated segment w_plain = torch.ones(B, T); w_plain[:, :2] = 0 # first 2 tokens = prompt (no loss) gamma = 5.0 w_ddpo = w_plain.clone(); w_ddpo[:, 6:8] = gamma # RLHF-V: corrected tokens count 5 times policy, ref = TinyVLM(), TinyVLM() ref.load_state_dict(policy.state_dict()); ref.requires_grad_(False) opt = torch.optim.AdamW(policy.parameters(), lr=1e-2) delta = 0.0 # BCO reward shift: moving average of rewards for step in range(100): _, lc = seq_logp(policy, img, yc, w_ddpo); _, lr = seq_logp(policy, img, yr, w_ddpo) with torch.no_grad(): _, lc0 = seq_logp(ref, img, yc, w_ddpo); _, lr0 = seq_logp(ref, img, yr, w_ddpo) l_sft = -(seq_logp(policy, img, yc, w_plain)[0] / w_plain[:, 1:].sum(-1)).mean() l_p, l_q = dpo_loss(lc, lr, lc0, lr0), bco_loss(lc, lr, lc0, lr0, delta) loss = 0.8 * l_p + 0.2 * l_q + 1.0 * l_sft # MPO weights from the paper opt.zero_grad(); loss.backward(); opt.step() with torch.no_grad(): delta = 0.9 * delta + 0.1 * (0.1 * ((lc - lc0) + (lr - lr0)) / 2).mean().item() if step in (0, 99): m = 0.1 * ((lc - lc0) - (lr - lr0)).mean().item() print(f"step {step:2d} dpo {l_p.item():.3f} bco {l_q.item():.3f} sft {l_sft.item():.3f} margin {m:.3f}") # ---- 3) GRPO pieces: verifiable rewards and group-relative advantages (190.9) ---- def exact_match(ans, gold): m = re.search(r"<answer>(.*?)</answer>", ans, re.S) if not m: return 0.0 norm = lambda s: re.sub(r"[^0-9a-z.]", "", s.lower()) return float(norm(m.group(1)) == norm(gold)) def format_reward(ans): return float(re.fullmatch(r"(?s)<think>.+?</think>\s*<answer>.+?</answer>", ans.strip()) is not None) def iou(a, b): # boxes as (x1, y1, x2, y2), formula (190.11) iw = max(0.0, min(a[2], b[2]) - max(a[0], b[0])); ih = max(0.0, min(a[3], b[3]) - max(a[1], b[1])) inter = iw * ih; union = (a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - inter return inter / union def group_advantages(r, eps=1e-6): r = torch.tensor(r, dtype=torch.float32) return (r - r.mean()) / (r.std(unbiased=False) + eps) answers = ["<think>3 red, 4 green</think><answer>7</answer>", "<think>count</think><answer> 7 </answer>", "<think>I see six</think><answer>6</answer>", "7 apples", "<think>rows of 2</think><answer>8</answer>", "<think>3+4</think><answer>seven</answer>", "<think>...</think><answer>7</answer>", "<answer>5</answer>"] r = [exact_match(a, "7") + 0.1 * format_reward(a) for a in answers] print("rewards ", [round(x, 2) for x in r]) print("advantages", [round(x, 2) for x in group_advantages(r).tolist()]) print("IoU", round(iou((2, 2, 6, 6), (3, 3, 7, 7)), 3), "whole-image box", iou((2, 2, 6, 6), (0, 0, 10, 10))) print("all correct ->", group_advantages([1.0] * 8).tolist()[:3], "(no learning signal)")
Output: at step 0 the DPO loss is \(\log 2 = 0.693\) and the BCO loss is \(2\log 2 = 1.386\), because the policy equals the reference. After 100 steps the margin is 0.418 and the DPO loss is 0.507. The rewards are [1.1, 1.1, 0.1, 0.0, 0.1, 0.1, 1.1, 0.0]: “seven” earns no task reward (our normalizer only accepts digits), and “7 apples” earns nothing because it has no answer tags. The advantages are \(+1.29\) for the three correct answers. The IoU line prints 0.391 and the hackable 0.16.
Where this is used
| Setting | Method and reward | Why |
|---|---|---|
| Captioning and visual chat assistants | DPO / mDPO / DDPO on corrected or AI-judged pairs | Object and attribute hallucination is the top complaint; no checker exists for free text |
| Visual math, geometry, science diagrams | MPO on correctness pairs, then GRPO with exact match | Answers are checkable; CoT quality is the weak point (MPO, Vision-R1) |
| Charts, tables and documents (DocVQA-style) | GRPO with numeric tolerance or normalized string match | Values are in the source table; small format slips should not count as errors [Likely] |
| Counting (retail shelves, crowds, inventory) | GRPO with exact match, optionally partial credit by distance | Counting is a classic VLM weakness; the label is one integer [Likely] |
| Grounding, referring expressions, open-vocabulary detection | GRPO with IoU (thresholded or penalized) + format reward (Visual-RFT) | Boxes give a continuous, cheap reward; few labeled examples needed |
| Video key-moment detection and temporal grounding | GRPO with temporal IoU on predicted start/end times | The same formula (190.12) on 1-D intervals [Likely] |
| Video QA and action recognition (multiple choice) | GRPO with exact match on the option letter | Long-video reasoning benefits from sampled CoT; answers are letters [Likely] |
| Video captioning and Video-RAG answers | DPO with Dropout-NTP-like negatives (drop frames, keep text) | Hallucinated events are the video version of hallucinated objects [Speculation] |
Common mistakes
- Pairs that can be judged without the image. If the rejected answer is always longer or always contains certain phrases, DPO learns style. Check with a text-only classifier; add image-conditioned pairs (190.4).
- Pure DPO with a long training run. DPO only cares about the gap, so both answers can lose probability and the model becomes less fluent. Add an SFT term (MPO) or an anchor (mDPO), and watch the chosen log-probability.
- Off-policy pairs. Pairs written by another model, or old pairs after the model has changed, teach little. Regenerate from the current model when you can.
- A reward that pays for hacks. Raw IoU pays for whole-image boxes; a lenient string match pays for “7 or 8”; a large format bonus pays for empty reasoning. Test your reward on deliberately bad answers before training.
- Too easy or too hard RL data. When most groups are all-correct or all-wrong, most compute gives zero gradient (190.11). Filter by the current model’s pass rate.
- Putting the loss on image or prompt tokens. Only answer tokens should carry log-probabilities in DPO and GRPO; the masks from session 188 apply here too.
- Forgetting the reference model must see the same image. \(\pi_0\) and \(\pi_{\text{ref}}\) must get the same visual tokens (same resolution and tiling) as the policy, or the log-ratios measure preprocessing differences, not learning.
- No general evaluation. RL on math can hurt OCR or captioning. Track a small set of general benchmarks and a hallucination benchmark (POPE, CHAIR) at every checkpoint (session 192).
Interview questions
Why do we need preference tuning or RL after SFT for a VLM?basic
Write the DPO loss for a VLM and explain its gradient.core
What is the “unconditional preference” problem and how does mDPO fix it?advanced
How does RLHF-V use fine-grained corrections, and why is that better than ranking?core
Explain MPO’s three losses and why each is there.core
How can you create VLM preference pairs without human annotators?core
Why does GRPO not need a value model, and why does that matter more for VLMs?core
A team runs GRPO on a counting dataset and the loss barely moves. What do you check?advanced
Design a reward for a referring-expression grounding task.advanced
<answer>[x1, y1, x2, y2]</answer>; unparsable output gets 0. Task reward: IoU with the gold box (190.12), which gives partial credit early. Guard against hacks: a whole-image box earns IoU = target area / image area for free (0.16 for a 4×4 target in 10×10), so use a threshold (0 below 0.5) later in training, or penalize boxes much larger than needed. Add a small format bonus (around 0.1). For several objects, match predicted and gold boxes one-to-one (for example by Hungarian matching) and average the IoUs, penalizing extra boxes.When would you choose DPO-style training over GRPO for a VLM, and the reverse?core
What does the KL term in GRPO do, and what is the estimator in (190.10)?advanced
Exercises
- A pair has \(\Delta_c = 0.4\) and \(\Delta_r = 0.9\) with \(\beta = 0.1\). Compute the margin, the DPO loss and the gradient weight. Is the model ranking the pair correctly?
- A corrected caption has 30 tokens, 2 of them edited. What share of the DDPO score (190.5) do the edited tokens have with \(\gamma = 1\) and with \(\gamma = 5\)?
- A GRPO group of \(G = 4\) has rewards \((1, 0, 0, 0)\). Compute the advantages with the population standard deviation.
- Your model answers a set of chart questions correctly 80% of the time. With \(G = 8\), what share of groups gives no signal? With \(G = 16\)?
- The true box is \((0, 0)\)–\((4, 4)\) and the predicted box is \((2, 0)\)–\((6, 4)\). Compute the IoU. Does it pass a 0.5 threshold?
- Give one way a model could hack the reward \(R = \text{exact match} + 0.5 \cdot \text{format}\) on a multiple-choice task, and fix it.
Answerscheck
Key takeaways
- SFT shows good answers; preference tuning and RL give feedback on the model’s own answers, which is what fixes hallucination and weak reasoning in VLMs.
- DPO (190.2) works for VLMs with the image in the condition, but pairs can often be judged without the image; mDPO’s image-conditioned pair (190.4) and an anchor force grounding.
- Fine-grained human corrections (RLHF-V) plus token weights \(\gamma = 5\) (190.5) put the signal on the wrong words: 1.4k samples cut response hallucination from 50.5% to 12.2% on Object HalBench.
- MPO mixes DPO, BCO and SFT (0.8 / 0.2 / 1.0, formula 190.6) on about 3M automatically built pairs, including Dropout NTP negatives; InternVL2-8B went from 58.3 to 67.0 on MathVista.
- GRPO (190.8–190.10) uses the group mean as the baseline, so no value model; groups that are all right or all wrong give no signal, with probability \(p^G + (1-p)^G\) (190.11).
- Vision rewards: normalized exact match, numeric tolerance, IoU and temporal IoU (190.12), plus a small format bonus (190.13). Test every reward against hacks such as whole-image boxes.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
- Wang et al. (2024). Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
- Yu et al. (2023). RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Wang et al. (2024). Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
- Yu et al. (2023). RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Wang, Zhou, Huang, Xu, Zhang, Poon & Chen (2024). mDPO: Conditional Preference Optimization for Multimodal Large Language Models
- Sun et al. (2023). Aligning Large Multimodal Models with Factually Augmented RLHF
- Yu et al. (2024). RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness
- Liu, Sun, Zang, Dong, Cao, Duan, Lin & Wang (2025). Visual-RFT: Visual Reinforcement Fine-Tuning
- Huang et al. (2025). Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models
- Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective
- Schulman et al. (2017). Proximal Policy Optimization Algorithms
- Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models