preference tuning a VLM against hallucination
In this lab you make a small vision-language model invent fewer objects. You build chosen and rejected answers that differ only in correct versus hallucinated objects, train with DPO (in TRL on a GPU, and in a toy model you can train live in your browser), sweep β, and measure the result with a POPE-style yes/no test, a paired significance test and a caption check, so that you do not ship a model that only learned to say less.
What you will learn
- How to build preference pairs against object hallucination: sample answers from your own model and score them against object labels, or edit a correct answer by inserting or swapping an invented object
- Why the pair design decides what DPO learns: pairs where the chosen answer is simply shorter teach the model to say less, not to look harder
- How the DPO loss of a VLM pair reduces to the objects where the two answers differ, and how to compute it by hand
- How to run DPO on a VLM with TRL and LoRA: dataset format with images, max_length=None, and what the logs should show
- How to sweep β and choose it with the adversarial POPE split, the yes ratio and the caption recall together
- How to run a POPE-style yes/no test correctly: the three negative-sampling splits, POPE's answer parser, and a paired test with an error bar
- Why POPE and a caption metric such as CHAIR can rank two models differently, and why you report both
Why this matters
You have a small vision-language model (VLM) that answers questions about images. It has one bad habit: it names objects that are not in the image. Ask it to describe a kitchen and it adds a sink that is not there, because kitchens usually have sinks. In this lab you use DPO to make the model prefer answers that name only what is really there. Then you check, with a yes/no test and a caption test, whether the model really looks at the image more, or whether it only learned to say less.
Session 193 explained where object hallucination comes from: the language model inside the VLM has a strong prior (“kitchens have sinks”), and when the visual evidence is weak, the prior wins. Session 190 showed the training fix in theory: DPO on pairs of a faithful answer and a hallucinated answer, plus the trap that a pair can sometimes be judged without looking at the image. Session 76 was the practical DPO lab for text. This lab joins them. Most of the work is not the loss. It is the design of the pairs, the choice of \(\beta\), and an honest measurement.
You will:
- Build chosen and rejected answers in three ways: sampled from your model and scored against object labels, or edited from a correct answer by inserting or by swapping an invented object (Figure 1).
- Compute the DPO loss of a VLM pair by hand, and see that it only depends on the objects where the two answers differ.
- Train
Qwen/Qwen2.5-VL-3B-Instructwith TRL’sDPOTrainerand LoRA. - Train a toy VLM live in your browser with the same loss, and see what each pair design does (Figure 2).
- Sweep \(\beta\) and choose it with several numbers, not one (Figure 3).
- Measure with a POPE-style yes/no test, its answer parser (Figure 4), a paired error bar (Figure 5) and a caption check.
This is how real systems are tuned. HA-DPO built pairs of non-hallucinated and hallucinated descriptions of the same image with a consistent style, and reported that POPE accuracy of MiniGPT-4 rose from 51.13% to 86.13% [Certain]. POVID kept the ground-truth answer as chosen and made rejected answers by letting GPT-4V inject plausible hallucinations into it, and also by distorting the image [Certain]. RLHF-V used human corrections of hallucinated segments [Certain], and RLAIF-V used AI feedback [Certain]. TRL’s documentation uses a formatted RLAIF-V dataset as its example for VLM DPO [Certain].
Setup
Hardware. The toy model and all figures need only a browser or a CPU. The real run trains LoRA adapters on a 3B-class VLM. Its model page lists about 4B parameters in BF16 [Certain], so the frozen weights take about 8 GB; with LoRA, a 24 GB GPU, batch size 1 to 2 and gradient checkpointing should be enough [Likely]. With LoRA, TRL does not need a second copy of the model as the reference: the reference is the base model with the adapter switched off [Likely]. On a smaller GPU, use a smaller VLM such as the one from session 194.
pip install "transformers>=4.57" trl peft datasets accelerate pillow
The TRL and Transformers code on this page follows the current documentation, but we did not run the GPU scripts for you, and this page reports no results from them: you will measure them. TRL changes often, so check the documentation of your installed version if an argument fails. We did run the toy model of Step 4, in pure Python and in PyTorch 2.8 on a CPU; all toy numbers on this page come from those runs, and the figures run the same code in your browser.
Data.
| Role | Data | Why |
|---|---|---|
| Images for pairs | COCO train2014 images and instances_train2014.json (80 object categories) | Object labels let you check every object an answer names |
| Test | POPE question files for COCO: coco_pope_random.json, coco_pope_popular.json, coco_pope_adversarial.json, on val2014 images | The standard yes/no probe for object hallucination |
| Optional ready-made pairs | HuggingFaceH4/rlaif-v_formatted: columns prompt, chosen, rejected, images; about 79k training rows [Certain] | A baseline: compare your own pairs with general AI-feedback pairs |
The table you will fill. Copy it now. Every empty cell is a number you measure. “Adv.” is POPE’s adversarial split; \(\text{CHAIR}_i\) and recall come from captions (Step 6).
| Model | Pairs | \(\beta\) | POPE adv. accuracy | POPE adv. F1 | Yes ratio | \(\text{CHAIR}_i\) | Caption recall |
|---|---|---|---|---|---|---|---|
| A: start model | none | none | |||||
| B: DPO | sampled and scored | best | |||||
| C: DPO | edited: swap | best | |||||
| D: DPO | edited: insert | best | |||||
| E: DPO | RLAIF-V (general) | 0.1 |
Step 1: build chosen and rejected answers
A preference example for a VLM has four parts: an image \(v\), a prompt \(x\) such as “Describe this image in detail.”, a chosen answer \(y_w\) and a rejected answer \(y_l\). Here “better” has a precise meaning: an answer is better if it names fewer objects that are not in the image, without dropping the objects that are. COCO object labels let you check this automatically. There are three common ways to make the pairs.
- Sampled and scored (on-policy). Sample two or more descriptions from your own model. Find the objects each one names, compare with the labels, and give each a score. The best one is chosen, the worst one is rejected. This is close to RLAIF-V, which also ranks the model’s own answers, but with an AI judge instead of labels [Likely].
- Edited: insert. Take a correct description (all labeled objects) as chosen, and add one invented object to make the rejected answer. This is cheap and fully controlled.
- Edited: swap. Take the same correct description, and replace one real object by an invented one. The two answers now name the same number of objects. HA-DPO stressed this kind of style consistency between the two answers [Certain]; POVID’s injected hallucinations are another editing method [Certain].
Which invented object should you use? The hardest and most useful one is an object that often appears together with the real objects (a sink in a kitchen), because that is exactly what the language prior pushes the model to say. POPE calls these negatives “adversarial” [Certain]. For sampled pairs, the score needs a choice too. Counting only hallucinations has a hidden problem: an answer that says nothing has no hallucinations. So we add a small reward for each correct object:
- Read the symbols
- \(n_\text{correct}(y)\)
- the number of objects the answer names that are in the image labels
- \(n_\text{halluc}(y)\)
- the number of objects it names that are not in the labels: hallucinated objects
- \(\alpha\)
- the reward for one correct object, relative to the cost of one hallucination; \(\alpha = 0\) means “only count mistakes”
- \(y_1, \dots, y_K\)
- \(K\) answers sampled from the model for the same image; pairs with equal scores are dropped
- input: images with object labels \(T(v)\), the model \(\pi_\text{ref}\), \(K = 4\), weight \(\alpha\), a synonym list for object names
- for each image \(v\):
- sample \(y_1, \dots, y_K \sim \pi_\text{ref}(\cdot \mid v, x)\) at temperature 0.7 // greedy gives K copies
- \(M_k \leftarrow\) objects named in \(y_k\) (match words and synonyms to the 80 categories)
- \(s_k \leftarrow \alpha\,|M_k \cap T(v)| - |M_k \setminus T(v)|\) // formula (195.1)
- if \(\max_k s_k \gt \min_k s_k\): keep (best, worst) as a sampled pair
- \(h \leftarrow\) the label that co-occurs most with \(T(v)\) but is not in \(T(v)\)
- in the best answer, replace one correct object word by \(h\): a swap pair // same length
- report: pairs kept, mean number of objects in chosen and rejected, share where chosen names fewer objects
Figure 1 shows the three kinds of pairs on the first eight training images of the toy model of Step 4. The toy “describes” an image by naming a set of objects, and the caption is written from that set.
How to read Figure 1
Start with “Sampled and scored” on image 1, a living room with a person, a chair, a dining table and a couch. Step through the images. Often the chosen answer is simply the one that names fewer objects: over all 354 sampled pairs (of 400 images, the rest are ties), the chosen answer names fewer objects in 71.5% of pairs, 4.71 objects on average against 6.00. Switch to “Edited: insert”: the rejected answer is always the chosen one plus one object, so the chosen answer is always shorter. Switch to “Edited: swap”: both answers name the same number of objects, and only one object differs. On image 1 the swap pair replaces the person by a tv. The readout shows how many objects differ and the start model’s log-probability gap, which is all that DPO will see (formula 195.4).
The length pattern is the most important thing in this figure. In text DPO, session 76 showed that chosen answers that are longer can teach a model to write longer. Here the risk is the mirror image: chosen answers that are shorter can teach the model to say less. A model that names nothing never hallucinates, and it also never answers “yes”. Step 4 shows how strong this effect is.
Here is the real version of Algorithm 1. It writes a dataset in TRL’s vision format: an images column and messages whose content is a list of {"type": "image"} and {"type": "text"} entries [Certain].
# make_pairs.py: sampled pairs and swap pairs from COCO train2014 (never from val2014) import json, random, re, torch from collections import Counter, defaultdict from PIL import Image from datasets import Dataset, Sequence, Image as HFImage from transformers import AutoProcessor, AutoModelForImageTextToText MODEL, ROOT, ALPHA, K = "Qwen/Qwen2.5-VL-3B-Instruct", "coco", 0.5, 4 PROMPT = "Describe this image in detail." proc = AutoProcessor.from_pretrained(MODEL, min_pixels=256 * 28 * 28, max_pixels=512 * 28 * 28) model = AutoModelForImageTextToText.from_pretrained(MODEL, dtype=torch.bfloat16, device_map="auto").eval() ann = json.load(open(f"{ROOT}/annotations/instances_train2014.json")) name = {c["id"]: c["name"] for c in ann["categories"]} labels = defaultdict(set) for a in ann["annotations"]: labels[a["image_id"]].add(name[a["category_id"]]) files = {im["id"]: im["file_name"] for im in ann["images"]} co = defaultdict(Counter) # co-occurrence counts between labels for s in labels.values(): for a in s: for b in s - {a}: co[a][b] += 1 SYN = {"person": ["person", "people", "man", "men", "woman", "women", "boy", "girl", "child"], "dining table": ["table", "dining table"], "couch": ["couch", "sofa"], "tv": ["tv", "television"], "cell phone": ["phone", "cell phone"], "motorcycle": ["motorcycle", "motorbike"]} WORDS = {c: SYN.get(c, [c]) for c in name.values()} def objects_in(text): # categories named in a text (simple CHAIR-style match) t = " " + re.sub(r"[^a-z ]", " ", text.lower()) + " " return {c for c, ws in WORDS.items() if any(f" {w} " in t or f" {w}s " in t for w in ws)} def describe(img, n): msg = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": PROMPT}]}] text = proc.apply_chat_template(msg, add_generation_prompt=True) inp = proc(text=[text], images=[img], return_tensors="pt").to(model.device) out = model.generate(**inp, do_sample=True, temperature=0.7, top_p=0.9, max_new_tokens=160, num_return_sequences=n) return proc.batch_decode(out[:, inp["input_ids"].shape[1]:], skip_special_tokens=True) def row(path, good, bad, kind): return {"images": [path], "kind": kind, "prompt": [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": PROMPT}]}], "chosen": [{"role": "assistant", "content": [{"type": "text", "text": good}]}], "rejected": [{"role": "assistant", "content": [{"type": "text", "text": bad}]}]} random.seed(0); torch.manual_seed(0) ids = random.sample(sorted(i for i in labels if labels[i]), 3000) rows, n_short = [], 0 for i in ids: path = f"{ROOT}/train2014/{files[i]}"; T = labels[i] ans = describe(Image.open(path).convert("RGB"), K) found = [objects_in(a) for a in ans] s = [ALPHA * len(M & T) - len(M - T) for M in found] # formula (195.1) w, l = max(range(K), key=s.__getitem__), min(range(K), key=s.__getitem__) if s[w] > s[l]: rows.append(row(path, ans[w], ans[l], "sampled")); n_short += len(found[w]) < len(found[l]) hits = sorted(found[w] & T) # swap: real object -> invented one cand = [c for c, _ in co[hits[0]].most_common() if c not in T] if hits else [] if hits and cand: word = next(x for x in WORDS[hits[0]] if re.search(rf"\b{x}s?\b", ans[w].lower())) bad = re.sub(rf"\b{word}\b", cand[0], ans[w], count=1, flags=re.I) if bad != ans[w]: rows.append(row(path, ans[w], bad, "swap")) n_s = sum(r["kind"] == "sampled" for r in rows) print(len(rows), "pairs;", n_s, "sampled; chosen names fewer objects in", round(n_short / max(1, n_s), 3)) Dataset.from_list(rows).cast_column("images", Sequence(HFImage())).save_to_disk("pairs_hallu")
The swap edit uses the best sampled answer, not a COCO caption, so both answers stay close to what your model actually writes. Read 30 swap pairs yourself: a word replacement can break grammar (“a oven”) or replace a word that was part of another phrase, and the model can learn to avoid such odd text instead of the invented object [Likely].
Step 2: the loss for one VLM pair, by hand
The DPO loss for a VLM is the text loss of session 68 with the image added to the condition. Both answers are scored for the same image and prompt, under the model you train \(\pi_\theta\) and under the frozen start model, the reference \(\pi_\text{ref}\):
- Read the symbols
- \(v, x\)
- the image and the text prompt; both answers share them
- \(y_w, y_l\)
- the chosen (faithful) and the rejected (hallucinated) answer
- \(\pi_\theta(y \mid v, x)\)
- the probability of the whole answer: the product of its token probabilities, so its log is a sum over answer tokens
- \(\beta\)
- how strongly the model is held near the reference; TRL’s default is 0.1 [Certain]
- \(\sigma\)
- the logistic function \(1 / (1 + e^{-u})\)
The image tokens are part of the prompt, so they get no loss, but they change every answer token’s probability. That is the whole point: the pair should be winnable only by looking at the image. For a swap pair, both answers are equally fluent and equally long, and the only way to tell them apart is to check whether the swapped object is in the picture.
The toy VLM
To see what DPO does inside, we use a toy VLM small enough to train in your browser. It knows 15 COCO objects in three kinds of scenes (kitchen, street, living room). For every object \(o\) in an image it computes a belief logit \(z_o\), a score that says how sure it is that the object is there:
- Read the symbols
- \(e_o\)
- visual evidence: \(+1\) plus noise if the object is in the image, \(-1\) plus noise if not (the noise has standard deviation 1, so some objects are hard to see)
- \(c_o\)
- the scene prior: how often this object appears in this kind of scene (0.5 for a sink in a kitchen, 0.03 for a sink on a street); this is the language prior
- \(g_o\)
- popularity: how often the object appears in any image (a person is the most common)
- \(\theta\)
- the four parameters \((w_\text{vis}, w_\text{scene}, w_\text{pop}, b)\); the start model has \(\theta = (0.7, 5.0, 1.5, -2.0)\): it trusts the scene prior much more than its eyes
The toy answers a yes/no question “Is there a o in the image?” with “yes” if \(z_o \gt 0\). It writes a description by naming each object with probability \(\sigma(z_o)\), independently. So the log-probability of a description that names the set of objects \(m\) (with \(m_o = 1\) if object \(o\) is named) is:
Since \(\log \sigma(z) - \log \sigma(-z) = z\), the log-probability gap between two descriptions is a sum over only the objects where they differ: \(\log \pi(y_w) - \log \pi(y_l) = \sum_o (m_{w,o} - m_{l,o})\, z_o\). The objects both answers share cancel. Real autoregressive models behave in a similar way: the tokens of a shared beginning cancel exactly in the log-ratio, and after the first difference the contexts differ, so the cancellation is only partial [Certain]. The toy is a simplification in other ways too: it has no word order, and it has no exposure bias, so it cannot show why SFT alone leaves hallucinations (session 190 covers that).
Training image 27 (not shown in Figure 1) is a street with a person, a bus and a bicycle. The swap pair keeps the person and the bicycle, and replaces the bus by a car, the object that co-occurs most with street scenes. For the bus, the toy sees \(e = 0.668\) (weak evidence), \(c = 0.35\), \(g = 0.137\), so \(z_\text{bus} = 0.7 \cdot 0.668 + 5 \cdot 0.35 + 1.5 \cdot 0.137 - 2 = 0.423\). For the car, \(e = -0.935\) (it is absent), but \(c = 0.6\) and \(g = 0.22\), so \(z_\text{car} = -0.654 + 3.0 + 0.33 - 2 = 0.676\).
The start model is more confident about the invented car than about the real bus. Its log-probability gap is \(\log \pi_\text{ref}(y_w) - \log \pi_\text{ref}(y_l) = z_\text{bus} - z_\text{car} = -0.253\): it prefers the hallucinated description. At the start, \(\pi_\theta = \pi_\text{ref}\), so the margin is 0, the loss is \(\log 2 = 0.693\), and the gradient weight is \(\sigma(0) = 0.5\).
Which way does DPO push? The margin is \(\beta\) times \((\theta - \theta_\text{ref}) \cdot \mathbf{d}\), with \(\mathbf{d} = \mathbf{f}_\text{bus} - \mathbf{f}_\text{car} = (1.603,\ -0.25,\ -0.083,\ 0)\). So this pair pushes \(w_\text{vis}\) up and the scene and popularity weights down, and it does not touch the bias \(b\) at all, because both answers name three objects. After 300 steps on all 398 swap pairs with \(\beta = 1\), the toy has \(\theta = (2.60, 2.60, 1.57, -2.00)\): \(z_\text{bus} = 0.863\) and \(z_\text{car} = -2.527\). The gap is now \(+3.390\), the margin is \(3.390 - (-0.253) = 3.643\), the loss is \(-\log\sigma(3.643) = 0.026\), and the pair’s gradient weight has fallen to \(\sigma(-3.643) = 0.026\): this pair is solved.
Step 3: train with TRL
Now the real run. TRL’s DPOTrainer accepts a VLM directly, and the dataset of Step 1 is already in its vision format [Certain]. Two settings matter for VLMs. First, max_length=None: TRL warns that truncation can remove image tokens and break training [Certain]. Second, LoRA: the documentation suggests a learning rate of about \(10^{-5}\) for adapters, higher than the full fine-tuning default of \(10^{-6}\) [Certain].
- input: a batch of \((v, x, y_w, y_l)\), policy \(\pi_\theta\) (base + LoRA), \(\beta\)
- encode each image once; build the sequences \([v; x; y_w]\) and \([v; x; y_l]\); loss mask \(= 1\) on answer tokens only
- \(\ell^\theta_w, \ell^\theta_l \leftarrow\) summed answer log-probs with the adapter on
- \(\ell^\text{ref}_w, \ell^\text{ref}_l \leftarrow\) the same with the adapter off, no gradient // the reference, at no extra memory
- loss \(\leftarrow\) mean of \(-\log\sigma\big(\beta[(\ell^\theta_w - \ell^\text{ref}_w) - (\ell^\theta_l - \ell^\text{ref}_l)]\big)\) // formula (195.2)
- backward into the LoRA weights only; clip the gradient norm to 1; optimizer step
- log: loss, rewards/chosen, rewards/rejected, margins, accuracies
# train_dpo.py: LoRA DPO on a VLM. Usage: python train_dpo.py 0.1 sampled out/B-b0.1 import sys, torch from datasets import load_from_disk from peft import LoraConfig from trl import DPOConfig, DPOTrainer beta, kind, out = float(sys.argv[1]), sys.argv[2], sys.argv[3] ds = load_from_disk("pairs_hallu").filter(lambda r: r["kind"] == kind).remove_columns("kind") ds = ds.train_test_split(test_size=200, seed=0) cfg = DPOConfig(output_dir=out, beta=beta, loss_type=["sigmoid"], # plain DPO learning_rate=1e-5, num_train_epochs=1, lr_scheduler_type="cosine", warmup_steps=10, per_device_train_batch_size=2, gradient_accumulation_steps=16, max_length=None, # never cut image tokens bf16=True, model_init_kwargs={"dtype": torch.bfloat16}, logging_steps=10, eval_strategy="steps", eval_steps=50, save_strategy="no", report_to="none", seed=0) lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, # LoRA on the language model only target_modules=r".*language_model.*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)") tr = DPOTrainer(model="Qwen/Qwen2.5-VL-3B-Instruct", args=cfg, peft_config=lora, train_dataset=ds["train"], eval_dataset=ds["test"]) tr.train(); tr.save_model(f"{out}/final")
The regular expression keeps LoRA on the language model and off the vision encoder. Module names differ between models and Transformers versions, so print [n for n, _ in model.named_modules()] once and check that the pattern matches what you expect [Likely]. Run it three times: sampled, swap, and, as a control, an insert set built the same way.
Read the logs while it runs. The loss starts at 0.693 (if not, the reference is not the start model). rewards/rejected should fall faster than rewards/chosen. Two warning signs are specific to this task. If rewards/accuracies jumps to almost 1.0 within a few dozen steps, the pairs are too easy, often because they can be separated by length alone. If rewards/chosen keeps falling far below zero, the model is moving probability away from all object words, including correct ones. Both are what the insert pairs do in Figure 2.
Step 4: the toy run, live
Figure 2 runs DPO on the toy VLM with the exact loss (195.2): 400 training images, all pairs in every step, Adam with learning rate 0.05, 300 steps. It then runs the POPE protocol of Step 6 on 500 different images: 3 “yes” and 3 “no” questions per image and split, so 3,000 questions per split, the same counts as POPE on COCO [Certain].
How to read Figure 2
Default: sampled pairs, \(\alpha = 0.5\), \(\beta = 1\), 300 steps. Adversarial accuracy rises from 0.729 to 0.819 and the yes ratio falls from 0.659 to 0.481, close to the 0.5 of a balanced test. Now pick “Edited: insert”: the loss falls to almost 0 (0.004), and the model answers “yes” to only 4.4% of the questions: adversarial accuracy 0.543, F1 0.159. It learned to deny everything. Move the step slider back to 10 with \(\beta = 0.5\): after 10 steps the insert model is briefly good (adversarial accuracy 0.824, yes ratio 0.448), and then it collapses. Now pick “Edited: swap” with \(\beta = 1\): adversarial accuracy 0.846, the best of the three, and the bias \(b\) does not move at all. Finally, set sampled pairs with \(\alpha = 0\): the model ends with a yes ratio of 0.316 and adversarial F1 of 0.717, worse than the start (0.766). Counting only mistakes rewards silence.
The parameter values tell the same story. Swap pairs moved \(w_\text{vis}\) from 0.7 to 2.60 and \(w_\text{scene}\) from 5.0 to 2.60: the model now trusts its eyes more and the prior less. Sampled pairs with \(\alpha = 0.5\) moved \(w_\text{vis}\) to 1.47 and the bias from \(-2.0\) to \(-3.81\): some real looking, and a lot of “say less”. Insert pairs moved the bias to \(-4.88\) and the scene weight down to 2.2. The figure only uses the toy, but the same three failure shapes are what you look for in the real logs and in the POPE yes ratio.
Here is the toy in PyTorch, with autograd and torch.optim.Adam instead of the hand-written gradient of the figures. It uses the same seeded generator, so its numbers match the figures:
# toy_dpo_pope.py: DPO against object hallucination on a toy VLM (CPU, a few seconds) import math, torch, torch.nn.functional as F torch.set_default_dtype(torch.float64) OBJ = ["person", "chair", "dining table", "cup", "bowl", "sink", "oven", "bottle", "car", "traffic light", "bus", "bicycle", "couch", "tv", "book"] P = {"kitchen": [.5, .4, .55, .55, .5, .5, .45, .5, .03, .03, .03, .03, .03, .03, .05], "street": [.8, .1, .05, .05, .03, .03, .03, .1, .6, .45, .35, .4, .03, .03, .03], "living": [.45, .55, .2, .2, .1, .03, .03, .15, .03, .03, .03, .03, .55, .5, .45]} SCENES, K = list(P), len(OBJ) G = [sum(P[s][o] for s in SCENES) / 3 for o in range(K)] # popularity REF = torch.tensor([0.7, 5.0, 1.5, -2.0]) # weights of: evidence, scene prior, popularity, bias def mulberry32(a): # tiny seeded RNG, same as in the figures def r(): nonlocal a a = (a + 0x6D2B79F5) & 0xFFFFFFFF t = ((a ^ (a >> 15)) * (a | 1)) & 0xFFFFFFFF t ^= (t + (((t ^ (t >> 7)) * (t | 61)) & 0xFFFFFFFF)) & 0xFFFFFFFF return ((t ^ (t >> 14)) & 0xFFFFFFFF) / 4294967296 return r def gauss(r): u = r() or 1e-12; v = r() return math.sqrt(-2 * math.log(u)) * math.cos(2 * math.pi * v) def make_images(n, seed, min_present=0): r, out = mulberry32(seed), [] while len(out) < n: s = SCENES[int(r() * 3)] pres = [1 if r() < P[s][o] else 0 for o in range(K)] e = [(1.0 if pres[o] else -1.0) + gauss(r) for o in range(K)] # visual evidence if sum(pres) >= min_present: out.append((pres, [[e[o], P[s][o], G[o], 1.0] for o in range(K)])) return out def logp(theta, f, m): # caption log-prob, formulas (195.3) and (195.4) zz = f @ theta # belief logit of every object return (m * F.logsigmoid(zz) + (1 - m) * F.logsigmoid(-zz)).sum(-1) def sampled_pairs(imgs, alpha, seed): # two captions from the reference, scored (195.1) r, out = mulberry32(seed), [] for pres, feats in imgs: p = torch.sigmoid(torch.tensor(feats) @ REF).tolist() a = [1 if r() < p[o] else 0 for o in range(K)] b = [1 if r() < p[o] else 0 for o in range(K)] sc = lambda m: alpha * sum(m[o] * pres[o] for o in range(K)) - sum(m[o] * (1 - pres[o]) for o in range(K)) if sc(a) != sc(b): # drop ties out.append((feats, a, b) if sc(a) > sc(b) else (feats, b, a)) return out def edited_pairs(imgs, mode, seed): # chosen = all true objects; rejected = edited r, out = mulberry32(seed), [] for pres, feats in imgs: if sum(pres) == 0: continue absent = [o for o in range(K) if not pres[o]] h = max(absent, key=lambda o: (feats[o][1], G[o], -o)) # most likely invented object w = pres[:]; l = pres[:]; l[h] = 1 if mode == "swap": # same number of objects true = [o for o in range(K) if pres[o]] l[true[int(r() * len(true))]] = 0 out.append((feats, w, l)) return out def train(pairs, beta, steps=300, lr=0.05): f = torch.tensor([p[0] for p in pairs]) # [pairs, objects, 4] w = torch.tensor([p[1] for p in pairs], dtype=torch.float64) # chosen mentions l = torch.tensor([p[2] for p in pairs], dtype=torch.float64) # rejected mentions ref_w, ref_l = logp(REF, f, w), logp(REF, f, l) # frozen reference theta = REF.clone().requires_grad_(True) opt = torch.optim.Adam([theta], lr=lr) for _ in range(steps): rw = logp(theta, f, w) - ref_w # log-ratio, chosen rl = logp(theta, f, l) - ref_l # log-ratio, rejected loss = -F.logsigmoid(beta * (rw - rl)).mean() # DPO, formula (195.2) opt.zero_grad(); loss.backward(); opt.step() return theta.detach() def pope_questions(imgs, seed): # 3 yes + 3 no questions per image and split r, Q = mulberry32(seed), {"random": [], "popular": [], "adversarial": []} for i, (pres, feats) in enumerate(imgs): yes = [o for o in range(K) if pres[o]]; no = [o for o in range(K) if not pres[o]] pos = [yes.pop(int(r() * len(yes))) for _ in range(3)] rnd = no[:]; rnd = [rnd.pop(int(r() * len(rnd))) for _ in range(3)] neg = {"random": rnd, "popular": sorted(no, key=lambda o: (-G[o], o))[:3], "adversarial": sorted(no, key=lambda o: (-feats[o][1], -G[o], o))[:3]} for k in Q: Q[k] += [(i, o, 1) for o in pos] + [(i, o, 0) for o in neg[k]] return Q def pope(theta, imgs, Q, split): # formula (195.5); answer "yes" if z > 0 f = torch.tensor([imgs[i][1][o] for i, o, _ in Q[split]]) yes = (f @ theta > 0).tolist(); lab = [q[2] for q in Q[split]] tp = sum(y and t for y, t in zip(yes, lab)); fp = sum(y and not t for y, t in zip(yes, lab)) fn = sum((not y) and t for y, t in zip(yes, lab)); tn = len(lab) - tp - fp - fn prec, rec = tp / max(1, tp + fp), tp / max(1, tp + fn) return (tp + tn) / len(Q[split]), 2 * prec * rec / max(1e-9, prec + rec), (tp + fp) / len(Q[split]) train_imgs, eval_imgs = make_images(400, seed=3), make_images(500, seed=7, min_present=3) Q = pope_questions(eval_imgs, seed=11) runs = {"reference": REF, "sampled, alpha 0, beta 1": train(sampled_pairs(train_imgs, 0.0, 5), 1.0), "sampled, alpha 0.5, beta 1": train(sampled_pairs(train_imgs, 0.5, 5), 1.0), "insert, beta 1": train(edited_pairs(train_imgs, "insert", 9), 1.0), "swap, beta 1": train(edited_pairs(train_imgs, "swap", 9), 1.0)} for name, th in runs.items(): acc, f1, yes = pope(th, eval_imgs, Q, "adversarial") print(f"{name:27s} theta {[round(x, 2) for x in th.tolist()]} adversarial acc {acc:.3f} F1 {f1:.3f} yes {yes:.3f}")
Output (PyTorch 2.8, CPU, under 5 seconds):
reference theta [0.7, 5.0, 1.5, -2.0] adversarial acc 0.729 F1 0.766 yes 0.659 sampled, alpha 0, beta 1 theta [1.58, 7.88, -0.01, -4.87] adversarial acc 0.769 F1 0.717 yes 0.316 sampled, alpha 0.5, beta 1 theta [1.47, 7.53, 1.45, -3.81] adversarial acc 0.819 F1 0.816 yes 0.481 insert, beta 1 theta [1.82, 2.2, -1.23, -4.88] adversarial acc 0.543 F1 0.159 yes 0.044 swap, beta 1 theta [2.6, 2.6, 1.57, -2.0] adversarial acc 0.846 F1 0.838 yes 0.455
Step 5: tune β
Recall from session 68 what \(\beta\) means: DPO fits the KL-regularized optimum \(\pi^* \propto \pi_\text{ref} \exp(r / \beta)\). A small \(\beta\) lets the model move far from the reference to fit the pairs; a large \(\beta\) keeps it close. So \(\beta\) answers “how much do I trust these pairs?”. If the pairs contain a shortcut (shorter chosen answers), a small \(\beta\) follows the shortcut further.
To measure how far the model moved, use the KL divergence to the reference. In the toy it has a closed form, a sum of Bernoulli KLs over the objects of each image:
For the real model, estimate it from sampled answers, as in session 76. Figure 3 trains the toy for seven values of \(\beta\) and shows four numbers for each run.
How to read Figure 3
Default, sampled pairs with \(\alpha = 0.5\): F1 peaks at \(\beta = 1\) (0.816). At \(\beta = 0.05\) the model moved far (KL 6.26), the yes ratio fell to 0.218, recall to 0.419, and F1 to 0.583, far below the start. At \(\beta = 5\) the model hardly moved (KL 0.10) and still says yes too often (0.601). Set \(\alpha = 1\): the pairs now reward coverage more, the shortcut is weaker, and smaller \(\beta\) values work (best F1 0.829 at \(\beta = 0.5\)). Pick “Edited: swap”: the yes ratio stays between 0.38 and 0.57 for every \(\beta\), because swap pairs cannot move the bias; the best F1 is 0.846 at \(\beta = 2\). Pick “Edited: insert”: every \(\beta\) from 0.05 to 2 ends below the start model; only \(\beta = 5\) slows the collapse enough to end at F1 0.714.
The scale of \(\beta\) in the toy is not the scale of a real model. A real answer has dozens of tokens, so its log-ratios are larger, and a good \(\beta\) is usually smaller; 0.1 is the TRL default and the value the DPO paper used for dialogue [Certain]. What carries over is the shape: a peak, with an over-cautious model on one side and an unchanged model on the other. For the real run, sweep \(\beta \in \{0.05, 0.1, 0.2, 0.5\}\) and choose it by three numbers together: adversarial F1, a yes ratio near 0.5, and caption recall (Step 6). Accuracy alone can hide a “no”-sayer: a model that always says “no” on a balanced test still gets 50%.
Step 6: measure with a POPE-style test
POPE (Polling-based Object Probing Evaluation) asks “Is there a object in the image?” and expects “yes” or “no” [Certain]. For each image it asks about 3 labeled objects (answer “yes”) and 3 absent objects (answer “no”). The absent objects are chosen in three ways, which gives three splits [Certain]:
- random: any object not in the image;
- popular: the most frequent objects in the whole dataset that are not in the image;
- adversarial: the objects that co-occur most often with the image’s real objects.
The adversarial split is the one that tests the language prior, so it is the one that should improve most. From the confusion matrix (TP: “yes” on a real object; FP: “yes” on an absent object; TN, FN likewise), POPE reports five numbers [Certain]:
On 3,000 adversarial questions the start model gives \(TP = 1{,}332\), \(FP = 644\), \(TN = 856\), \(FN = 168\). Accuracy \(= 2{,}188 / 3{,}000 = 0.729\). Precision \(= 1{,}332 / 1{,}976 = 0.674\), recall \(= 1{,}332 / 1{,}500 = 0.888\), \(F_1 = 0.766\). The yes ratio is \(1{,}976 / 3{,}000 = 0.659\): on a test with exactly half “yes” answers, it says “yes” two times out of three. On the random split the same model has accuracy 0.876 and a yes ratio of 0.512. The gap between the splits, 0.876 against 0.729, is the language prior at work.
Parse the answers the way POPE does
A VLM answers in words, not in labels. POPE’s evaluate.py keeps the text before the first period, removes commas, splits on spaces, and labels the answer “no” if one of the words is No, no or not; every other answer counts as “yes” [Certain]. That rule matters when you train a model to be cautious: “I cannot tell.” counts as “yes”, and so does an empty answer. Try it in Figure 4.
evaluate.py, next to a strict rule that accepts only a first word of “yes” or “no” and marks everything else as unparsed. Type any answer. The last example button is an empty answer.How to read Figure 4
“There is no dog in the image.” is “no” under POPE’s rule, but “unparsed” under the strict rule, which only reads the first word. “I cannot tell.” is “yes” under POPE’s rule, because it contains none of the three words, but “unparsed” under the strict rule. “Nope.”, “Not sure.” and “NO” are also “yes” under POPE’s rule, since the match is on exact words and is case-sensitive (“No” and “no” count, “NO” and “Not” do not). “Yes. No other animals.” is “yes”, because only the text before the first period counts. Report the share of unparsed answers next to the POPE numbers: if it changes after DPO, your accuracy change may be a parsing change.
# pope_eval.py: python pope_eval.py out/B-b0.1/final coco_pope_adversarial.json val2014 B-adv.jsonl import sys, json, torch from PIL import Image from peft import PeftModel from transformers import AutoProcessor, AutoModelForImageTextToText BASE = "Qwen/Qwen2.5-VL-3B-Instruct" ckpt, qfile, imgdir, outf = sys.argv[1:5] proc = AutoProcessor.from_pretrained(BASE, min_pixels=256 * 28 * 28, max_pixels=512 * 28 * 28) model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto") if ckpt != "base": model = PeftModel.from_pretrained(model, ckpt).merge_and_unload() model.eval() def pope_label(ans): # the rule of POPE's evaluate.py words = ans.split(".")[0].replace(",", "").split() return "no" if any(w in ("No", "not", "no") for w in words) else "yes" def strict_label(ans): w = ans.strip().lower().split() return w[0].strip(".,!") if w and w[0].strip(".,!") in ("yes", "no") else "unparsed" qs = [json.loads(l) for l in open(qfile) if l.strip()] # one JSON object per line rows = [] for q in qs: msg = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": q["text"] + " Answer yes or no."}]}] text = proc.apply_chat_template(msg, add_generation_prompt=True) img = Image.open(f"{imgdir}/{q['image']}").convert("RGB") inp = proc(text=[text], images=[img], return_tensors="pt").to(model.device) with torch.no_grad(): out = model.generate(**inp, max_new_tokens=10, do_sample=False) # greedy ans = proc.batch_decode(out[:, inp["input_ids"].shape[1]:], skip_special_tokens=True)[0].strip() rows.append({"id": q["question_id"], "label": q["label"], "answer": ans, "pred": pope_label(ans), "strict": strict_label(ans)}) tp = sum(r["pred"] == "yes" and r["label"] == "yes" for r in rows) fp = sum(r["pred"] == "yes" and r["label"] == "no" for r in rows) fn = sum(r["pred"] == "no" and r["label"] == "yes" for r in rows) n = len(rows); tn = n - tp - fp - fn prec, rec = tp / max(1, tp + fp), tp / max(1, tp + fn) # formula (195.5) print(f"acc {(tp + tn) / n:.4f} prec {prec:.4f} rec {rec:.4f} F1 {2 * prec * rec / max(1e-9, prec + rec):.4f} " f"yes {(tp + fp) / n:.4f} unparsed {sum(r['strict'] == 'unparsed' for r in rows) / n:.4f}") with open(outf, "w") as f: for r in rows: f.write(json.dumps(r) + "\n")
Is the difference real? A paired test
Two models answer the same 3,000 questions, so compare them question by question. Count \(b\), the questions model A got wrong and model B got right, and \(c\), the questions A got right and B got wrong. Questions both got right or both got wrong say nothing about the difference. The accuracy difference and its standard error are:
- Read the symbols
- \(N\)
- the number of questions both models answered
- \(b, c\)
- the discordant counts: only B right, only A right
- \(\Delta \pm 1.96\,\mathrm{SE}\)
- an approximate 95% interval for the accuracy difference; if it excludes 0, the difference is unlikely to be noise
This is the paired standard error behind McNemar’s test. It treats the questions as independent. The 6 questions of one image are not fully independent, so with few images the true error bar is a little wider; resampling whole images (a bootstrap over images) handles that [Likely].
How to read Figure 5
Default: start model (A) against sampled DPO (B) on 500 adversarial images. \(b = 402\), \(c = 132\), \(\Delta = +0.090 \pm 0.015\): a clear gain. Now set A to “sampled” and B to “swap, \(\beta = 1\)”. On 500 images, \(\Delta = +0.026\), interval 0.015 to 0.038: swap is better on POPE. Reduce the test to 50 images: \(\Delta = +0.010\), interval \(-0.028\) to \(0.048\); with 300 questions you cannot tell the two apart. Now look at the caption line: the swap model has more hallucinated objects in its captions (\(\text{CHAIR}_i\) 0.188 against 0.140), but it also finds more of the real ones (recall 0.801 against 0.793). And switch to the random split with A = start, B = swap, \(\beta = 1\): the swap model is slightly worse there (0.876 to 0.861, interval \(-0.029\) to \(0.000\)). One number never tells the whole story.
Check the captions too: CHAIR
POPE asks about one object at a time and only in yes/no form. A model can learn the yes/no format without changing how it describes images, or it can learn to describe less. So also generate a description for 500 test images and compute CHAIR (Caption Hallucination Assessment with Image Relevance) [Certain]:
Lower is better for both, and both are trivially 0 for a model that names nothing. So always report recall next to them: the share of labeled objects that the captions name. In the toy, the start model’s captions have \(\text{CHAIR}_i = 0.280\), \(\text{CHAIR}_s = 0.790\) and recall 0.884. The insert model reaches \(\text{CHAIR}_i = 0.015\), which looks excellent, but its recall is 0.091: it names 0.39 objects per caption. The objects_in function of Step 1 is a simple version of CHAIR’s word matching; the original uses a longer synonym list [Certain].
Step 7: decide and write it down
Fill the table from the Setup. Then write a half-page memo:
- Goal. Example: “fewer invented objects in product photo descriptions, without losing objects that are there”.
- Results. The table, with the paired 95% interval of each model against A on the adversarial split, and the number of test images.
- Decision. The pair type and \(\beta\). Reject any model whose yes ratio fell far below 0.5 or whose caption recall fell a lot, even if its adversarial accuracy rose.
- Checks. Read 20 descriptions of each model yourself. Run a general benchmark (for example a VQA set from session 192) to check that answers outside the preference data did not get worse. Check that the unparsed share of Figure 4 did not change.
What earlier work suggests, so you know what to look for: HA-DPO reported large POPE gains on MiniGPT-4, and its authors argued that a consistent style between chosen and rejected answers matters [Certain]. mDPO argued that VLM DPO can ignore the image when the pair can be judged from text alone, and added an image-conditioned term [Certain]. The toy shows a mechanism that fits both observations: pairs that differ in length or style can be won without looking. Your model, data and judge differ, so measure instead of assuming.
Where this is used
| Setting | How the pairs are made | What to watch |
|---|---|---|
| General VLM assistants | AI-feedback or human-corrected pairs (RLAIF-V, RLHF-V) mixed with other preference data | POPE and CHAIR together; general benchmarks for regressions |
| Product and e-commerce image descriptions | Sampled descriptions scored against catalog attributes; swap edits of colors or parts | Recall of real attributes; a model that stops naming attributes looks “safe” |
| Document and chart question answering | Answers checked against extracted text or table values; swap edits of numbers | Exact-match rewards may suit better than DPO (session 190) [Likely] |
| Accessibility captions for blind users | Human corrections of invented objects | Silence is also harmful here: track recall and \(\text{CHAIR}_s\) |
| Video captions and key-moment descriptions for Video-RAG | Captions checked against per-frame detector labels; swap edits of objects or actions | Temporal hallucination (events that did not happen) needs time-aware checks, not only object lists [Likely] |
| Video question answering and action recognition | Yes/no probes such as “Does the person open the door?” with POPE-style negative sampling of co-occurring actions | The adversarial idea carries over: sample actions that often co-occur in the same scene type [Likely] |
| Video retrieval indexes built from generated captions | DPO on captions that name only visible objects | Invented objects in captions become false search hits; measure retrieval precision after tuning [Likely] |
Common mistakes
- Pairs that can be won by saying less. If the chosen answer is usually shorter (insert edits, or a score that only counts mistakes), DPO learns to name fewer objects. Measure the share of pairs where chosen is shorter; prefer swap edits or a coverage reward \(\alpha \gt 0\).
- Reading POPE accuracy alone. A model that drifts toward “no” can gain accuracy on the adversarial split while recall collapses. Always report the yes ratio and recall.
- Training on POPE images. POPE uses COCO
val2014; most of those images are intrain2017. Build pairs fromtrain2014and check the ids. - Truncating VLM sequences. A
max_lengththat cuts image tokens breaks training. Usemax_length=None, or check every example’s length first. - Choosing \(\beta\) by one number. Small \(\beta\) follows the shortcuts in the pairs further. Choose with adversarial F1, the yes ratio and caption recall together.
- Ignoring the answer parser. Under POPE’s rule, every answer without “no”, “No” or “not” before the first period counts as “yes”. A change in answer style after DPO can look like a change in accuracy.
- No error bar. On 50 images, a 1-point difference between two DPO models is noise (Figure 5). Use the paired test (195.7) and at least the full 500 images.
- Evaluating with the same prompt and format as the training pairs only. If the pairs were all “Describe this image”, also test yes/no and open questions; a gain on one format may not transfer to another [Likely].
Interview questions
How would you build preference data to reduce object hallucination in a VLM?basic
What are POPE’s three splits, and why is the adversarial one the most important for this work?basic
Your DPO model improved POPE adversarial accuracy by 8 points. How do you check that it did not just learn to say “no”?core
Why do insert-style pairs (rejected = chosen plus one invented object) tend to make the model over-cautious?core
Write the DPO loss for a VLM pair. What is different from text DPO?core
max_length=None); the processor must place the image tokens the same way for both answers; with LoRA the reference is the base with the adapter off; and the pair must require the image to judge, otherwise the model can learn a text shortcut (the mDPO argument).What does \(\beta\) control here, and what do you expect at a very small \(\beta\)?core
How do you decide if model B is really better than model A on POPE?core
POPE says model C is better, CHAIR says model B is better. Which do you trust?advanced
Why sample the pairs from your own model rather than use GPT-written answers?advanced
How would you extend this lab to video?advanced
Exercises
- A VLM pair has \(\log\pi_\theta(y_w) = -20\), \(\log\pi_\text{ref}(y_w) = -22\), \(\log\pi_\theta(y_l) = -30\), \(\log\pi_\text{ref}(y_l) = -27\), \(\beta = 0.1\). Compute the implicit rewards, the margin, the loss and the gradient weight \(\sigma(-m)\).
- A POPE run gives \(TP = 1{,}400\), \(FP = 300\), \(TN = 1{,}200\), \(FN = 100\). Compute accuracy, precision, recall, F1 and the yes ratio. Is the model too often “yes” or too often “no”?
- Using formula (195.4), show that for a swap pair the toy’s bias \(b\) gets zero gradient. What about an insert pair?
- In Figure 3, set sampled pairs and \(\alpha = 0\). Which \(\beta\) gives the best adversarial F1, and does any \(\beta\) beat the start model’s 0.766? Explain with the yes ratio.
- Model B beats model A with \(b = 30\), \(c = 18\) on \(N = 600\) questions. Compute \(\Delta\), its standard error and a 95% interval. Would you ship B on this evidence?
- In Figure 2, choose insert pairs and \(\beta = 0.5\), and move the step slider. At which step is the adversarial accuracy best, and what does that suggest about early stopping with such pairs?
Answerscheck
Key takeaways
- Hallucination pairs need a precise notion of “better”: fewer invented objects without losing real ones. Score with \(\alpha\,n_\text{correct} - n_\text{halluc}\) (195.1), not with mistakes alone.
- The pair design decides what DPO learns. Pairs that differ in length can be won by saying less; swap pairs, which differ in one object only, force the model to look (Figure 2).
- The DPO loss of a VLM pair (195.2) only sees where the two answers differ; in the toy, exactly the differing objects (195.4).
- Sweep \(\beta\): small values follow the shortcuts in the pairs, large values change nothing. Choose with adversarial F1, the yes ratio and caption recall together.
- Measure with all three POPE splits, POPE’s own parser, a paired error bar (195.7), and CHAIR with recall (195.8). Different metrics can rank models differently, so report all of them.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
- Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 5: Post-Training Vision Language Models
- Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models
- Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Hugging Face (code). TRL library
- Yu et al. (2023). RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
- Yu et al. (2024). RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness
- Wang, Zhou, Huang, Xu, Zhang, Poon & Chen (2024). mDPO: Conditional Preference Optimization for Multimodal Large Language Models
- Zhao, Wang, Ouyang, Dong, Wang & He (2023). Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
- Zhou, Cui, Rafailov, Finn & Yao (2024). Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
- Rohrbach, Hendricks, Burns, Darrell & Saenko (2018). Object Hallucination in Image Captioning
- Wang et al. (2024). Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models
- Azar et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences