VLM failure modes
A vision-language model can describe a photo well and still say there is a fork that is not there, count seven apples as six, miss the total on a receipt, or mix up what happened first in a video. This session takes the four big failure families one by one: object hallucination, counting and spatial relations, small text lost by resizing, and temporal confusion in video. For each one we find the cause, measure it with a simple test, and fix it.
What you will learn
- Why the language prior can beat the image: the Bayes view of a VLM answer, and how co-occurrence drives object hallucination
- How to measure hallucination with CHAIR and POPE, and why a model that always says “yes” gets F1 = 66.7 on POPE
- Visual contrastive decoding (VCD): subtract the answer given a noised image, and why this amplifies only the visual evidence
- Why counting and spatial relations fail: bag-of-words contrastive encoders, CLIP-blind pairs (MMVP), coarse token grids, and the p to the power n effect
- How resizing destroys small text: the effective text height after resizing, and how tiling, native resolution and zoom-in search fix it
- Why video models confuse time: sparse frame sampling, the hit probability d N / T, bag-of-frames encoders, and absolute-time position ids
- A diagnosis protocol: blind, noised-image, crop, resize and frame-shuffle tests that tell you which failure you have
Why this matters
A vision-language model (VLM) can write a fluent paragraph about a photo and still be wrong in ways a child would not be. It says there is a fork next to the plate when there is none. It counts seven apples as six. It cannot read the total on a receipt. In a video, it says the man sat down and then stood up when he did the opposite. This session is about these failures: what causes each one, how to measure it, and how to fix it.
In session 192 we saw how to score a model on benchmarks. A single score hides why a model fails. In practice, almost all VLM errors fall into four families, and each has a different root cause in the pipeline you met in session 177:
- Object hallucination. The model names things that are not in the image. Cause: the language prior, which is what the language model expects from text alone, is stronger than the visual evidence.
- Counting and spatial relations. “How many?” and “Is the cup left of the laptop?” fail. Cause: vision encoders trained with image–text matching learn what is in the image much better than how many or where.
- Small text and fine details. The model cannot read small print or see a small object. Cause: the image is resized to a few hundred pixels before the encoder sees it.
- Temporal confusion in video. The model mixes up order, misses short events, or answers from one frame. Cause: only a few frames are sampled, and the model often treats them as an unordered bag.
These failures matter in real products. A retail shelf audit needs counts. An invoice or form pipeline needs small digits. A content moderation system must not invent objects. A video search system must find a 3-second moment in a 10-minute clip. The good news: each failure has a clear cause, and the cause tells you the fix. Figure 1 shows where each one enters the pipeline.
Object hallucination: the language prior beats the image
Object hallucination means the model mentions an object that is not in the image. It is the most common VLM failure and the most studied one. To see why it happens, write the answer probability with Bayes’ rule. Let \(I\) be the image, \(x\) the question and \(y\) an answer (for example the word “fork”):
- Read the symbols
- \(p(y \mid I, x)\)
- what the VLM outputs: the probability of the answer given the image and the question
- \(p(y \mid x)\)
- the language prior: how likely the answer is from the text alone, with no image
- \(\log \frac{p(I \mid y, x)}{p(I \mid x)}\)
- the visual evidence: how much the image supports this answer more than an average answer; positive if the image supports \(y\), negative if it argues against it
This is an exact identity, not a model. Its message is simple: the answer is the sum of a text term and an image term. A VLM is built on a language model that has read trillions of words. Its prior is strong and sharp: after “a kitchen table with a plate and a knife”, the word “fork” is very likely. The visual term comes from a few hundred visual tokens that the model learned to read on far less data. When the image is clear, the visual term wins. When the image is unclear, small or busy, the visual term is weak, and the prior decides. That is hallucination.
For a yes/no question such as “Is there a fork in the image?” it helps to write the same idea as log-odds (the log of “yes” probability over “no” probability):
- Read the symbols
- \(a\)
- the prior log-odds of “yes” from the text and scene type: large for objects that often co-occur with what is in the image (fork near a plate)
- \(e\)
- the visual evidence: positive if the encoder sees the object, negative if it sees that it is absent
- \(w\)
- how much the model actually uses the image; small when the model has learned to rely on text
- \(\sigma\)
- the logistic (sigmoid) function that turns log-odds into a probability
This is a simplified model for thinking, not a formula from a paper. But it explains the three main causes that the literature reports:
- Co-occurrence. POPE (Li et al.) found that LVLMs (large VLMs) tend to hallucinate objects that appear often in the instruction data, or that often appear together with the objects that are really in the image [Certain]. In (193.2), this is a large \(a\).
- Weak or missing visual evidence. If the encoder cannot see the object (too small, hidden, low resolution), \(e\) is near 0 and the prior wins. This links hallucination to the resizing problem below.
- Training data that teaches the model to talk beyond the image. Many early instruction datasets were written by text-only LLMs from captions and boxes. Long, detailed answers reward plausible extra details. The POPE authors suggest this as one reason why some models hallucinated more [Likely]. Longer answers also hallucinate more: in POPE’s CHAIR test, captions from the longer “brief description” prompt had much higher hallucination rates than short captions for most models (Table 1 of the paper) [Certain].
An image shows a dinner table with a plate and a knife. There is no fork. The question is “Is there a fork in the image?”. The prior is strong: \(a = 2.0\), so the text alone says “yes” with \(\sigma(2.0) = 0.881\). The encoder weakly sees that there is no fork: \(w e = -1.5\).
Then \(z = 2.0 - 1.5 = 0.5\) and \(p(\text{yes}) = \sigma(0.5) = 0.622\). The model says “yes”. It hallucinated, even though its visual evidence pointed the right way. To answer “no”, the visual term must be stronger than the prior: \(w e \lt -2.0\). Below we will see a decoding trick that, with \(\alpha = 1\), doubles the visual term to \(-3.0\), giving \(z = -1.0\) and \(p(\text{yes}) = 0.269\).
How to read Figure 2
Each row starts at 0. The first arrow is the prior, the second is the visual evidence; the dot is where the sum \(z\) lands. Right of the dashed line the model says “yes”. At the defaults, the top row lands at \(z = 0.5\) (“yes”, a hallucination) and the bottom row at \(z = -1.0\) (“no”). Move the prior \(a\) up to 4: even VCD with \(\alpha = 1\) now says “yes”; raise \(\alpha\) to 2 to win again. Move \(s\) toward 1: the noised image keeps the evidence, so the contrast removes nothing and VCD falls back to normal decoding. Finally set \(we\) to a positive value (the fork is really there): VCD makes the correct “yes” even more confident.
Measuring it: CHAIR and POPE
The first standard metric is CHAIR (Caption Hallucination Assessment with Image Relevance, Rohrbach et al., 2018). Generate captions, find the object words in them, and compare them with the ground-truth objects of the image (for example the COCO labels):
\(\text{CHAIR}_i\) counts at the object level, \(\text{CHAIR}_s\) at the caption (sentence) level. CHAIR has two problems. It needs a parser and synonym lists to find objects in free text. And it depends on the prompt and the caption length. In POPE’s tests, LLaVA had \(\text{CHAIR}_s = 32.7\) with “Generate a short caption of the image” and 62.7 with “Provide a brief description of the given image”, with average caption lengths of 64.3 and 90.7 [Certain].
POPE (Polling-based Object Probing Evaluation, Li et al., 2023) turns the problem into yes/no questions: “Is there a <object> in the image?”. Half of the questions ask about objects that are really there (answer “yes”) and half about objects that are not (answer “no”). The trick is how the absent objects are chosen. There are three settings [Certain]:
- Random: any object that is not in the image.
- Popular: the most frequent objects in the whole dataset that are not in this image.
- Adversarial: the objects that co-occur most often with the objects that are in the image (a fork for a table with a plate). This attacks the prior \(a\) directly.
On COCO, POPE used 500 images with more than 3 labeled objects and 6 questions per image (3 “yes”, 3 “no”), so 3,000 questions per setting [Certain]. It reports accuracy, precision, recall, F1 and the yes ratio, the share of questions answered “yes”:
- Read the symbols
- \(TP\)
- true positives: the object is there and the model says “yes”
- \(FP\)
- false positives: the object is not there and the model says “yes”; each one is a hallucination
- \(FN\)
- false negatives: the object is there and the model says “no”
- \(N\)
- the number of questions (3,000 per setting); a balanced test has a correct yes ratio of 50%
The yes ratio is the key diagnostic. In the POPE paper, LLaVA (the first version), mPLUG-Owl and MultiModal-GPT answered “yes” to 95–100% of the questions: they had almost stopped looking at the image [Certain]. A model that always says “yes” has precision 0.5 and recall 1, so its F1 is \(2 \cdot 0.5 \cdot 1 / 1.5 = 0.667\). Several of those models scored F1 between 66.7 and 68.7, which is no better than “always yes”. InstructBLIP was the best model in the paper, but even its F1 fell from 89.29 (random) to 78.45 (adversarial) [Certain]. The fall from random to adversarial is the prior at work. Explore the real rows in Figure 3.
How to read Figure 3
The left square is the confusion matrix; the red cell counts hallucinations (FP). The bars on the right are the metrics of (193.4). Start with “Always yes”: F1 is 0.667 and accuracy 0.5. Now pick “LLaVA, random”: the numbers are almost the same as “always yes”, and the yes ratio is 95.4%. Compare “InstructBLIP, random” and “InstructBLIP, adversarial”: recall barely moves (93.9% vs 93.3%), but the number of hallucinations nearly triples (247 vs 669). That is co-occurrence: adversarial questions ask about objects the prior expects. Always look at the yes ratio and FP next to F1.
Fixing hallucination
The fixes follow from (193.1): make the visual term stronger, make the prior weaker, or teach the model that the prior is not enough.
Visual contrastive decoding (VCD) (Leng et al., 2023) is a training-free fix at decoding time. Run the model twice at each step: once with the real image \(v\) and once with a distorted image \(v'\) (the paper adds Gaussian noise in steps, like the forward process of a diffusion model). The distorted run keeps the prior but loses most of the visual evidence. Then subtract [Certain]:
- Read the symbols
- \(\ell_\theta(y \mid v, x)\)
- the logits (scores before softmax) of the next token with the real image
- \(\ell_\theta(y \mid v', x)\)
- the logits with the noised image: mostly the language prior
- \(\alpha\)
- contrast strength; \(\alpha = 0\) is normal decoding; the paper’s default is \(\alpha = 1\)
Subtracting can also give a high score to junk tokens that are unlikely in both runs. So VCD keeps only tokens that are already plausible with the real image, an adaptive plausibility constraint [Certain]:
with default \(\beta = 0.1\). Why does this work? Put the yes/no model (193.2) into (193.5). With the real image the log-odds are \(a + we\). With the noised image a share \(s\) of the evidence survives: \(a + s\,we\). Then:
The prior \(a\) cancels out of the contrast and stays as it is, while the visual term is multiplied by \(1 + \alpha(1 - s)\). With full noise (\(s = 0\)) and \(\alpha = 1\), the visual evidence counts twice. If the noise does not remove the evidence (\(s = 1\)), VCD does nothing. This is what Figure 2 shows. On POPE (MSCOCO) with LLaVA-1.5, the paper reports F1 going from 81.33 to 87.16 (random) and from 77.57 to 81.33 (adversarial) [Certain]. The cost: two forward passes per token, so decoding is about twice as slow [Likely].
Other fixes, from cheap to expensive:
- Ask shorter, closed questions. Long free descriptions drift away from the image; ask “Is there X?” or “List only objects you can see” when you can [Likely].
- Better visual evidence. Higher resolution and a stronger encoder raise \(e\) (sections below).
- Preference tuning. Train on pairs where the rejected answer has a hallucinated object, so the model learns to push down its own prior-driven words: see session 190 (mDPO, RLHF-V) and the lab session 195.
- Better instruction data. Include questions whose correct answer is “no”, and avoid answers that mention things not in the image.
Counting and spatial relations
“How many people are in the photo?” and “Is the mug to the left of the laptop?” look easy. VLMs are often bad at both. There are three causes, and they stack.
Cause 1: the encoder was trained to know “what”, not “how many” or “where”. Most VLMs use a CLIP- or SigLIP-style encoder (see session 166). Contrastive training only asks: does this image match this caption better than the other captions in the batch? Captions rarely depend on exact counts or on left versus right, so the encoder gets little reward for keeping them. Yuksekgonul et al. showed that such models often behave like a bag of words: they are poor at relations and at matching attributes to the right object, and barely notice word order [Certain]. They built the ARO benchmark (Attribution, Relation, Order) with more than 50,000 test cases [Certain]. Kamath et al. built What’sUp, photos that differ only in the spatial relation between two objects; all 18 models they tested performed poorly, far below humans [Certain].
Cause 2: the language model cannot recover what the encoder dropped. Tong et al. (“Eyes Wide Shut”) searched for CLIP-blind pairs: two images that CLIP sees as almost the same (cosine similarity above 0.95) but a self-supervised model, DINOv2, sees as different (below 0.6) [Certain]. From these pairs they built MMVP: 150 image pairs, 300 questions, scored per pair (both answers must be right, so random guessing gets \(0.5 \times 0.5 = 25\%\)). Humans scored 95.7%, GPT-4V 38.7% and LLaVA-1.5 24.7% [Certain]. “Quantity and count” and “positional and relational context” are two of their nine failure patterns [Certain]. Their fix, Mixture of Features, adds DINOv2 features next to CLIP features [Certain].
Cause 3: a coarse token grid, and many objects. A 336-pixel image with 14-pixel patches becomes a 24 × 24 grid of 576 tokens. Connectors often pool or merge these further (see session 178). Small objects that fall into the same token become one blob. And even if each object is seen with high probability, the chance to get the whole count right falls fast with the number of objects. If each of \(n\) objects is found independently with probability \(p\) and nothing is counted twice:
With \(p = 0.97\), the count is exactly right with probability 0.859 for 5 objects, 0.737 for 10 and 0.544 for 20. This is why counting errors grow with crowd size even for good models.
The token grid adds a second effect. Suppose, as a toy model, that the model can tell whether a cell of the grid contains the object, but not how many copies are inside one cell. With \(M\) cells and \(n\) objects placed at random, the expected number of occupied cells (the classic occupancy formula) is:
- Read the symbols
- \(M\)
- the number of cells that the model can tell apart, for example \(6 \times 6 = 36\) after strong pooling, or \(24 \times 24 = 576\) patch tokens
- \(n\)
- the true number of objects
- \(\big(1 - \tfrac{1}{M}\big)^{n}\)
- the chance that one given cell gets none of the \(n\) objects
With 36 cells and 12 objects, the expected count is 10.33: the toy model undercounts by almost 2. With 576 cells and the same 12 objects it is 11.89. Real models are not this simple, but the direction is right: coarse grids and crowded scenes lead to undercounting. Try it in Figure 4.
How to read Figure 4
At the defaults (12 objects, 6 × 6 cells) the expected toy count is 10.33. Press “New layout” a few times: the count of this layout moves around that value. Now raise the grid to 24 cells per side (the 576-token grid of a 336-pixel image): merges become rare and the count is nearly exact. Then push \(n\) to 40 with a 6 × 6 grid: the toy count saturates near the number of cells. This is why high resolution, less pooling and point-then-count prompts help counting.
Fixing counting and spatial errors
- Count by pointing or detecting. Ask the model to output one point or box per object first, then count the list. Molmo was trained with a large 2D pointing dataset (PixMo) [Certain] and counts by pointing [Likely]. Qwen2.5-VL reports 93.6 on CountBench (72B model) with a “detect then count” style prompt [Certain]. Each object then gets its own output, so errors no longer hide inside one blob.
- Use a detector as a tool. For production counting (shelves, parking lots), an open-vocabulary detector (session 174) plus code is usually more reliable than asking the VLM for a number [Likely].
- Better features. Add self-supervised features (DINOv2, see session 165) next to CLIP (MMVP’s Mixture of Features), or use a less pooled, higher-resolution feature map.
- Hard negatives and spatial data. ARO showed that contrastive training with composition-aware hard negatives (for example, captions with swapped words) improves order and composition tasks [Certain]. For the VLM itself, add instruction data with counts, left/right and boxes.
- Absolute coordinates. Train and ask for boxes in pixel coordinates of the real image; Qwen2.5-VL represents boxes and points in the actual image size, not normalized values [Certain].
Cambrian-1 (Tong et al., 2024) made these skills measurable. It introduced CV-Bench, a vision-centric benchmark of 2,638 manually checked examples, with 2D tasks (spatial relationship: 650, object count: 788, from ADE20K and COCO) and 3D tasks (depth order and relative distance: 600 each, from Omni3D) [Certain]. It also warned that several popular benchmarks (for example MMMU, MathVista and AI2D) show less than 5% difference between vision enabled and disabled: a model can do well on them mostly with its language prior [Certain]. Always include a vision-centric test when you study these failures.
Small text and fine details lost by resizing
Many VLMs resize every image to a fixed square: 224 or 336 pixels for classic CLIP encoders (LLaVA-1.5 uses 336 × 336, so 576 tokens) [Certain]. A phone photo is 3024 × 4032 pixels. If we keep the aspect ratio and fit the long side to the model input of size \(S\), every length in the image shrinks by the same factor:
- Read the symbols
- \(W, H\)
- the original width and height of the image in pixels
- \(S_W, S_H\)
- the width and height the encoder receives (336 and 336 for LLaVA-1.5)
- \(h\)
- the height of a line of text (or of a small object) in the original image
- \(h'\)
- its height after resizing
- \(P\)
- the patch size of the ViT, usually 14 pixels; one token sees one \(P \times P\) patch
A receipt is photographed at 3024 × 4032. The line with the total is 40 pixels tall. Fit to 336: the factor is \(336 / 4032 = 0.0833\), so the text becomes \(40 \times 0.0833 = 3.33\) pixels tall, about a quarter of one 14-pixel patch (0.24). No model can read digits that are 3 pixels tall. At 224 it is 2.22 pixels, and with a 672-pixel canvas (2 × 2 tiles of 336) it is 6.67 pixels.
Now use native resolution as in Qwen2-VL and Qwen2.5-VL, where each visual token covers 28 × 28 pixels after a 2 × 2 merge of 14-pixel patches. The default processor limit is max_pixels = 12845056 \(= 16384 \times 28 \times 28\) [Certain], and \(3024 \times 4032 = 12{,}192{,}768\) is below it. So the photo is kept as it is: the text stays 40 pixels tall, at the cost of \(108 \times 144 = 15{,}552\) visual tokens. With a cap of 1280 tokens (max_pixels = 1280 × 28 × 28, a common setting [Likely]), the processor resizes the photo to 840 × 1148, which is 1,230 tokens, and the text becomes 11.39 pixels tall.
This is the core trade-off of session 180: legibility costs tokens, and tokens cost latency and memory (session 183). Squashing to a square without keeping the aspect ratio is even worse for text: it also distorts the letter shapes. Figure 5 lets you see the effect.
How to read Figure 5
Start at the receipt and 336: the total is 3.33 pixels tall and unreadable. Click “2 × 2 tiles”: it doubles to 6.67 pixels and starts to be readable, for 5 times the tokens. Click “native, ≤ 1280 tokens”: 11.39 pixels with 1,230 tokens, a better trade than tiles here. Switch to the screenshot with 12-pixel text: at 336 it shrinks to 2.10 pixels, while native resolution with the default limit keeps it at about 12 pixels (2,691 tokens). The “readable?” label uses a rough rule for this demo (below 6 pixels: unreadable; 6 to 10: risky), not a measured threshold.
Fixing small text and small objects
- Tiling. Split the image into tiles at the encoder’s size, plus a low-resolution thumbnail for the global view (LLaVA-NeXT, InternVL). Cambrian-1 lists 2,880 visual tokens for LLaVA-NeXT [Certain].
- Native resolution. Patchify the image at (almost) its real size, with 2-D positions (Qwen2-VL, Qwen2.5-VL). Set
max_pixelsfrom the smallest text you must read, using (193.10) backwards. - Better connectors for high resolution. Cambrian-1 found that high-resolution encoders greatly improve chart and vision-centric tasks, and that ConvNet encoders handle high resolution well [Certain]. Its Spatial Vision Aggregator (SVA) uses a grid of learnable queries, where each query attends only to its own sub-region of the encoder features; the final models use 576 visual tokens [Certain].
- Zoom in on demand. V* (Wu and Xie) lets the LLM search the image for the region it needs and look again at high resolution, for crowded high-resolution images [Certain]. A simple version is a crop tool: the model names a box, the system crops it at full resolution and feeds it back (see session 154).
- OCR as a tool. For documents, run a dedicated OCR model and give its text (with positions) to the VLM. This is often cheaper than very high resolution [Likely].
Temporal confusion in video
A Video-LLM (see session 205) usually samples a fixed number of frames \(N\), encodes each frame like an image, and puts all the visual tokens into the language model. A 10-minute video at 30 frames per second has 18,000 frames; the model may see 32. Three things go wrong.
Short events fall between frames. With uniform sampling the gap between frames is \(T/N\). An event of length \(d\) is seen in at least one frame with probability (assuming a random start of the sampling grid):
- Read the symbols
- \(T\)
- video length in seconds
- \(N\)
- number of sampled frames; \(T/N\) is the stride, the time between two frames
- \(d\)
- duration of the event, for example a 3-second goal or a fall
For a 600-second video with 32 frames, the stride is 18.75 seconds and a 3-second event is seen with probability \(3 \times 32 / 600 = 0.16\). With 64 frames it is 0.32. A model that “misses” the event never had a chance. Worse, the language prior fills the gap: it answers with what usually happens in such a video.
Order is lost. Many encoders process each frame on its own. If the frame tokens carry weak or no time positions, the model sees a bag of frames and cannot tell “sat down, then stood up” from the reverse. TempCompass (Liu et al., 2024) tested this with “conflicting videos” that share the same static content but differ in one temporal aspect such as direction or speed, so that single-frame cues and language priors do not help; the 8 Video LLMs and 3 image LLMs they tested showed notably poor temporal perception [Certain].
Time is in frames, not seconds. If position ids count frames, “frame 10” means 5 seconds at 2 fps and 50 seconds at 0.2 fps. The model cannot answer “when?” or compare speeds across videos sampled differently. Qwen2.5-VL aligns the temporal part of its M-RoPE positions with absolute time, so the gaps between temporal ids reflect real time, and it samples the frame rate dynamically during training [Certain]. It also groups two consecutive frames into one temporal step to save tokens [Certain], which means one temporal token already spans two frames.
How to read Figure 6
At the defaults (600 s, 32 frames, 3 s events) the stride is 18.75 s and each event is seen with probability 0.16 by (193.11); here all five events are missed. Raise \(N\) to 128: the stride is 4.69 s and the chance is 0.64. Now raise \(d\) to 20 s: every event is caught even with 32 frames. The lesson: the frame budget must be chosen from the shortest event you care about, \(N \ge T/d\), or you must pick frames smartly (key frames, scene cuts, or retrieval).
Fixing temporal confusion
- More frames, fewer tokens per frame. Raise \(N\) and compress each frame (pooling, token merging; see session 183). Qwen2.5-VL was evaluated with up to 768 frames and at most 24,576 video tokens [Certain].
- Smart frame selection. Sample at scene cuts, at motion peaks, or retrieve frames that match the question with a CLIP-style score (see session 206).
- Time in the input. Use absolute-time position ids (Qwen2.5-VL) or write timestamps as text next to each frame, so “when” questions have an anchor [Likely].
- Temporal training data. Include questions that need order, direction and speed, and conflicting videos (reversed or reordered clips with flipped answers) so single-frame shortcuts stop working.
- Test the shortcut. Ask the same question with the frames shuffled or reversed. If the answer does not change on an order question, the model is using a bag of frames.
A diagnosis protocol
When a VLM fails, first find which failure you have. Each test removes or changes one input and watches the answer. The table maps symptoms to tests and fixes.
| Failure | Typical symptom | Test that confirms the cause | Fix that targets it |
|---|---|---|---|
| Object hallucination | Names typical but absent objects; yes ratio far above 50% | Blind test (no image or a blank image): the same answer means the prior decides. POPE adversarial vs random | VCD, preference tuning with hallucinated negatives, “no” questions in data |
| Counting | Undercounts crowds; right for 2–3 objects, wrong for 10+ | Accuracy vs true count; same image at higher resolution; ask for points first | Point/detect then count, detector tool, less pooling |
| Spatial relations | Left/right and above/below at chance | Mirror the image horizontally: the answer should flip; CV-Bench, What’sUp | Spatial instruction data, absolute boxes, SSL features |
| Small text / details | Wrong digits, “I cannot read it”, guesses typical values | Crop the region and ask again at full resolution; log the encoder input size | Tiling, native resolution, zoom-in tool, OCR tool |
| Temporal confusion | Misses short events; wrong order; answers from one frame | Shuffle or reverse frames; double \(N\); compare with a single middle frame | More frames with compression, smart sampling, absolute time ids |
- input: model \(f\), image or video \(I\), question \(x\), wrong answer \(\hat y = f(I, x)\)
- \(y_0 \leftarrow f(\varnothing, x)\) // blind test: no image, or a blank gray image
- if \(y_0 = \hat y\) then report “language prior” and stop // the image did not change the answer
- \(y_c \leftarrow f(\text{crop}(I, \text{region}), x)\) at full resolution // zoom on the relevant region
- if \(y_c\) is correct then report “resolution” and stop
- if \(x\) asks for a count then ask for one point per object, count the points // pointing splits the blob
- if the point count is correct then report “counting by direct read-out” and stop
- if \(x\) asks about left/right then \(y_m \leftarrow f(\text{mirror}(I), x)\); if \(y_m = \hat y\) then report “spatial relation ignored”
- if \(I\) is a video then
- \(y_s \leftarrow f(\text{shuffle}(I), x)\); if \(x\) needs order and \(y_s = \hat y\) then report “bag of frames”
- \(y_{2N} \leftarrow f(I \text{ with } 2N \text{ frames}, x)\); if \(y_{2N}\) is correct then report “sparse sampling”
- else report “encoder cannot see it” // try a stronger encoder or SSL features
Run this on a few hundred failures, not one. The share of each family tells you where to spend effort. The second algorithm is the decoding fix for the first family.
- input: image \(v\), prompt and answer so far \(x, y_{\lt t}\), strength \(\alpha\), plausibility \(\beta\), noise steps
- \(v' \leftarrow \operatorname{noise}(v)\) // once per image, reused for every step
- \(\mathbf{u} \leftarrow \text{model}(v, x, y_{\lt t})\), \(\;\mathbf{u}' \leftarrow \text{model}(v', x, y_{\lt t})\) // two passes, two KV caches
- \(\mathbf{p} \leftarrow \softmax(\mathbf{u})\)
- \(\mathcal{V}_{\text{head}} \leftarrow \{y : p_y \ge \beta \max_w p_w\}\) // (193.6)
- \(\mathbf{c} \leftarrow (1 + \alpha)\mathbf{u} - \alpha \mathbf{u}'\); set \(c_y = -\infty\) for \(y \notin \mathcal{V}_{\text{head}}\) // (193.5)
- \(y_t \leftarrow\) sample from \(\softmax(\mathbf{c})\) (or take the argmax)
- return \(y_t\)
Here \(\mathbf{u}\) and \(\mathbf{u}'\) are the two logit vectors \(\ell_\theta(\cdot \mid v, x)\) and \(\ell_\theta(\cdot \mid v', x)\) of (193.5). Each failure family also has a standard benchmark: POPE and CHAIR for hallucination, CV-Bench and MMVP for counting and spatial skills, text-heavy sets such as TextVQA and DocVQA for small text, and Video-MME or TempCompass for time (see session 192).
The code: POPE metrics, VCD, resizing and frame sampling
The code below implements the four measurements of this session in plain PyTorch, with no model download. It computes POPE metrics (193.4), one VCD step (193.5–193.6), the effective text height under fixed and native resizing (193.10), and checks the frame hit formula (193.11) by simulation. It is deterministic.
import math, torch torch.manual_seed(0) # 1) POPE metrics from yes/no answers (1 = "yes") def pope_metrics(pred, gold): tp = ((pred == 1) & (gold == 1)).sum().item() fp = ((pred == 1) & (gold == 0)).sum().item() fn = ((pred == 0) & (gold == 1)).sum().item() acc = (pred == gold).float().mean().item() prec = tp / max(tp + fp, 1) rec = tp / max(tp + fn, 1) f1 = 2 * prec * rec / max(prec + rec, 1e-9) return dict(acc=round(acc, 4), prec=round(prec, 4), rec=round(rec, 4), f1=round(f1, 4), yes=round(pred.float().mean().item(), 4)) gold = torch.tensor([1, 1, 1, 0, 0, 0] * 500) # 500 images x 6 questions always_yes = torch.ones_like(gold) print("always yes:", pope_metrics(always_yes, gold)) # a model that sees real objects 90% of the time and says "yes" to 30% of absent ones g = torch.Generator().manual_seed(0) u = torch.rand(gold.shape, generator=g) pred = torch.where(gold == 1, (u < 0.9).long(), (u < 0.3).long()) print("biased model:", pope_metrics(pred, gold)) # 2) Visual contrastive decoding (VCD) for one next-token step def vcd_logits(logit_img, logit_noisy, alpha=1.0, beta=0.1): """(1+alpha)*l(y|v,x) - alpha*l(y|v',x), restricted to plausible tokens.""" p = logit_img.softmax(-1) keep = p >= beta * p.max(-1, keepdim=True).values # head set V_head out = (1 + alpha) * logit_img - alpha * logit_noisy return out.masked_fill(~keep, float("-inf")) vocab = ["yes", "no", "maybe", "a"] l_img = torch.tensor([0.5, 0.0, -1.5, -3.0]) # with the real image l_noisy = torch.tensor([2.0, 0.0, -1.5, -3.0]) # with a noised image: prior only for name, l in [("regular", l_img), ("vcd", vcd_logits(l_img, l_noisy))]: pr = l.softmax(-1) print(name, {w: round(pr[i].item(), 3) for i, w in enumerate(vocab)}) # 3) Effective text height after resizing (aspect ratio kept) def smart_resize(h, w, factor=28, min_pixels=56 * 56, max_pixels=1280 * 28 * 28): hb, wb = round(h / factor) * factor, round(w / factor) * factor if hb * wb > max_pixels: b = math.sqrt(h * w / max_pixels) hb, wb = math.floor(h / b / factor) * factor, math.floor(w / b / factor) * factor elif hb * wb < min_pixels: b = math.sqrt(min_pixels / (h * w)) hb, wb = math.ceil(h * b / factor) * factor, math.ceil(w * b / factor) * factor return hb, wb H, W, text_h = 4032, 3024, 40 # phone photo of a receipt for side in (224, 336, 672): s = min(side / H, side / W) print(f"fixed {side}: text {text_h * s:.2f} px") hb, wb = smart_resize(H, W) print("native 1280-token cap:", (hb, wb), "tokens", (hb // 28) * (wb // 28), f"text {text_h * hb / H:.2f} px") # 4) Uniform frame sampling: chance to see a short event def hit_rate(T, N, d, trials=20000): g = torch.Generator().manual_seed(1) stride = T / N offset = torch.rand(trials, generator=g) * stride # random phase start = torch.rand(trials, generator=g) * (T - d) # event start t = offset[:, None] + stride * torch.arange(N)[None, :] # frame times seen = ((t >= start[:, None]) & (t <= start[:, None] + d)).any(1) return seen.float().mean().item() for N in (8, 32, 64): print(f"T=600 s, N={N}: simulated {hit_rate(600, N, 3):.3f}, formula {min(1, 3 * N / 600):.3f}")
Output (the same on every run):
always yes: {'acc': 0.5, 'prec': 0.5, 'rec': 1.0, 'f1': 0.6667, 'yes': 1.0}
biased model: {'acc': 0.7953, 'prec': 0.7445, 'rec': 0.8993, 'f1': 0.8146, 'yes': 0.604}
regular {'yes': 0.564, 'no': 0.342, 'maybe': 0.076, 'a': 0.017}
vcd {'yes': 0.231, 'no': 0.629, 'maybe': 0.14, 'a': 0.0}
fixed 224: text 2.22 px
fixed 336: text 3.33 px
fixed 672: text 6.67 px
native 1280-token cap: (1148, 840) tokens 1230 text 11.39 px
T=600 s, N=8: simulated 0.038, formula 0.040
T=600 s, N=32: simulated 0.160, formula 0.160
T=600 s, N=64: simulated 0.322, formula 0.320
Read the output. “Always yes” gets F1 0.667, as computed above. In the VCD step, “yes” was the top token (0.564) and becomes 0.231, while “no” rises to 0.629; the ratio of “yes” to “no” is \(e^{-1}\), the same log-odds of \(-1.0\) as in the worked example. The token “a” had probability 0.017, below \(\beta \times 0.564 = 0.056\), so the plausibility constraint removes it. The resize lines repeat the receipt example. The simulated hit rates match (193.11) up to sampling noise. To use VCD with a real model, compute the two logit vectors with two forward passes (one per image) and call vcd_logits at each step.
Where this is used
| Application | Main failure risk | What teams do |
|---|---|---|
| Image captioning and alt text | Object hallucination, especially in long captions | Short captions, CHAIR / POPE in evaluation, preference tuning, VCD-style decoding [Likely] |
| Retail shelf audit, crowd and inventory counting | Undercounting crowded scenes | Detector or point-then-count, high resolution, count by tiles [Likely] |
| Invoices, receipts, forms, screenshots (Document AI) | Small text lost by resizing | Native resolution with a chosen max_pixels, tiling, OCR as a tool |
| Robotics and UI agents | Spatial relations, exact positions | Absolute coordinates, pointing, crop-and-zoom loops |
| Video key-moment detection and temporal grounding | Short events between frames | Dense sampling with token compression, shot-cut sampling, absolute time ids |
| Video action recognition and video QA | Bag-of-frames answers, single-frame shortcuts | Reverse/shuffle tests, temporal training data (TempCompass-style) |
| Video-RAG and text-to-video retrieval | The retrieved clip misses the moment; the answer is filled in by the prior | Retrieve frames by question, cite timestamps, check answers against frames [Speculation] |
| Content moderation | Hallucinated objects lead to false flags | Ask closed questions per policy item, check yes ratio on clean data |
Common mistakes
- Reporting only F1 or accuracy on POPE. A model can win by changing its yes ratio. Always report the yes ratio and FP, and test the adversarial setting.
- Evaluating on benchmarks that do not need the image. Cambrian-1 found less than 5% gap with and without vision on several popular sets. Run a blind baseline on every benchmark you use.
- Blaming the language model for what the resize destroyed. Before you fine-tune, check the real encoder input size. Many “OCR failures” are 3-pixel text.
- Tiling without a thumbnail. Tiles cut objects and text lines at tile borders; the global view is needed to put them back together.
- Asking for a number when you can ask for a list. “How many?” forces one read-out. “Point to each X” gives one output per object and can be checked.
- Choosing the frame count from the token budget only. Choose it from the shortest event you must catch, \(N \ge T/d\), then compress tokens to fit the budget.
- Forgetting the time unit. Training at 2 fps and serving at 0.5 fps changes what “frame 10” means if positions count frames.
- Using VCD without the plausibility constraint. The contrast alone can push up rare junk tokens; keep \(\beta\) around 0.1.
Interview questions
Why do VLMs hallucinate objects?basic
Explain POPE and why the adversarial setting matters.core
A model scores F1 = 67 on POPE. Is that good?core
How does visual contrastive decoding work, and why does it reduce hallucination?advanced
Why are VLMs bad at counting?core
What are CLIP-blind pairs and what do they tell us about VLMs?advanced
Our invoice model gets digits wrong. What do you check first?basic
max_pixels (1280 tokens gives 11.39 px here), tiling with a thumbnail, a zoom/crop tool, or OCR as a tool. Also check that serving uses the same resolution settings as fine-tuning.How do you pick the number of frames for a video task?core
How can you tell if a Video-LLM uses temporal order at all?advanced
Design an evaluation suite that separates the four failure families.advanced
Exercises
- In the log-odds model (193.2), the prior is \(a = 1.0\) and the visual term is \(we = -2.0\). What is \(p(\text{yes})\)? Now apply VCD with \(\alpha = 0.5\) and \(s = 0\). What is \(p(\text{yes})\) now?
- On balanced POPE (1,500 real, 1,500 absent), a model says “yes” to 1,350 real and 450 absent objects. Compute precision, recall, accuracy, F1 and the yes ratio.
- A model finds each object with probability 0.98. What is the chance of an exact count for 15 objects? For 30?
- A 1920 × 1080 screenshot has 12-pixel text. How tall is the text after fitting to 336 (aspect ratio kept)? To 448? How large must the long side be for the text to stay at least 8 pixels tall?
- A 90-second clip is sampled at 16 frames. What is the chance to see a 1.5-second event? How many frames do you need to be sure to see it?
- On MMVP, a model answers each question right with probability 0.7, independently. What MMVP score do you expect? Why is random guessing 25%?
Answerscheck
Key takeaways
- Most VLM errors fall into four families with four different causes: the language prior (hallucination), the encoder (counting and spatial), resizing (small text), and frame sampling (video time).
- Bayes’ rule (193.1) splits an answer into prior plus visual evidence; when evidence is weak, co-occurring objects get hallucinated. POPE’s adversarial setting and the yes ratio expose this; “always yes” gets F1 0.667.
- VCD (193.5) subtracts the logits of a noised image; in the log-odds view the prior stays and the visual term is multiplied by \(1 + \alpha(1 - s)\) (193.7). LLaVA-1.5 POPE F1 (random) rose from 81.33 to 87.16.
- Counting fails because contrastive encoders keep “what” not “how many”, grids are coarse (193.9), and exact counts need every object (\(p^n\)). Point or detect, then count.
- Resizing shrinks text by \(\min(S_W/W, S_H/H)\) (193.10): 40 px becomes 3.33 px at 336. Tiling, native resolution with a chosen
max_pixels, and zoom-in tools fix it at a token cost. - Video models miss events with probability \(1 - \min(1, dN/T)\) (193.11) and lose order when frames are a bag. Choose \(N\) from the shortest event, add absolute time, and test with reversed clips.
- Diagnose before fixing: blind, crop, mirror, shuffle and more-frames tests (Algorithm 1) tell you which family you have.
Key formulas
References for this session
- Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models
- Tong et al. (2024). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 11: Advanced Topics and Cutting-Edge Research
- Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models
- Tong et al. (2024). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- Rohrbach, Hendricks, Burns, Darrell & Saenko (2018). Object Hallucination in Image Captioning
- Leng, Zhang, Chen, Li, Lu, Miao & Bing (2023). Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
- Tong, Liu, Zhai, Ma, LeCun & Xie (2024). Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
- Yuksekgonul, Bianchi, Kalluri, Jurafsky & Zou (2022). When and why vision-language models behave like bags-of-words, and what to do about it?
- Kamath, Hessel & Chang (2023). What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- Wu & Xie (2023). V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
- Deitke, Clark et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
- Liu, Li, Liu, Wang, Ren, Li, Chen, Sun & Hou (2024). TempCompass: Do Video LLMs Really Understand Videos?
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning