LLM & VLM courseBlog
193Track 7: Vision and VLMs · session 31 of 38

VLM failure modes

A vision-language model can describe a photo well and still say there is a fork that is not there, count seven apples as six, miss the total on a receipt, or mix up what happened first in a video. This session takes the four big failure families one by one: object hallucination, counting and spatial relations, small text lost by resizing, and temporal confusion in video. For each one we find the cause, measure it with a simple test, and fix it.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on the anatomy of a VLM (177), high resolution (180), VLM preference optimization (190) and evaluating VLMs (192)

What you will learn

  • Why the language prior can beat the image: the Bayes view of a VLM answer, and how co-occurrence drives object hallucination
  • How to measure hallucination with CHAIR and POPE, and why a model that always says “yes” gets F1 = 66.7 on POPE
  • Visual contrastive decoding (VCD): subtract the answer given a noised image, and why this amplifies only the visual evidence
  • Why counting and spatial relations fail: bag-of-words contrastive encoders, CLIP-blind pairs (MMVP), coarse token grids, and the p to the power n effect
  • How resizing destroys small text: the effective text height after resizing, and how tiling, native resolution and zoom-in search fix it
  • Why video models confuse time: sparse frame sampling, the hit probability d N / T, bag-of-frames encoders, and absolute-time position ids
  • A diagnosis protocol: blind, noised-image, crop, resize and frame-shuffle tests that tell you which failure you have

Why this matters

A vision-language model (VLM) can write a fluent paragraph about a photo and still be wrong in ways a child would not be. It says there is a fork next to the plate when there is none. It counts seven apples as six. It cannot read the total on a receipt. In a video, it says the man sat down and then stood up when he did the opposite. This session is about these failures: what causes each one, how to measure it, and how to fix it.

In session 192 we saw how to score a model on benchmarks. A single score hides why a model fails. In practice, almost all VLM errors fall into four families, and each has a different root cause in the pipeline you met in session 177:

  1. Object hallucination. The model names things that are not in the image. Cause: the language prior, which is what the language model expects from text alone, is stronger than the visual evidence.
  2. Counting and spatial relations. “How many?” and “Is the cup left of the laptop?” fail. Cause: vision encoders trained with image–text matching learn what is in the image much better than how many or where.
  3. Small text and fine details. The model cannot read small print or see a small object. Cause: the image is resized to a few hundred pixels before the encoder sees it.
  4. Temporal confusion in video. The model mixes up order, misses short events, or answers from one frame. Cause: only a few frames are sampled, and the model often treats them as an unordered bag.

These failures matter in real products. A retail shelf audit needs counts. An invoice or form pipeline needs small digits. A content moderation system must not invent objects. A video search system must find a 3-second moment in a 10-minute clip. The good news: each failure has a clear cause, and the cause tells you the fix. Figure 1 shows where each one enters the pipeline.

image e.g. 3024 × 4032 video e.g. 600 s at 30 fps resize / sample frames 336 × 336, 32 frames vision encoder CLIP / SigLIP ViT + connector 576 tokens language model visual tokens + question → answer “There is a fork.” 3. small text lost 40 px text → 3.3 px 4. temporal confusion a 3 s event between frames 2. counting, spatial “what”, not “how many” 1. object hallucination text prior beats the image Each failure enters at a different stage, so each needs a different test and a different fix
Figure 1. Where the four failure families enter a VLM. Resizing and frame sampling throw information away before the model sees anything. The vision encoder keeps “what” better than “how many” and “where”. The language model can override weak visual evidence with what text usually says.

Object hallucination: the language prior beats the image

Object hallucination means the model mentions an object that is not in the image. It is the most common VLM failure and the most studied one. To see why it happens, write the answer probability with Bayes’ rule. Let \(I\) be the image, \(x\) the question and \(y\) an answer (for example the word “fork”):

\[ \log p(y \mid I, x) = \underbrace{\log p(y \mid x)}_{\text{language prior}} + \underbrace{\log \frac{p(I \mid y, x)}{p(I \mid x)}}_{\text{visual evidence}} \tag{193.1} \]
Read the symbols
\(p(y \mid I, x)\)
what the VLM outputs: the probability of the answer given the image and the question
\(p(y \mid x)\)
the language prior: how likely the answer is from the text alone, with no image
\(\log \frac{p(I \mid y, x)}{p(I \mid x)}\)
the visual evidence: how much the image supports this answer more than an average answer; positive if the image supports \(y\), negative if it argues against it

This is an exact identity, not a model. Its message is simple: the answer is the sum of a text term and an image term. A VLM is built on a language model that has read trillions of words. Its prior is strong and sharp: after “a kitchen table with a plate and a knife”, the word “fork” is very likely. The visual term comes from a few hundred visual tokens that the model learned to read on far less data. When the image is clear, the visual term wins. When the image is unclear, small or busy, the visual term is weak, and the prior decides. That is hallucination.

For a yes/no question such as “Is there a fork in the image?” it helps to write the same idea as log-odds (the log of “yes” probability over “no” probability):

\[ z = \log \frac{p(\text{yes} \mid I, x)}{p(\text{no} \mid I, x)} = a + w\,e, \qquad p(\text{yes}) = \sigma(z) = \frac{1}{1 + e^{-z}} \tag{193.2} \]
Read the symbols
\(a\)
the prior log-odds of “yes” from the text and scene type: large for objects that often co-occur with what is in the image (fork near a plate)
\(e\)
the visual evidence: positive if the encoder sees the object, negative if it sees that it is absent
\(w\)
how much the model actually uses the image; small when the model has learned to rely on text
\(\sigma\)
the logistic (sigmoid) function that turns log-odds into a probability

This is a simplified model for thinking, not a formula from a paper. But it explains the three main causes that the literature reports:

  • Co-occurrence. POPE (Li et al.) found that LVLMs (large VLMs) tend to hallucinate objects that appear often in the instruction data, or that often appear together with the objects that are really in the image [Certain]. In (193.2), this is a large \(a\).
  • Weak or missing visual evidence. If the encoder cannot see the object (too small, hidden, low resolution), \(e\) is near 0 and the prior wins. This links hallucination to the resizing problem below.
  • Training data that teaches the model to talk beyond the image. Many early instruction datasets were written by text-only LLMs from captions and boxes. Long, detailed answers reward plausible extra details. The POPE authors suggest this as one reason why some models hallucinated more [Likely]. Longer answers also hallucinate more: in POPE’s CHAIR test, captions from the longer “brief description” prompt had much higher hallucination rates than short captions for most models (Table 1 of the paper) [Certain].
Worked example: a hallucinated fork, and how much the prior must lose

An image shows a dinner table with a plate and a knife. There is no fork. The question is “Is there a fork in the image?”. The prior is strong: \(a = 2.0\), so the text alone says “yes” with \(\sigma(2.0) = 0.881\). The encoder weakly sees that there is no fork: \(w e = -1.5\).

Then \(z = 2.0 - 1.5 = 0.5\) and \(p(\text{yes}) = \sigma(0.5) = 0.622\). The model says “yes”. It hallucinated, even though its visual evidence pointed the right way. To answer “no”, the visual term must be stronger than the prior: \(w e \lt -2.0\). Below we will see a decoding trick that, with \(\alpha = 1\), doubles the visual term to \(-3.0\), giving \(z = -1.0\) and \(p(\text{yes}) = 0.269\).

Figure 2. The answer to “Is there a fork?” as a sum of two arrows on the log-odds axis: the language prior \(a\) and the visual term \(we\). The top row is normal decoding (193.2). The bottom row is visual contrastive decoding (193.5), which makes the visual arrow longer and leaves the prior alone.
How to read Figure 2

Each row starts at 0. The first arrow is the prior, the second is the visual evidence; the dot is where the sum \(z\) lands. Right of the dashed line the model says “yes”. At the defaults, the top row lands at \(z = 0.5\) (“yes”, a hallucination) and the bottom row at \(z = -1.0\) (“no”). Move the prior \(a\) up to 4: even VCD with \(\alpha = 1\) now says “yes”; raise \(\alpha\) to 2 to win again. Move \(s\) toward 1: the noised image keeps the evidence, so the contrast removes nothing and VCD falls back to normal decoding. Finally set \(we\) to a positive value (the fork is really there): VCD makes the correct “yes” even more confident.

Measuring it: CHAIR and POPE

The first standard metric is CHAIR (Caption Hallucination Assessment with Image Relevance, Rohrbach et al., 2018). Generate captions, find the object words in them, and compare them with the ground-truth objects of the image (for example the COCO labels):

\[ \text{CHAIR}_i = \frac{|\{\text{hallucinated objects}\}|}{|\{\text{all mentioned objects}\}|}, \qquad \text{CHAIR}_s = \frac{|\{\text{captions with a hallucinated object}\}|}{|\{\text{all captions}\}|} \tag{193.3} \]

\(\text{CHAIR}_i\) counts at the object level, \(\text{CHAIR}_s\) at the caption (sentence) level. CHAIR has two problems. It needs a parser and synonym lists to find objects in free text. And it depends on the prompt and the caption length. In POPE’s tests, LLaVA had \(\text{CHAIR}_s = 32.7\) with “Generate a short caption of the image” and 62.7 with “Provide a brief description of the given image”, with average caption lengths of 64.3 and 90.7 [Certain].

POPE (Polling-based Object Probing Evaluation, Li et al., 2023) turns the problem into yes/no questions: “Is there a <object> in the image?”. Half of the questions ask about objects that are really there (answer “yes”) and half about objects that are not (answer “no”). The trick is how the absent objects are chosen. There are three settings [Certain]:

  • Random: any object that is not in the image.
  • Popular: the most frequent objects in the whole dataset that are not in this image.
  • Adversarial: the objects that co-occur most often with the objects that are in the image (a fork for a table with a plate). This attacks the prior \(a\) directly.

On COCO, POPE used 500 images with more than 3 labeled objects and 6 questions per image (3 “yes”, 3 “no”), so 3,000 questions per setting [Certain]. It reports accuracy, precision, recall, F1 and the yes ratio, the share of questions answered “yes”:

\[ \text{Prec} = \frac{TP}{TP + FP}, \quad \text{Rec} = \frac{TP}{TP + FN}, \quad F_1 = \frac{2\,\text{Prec}\,\text{Rec}}{\text{Prec} + \text{Rec}}, \quad \text{yes ratio} = \frac{TP + FP}{N} \tag{193.4} \]
Read the symbols
\(TP\)
true positives: the object is there and the model says “yes”
\(FP\)
false positives: the object is not there and the model says “yes”; each one is a hallucination
\(FN\)
false negatives: the object is there and the model says “no”
\(N\)
the number of questions (3,000 per setting); a balanced test has a correct yes ratio of 50%

The yes ratio is the key diagnostic. In the POPE paper, LLaVA (the first version), mPLUG-Owl and MultiModal-GPT answered “yes” to 95–100% of the questions: they had almost stopped looking at the image [Certain]. A model that always says “yes” has precision 0.5 and recall 1, so its F1 is \(2 \cdot 0.5 \cdot 1 / 1.5 = 0.667\). Several of those models scored F1 between 66.7 and 68.7, which is no better than “always yes”. InstructBLIP was the best model in the paper, but even its F1 fell from 89.29 (random) to 78.45 (adversarial) [Certain]. The fall from random to adversarial is the prior at work. Explore the real rows in Figure 3.

Figure 3. POPE metrics from a confusion matrix of 3,000 questions (1,500 with a real object, 1,500 with an absent one). The presets rebuild real rows of Table 3 of the POPE paper (MSCOCO) from their recall and yes ratio.
How to read Figure 3

The left square is the confusion matrix; the red cell counts hallucinations (FP). The bars on the right are the metrics of (193.4). Start with “Always yes”: F1 is 0.667 and accuracy 0.5. Now pick “LLaVA, random”: the numbers are almost the same as “always yes”, and the yes ratio is 95.4%. Compare “InstructBLIP, random” and “InstructBLIP, adversarial”: recall barely moves (93.9% vs 93.3%), but the number of hallucinations nearly triples (247 vs 669). That is co-occurrence: adversarial questions ask about objects the prior expects. Always look at the yes ratio and FP next to F1.

Fixing hallucination

The fixes follow from (193.1): make the visual term stronger, make the prior weaker, or teach the model that the prior is not enough.

Visual contrastive decoding (VCD) (Leng et al., 2023) is a training-free fix at decoding time. Run the model twice at each step: once with the real image \(v\) and once with a distorted image \(v'\) (the paper adds Gaussian noise in steps, like the forward process of a diffusion model). The distorted run keeps the prior but loses most of the visual evidence. Then subtract [Certain]:

\[ p_{\text{vcd}}(y \mid v, v', x) = \softmax\Big[ (1 + \alpha)\, \ell_\theta(y \mid v, x) - \alpha\, \ell_\theta(y \mid v', x) \Big] \tag{193.5} \]
Read the symbols
\(\ell_\theta(y \mid v, x)\)
the logits (scores before softmax) of the next token with the real image
\(\ell_\theta(y \mid v', x)\)
the logits with the noised image: mostly the language prior
\(\alpha\)
contrast strength; \(\alpha = 0\) is normal decoding; the paper’s default is \(\alpha = 1\)

Subtracting can also give a high score to junk tokens that are unlikely in both runs. So VCD keeps only tokens that are already plausible with the real image, an adaptive plausibility constraint [Certain]:

\[ \mathcal{V}_{\text{head}} = \big\{\, y : p_\theta(y \mid v, x) \ge \beta \max_{w} p_\theta(w \mid v, x) \,\big\}, \qquad p_{\text{vcd}}(y) = 0 \text{ for } y \notin \mathcal{V}_{\text{head}} \tag{193.6} \]

with default \(\beta = 0.1\). Why does this work? Put the yes/no model (193.2) into (193.5). With the real image the log-odds are \(a + we\). With the noised image a share \(s\) of the evidence survives: \(a + s\,we\). Then:

\[ z_{\text{vcd}} = (1 + \alpha)(a + we) - \alpha(a + s\,we) = a + (1 + \alpha - \alpha s)\, w e \tag{193.7} \]

The prior \(a\) cancels out of the contrast and stays as it is, while the visual term is multiplied by \(1 + \alpha(1 - s)\). With full noise (\(s = 0\)) and \(\alpha = 1\), the visual evidence counts twice. If the noise does not remove the evidence (\(s = 1\)), VCD does nothing. This is what Figure 2 shows. On POPE (MSCOCO) with LLaVA-1.5, the paper reports F1 going from 81.33 to 87.16 (random) and from 77.57 to 81.33 (adversarial) [Certain]. The cost: two forward passes per token, so decoding is about twice as slow [Likely].

Other fixes, from cheap to expensive:

  • Ask shorter, closed questions. Long free descriptions drift away from the image; ask “Is there X?” or “List only objects you can see” when you can [Likely].
  • Better visual evidence. Higher resolution and a stronger encoder raise \(e\) (sections below).
  • Preference tuning. Train on pairs where the rejected answer has a hallucinated object, so the model learns to push down its own prior-driven words: see session 190 (mDPO, RLHF-V) and the lab session 195.
  • Better instruction data. Include questions whose correct answer is “no”, and avoid answers that mention things not in the image.

Counting and spatial relations

“How many people are in the photo?” and “Is the mug to the left of the laptop?” look easy. VLMs are often bad at both. There are three causes, and they stack.

Cause 1: the encoder was trained to know “what”, not “how many” or “where”. Most VLMs use a CLIP- or SigLIP-style encoder (see session 166). Contrastive training only asks: does this image match this caption better than the other captions in the batch? Captions rarely depend on exact counts or on left versus right, so the encoder gets little reward for keeping them. Yuksekgonul et al. showed that such models often behave like a bag of words: they are poor at relations and at matching attributes to the right object, and barely notice word order [Certain]. They built the ARO benchmark (Attribution, Relation, Order) with more than 50,000 test cases [Certain]. Kamath et al. built What’sUp, photos that differ only in the spatial relation between two objects; all 18 models they tested performed poorly, far below humans [Certain].

Cause 2: the language model cannot recover what the encoder dropped. Tong et al. (“Eyes Wide Shut”) searched for CLIP-blind pairs: two images that CLIP sees as almost the same (cosine similarity above 0.95) but a self-supervised model, DINOv2, sees as different (below 0.6) [Certain]. From these pairs they built MMVP: 150 image pairs, 300 questions, scored per pair (both answers must be right, so random guessing gets \(0.5 \times 0.5 = 25\%\)). Humans scored 95.7%, GPT-4V 38.7% and LLaVA-1.5 24.7% [Certain]. “Quantity and count” and “positional and relational context” are two of their nine failure patterns [Certain]. Their fix, Mixture of Features, adds DINOv2 features next to CLIP features [Certain].

Cause 3: a coarse token grid, and many objects. A 336-pixel image with 14-pixel patches becomes a 24 × 24 grid of 576 tokens. Connectors often pool or merge these further (see session 178). Small objects that fall into the same token become one blob. And even if each object is seen with high probability, the chance to get the whole count right falls fast with the number of objects. If each of \(n\) objects is found independently with probability \(p\) and nothing is counted twice:

\[ P(\text{exact count}) = p^{\,n} \tag{193.8} \]

With \(p = 0.97\), the count is exactly right with probability 0.859 for 5 objects, 0.737 for 10 and 0.544 for 20. This is why counting errors grow with crowd size even for good models.

The token grid adds a second effect. Suppose, as a toy model, that the model can tell whether a cell of the grid contains the object, but not how many copies are inside one cell. With \(M\) cells and \(n\) objects placed at random, the expected number of occupied cells (the classic occupancy formula) is:

\[ \E[\text{occupied cells}] = M \Big( 1 - \big(1 - \tfrac{1}{M}\big)^{n} \Big) \tag{193.9} \]
Read the symbols
\(M\)
the number of cells that the model can tell apart, for example \(6 \times 6 = 36\) after strong pooling, or \(24 \times 24 = 576\) patch tokens
\(n\)
the true number of objects
\(\big(1 - \tfrac{1}{M}\big)^{n}\)
the chance that one given cell gets none of the \(n\) objects

With 36 cells and 12 objects, the expected count is 10.33: the toy model undercounts by almost 2. With 576 cells and the same 12 objects it is 11.89. Real models are not this simple, but the direction is right: coarse grids and crowded scenes lead to undercounting. Try it in Figure 4.

Figure 4. A toy model of counting on a token grid. Each dot is an object; the model sees only which cells are occupied. Orange cells hold two or more objects and are counted once. The right panel compares the true count, this layout’s count and the expectation (193.9).
How to read Figure 4

At the defaults (12 objects, 6 × 6 cells) the expected toy count is 10.33. Press “New layout” a few times: the count of this layout moves around that value. Now raise the grid to 24 cells per side (the 576-token grid of a 336-pixel image): merges become rare and the count is nearly exact. Then push \(n\) to 40 with a 6 × 6 grid: the toy count saturates near the number of cells. This is why high resolution, less pooling and point-then-count prompts help counting.

Fixing counting and spatial errors

  • Count by pointing or detecting. Ask the model to output one point or box per object first, then count the list. Molmo was trained with a large 2D pointing dataset (PixMo) [Certain] and counts by pointing [Likely]. Qwen2.5-VL reports 93.6 on CountBench (72B model) with a “detect then count” style prompt [Certain]. Each object then gets its own output, so errors no longer hide inside one blob.
  • Use a detector as a tool. For production counting (shelves, parking lots), an open-vocabulary detector (session 174) plus code is usually more reliable than asking the VLM for a number [Likely].
  • Better features. Add self-supervised features (DINOv2, see session 165) next to CLIP (MMVP’s Mixture of Features), or use a less pooled, higher-resolution feature map.
  • Hard negatives and spatial data. ARO showed that contrastive training with composition-aware hard negatives (for example, captions with swapped words) improves order and composition tasks [Certain]. For the VLM itself, add instruction data with counts, left/right and boxes.
  • Absolute coordinates. Train and ask for boxes in pixel coordinates of the real image; Qwen2.5-VL represents boxes and points in the actual image size, not normalized values [Certain].

Cambrian-1 (Tong et al., 2024) made these skills measurable. It introduced CV-Bench, a vision-centric benchmark of 2,638 manually checked examples, with 2D tasks (spatial relationship: 650, object count: 788, from ADE20K and COCO) and 3D tasks (depth order and relative distance: 600 each, from Omni3D) [Certain]. It also warned that several popular benchmarks (for example MMMU, MathVista and AI2D) show less than 5% difference between vision enabled and disabled: a model can do well on them mostly with its language prior [Certain]. Always include a vision-centric test when you study these failures.

Small text and fine details lost by resizing

Many VLMs resize every image to a fixed square: 224 or 336 pixels for classic CLIP encoders (LLaVA-1.5 uses 336 × 336, so 576 tokens) [Certain]. A phone photo is 3024 × 4032 pixels. If we keep the aspect ratio and fit the long side to the model input of size \(S\), every length in the image shrinks by the same factor:

\[ h' = h \cdot \min\Big( \frac{S_W}{W}, \frac{S_H}{H} \Big), \qquad \text{text height in patches} = \frac{h'}{P} \tag{193.10} \]
Read the symbols
\(W, H\)
the original width and height of the image in pixels
\(S_W, S_H\)
the width and height the encoder receives (336 and 336 for LLaVA-1.5)
\(h\)
the height of a line of text (or of a small object) in the original image
\(h'\)
its height after resizing
\(P\)
the patch size of the ViT, usually 14 pixels; one token sees one \(P \times P\) patch
Worked example: a receipt photo

A receipt is photographed at 3024 × 4032. The line with the total is 40 pixels tall. Fit to 336: the factor is \(336 / 4032 = 0.0833\), so the text becomes \(40 \times 0.0833 = 3.33\) pixels tall, about a quarter of one 14-pixel patch (0.24). No model can read digits that are 3 pixels tall. At 224 it is 2.22 pixels, and with a 672-pixel canvas (2 × 2 tiles of 336) it is 6.67 pixels.

Now use native resolution as in Qwen2-VL and Qwen2.5-VL, where each visual token covers 28 × 28 pixels after a 2 × 2 merge of 14-pixel patches. The default processor limit is max_pixels = 12845056 \(= 16384 \times 28 \times 28\) [Certain], and \(3024 \times 4032 = 12{,}192{,}768\) is below it. So the photo is kept as it is: the text stays 40 pixels tall, at the cost of \(108 \times 144 = 15{,}552\) visual tokens. With a cap of 1280 tokens (max_pixels = 1280 × 28 × 28, a common setting [Likely]), the processor resizes the photo to 840 × 1148, which is 1,230 tokens, and the text becomes 11.39 pixels tall.

This is the core trade-off of session 180: legibility costs tokens, and tokens cost latency and memory (session 183). Squashing to a square without keeping the aspect ratio is even worse for text: it also distorts the letter shapes. Figure 5 lets you see the effect.

Figure 5. What resizing does to a line of text. The string is drawn at its size after resizing (193.10) and then blown up with no smoothing, so each square is one pixel the encoder actually gets. Token counts: 256 at 224, 576 at 336, \(5 \times 576 = 2{,}880\) for 2 × 2 tiles plus a thumbnail, and Qwen2-VL’s rule for native resolution.
How to read Figure 5

Start at the receipt and 336: the total is 3.33 pixels tall and unreadable. Click “2 × 2 tiles”: it doubles to 6.67 pixels and starts to be readable, for 5 times the tokens. Click “native, ≤ 1280 tokens”: 11.39 pixels with 1,230 tokens, a better trade than tiles here. Switch to the screenshot with 12-pixel text: at 336 it shrinks to 2.10 pixels, while native resolution with the default limit keeps it at about 12 pixels (2,691 tokens). The “readable?” label uses a rough rule for this demo (below 6 pixels: unreadable; 6 to 10: risky), not a measured threshold.

Fixing small text and small objects

  • Tiling. Split the image into tiles at the encoder’s size, plus a low-resolution thumbnail for the global view (LLaVA-NeXT, InternVL). Cambrian-1 lists 2,880 visual tokens for LLaVA-NeXT [Certain].
  • Native resolution. Patchify the image at (almost) its real size, with 2-D positions (Qwen2-VL, Qwen2.5-VL). Set max_pixels from the smallest text you must read, using (193.10) backwards.
  • Better connectors for high resolution. Cambrian-1 found that high-resolution encoders greatly improve chart and vision-centric tasks, and that ConvNet encoders handle high resolution well [Certain]. Its Spatial Vision Aggregator (SVA) uses a grid of learnable queries, where each query attends only to its own sub-region of the encoder features; the final models use 576 visual tokens [Certain].
  • Zoom in on demand. V* (Wu and Xie) lets the LLM search the image for the region it needs and look again at high resolution, for crowded high-resolution images [Certain]. A simple version is a crop tool: the model names a box, the system crops it at full resolution and feeds it back (see session 154).
  • OCR as a tool. For documents, run a dedicated OCR model and give its text (with positions) to the VLM. This is often cheaper than very high resolution [Likely].

Temporal confusion in video

A Video-LLM (see session 205) usually samples a fixed number of frames \(N\), encodes each frame like an image, and puts all the visual tokens into the language model. A 10-minute video at 30 frames per second has 18,000 frames; the model may see 32. Three things go wrong.

Short events fall between frames. With uniform sampling the gap between frames is \(T/N\). An event of length \(d\) is seen in at least one frame with probability (assuming a random start of the sampling grid):

\[ P(\text{event seen}) = \min\Big( 1, \frac{d\,N}{T} \Big) \tag{193.11} \]
Read the symbols
\(T\)
video length in seconds
\(N\)
number of sampled frames; \(T/N\) is the stride, the time between two frames
\(d\)
duration of the event, for example a 3-second goal or a fall

For a 600-second video with 32 frames, the stride is 18.75 seconds and a 3-second event is seen with probability \(3 \times 32 / 600 = 0.16\). With 64 frames it is 0.32. A model that “misses” the event never had a chance. Worse, the language prior fills the gap: it answers with what usually happens in such a video.

Order is lost. Many encoders process each frame on its own. If the frame tokens carry weak or no time positions, the model sees a bag of frames and cannot tell “sat down, then stood up” from the reverse. TempCompass (Liu et al., 2024) tested this with “conflicting videos” that share the same static content but differ in one temporal aspect such as direction or speed, so that single-frame cues and language priors do not help; the 8 Video LLMs and 3 image LLMs they tested showed notably poor temporal perception [Certain].

Time is in frames, not seconds. If position ids count frames, “frame 10” means 5 seconds at 2 fps and 50 seconds at 0.2 fps. The model cannot answer “when?” or compare speeds across videos sampled differently. Qwen2.5-VL aligns the temporal part of its M-RoPE positions with absolute time, so the gaps between temporal ids reflect real time, and it samples the frame rate dynamically during training [Certain]. It also groups two consecutive frames into one temporal step to save tokens [Certain], which means one temporal token already spans two frames.

Figure 6. Uniform frame sampling on a video timeline. Ticks are sampled frames; the boxes A to E are events of length \(d\) at fixed places in the video. A filled box was hit by at least one frame; an empty box is invisible to the model.
How to read Figure 6

At the defaults (600 s, 32 frames, 3 s events) the stride is 18.75 s and each event is seen with probability 0.16 by (193.11); here all five events are missed. Raise \(N\) to 128: the stride is 4.69 s and the chance is 0.64. Now raise \(d\) to 20 s: every event is caught even with 32 frames. The lesson: the frame budget must be chosen from the shortest event you care about, \(N \ge T/d\), or you must pick frames smartly (key frames, scene cuts, or retrieval).

Fixing temporal confusion

  • More frames, fewer tokens per frame. Raise \(N\) and compress each frame (pooling, token merging; see session 183). Qwen2.5-VL was evaluated with up to 768 frames and at most 24,576 video tokens [Certain].
  • Smart frame selection. Sample at scene cuts, at motion peaks, or retrieve frames that match the question with a CLIP-style score (see session 206).
  • Time in the input. Use absolute-time position ids (Qwen2.5-VL) or write timestamps as text next to each frame, so “when” questions have an anchor [Likely].
  • Temporal training data. Include questions that need order, direction and speed, and conflicting videos (reversed or reordered clips with flipped answers) so single-frame shortcuts stop working.
  • Test the shortcut. Ask the same question with the frames shuffled or reversed. If the answer does not change on an order question, the model is using a bag of frames.

A diagnosis protocol

When a VLM fails, first find which failure you have. Each test removes or changes one input and watches the answer. The table maps symptoms to tests and fixes.

FailureTypical symptomTest that confirms the causeFix that targets it
Object hallucinationNames typical but absent objects; yes ratio far above 50%Blind test (no image or a blank image): the same answer means the prior decides. POPE adversarial vs randomVCD, preference tuning with hallucinated negatives, “no” questions in data
CountingUndercounts crowds; right for 2–3 objects, wrong for 10+Accuracy vs true count; same image at higher resolution; ask for points firstPoint/detect then count, detector tool, less pooling
Spatial relationsLeft/right and above/below at chanceMirror the image horizontally: the answer should flip; CV-Bench, What’sUpSpatial instruction data, absolute boxes, SSL features
Small text / detailsWrong digits, “I cannot read it”, guesses typical valuesCrop the region and ask again at full resolution; log the encoder input sizeTiling, native resolution, zoom-in tool, OCR tool
Temporal confusionMisses short events; wrong order; answers from one frameShuffle or reverse frames; double \(N\); compare with a single middle frameMore frames with compression, smart sampling, absolute time ids
Algorithm 1: find the failure family of one wrong answer
  1. input: model \(f\), image or video \(I\), question \(x\), wrong answer \(\hat y = f(I, x)\)
  2. \(y_0 \leftarrow f(\varnothing, x)\) // blind test: no image, or a blank gray image
  3. if \(y_0 = \hat y\) then report “language prior” and stop // the image did not change the answer
  4. \(y_c \leftarrow f(\text{crop}(I, \text{region}), x)\) at full resolution // zoom on the relevant region
  5. if \(y_c\) is correct then report “resolution” and stop
  6. if \(x\) asks for a count then ask for one point per object, count the points // pointing splits the blob
  7. if the point count is correct then report “counting by direct read-out” and stop
  8. if \(x\) asks about left/right then \(y_m \leftarrow f(\text{mirror}(I), x)\); if \(y_m = \hat y\) then report “spatial relation ignored”
  9. if \(I\) is a video then
  10. \(y_s \leftarrow f(\text{shuffle}(I), x)\); if \(x\) needs order and \(y_s = \hat y\) then report “bag of frames”
  11. \(y_{2N} \leftarrow f(I \text{ with } 2N \text{ frames}, x)\); if \(y_{2N}\) is correct then report “sparse sampling”
  12. else report “encoder cannot see it” // try a stronger encoder or SSL features

Run this on a few hundred failures, not one. The share of each family tells you where to spend effort. The second algorithm is the decoding fix for the first family.

Algorithm 2: one decoding step of visual contrastive decoding (VCD)
  1. input: image \(v\), prompt and answer so far \(x, y_{\lt t}\), strength \(\alpha\), plausibility \(\beta\), noise steps
  2. \(v' \leftarrow \operatorname{noise}(v)\) // once per image, reused for every step
  3. \(\mathbf{u} \leftarrow \text{model}(v, x, y_{\lt t})\), \(\;\mathbf{u}' \leftarrow \text{model}(v', x, y_{\lt t})\) // two passes, two KV caches
  4. \(\mathbf{p} \leftarrow \softmax(\mathbf{u})\)
  5. \(\mathcal{V}_{\text{head}} \leftarrow \{y : p_y \ge \beta \max_w p_w\}\) // (193.6)
  6. \(\mathbf{c} \leftarrow (1 + \alpha)\mathbf{u} - \alpha \mathbf{u}'\); set \(c_y = -\infty\) for \(y \notin \mathcal{V}_{\text{head}}\) // (193.5)
  7. \(y_t \leftarrow\) sample from \(\softmax(\mathbf{c})\) (or take the argmax)
  8. return \(y_t\)

Here \(\mathbf{u}\) and \(\mathbf{u}'\) are the two logit vectors \(\ell_\theta(\cdot \mid v, x)\) and \(\ell_\theta(\cdot \mid v', x)\) of (193.5). Each failure family also has a standard benchmark: POPE and CHAIR for hallucination, CV-Bench and MMVP for counting and spatial skills, text-heavy sets such as TextVQA and DocVQA for small text, and Video-MME or TempCompass for time (see session 192).

The code: POPE metrics, VCD, resizing and frame sampling

The code below implements the four measurements of this session in plain PyTorch, with no model download. It computes POPE metrics (193.4), one VCD step (193.5–193.6), the effective text height under fixed and native resizing (193.10), and checks the frame hit formula (193.11) by simulation. It is deterministic.

import math, torch

torch.manual_seed(0)

# 1) POPE metrics from yes/no answers (1 = "yes")
def pope_metrics(pred, gold):
    tp = ((pred == 1) & (gold == 1)).sum().item()
    fp = ((pred == 1) & (gold == 0)).sum().item()
    fn = ((pred == 0) & (gold == 1)).sum().item()
    acc = (pred == gold).float().mean().item()
    prec = tp / max(tp + fp, 1)
    rec = tp / max(tp + fn, 1)
    f1 = 2 * prec * rec / max(prec + rec, 1e-9)
    return dict(acc=round(acc, 4), prec=round(prec, 4), rec=round(rec, 4),
                f1=round(f1, 4), yes=round(pred.float().mean().item(), 4))

gold = torch.tensor([1, 1, 1, 0, 0, 0] * 500)          # 500 images x 6 questions
always_yes = torch.ones_like(gold)
print("always yes:", pope_metrics(always_yes, gold))
# a model that sees real objects 90% of the time and says "yes" to 30% of absent ones
g = torch.Generator().manual_seed(0)
u = torch.rand(gold.shape, generator=g)
pred = torch.where(gold == 1, (u < 0.9).long(), (u < 0.3).long())
print("biased model:", pope_metrics(pred, gold))

# 2) Visual contrastive decoding (VCD) for one next-token step
def vcd_logits(logit_img, logit_noisy, alpha=1.0, beta=0.1):
    """(1+alpha)*l(y|v,x) - alpha*l(y|v',x), restricted to plausible tokens."""
    p = logit_img.softmax(-1)
    keep = p >= beta * p.max(-1, keepdim=True).values      # head set V_head
    out = (1 + alpha) * logit_img - alpha * logit_noisy
    return out.masked_fill(~keep, float("-inf"))

vocab = ["yes", "no", "maybe", "a"]
l_img   = torch.tensor([0.5, 0.0, -1.5, -3.0])   # with the real image
l_noisy = torch.tensor([2.0, 0.0, -1.5, -3.0])   # with a noised image: prior only
for name, l in [("regular", l_img), ("vcd", vcd_logits(l_img, l_noisy))]:
    pr = l.softmax(-1)
    print(name, {w: round(pr[i].item(), 3) for i, w in enumerate(vocab)})

# 3) Effective text height after resizing (aspect ratio kept)
def smart_resize(h, w, factor=28, min_pixels=56 * 56, max_pixels=1280 * 28 * 28):
    hb, wb = round(h / factor) * factor, round(w / factor) * factor
    if hb * wb > max_pixels:
        b = math.sqrt(h * w / max_pixels)
        hb, wb = math.floor(h / b / factor) * factor, math.floor(w / b / factor) * factor
    elif hb * wb < min_pixels:
        b = math.sqrt(min_pixels / (h * w))
        hb, wb = math.ceil(h * b / factor) * factor, math.ceil(w * b / factor) * factor
    return hb, wb

H, W, text_h = 4032, 3024, 40                     # phone photo of a receipt
for side in (224, 336, 672):
    s = min(side / H, side / W)
    print(f"fixed {side}: text {text_h * s:.2f} px")
hb, wb = smart_resize(H, W)
print("native 1280-token cap:", (hb, wb), "tokens", (hb // 28) * (wb // 28),
      f"text {text_h * hb / H:.2f} px")

# 4) Uniform frame sampling: chance to see a short event
def hit_rate(T, N, d, trials=20000):
    g = torch.Generator().manual_seed(1)
    stride = T / N
    offset = torch.rand(trials, generator=g) * stride           # random phase
    start = torch.rand(trials, generator=g) * (T - d)           # event start
    t = offset[:, None] + stride * torch.arange(N)[None, :]     # frame times
    seen = ((t >= start[:, None]) & (t <= start[:, None] + d)).any(1)
    return seen.float().mean().item()

for N in (8, 32, 64):
    print(f"T=600 s, N={N}: simulated {hit_rate(600, N, 3):.3f}, formula {min(1, 3 * N / 600):.3f}")

Output (the same on every run):

always yes: {'acc': 0.5, 'prec': 0.5, 'rec': 1.0, 'f1': 0.6667, 'yes': 1.0}
biased model: {'acc': 0.7953, 'prec': 0.7445, 'rec': 0.8993, 'f1': 0.8146, 'yes': 0.604}
regular {'yes': 0.564, 'no': 0.342, 'maybe': 0.076, 'a': 0.017}
vcd {'yes': 0.231, 'no': 0.629, 'maybe': 0.14, 'a': 0.0}
fixed 224: text 2.22 px
fixed 336: text 3.33 px
fixed 672: text 6.67 px
native 1280-token cap: (1148, 840) tokens 1230 text 11.39 px
T=600 s, N=8: simulated 0.038, formula 0.040
T=600 s, N=32: simulated 0.160, formula 0.160
T=600 s, N=64: simulated 0.322, formula 0.320

Read the output. “Always yes” gets F1 0.667, as computed above. In the VCD step, “yes” was the top token (0.564) and becomes 0.231, while “no” rises to 0.629; the ratio of “yes” to “no” is \(e^{-1}\), the same log-odds of \(-1.0\) as in the worked example. The token “a” had probability 0.017, below \(\beta \times 0.564 = 0.056\), so the plausibility constraint removes it. The resize lines repeat the receipt example. The simulated hit rates match (193.11) up to sampling noise. To use VCD with a real model, compute the two logit vectors with two forward passes (one per image) and call vcd_logits at each step.

Where this is used

ApplicationMain failure riskWhat teams do
Image captioning and alt textObject hallucination, especially in long captionsShort captions, CHAIR / POPE in evaluation, preference tuning, VCD-style decoding [Likely]
Retail shelf audit, crowd and inventory countingUndercounting crowded scenesDetector or point-then-count, high resolution, count by tiles [Likely]
Invoices, receipts, forms, screenshots (Document AI)Small text lost by resizingNative resolution with a chosen max_pixels, tiling, OCR as a tool
Robotics and UI agentsSpatial relations, exact positionsAbsolute coordinates, pointing, crop-and-zoom loops
Video key-moment detection and temporal groundingShort events between framesDense sampling with token compression, shot-cut sampling, absolute time ids
Video action recognition and video QABag-of-frames answers, single-frame shortcutsReverse/shuffle tests, temporal training data (TempCompass-style)
Video-RAG and text-to-video retrievalThe retrieved clip misses the moment; the answer is filled in by the priorRetrieve frames by question, cite timestamps, check answers against frames [Speculation]
Content moderationHallucinated objects lead to false flagsAsk closed questions per policy item, check yes ratio on clean data

Common mistakes

  • Reporting only F1 or accuracy on POPE. A model can win by changing its yes ratio. Always report the yes ratio and FP, and test the adversarial setting.
  • Evaluating on benchmarks that do not need the image. Cambrian-1 found less than 5% gap with and without vision on several popular sets. Run a blind baseline on every benchmark you use.
  • Blaming the language model for what the resize destroyed. Before you fine-tune, check the real encoder input size. Many “OCR failures” are 3-pixel text.
  • Tiling without a thumbnail. Tiles cut objects and text lines at tile borders; the global view is needed to put them back together.
  • Asking for a number when you can ask for a list. “How many?” forces one read-out. “Point to each X” gives one output per object and can be checked.
  • Choosing the frame count from the token budget only. Choose it from the shortest event you must catch, \(N \ge T/d\), then compress tokens to fit the budget.
  • Forgetting the time unit. Training at 2 fps and serving at 0.5 fps changes what “frame 10” means if positions count frames.
  • Using VCD without the plausibility constraint. The contrast alone can push up rare junk tokens; keep \(\beta\) around 0.1.

Interview questions

Why do VLMs hallucinate objects?basic
By Bayes’ rule (193.1), \(\log p(y \mid I, x)\) is the language prior \(\log p(y \mid x)\) plus a visual-evidence term. The language model’s prior is strong, learned from trillions of tokens; the visual term comes from a few hundred visual tokens learned on much less data. When the evidence is weak (small, hidden or blurry object, low resolution, long answer far from the image), the prior wins and the model names objects that usually co-occur with the scene. Instruction data written by text-only LLMs and long detailed answers make it worse.
Explain POPE and why the adversarial setting matters.core
POPE asks “Is there a <object> in the image?” with half real and half absent objects (COCO: 500 images, 6 questions each, 3,000 per setting). Absent objects are sampled at random, from the most popular objects, or from the objects that co-occur most with the ones present (adversarial). Adversarial questions attack the language prior directly: InstructBLIP’s F1 fell from 89.29 (random) to 78.45 (adversarial) while recall stayed near 93%, so the drop is almost all extra false “yes” answers. Report accuracy, precision, recall, F1 and the yes ratio.
A model scores F1 = 67 on POPE. Is that good?core
No. A model that always says “yes” has precision 0.5, recall 1 and F1 \(= 2 \cdot 0.5 / 1.5 = 0.667\). Check the yes ratio: early LLaVA, mPLUG-Owl and MultiModal-GPT answered “yes” 95–100% of the time and scored F1 66.7–68.7, so they were barely using the image. A good model has a yes ratio near 50% on balanced POPE and high precision.
How does visual contrastive decoding work, and why does it reduce hallucination?advanced
At each step compute logits with the real image and with a noised image, and decode from \(\softmax[(1+\alpha)\ell(y \mid v, x) - \alpha\ell(y \mid v', x)]\), restricted to tokens with probability at least \(\beta\) times the top token (default \(\alpha = 1, \beta = 0.1\)). The noised image keeps the language prior but loses visual evidence. In the log-odds model the result is \(a + (1 + \alpha - \alpha s)we\) (193.7): the prior is unchanged and the visual term is amplified. It needs no training but doubles the decoding compute. On LLaVA-1.5, POPE F1 (random) went from 81.33 to 87.16.
Why are VLMs bad at counting?core
(1) Contrastive encoders are trained to match captions, which rarely depend on exact counts, so count information is weak in the features (bag-of-words behaviour, MMVP “quantity” pattern). (2) The token grid is coarse, and connectors pool it further, so nearby small objects merge (toy occupancy model 193.9: 12 objects on 36 cells give 10.33 expected cells). (3) Exact counts need every object to be found: \(p^n\) falls fast (0.97 per object gives 0.544 for 20 objects). Fixes: point or detect each object then count, a detector tool, higher resolution and less pooling.
What are CLIP-blind pairs and what do they tell us about VLMs?advanced
Pairs of images that CLIP embeds almost identically (cosine above 0.95) while DINOv2 sees them as different (below 0.6). Tong et al. built MMVP from 150 such pairs (300 questions, scored per pair, random = 25%): humans got 95.7%, GPT-4V 38.7%, LLaVA-1.5 24.7%. Errors cluster in patterns such as orientation, counts, position and text. Since the language model only sees what the encoder gives, these failures cannot be fixed by the LLM; the fix is better visual features, such as adding self-supervised DINOv2 features (Mixture of Features).
Our invoice model gets digits wrong. What do you check first?basic
The size of the image the encoder actually receives. By (193.10), a 40-pixel line on a 3024 × 4032 photo becomes 3.33 pixels at 336. Crop the total and ask again at full resolution: if that is right, it is a resolution problem. Fixes: native resolution with a larger max_pixels (1280 tokens gives 11.39 px here), tiling with a thumbnail, a zoom/crop tool, or OCR as a tool. Also check that serving uses the same resolution settings as fine-tuning.
How do you pick the number of frames for a video task?core
From the shortest event you must catch. With uniform sampling the hit probability is \(\min(1, dN/T)\); a 3-second event in a 600-second video is seen with probability 0.16 at 32 frames. To be sure, \(N \ge T/d\) (200 frames here). If that is too many tokens, compress each frame (pooling, merging) or select frames smartly (shot cuts, motion, question-based retrieval). Add absolute timestamps so the model can answer “when”.
How can you tell if a Video-LLM uses temporal order at all?advanced
Ask order questions (“what happened first?”, “is the door opening or closing?”) on the original clip and on the reversed or shuffled clip; the correct answer should flip or change. If the model’s answer stays the same, it uses a bag of frames or single-frame cues. Also compare with a single middle frame: if accuracy is the same, the benchmark or the model does not need time. TempCompass builds conflicting videos for exactly this purpose. Fixes: time-aware positions (Qwen2.5-VL aligns M-RoPE temporal ids with absolute time), temporal training data with reversed clips.
Design an evaluation suite that separates the four failure families.advanced
Hallucination: POPE (all three settings, with yes ratio) plus CHAIR on free captions, and a blind baseline. Counting and spatial: CV-Bench (count, relation, depth, distance), MMVP, and a mirror test for left/right. Small text: a text-heavy set (TextVQA, DocVQA, charts) evaluated at two resolutions, so the gain from resolution is visible. Video: Video-MME plus TempCompass-style reversed or conflicting clips, and accuracy vs number of frames. For each, report the result with and without the image to see how much the prior explains (Cambrian-1 found less than 5% gap on some popular benchmarks).

Exercises

  1. In the log-odds model (193.2), the prior is \(a = 1.0\) and the visual term is \(we = -2.0\). What is \(p(\text{yes})\)? Now apply VCD with \(\alpha = 0.5\) and \(s = 0\). What is \(p(\text{yes})\) now?
  2. On balanced POPE (1,500 real, 1,500 absent), a model says “yes” to 1,350 real and 450 absent objects. Compute precision, recall, accuracy, F1 and the yes ratio.
  3. A model finds each object with probability 0.98. What is the chance of an exact count for 15 objects? For 30?
  4. A 1920 × 1080 screenshot has 12-pixel text. How tall is the text after fitting to 336 (aspect ratio kept)? To 448? How large must the long side be for the text to stay at least 8 pixels tall?
  5. A 90-second clip is sampled at 16 frames. What is the chance to see a 1.5-second event? How many frames do you need to be sure to see it?
  6. On MMVP, a model answers each question right with probability 0.7, independently. What MMVP score do you expect? Why is random guessing 25%?
Answerscheck
1. \(z = 1.0 - 2.0 = -1.0\), \(p = \sigma(-1) = 0.269\): the model correctly says “no”. With VCD: \(z = 1.0 + (1 + 0.5) \times (-2.0) = -2.0\), \(p = \sigma(-2) = 0.119\): more confident “no”. 2. \(TP = 1350\), \(FP = 450\), \(FN = 150\). Precision \(1350/1800 = 0.75\), recall \(1350/1500 = 0.9\), accuracy \((1350 + 1050)/3000 = 0.8\), \(F_1 = 2 \times 0.75 \times 0.9 / 1.65 = 0.818\), yes ratio \(1800/3000 = 0.6\). 3. \(0.98^{15} = 0.739\); \(0.98^{30} = 0.545\). 4. Factor \(336/1920 = 0.175\): text 2.10 px. Factor \(448/1920 = 0.233\): text 2.80 px. Need factor \(8/12 = 0.667\), so the long side must be at least \(0.667 \times 1920 = 1280\) px. 5. Stride \(90/16 = 5.625\) s; \(P = 1.5/5.625 = 0.267\). To be sure: \(N \ge 90/1.5 = 60\) frames. 6. \(0.7^2 = 0.49\), so 49%. A pair counts only if both questions are right; random guessing gets each yes/no-style question right half the time, so \(0.5^2 = 0.25\).

Key takeaways

  • Most VLM errors fall into four families with four different causes: the language prior (hallucination), the encoder (counting and spatial), resizing (small text), and frame sampling (video time).
  • Bayes’ rule (193.1) splits an answer into prior plus visual evidence; when evidence is weak, co-occurring objects get hallucinated. POPE’s adversarial setting and the yes ratio expose this; “always yes” gets F1 0.667.
  • VCD (193.5) subtracts the logits of a noised image; in the log-odds view the prior stays and the visual term is multiplied by \(1 + \alpha(1 - s)\) (193.7). LLaVA-1.5 POPE F1 (random) rose from 81.33 to 87.16.
  • Counting fails because contrastive encoders keep “what” not “how many”, grids are coarse (193.9), and exact counts need every object (\(p^n\)). Point or detect, then count.
  • Resizing shrinks text by \(\min(S_W/W, S_H/H)\) (193.10): 40 px becomes 3.33 px at 336. Tiling, native resolution with a chosen max_pixels, and zoom-in tools fix it at a token cost.
  • Video models miss events with probability \(1 - \min(1, dN/T)\) (193.11) and lose order when frames are a bag. Choose \(N\) from the shortest event, add absolute time, and test with reversed clips.
  • Diagnose before fixing: blind, crop, mirror, shuffle and more-frames tests (Algorithm 1) tell you which family you have.

Key formulas

\[ \log p(y \mid I, x) = \log p(y \mid x) + \log \frac{p(I \mid y, x)}{p(I \mid x)}, \qquad z_{\text{vcd}} = a + (1 + \alpha - \alpha s)\, w e \]
\[ p_{\text{vcd}} = \softmax\big[(1+\alpha)\ell(y \mid v, x) - \alpha\,\ell(y \mid v', x)\big], \qquad F_1 = \frac{2\,\text{Prec}\,\text{Rec}}{\text{Prec} + \text{Rec}} \]
\[ P(\text{exact}) = p^n, \quad \E[\text{cells}] = M\big(1 - (1 - \tfrac1M)^n\big), \quad h' = h \min\Big(\frac{S_W}{W}, \frac{S_H}{H}\Big), \quad P(\text{seen}) = \min\Big(1, \frac{dN}{T}\Big) \]

References for this session

  1. Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models
  2. Tong et al. (2024). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
  5. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  6. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 11: Advanced Topics and Cutting-Edge Research
  7. Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models
  8. Tong et al. (2024). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
  9. Rohrbach, Hendricks, Burns, Darrell & Saenko (2018). Object Hallucination in Image Captioning
  10. Leng, Zhang, Chen, Li, Lu, Miao & Bing (2023). Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
  11. Tong, Liu, Zhai, Ma, LeCun & Xie (2024). Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
  12. Yuksekgonul, Bianchi, Kalluri, Jurafsky & Zou (2022). When and why vision-language models behave like bags-of-words, and what to do about it?
  13. Kamath, Hessel & Chang (2023). What's "up" with vision-language models? Investigating their struggle with spatial reasoning
  14. Wu & Xie (2023). V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
  15. Deitke, Clark et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
  16. Liu, Li, Liu, Wang, Ren, Li, Chen, Sun & Hou (2024). TempCompass: Do Video LLMs Really Understand Videos?
  17. Bai et al. (2025). Qwen2.5-VL Technical Report
  18. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  19. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning