LLM & VLM courseBlog
192Track 7: Vision and VLMs · session 30 of 38

Evaluating VLMs

A VLM answers in free text, and every score you see is the result of a pipeline: the prompt, the image settings, how the answer is parsed, the metric and the test set. This session explains the metrics behind the numbers on model cards (VQA accuracy, ANLS, CIDEr, POPE), the main benchmarks for images, documents, hallucination and video, how to build a test set from your own data, and when a judge model can score open answers.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on evaluating base models (40), metrics for text generation (149), evaluation of LLM systems (155) and VLM tasks (191)

What you will learn

  • The evaluation pipeline (prompt, model settings, parsing, metric, aggregation) and where each step changes the score
  • VQA accuracy with 10 human answers: the min(1, k/3) rule, the leave-one-out average, and answer normalization
  • ANLS for documents: normalized edit distance with the 0.5 threshold, and why near-miss numbers need exact checks
  • CIDEr and CIDEr-D for captions: TF-IDF n-grams, clipping, the length penalty, and why scores depend on the test set
  • MMMU, DocVQA, VQAv2, POPE, MVBench and Video-MME: what each tests, its size and metric; POPE's yes ratio and blind baselines (MMStar MG and ML)
  • A domain test set from your own data: stratified sampling, labeling guidelines, Cohen's kappa, sample size and paired tests (McNemar)
  • Judge-based scoring for open answers: setups, position and verbosity bias, the image problem for judges, and calibration against humans

Why this matters

A vision-language model (VLM) answers in free text. To say “model A is better than model B” you must turn that free text into a number. Every step of that conversion (the prompt, how you read the answer, the metric, the test set) can change the number by several points.

Model cards list scores such as MMMU, DocVQA, POPE and Video-MME. Teams choose models, and sometimes ship them, because of a two-point gap in such a table. If you do not know how each number is computed, you cannot tell a real gain from a change in the answer parser. And public benchmarks rarely look like your own data: a model that is best on MMMU (college exam questions) may be worse at reading your shipping labels.

In session 191 we saw the main tasks (captioning, VQA, documents, charts, grounding, video) and the prompt formats for each. This session shows how each task is scored. We go step by step:

  1. The evaluation pipeline, and where it can go wrong.
  2. Three metrics in depth: VQA accuracy (short answers with many human labels), ANLS (text read from documents), CIDEr (captions).
  3. The main benchmarks: VQAv2, DocVQA, MMMU, POPE (hallucination), MVBench and Video-MME (video), and how to check that a benchmark really needs the image.
  4. Your own domain test set: how to sample it, label it, and how big it must be.
  5. Judge-based scoring for open answers, and its limits.

The ideas are the same as for text-only models (session 40, session 149, session 155). What is new is the image: answers can be checked against pixels, models can ignore the pixels, and some answers (dates, totals, names) are read from the image, so small reading errors need their own metric.

The evaluation pipeline

An evaluation runs a fixed set of test items through the model and summarizes the results in one or a few numbers. Each test item has an image (or video), a question, and one or more reference answers. Figure 1 shows the steps.

Test itemimage + question Prompttemplate, options Modelresolution, frames Parseextract, normalize Metricper item score Aggregatemean ± interval leaked intotraining data?needs the image? wording changesthe scoreoption order bias image resized:small text lost8 vs 64 frames “Two.” vs “2”“(B)” vs “B”regex misses exact vs softmatch; judgebias too few items:noise lookslike a gain The same model and benchmark can give different scores in two harnesses: report the prompt, the parser, the image settings and the harness version with every number.
Figure 1. The evaluation pipeline. Above: the six steps from a test item to a final score. Below each step in red: the most common way it goes wrong. Most surprising “gains” come from the parse and aggregate steps, not from the model.

Answers come in a few types, and each type has its scoring rule:

Answer typeExampleHow it is scored
Short open answer, many human labels“How many dogs?” → “2”VQA accuracy after normalization (formula 192.1)
Text read from the image“What is the invoice date?” → “03/12/2021”ANLS, a soft string match (formula 192.4)
Multiple choiceMMMU, MVBench, Video-MME → “(B)”Extract the letter, then exact match; accuracy
Yes / no probePOPE: “Is there a car in the image?”Accuracy, precision, recall, F1, yes ratio (192.8)
Free descriptionCaption, detailed descriptionCIDEr against references (192.7), or a judge model
Box, point, time spanGrounding, key moments in videoIoU above a threshold, distance to a point

VQA accuracy: short answers with ten human labels

In visual question answering (VQA), a model gets an image and a question and gives a short answer. The VQA datasets asked 10 different people to answer each question. People do not always agree: for “What color is the bus?” some say “red”, others “red and white”. So the metric gives partial credit when only some humans gave your answer.

The rule is: an answer is fully correct if at least 3 of the humans gave it. To make the score stable, the official code averages this rule over the 10 ways of leaving one human out:

\[ \mathrm{Acc}(a) = \frac{1}{10} \sum_{i=1}^{10} \min\!\left(1,\ \frac{\#\{\, j \ne i : h_j = a \,\}}{3}\right) \tag{192.1} \]
Read the symbols
\(a\)
the model’s answer, after normalization (below)
\(h_1, \dots, h_{10}\)
the 10 human answers, also normalized
\(\#\{ j \ne i : h_j = a \}\)
how many of the other 9 humans (all except human \(i\)) gave exactly \(a\)
\(\min(1, \cdot / 3)\)
3 or more matches give full credit; 1 match gives 1/3; 2 matches give 2/3

If \(k\) of the 10 humans gave the answer \(a\), the sum has a simple closed form: \(k\) of the left-out humans are matches (then \(k-1\) matches remain), and \(10-k\) are not (then \(k\) remain):

\[ \mathrm{Acc}(k) = \frac{k \cdot \min\!\left(1, \frac{k-1}{3}\right) + (10 - k) \cdot \min\!\left(1, \frac{k}{3}\right)}{10} \tag{192.2} \]

This gives 0.3 for \(k = 1\), 0.6 for \(k = 2\), 0.9 for \(k = 3\), and 1.0 for \(k \ge 4\). Note that two different answers can both get full credit: if 6 people said “yes” and 4 said “no”, both “yes” and “no” score 1.0.

Normalization: the hidden half of the metric

Before comparing strings, the official script normalizes answers: it lower-cases, removes punctuation (but keeps the period in “2.5”), removes the articles “a”, “an”, “the”, turns number words into digits (“two” → “2”, “none” → “0”), and fixes common contractions (“dont” → “don’t”). [Certain] One quirk: the script normalizes only when the 10 human answers are not all the same string. If all 10 humans wrote “2” and the model writes “Two.”, the model’s answer is not normalized and scores 0. Many newer harnesses always normalize, so their numbers differ slightly. [Likely]

Normalization cannot fix long answers. Chat models like to answer in sentences: “There are two dogs in the picture.” That string matches no human answer and scores 0. This is why VQA prompts add “Answer the question using a single word or phrase.” Without it, a strong chat model can look weak.

Question: “How many dogs are in the picture?” The 10 human answers:
Model answer
Figure 2. VQA accuracy for one question. Green chips are the human answers that match the model answer after normalization. Each bar is one term of formula (192.1): the score when human \(i\) is left out. The final score is the mean of the 10 bars.
How to read Figure 2

Pick “Two.”: normalization turns it into “2”, which 7 humans gave (the “two” chip also becomes “2”), so every bar is full and the score is 1.0. Pick “3”: 2 humans gave it, so the bars are 1/3 when a “3” is left out and 2/3 otherwise, mean 0.6, as formula (192.2) says for \(k = 2\). Turn normalization off and “Two.” drops to 0. Then switch on “all 10 humans write 2”: now the official script skips normalization even when the box is checked, and “Two.” scores 0. The long chat answer scores 0 in every case.

Worked example: three model answers to one question

Human answers: “2” ×6, “two”, “2 dogs”, “3” ×2. After normalization: “2” ×7, “2 dogs” ×1, “3” ×2.

Model says “Two.” → normalized “2”, \(k = 7\): each leave-one-out term has at least 6 matches, so \(\mathrm{Acc} = 1.0\). Model says “3” → \(k = 2\): two terms are \(\min(1, 1/3) = 0.333\) and eight are \(\min(1, 2/3) = 0.667\); mean \((2 \times 0.333 + 8 \times 0.667)/10 = 0.6\). Model says “2 dogs” → \(k = 1\): one term is 0, nine are 0.333; mean 0.3. A dataset score is the mean over all questions.

VQA accuracy is used by VQAv2, OK-VQA, TextVQA and VizWiz, among others. [Likely] For TextVQA (questions about text in natural images), the same formula is used even though the answer is read, not reasoned: one wrong letter scores 0. Documents need a softer rule.

ANLS: forgive small reading errors in documents

On a scanned invoice, the answer is usually a string copied from the page: a date, a total, a name. A model may read the text almost right and make one character error (“2O21” with a letter O instead of the digit 0). Exact match gives 0 for this, the same as a completely wrong answer. ANLS (Average Normalized Levenshtein Similarity), introduced for scene-text VQA (ST-VQA) and used as the main metric of DocVQA, gives partial credit instead.

The Levenshtein distance \(\mathrm{lev}(a, o)\) is the smallest number of single-character edits (insert, delete, substitute) that turn string \(o\) into string \(a\). Divide by the longer length to get a number between 0 and 1:

\[ \mathrm{NL}(a, o) = \frac{\mathrm{lev}(a, o)}{\max(|a|, |o|)}, \qquad s(a, o) = \begin{cases} 1 - \mathrm{NL}(a, o) & \text{if } \mathrm{NL}(a, o) \lt \tau \\ 0 & \text{otherwise} \end{cases} \tag{192.3} \]
\[ \mathrm{ANLS} = \frac{1}{N} \sum_{i=1}^{N} \max_{j}\ s(a_{ij}, o_i), \qquad \tau = 0.5 \tag{192.4} \]
Read the symbols
\(o_i\)
the model’s answer to question \(i\)
\(a_{ij}\)
the \(j\)-th accepted reference answer for question \(i\) (a question can have several, such as two date formats)
\(|a|\)
the length of string \(a\) in characters
\(\tau\)
the threshold. If more than half of the string is wrong, we assume the model picked the wrong text, and the score is 0
\(N\)
the number of questions

Standard implementations lower-case and trim both strings before comparing. [Likely] The threshold is the key design choice: below it the score falls smoothly with each error (“nearly read it”); above it the answer counts as wrong.

Question: “What is the invoice date?” Accepted answers: 03/12/2021 march 12, 2021
Edits from the model answer to the closest accepted answer:
Figure 3. ANLS for one question with two accepted answers. The edit line shows matched characters in green, substitutions in orange, characters the model should have written in pink, and extra characters struck out in red. A space character is drawn as a small shaded blank box. The score uses the closest accepted answer.
How to read Figure 3

“03/12/2O21” has one substitution in 10 characters: \(\mathrm{NL} = 0.1\), score 0.9. This is the case ANLS was made for. Now try “03/21/2021”: it is the wrong date, yet it scores 0.8, because only 2 characters differ. And “12 March 2021” is the right date in a third format, but it is exactly 7 edits from “march 12, 2021” (\(\mathrm{NL} = 7/14 = 0.5\), not below \(\tau\)) and scores 0. Move \(\tau\) to see how strict the cut is. Type your own answers in the box.

Worked example: what ANLS rewards and what it misses

“03/12/2O21” vs “03/12/2021”: 1 edit, length 10, \(\mathrm{NL} = 0.1\), \(s = 0.9\). Against “march 12, 2021” it needs 9 edits of 14 (\(\mathrm{NL} = 0.643 \gt 0.5\)), so it scores 0 there; the max over references is 0.9.

“03/21/2021”: 2 edits, \(s = 0.8\). For a person paying this invoice, the answer is simply wrong. “2021”: 6 edits of 10, \(\mathrm{NL} = 0.6\), score 0. So ANLS is a good measure of reading quality, but for numbers, dates and IDs in a product, also report exact match after normalizing formats (for example, parse dates and amounts and compare the values).

DocVQA has 50,000 questions on 12,767 document images (39,463 / 5,349 / 5,188 questions in train / validation / test). [Certain] Humans scored 0.981 ANLS (94.36% exact accuracy) on the test set; the best baseline in the paper, a BERT-large reader fine-tuned on SQuAD and DocVQA, reached 0.665. [Certain] Recent VLMs report DocVQA scores above 0.9 on model cards. [Likely] Chart and infographic benchmarks (ChartQA, InfographicVQA) use ANLS or “relaxed accuracy” (numbers within 5% count as right). [Likely]

CIDEr: scoring captions by consensus

A caption can be correct in many ways, so captioning datasets have several human reference captions per image (COCO has 5). Metrics compare the model caption to all of them. You met BLEU, ROUGE and METEOR in session 149. CIDEr (Consensus-based Image Description Evaluation) is the standard captioning metric. Its idea: an n-gram (a sequence of \(n\) words) that matches the references counts more if it is rare across the whole dataset. “a” and “on the” appear in almost every caption and tell you nothing about this image; “dog” and “the beach” do.

Step 1: turn each caption into a vector of TF-IDF weights, one entry per n-gram, for each \(n = 1, \dots, 4\):

\[ g_k(s) = h_k(s) \cdot \log \frac{|I|}{\max\!\left(1,\ \mathrm{df}(k)\right)} \tag{192.5} \]
Read the symbols
\(k\)
one n-gram, such as “the beach”
\(h_k(s)\)
how many times \(k\) occurs in caption \(s\) (term frequency, TF)
\(|I|\)
the number of images in the test set
\(\mathrm{df}(k)\)
document frequency: in how many images at least one reference caption contains \(k\)
\(\log(\cdot)\)
inverse document frequency (IDF): 0 for an n-gram found in every image, \(\log |I|\) for one found in a single image

Step 2: compare the candidate \(c\) with each of the \(m\) references \(s_j\) by cosine similarity, separately for each \(n\), and average:

\[ \mathrm{CIDEr}_n(c, S) = \frac{1}{m} \sum_{j=1}^{m} \frac{\mathbf{g}^n(c) \cdot \mathbf{g}^n(s_j)}{\lVert \mathbf{g}^n(c) \rVert\, \lVert \mathbf{g}^n(s_j) \rVert}, \qquad \mathrm{CIDEr} = \frac{1}{4} \sum_{n=1}^{4} \mathrm{CIDEr}_n \tag{192.6} \]

Plain CIDEr can be gamed: repeat a good word many times, or write a very long caption. The version everyone reports, CIDEr-D, adds two fixes and a scale factor of 10:

\[ \mathrm{CIDEr\text{-}D}_n = \frac{10}{m} \sum_{j=1}^{m} e^{-\frac{(\ell(c) - \ell(s_j))^2}{2 \sigma^2}} \cdot \frac{\min\!\left(\mathbf{g}^n(c), \mathbf{g}^n(s_j)\right) \cdot \mathbf{g}^n(s_j)}{\lVert \mathbf{g}^n(c) \rVert\, \lVert \mathbf{g}^n(s_j) \rVert}, \qquad \sigma = 6 \tag{192.7} \]
Read the symbols
\(\min(\mathbf{g}^n(c), \mathbf{g}^n(s_j))\)
clipping: element by element, the candidate’s weight cannot be larger than the reference’s, so repeating “dog” five times earns no more than the references allow
\(\ell(c), \ell(s_j)\)
caption lengths in words
\(e^{-(\cdot)^2 / 2\sigma^2}\)
a Gaussian length penalty: a caption 6 words longer or shorter than a reference keeps \(e^{-0.5} = 0.61\) of its score
10
a scale factor, so scores look like other metrics; this is why CIDEr is often above 1 (reported as, for example, 120 = 1.20)

One important consequence: the IDF part is computed over the whole test set. The same caption for the same image gets a different CIDEr in a different test set. Never compare CIDEr numbers computed on different splits or with different tokenizers.

A tiny test set of 4 images, 3 references each. We score a caption for image A.
Words of the caption, shaded by IDF (darker = rarer in the test set):
Figure 4. CIDEr-D computed live with the same rules as the standard coco-caption code (formula 192.7). Bars show the score for 1- to 4-grams; the final score is their mean. With both boxes off you get plain CIDEr (×10).
How to read Figure 4

“a”, “on” and “the” appear in all 4 images, so their IDF is \(\log(4/4) = 0\): they add nothing. “dog” and “beach” appear only in image A, so they carry all the weight. The exact reference scores highest (4.367). The paraphrase shares few rare n-grams and drops to 2.742. The word salad has the right rare words but no matching 2-grams; clipping cuts it further (switch clipping off and watch it jump from 0.479 to a higher value). The repetition caption is punished by clipping and by the length penalty. Type any caption, for example one about image B, which scores 0.

Worked example: IDF in the tiny test set

With \(|I| = 4\) images: “dog” is in the references of 1 image, so its IDF is \(\log(4/1) = 1.386\). “on the” is in all 4, so its IDF is \(\log(4/4) = 0\). The caption “a dog runs on the beach” scores \(\mathrm{CIDEr\text{-}D}_{1..4} = 5.801, 4.445, 3.889, 3.333\), mean 4.367. “a photo of a dog” keeps only the 1-gram and 2-gram matches (“a dog” is in a reference): 3.112, 1.326, 0, 0, mean 1.110. The numbers are large because the test set is tiny; on COCO, good models score about 1.2 to 1.4 (“120 to 140”). [Likely]

CIDEr has clear limits. It rewards wording that matches the references, not truth: “a dog runs on the beach” scores high even if the dog is sitting. It punishes good captions that are longer or more detailed than the short COCO references. Modern VLMs write long, rich descriptions, so their CIDEr is often lower than that of small models fine-tuned on COCO style. [Likely] For detailed descriptions, teams use a judge model (below) or hallucination metrics such as CHAIR, the fraction of mentioned objects that are not in the image. [Certain]

The main benchmarks

A benchmark is a fixed public test set plus a scoring rule. No single benchmark covers a VLM, so model cards report a suite. The table lists the ones from this session’s key points and what each one tests.

BenchmarkWhat it testsSize and formatMetric
VQAv2General visual questions on COCO photosAbout 1.1M image–question pairs, about 13M answers, about 200K images; 10 answers per questionVQA accuracy (192.1)
DocVQAReading and understanding scanned documents50,000 questions, 12,767 images; open answers copied from the pageANLS (192.4)
MMMUCollege-level knowledge and reasoning with images (charts, diagrams, chemical structures, music sheets…)11.5K questions; 6 disciplines, 30 subjects, 183 subfields, 30 image types; 150 / 900 / 10,500 dev / validation / test; 94% multiple choiceAccuracy
POPEObject hallucination500 COCO images × 6 yes/no questions = 3,000 per setting; 3 settingsAccuracy, precision, recall, F1, yes ratio (192.8)
MVBenchTemporal understanding in short videos (action order, motion direction, counting over time…)20 tasks that “cannot be effectively solved with a single frame”, 200 multiple-choice questions each, 3 to 5 optionsAccuracy
Video-MMEVideo understanding from 11 seconds to 1 hour900 videos (254 hours), 2,700 questions with 4 options; short (under 2 min), medium (4–15 min), long (30–60 min); with and without subtitlesAccuracy

All numbers in the table are from the papers. [Certain] At release, GPT-4V scored 56% and Gemini Ultra 59% on MMMU. [Certain] On Video-MME, the best model in the paper, Gemini 1.5 Pro, scored 75.0% without subtitles and 81.3% with them; its score fell from 81.7% on short videos to 67.4% on long ones (without subtitles). [Certain]

VQAv2: why “balanced” matters

The first VQA dataset had a strong language prior: the question alone often gave away the answer. For questions that start with “Do you see a …”, answering “yes” without looking at the image scored 87% VQA accuracy. [Certain] VQAv2 fixed this by collecting, for each question, a second similar image where the answer is different. A model that ignores the image now gets one of the pair wrong. This idea, “make the image necessary”, returns in every good benchmark.

POPE: measuring object hallucination

Object hallucination means the model mentions objects that are not in the image (a strong language prior: “kitchen” suggests “refrigerator”). POPE (Polling-based Object Probing Evaluation) asks yes/no questions “Is there a <object> in the image?”. For each image, it asks about 3 objects that are there (answer yes) and 3 that are not (answer no), so the labels are balanced 1 : 1. The “no” objects are chosen in three ways, from easy to hard:

  • Random: any object not in the image.
  • Popular: the most frequent objects in the dataset that are not in the image.
  • Adversarial: objects that often co-occur with the objects in the image (a “fork” in a picture of a dining table), but are not there.

Treat “yes” as the positive class. With true positives TP (said yes, object present), false positives FP (said yes, object absent), false negatives FN (said no, object present) and \(N\) questions:

\[ P = \frac{TP}{TP + FP}, \quad R = \frac{TP}{TP + FN}, \quad F_1 = \frac{2PR}{P + R}, \quad \text{yes ratio} = \frac{TP + FP}{N} \tag{192.8} \]

The yes ratio matters because a model can “pass” with bias alone. A model that always says yes has recall 1, precision 0.5, accuracy 50% and \(F_1 = 2/3 = 66.7\%\). In the POPE paper, MultiModal-GPT answered yes to 100% of questions and got exactly that; LLaVA (the first version) answered yes to 99.1% in the adversarial setting. [Certain] A healthy model has a yes ratio near 50% on POPE.

Figure 5. POPE metrics for one setting (3,000 questions, half with the object present). Left: the confusion matrix. Right: the five numbers POPE reports. The first preset reproduces InstructBLIP’s adversarial row from the POPE paper.
How to read Figure 5

The InstructBLIP preset gives accuracy 74.37, precision 67.67, recall 93.33, F1 78.45 and yes ratio 68.97, the published numbers. Its errors are almost all false “yes” answers: it hallucinates 669 of 1,500 absent objects. Now press “Always yes”: F1 is still 66.67 even though the model looked at nothing. Move the second slider down: fewer hallucinations raise precision, accuracy and F1 together. This is why you should always read F1 together with the yes ratio.

Worked example: getting the counts back from a published row

InstructBLIP, adversarial: recall 93.33% and yes ratio 68.97% on 3,000 questions (1,500 per label). TP \(= 0.9333 \times 1500 = 1400\), FN \(= 100\). All yes answers \(= 0.6897 \times 3000 = 2069\), so FP \(= 2069 - 1400 = 669\) and TN \(= 1500 - 669 = 831\). Check: accuracy \(= (1400 + 831)/3000 = 74.37\%\), precision \(= 1400/2069 = 67.67\%\), \(F_1 = 2 \cdot 0.6767 \cdot 0.9333 / (0.6767 + 0.9333) = 78.45\%\). All match the paper.

POPE is simple and stable: in the paper, four different wordings of the question changed LLaVA’s F1 by a standard deviation of only 0.78 points, while four caption prompts changed CHAIR by 3.22. [Certain] Its limits: it only tests objects (not attributes, counts or relations), only on COCO categories, and only in yes/no form. A model can learn to say “no” more often and look better on POPE while its free descriptions still hallucinate. [Likely] We study the causes and fixes in session 193.

Multiple choice: reading the letter correctly

MMMU, MVBench and Video-MME are mostly multiple choice, which avoids open-answer scoring. But two problems remain. First, extraction: the model may answer “The answer is (B) because…”, “b”, or the option text instead of the letter. MVBench solves this by ending the prompt with “Best Option: (” so the next token is the letter; it reports 100% extraction success. [Certain] Other harnesses use rules, then an LLM to map the text to an option when rules fail. Second, position bias: models prefer some letters. MMBench (3,217 questions) uses CircularEval: each question is asked several times with the options rotated, and it counts as correct only if the model is right every time. [Certain] For 4 options, random guessing then succeeds with probability \((1/4)^4 = 0.39\%\) instead of 25%.

Does the benchmark need the image?

Many questions can be answered from the text alone, either because the question gives the answer away, or because the benchmark leaked into training data. MMStar tested this: the text-only Gemini Pro scored 42.9% on MMMU without seeing any image, and some VLMs answered “vision-necessary” questions with the image removed. [Certain] MMStar then kept 1,500 questions that need the image, and defined two diagnostics:

\[ \mathrm{MG} = S_v - S_{wv}, \qquad \mathrm{ML} = \max\!\left(0,\ S_{wv} - S_t\right) \tag{192.9} \]
Read the symbols
\(S_v\)
the VLM’s score with the images
\(S_{wv}\)
the same VLM’s score with the images removed
\(S_t\)
the score of the VLM’s original language model (before any multimodal training), text only
MG
multimodal gain: how much the image actually helps
ML
multimodal leakage: how much the VLM “knows” without the image beyond what its LLM knew; a sign that test questions were in the multimodal training data

Video-MME used the same idea during construction: questions that Gemini 1.5 Pro could answer from the text alone were sent back for rewriting, and the final text-only accuracy was below 15%. [Certain] Always run this blind baseline on your own test set too. If a model scores well without the image, the test set measures language guessing.

Video benchmarks: settings change the score

For video, the number of sampled frames, their resolution, and whether subtitles are given are part of the test. On Video-MME, adding subtitles raised Gemini 1.5 Pro from 75.0% to 81.3%, and the gain was largest on long videos (67.4% → 77.4%). [Certain] A score reported with 64 frames and subtitles is not comparable with one at 8 frames without. MVBench builds its tasks so a single frame is not enough (for example, the order of actions); a strong single-image model that does well there may be guessing from the scene. [Likely] Video-specific tasks such as temporal grounding are covered in session 213.

Your own domain test set

Public benchmarks tell you about general skill. Your product needs a test set that looks like your data: your document layouts, your camera angles, your languages, your questions. This is the most useful evaluation work you can do, and it is often skipped.

Sample it from real data

Take items from the real input stream (production logs, or the data the product will see), not from the training set and not from what is easy to collect. Use stratified sampling: define slices that matter (document type, language, image quality, question type, video length) and sample enough items from each slice, including rare but important ones. Remove near-duplicates of training items (perceptual hashes or embedding similarity, as in session 209), or the test set measures memory. Add items that test failure modes on purpose: questions about objects that are not there, unanswerable questions (“the total is not visible”), blurry images, very small text.

Label it clearly

Write labeling guidelines with examples before labeling: the exact answer format (dates as YYYY-MM-DD, amounts without currency sign), what to do when the image is unclear, when “unanswerable” is correct. Have two people label a subset (for example 100 to 200 items) independently and measure their agreement with Cohen’s kappa (formula 192.12). Low agreement means the guidelines are unclear, not that people are careless. Decide the scoring rule per item type (exact match on parsed values, ANLS, IoU), store it with the item, then freeze and version the set. Keep it hidden from training, and never tune prompts on it; use a separate development set for that.

Algorithm 1: build a domain test set
  1. input: real data stream \(D\), slice definitions, error budget \(E\) (for example \(\pm 3\) points)
  2. \(n \leftarrow \lceil 1.96^2\, p (1 - p) / E^2 \rceil\) per slice you want to report // formula (192.10); use \(p = 0.5\) if unknown
  3. for each slice: sample \(n\) items from \(D\) at random // stratified, not “easy” items
  4. remove items that are near-duplicates of training data
  5. add probe items: absent objects, unanswerable, low quality // tests hallucination and abstention
  6. write guidelines; two labelers on a subset; compute \(\kappa\) // formula (192.12)
  7. if \(\kappa\) is low: fix guidelines and relabel
  8. store per item: input, accepted answers, answer type, scoring rule, slice
  9. run a blind baseline (no image) // must score low; else fix items
  10. freeze, version, and keep out of training and prompt tuning

How big must it be?

A test set is a sample, so its accuracy is noisy. If the true accuracy is \(p\) and the set has \(n\) independent items, the measured accuracy has standard error

\[ \mathrm{SE} = \sqrt{\frac{p(1 - p)}{n}}, \qquad \text{95\% interval: } \hat p \pm 1.96\,\mathrm{SE}, \qquad n_{\text{needed}} = \frac{1.96^2\, p(1 - p)}{E^2} \tag{192.10} \]

At \(p = 0.8\): 200 items give \(\pm 5.5\) points, 500 items \(\pm 3.5\), 1,000 items \(\pm 2.5\). To get \(\pm 2\) points you need about 1,537 items. Small slices are much noisier than the full set.

When you compare two models, use the same items (a paired comparison). Only the items where the models disagree carry information. Let \(b\) be the number of items model A gets right and B wrong, and \(c\) the reverse:

\[ \hat\Delta = \frac{c - b}{n}, \qquad \mathrm{SE}(\hat\Delta) = \frac{1}{n}\sqrt{\,b + c - \frac{(c - b)^2}{n}\,}, \qquad \chi^2_{\text{McNemar}} = \frac{(c - b)^2}{b + c} \tag{192.11} \]
Read the symbols
\(\hat\Delta\)
accuracy of B minus accuracy of A
\(b, c\)
the discordant counts (items where exactly one model is right)
\(\chi^2_{\text{McNemar}}\)
McNemar’s test; a value above 3.84 means the difference is significant at the 5% level
Figure 6. Top: the half-width of the 95% interval for one model’s accuracy as the test set grows (formula 192.10); the dot is your setting. Bottom: the 95% interval for “B minus A” on the same items (formula 192.11). If the bar crosses zero, the test set cannot tell the two models apart.
How to read Figure 6

At the start, A is right on 400 of 500 items and B on 415: B looks 3 points better. But the interval for the difference is \(3.0 \pm 3.8\) points and crosses zero, and McNemar’s \(\chi^2 = 2.37\) is below 3.84. You cannot claim B is better. Double \(n\) (and double \(b\) and \(c\)) and the same 3-point gap becomes significant. Also try \(b = 5\), \(c = 20\): the same 3-point gap with few disagreements is clearly significant, because the models agree almost everywhere else.

Worked example: is B really better?

On 500 items: both right 360, A only 40, B only 55, both wrong 45. \(A = 400/500 = 80\%\), \(B = 415/500 = 83\%\). \(\hat\Delta = 15/500 = 3.0\) points. \(\mathrm{SE} = \tfrac{1}{500}\sqrt{95 - 225/500} = 0.0194\), so the 95% interval is \(3.0 \pm 3.8\) points, from −0.8 to +6.8. A paired bootstrap (resample the 500 items 10,000 times, code below) gives the same interval. If you had treated the two scores as independent, the interval would be wider (\(\pm 4.8\) points): pairing helps, but here it is still not enough. Decision: collect more items or look at the 95 disagreements by hand before shipping.

Judge-based scoring for open answers

Some answers cannot be checked by string rules: a detailed description of a scene, an explanation of a chart, a step-by-step solution, a chat reply. References cover only a few of the many good answers. Here we use a judge: a strong model that reads the question, the answer, often a reference answer, and ideally the image, and gives a score or a verdict. This is the “LLM-as-a-judge” method from session 155; for VLMs the judge may be a text-only LLM or a VLM.

Three common setups:

  • Single-answer grading: the judge gives a score (1–10, or pass/fail) using a rubric.
  • Reference-guided grading: the judge compares the answer with a reference answer. More reliable for factual questions and math.
  • Pairwise comparison: the judge sees two answers and picks the better one (or a tie). Good for comparing two models.

Its limits

Zheng et al. found that strong judges (GPT-4) agreed with human experts over 80% of the time, the same level as humans agree with each other. [Certain] But they also measured clear biases:

  • Position bias: when the two answers were swapped, GPT-4 gave a consistent verdict in only 65.0% of cases; Claude-v1 in 23.8% (it preferred the first answer 75.0% of the time). [Certain]
  • Verbosity bias: answers made longer by repeating a list (no new content) fooled Claude-v1 and GPT-3.5 in 91.3% of 23 cases; GPT-4 in 8.7%. [Certain]
  • Self-enhancement: a judge may prefer answers written by itself or its own model family. [Likely]
  • Limited ability: a judge can be wrong on math and reasoning questions it cannot solve itself; giving it a reference answer helps. [Certain]

For VLMs there are extra problems. A text-only judge cannot see the image: it can only check fluency and agreement with the reference, so it misses hallucinated details that the reference does not mention. A VLM judge sees the image but has the same weaknesses we are trying to measure (hallucination, small text, counting), and it may share them with the model under test. [Likely] And judge scores change when the judge model is updated, so old and new numbers are not comparable.

Algorithm 2: pairwise judging with position swap and calibration
  1. input: items \((x, q, \text{ref})\), answers \(o_A, o_B\), judge \(J\), rubric, human labels on a calibration subset
  2. for each item:
  3. \(v_1 \leftarrow J(x, q, \text{ref}, o_A, o_B)\) // A shown first
  4. \(v_2 \leftarrow J(x, q, \text{ref}, o_B, o_A)\) // B shown first
  5. if \(v_1\) and \(v_2\) pick the same answer: record a win for it; else record a tie
  6. judge settings: fixed model version, temperature 0, short reason before the verdict
  7. on the calibration subset: compute \(\kappa\) between judge and humans // formula (192.12)
  8. if \(\kappa\) is low: improve the rubric (atomic checks such as “mentions an object not in the image?”) and repeat
  9. report win rate with a 95% interval, the judge name and version, and \(\kappa\)

Check the judge against humans

Before you trust a judge, label 100 to 300 items by hand and compare. Raw agreement can look high just because most answers are correct. Cohen’s kappa corrects for agreement by chance:

\[ \kappa = \frac{p_o - p_e}{1 - p_e}, \qquad p_e = \sum_{c} p_{\text{judge}}(c)\; p_{\text{human}}(c) \tag{192.12} \]
Read the symbols
\(p_o\)
observed agreement: the fraction of items where judge and human give the same label
\(p_e\)
agreement expected by chance if both labeled at random with their own label rates
\(c\)
a label, for example “correct” or “wrong”
\(\kappa\)
1 is perfect agreement, 0 is chance level; values around 0.6 to 0.8 are usually called substantial
Worked example: a lenient judge

200 answers. Both say correct: 120. Both say wrong: 50. Judge correct, human wrong: 20. Judge wrong, human correct: 10. So \(p_o = 170/200 = 0.85\). The judge says “correct” for 140/200 = 0.70, humans for 130/200 = 0.65. \(p_e = 0.70 \times 0.65 + 0.30 \times 0.35 = 0.56\). \(\kappa = (0.85 - 0.56)/(1 - 0.56) = 0.66\). The judge is useful, but it reports 70% accuracy where humans see 65%: it is 5 points too lenient. Report the human-calibrated number, or fix the rubric.

Putting it together: an evaluation harness

Algorithm 3: score a VLM on one benchmark
  1. input: model \(M\), items \(\{(x_i, q_i, A_i, \text{type}_i)\}_{i=1}^{N}\), prompt template, image or frame settings
  2. for \(i = 1 \dots N\):
  3. \(p \leftarrow \text{template}(q_i)\) // e.g. “Answer with a single word or phrase.”
  4. \(o \leftarrow M(x_i, p)\), greedy decoding // deterministic
  5. \(o \leftarrow \text{parse}(o, \text{type}_i)\) // option letter, number, string
  6. \(s_i \leftarrow \text{metric}(o, A_i)\) // VQA acc, ANLS, exact, IoU, judge
  7. log \((i, \text{raw output}, o, s_i)\) // you will need to read failures
  8. \(\bar s \leftarrow \frac{1}{N}\sum_i s_i\); interval by formula (192.10) or bootstrap
  9. repeat lines 2–8 with the image removed // blind baseline, formula (192.9)
  10. report \(\bar s\), interval, per-slice scores, blind score, parse failure rate, all settings

The code below implements the metrics of this session in plain Python and PyTorch, and checks them on the worked examples. It is deterministic.

import math, re, torch
from collections import defaultdict

# 1) VQA accuracy, formula (192.1), with a short version of the official answer normalization
NUM = {"none": "0", "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
       "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9", "ten": "10"}
def norm(a):
    a = a.lower().strip()
    a = re.sub(r"(?<!\d)\.(?!\d)", "", a)              # drop periods that are not decimal points
    a = re.sub(r"[?!,;:\"']", "", a)
    return " ".join(NUM.get(w, w) for w in a.split() if w not in ("a", "an", "the"))

def vqa_acc(pred, gts):                                # gts: the 10 human answers
    if len(set(gts)) > 1:                              # the official script normalizes only in this case
        pred, gts = norm(pred), [norm(g) for g in gts]
    per = [min(1.0, sum(o == pred for o in gts[:i] + gts[i + 1:]) / 3) for i in range(len(gts))]
    return sum(per) / len(per)

# 2) ANLS, formulas (192.3) and (192.4)
def lev(a, b):
    d = list(range(len(b) + 1))
    for i, ca in enumerate(a, 1):
        prev, d[0] = d[0], i
        for j, cb in enumerate(b, 1):
            prev, d[j] = d[j], min(d[j] + 1, d[j - 1] + 1, prev + (ca != cb))
    return d[-1]

def anls_one(pred, gts, tau=0.5):
    best = 0.0
    for g in gts:
        p, g = pred.lower().strip(), g.lower().strip()
        nl = lev(p, g) / max(len(p), len(g), 1)
        best = max(best, 1 - nl if nl < tau else 0.0)
    return best

# 3) CIDEr-D, formulas (192.5) to (192.7), following the coco-caption code
def ngrams(s, n=4):
    w, c = s.split(), defaultdict(int)
    for k in range(1, n + 1):
        for i in range(len(w) - k + 1):
            c[tuple(w[i:i + k])] += 1
    return c

def cider_d(cands, refs, sigma=6.0):                   # cands: {img: str}, refs: {img: [str]}
    R = {k: [ngrams(r) for r in v] for k, v in refs.items()}
    df = defaultdict(float)
    for rs in R.values():
        for g in set(g for r in rs for g in r):
            df[g] += 1                                 # document frequency: in how many images
    logN = math.log(len(R))
    def vec(c):
        v, nrm = [dict() for _ in range(4)], [0.0] * 4
        for g, tf in c.items():
            v[len(g) - 1][g] = tf * (logN - math.log(max(1.0, df[g])))
            nrm[len(g) - 1] += v[len(g) - 1][g] ** 2
        return v, [math.sqrt(x) for x in nrm], sum(tf for g, tf in c.items() if len(g) == 1)
    scores = {}
    for img, cand in cands.items():
        vc, nc, lc = vec(ngrams(cand))
        total = 0.0
        for r in R[img]:
            vr, nr, lr = vec(r)
            pen = math.exp(-((lc - lr) ** 2) / (2 * sigma ** 2))     # length penalty
            for n in range(4):
                dot = sum(min(x, vr[n].get(g, 0.0)) * vr[n].get(g, 0.0) for g, x in vc[n].items())  # clipping
                if nc[n] > 0 and nr[n] > 0:
                    total += pen * dot / (nc[n] * nr[n]) / 4
        scores[img] = 10.0 * total / len(R[img])
    return scores

# 4) POPE metrics, formula (192.8); "yes" is the positive class
def pope(pred_yes, label_yes):
    tp = (pred_yes & label_yes).sum().item(); fp = (pred_yes & ~label_yes).sum().item()
    fn = (~pred_yes & label_yes).sum().item(); n = label_yes.numel()
    P, R = tp / max(1, tp + fp), tp / max(1, tp + fn)
    return dict(acc=(pred_yes == label_yes).float().mean().item(), precision=P, recall=R,
                f1=2 * P * R / max(1e-12, P + R), yes_ratio=pred_yes.float().mean().item())

# 5) Paired bootstrap for the difference of two models on the same items, formula (192.11)
def paired_bootstrap(a, b, iters=10000, seed=0):
    g = torch.Generator().manual_seed(seed)
    idx = torch.randint(0, a.numel(), (iters, a.numel()), generator=g)
    diff = b.float()[idx].mean(1) - a.float()[idx].mean(1)
    return diff.quantile(torch.tensor([0.025, 0.975])).tolist()

if __name__ == "__main__":
    gts = ["2", "2", "2", "two", "2 dogs", "3", "2", "2", "3", "2"]
    print([round(vqa_acc(p, gts), 3) for p in ["Two.", "3", "2 dogs", "4"]])        # [1.0, 0.6, 0.3, 0.0]
    print([round(anls_one(p, ["03/12/2021", "march 12, 2021"]), 4)
           for p in ["03/12/2O21", "03/21/2021", "12 March 2021"]])                  # [0.9, 0.8, 0.0]
    refs = {"A": ["a dog runs on the beach", "a brown dog running on the sand",
                  "a dog plays on the beach near the water"],
            "B": ["a man rides a bike on the street", "a person riding a bicycle down the road",
                  "a man on a bike in the city"],
            "C": ["two cats sleep on a bed", "two cats lying on a white bed", "a pair of cats asleep on the bed"],
            "D": ["a plate of food on a table", "a table with a plate of pasta", "food on a white plate on the table"]}
    print({k: round(v, 3) for k, v in cider_d({"A": "a dog runs on the beach"}, refs).items()})  # {'A': 4.367}
    label = torch.tensor([True] * 1500 + [False] * 1500)
    pred = torch.tensor([True] * 1400 + [False] * 100 + [True] * 669 + [False] * 831)
    print({k: round(100 * v, 2) for k, v in pope(pred, label).items()})
    a = torch.tensor([1] * 360 + [1] * 40 + [0] * 55 + [0] * 45)   # model A: 400 / 500 right
    b = torch.tensor([1] * 360 + [0] * 40 + [1] * 55 + [0] * 45)   # model B: 415 / 500 right
    print([round(x, 4) for x in paired_bootstrap(a, b)])

Real harnesses such as VLMEvalKit and lmms-eval implement hundreds of benchmarks with these rules. [Likely] Even so, the same model can get different scores in two harnesses because of prompts, image resolution and parsing. When you compare with a number from a paper, re-run the baseline in your own harness.

Where this is used

SettingWhat to measureNotes
Choosing a base VLM for a productPublic suite (MMMU, DocVQA, POPE, Video-MME) as a first filter, then your domain test setDecide on the domain set; public scores only shortlist
Document extraction (invoices, forms, IDs)ANLS for reading; exact match on parsed values (dates, amounts) per fieldA wrong total with small edit distance must count as wrong
Alt text and captioningCIDEr for short captions; judge with rubric plus hallucination checks for long descriptionsCIDEr is unfair to long, detailed captions
Safety-critical visual QA (medical, industrial inspection)POPE-style probes on your own images, yes ratio, abstention on unanswerable itemsA false “yes” may cost more than a false “no”: report precision and recall, not only F1
Video question answering and Video-RAGMultiple-choice accuracy by video length (Video-MME style), with and without subtitles; for RAG also whether the right clip was retrievedFix frame count and resolution in the report
Key-moment detection and temporal groundingRecall at temporal IoU thresholds (for example R@1, \(\text{IoU} \ge 0.5\)) [Likely]Covered in session 213
Text-to-video retrievalRecall@K, median rankNot generated text: no parsing problems
Fine-tuning a VLM (session 194)Domain set before and after, plus a few public benchmarks to detect forgettingPaired comparison on the same items (192.11)
Preference tuning against hallucination (session 190)POPE, CHAIR, and a judge on detailed descriptionsWatch the yes ratio: “say no more often” is a cheap fake gain

Common mistakes

  • Letting a chat model answer in sentences on VQA. “There are two dogs.” scores 0. Use the benchmark’s answer instruction (“single word or phrase”) and check the parse failure rate.
  • Comparing numbers from different harnesses. Prompt wording, normalization, image resolution and number of video frames all change scores. Re-run baselines yourself.
  • Reading POPE F1 without the yes ratio. Always-yes gets F1 66.7. A big jump in F1 with a yes ratio far from 50% is a bias change, not a fix.
  • Using ANLS for values that must be exact. A swapped day and month scores 0.8. Parse and compare values for dates, amounts and IDs.
  • Comparing CIDEr across test sets or tokenizers. The IDF weights come from the test set itself.
  • No blind baseline. If the model scores well without the image, the test measures language priors or leakage (formula 192.9).
  • Declaring a winner from a 2-point gap on a few hundred items. Compute the paired interval (192.11) first.
  • Trusting a judge without calibration. Swap answer order, fix the judge version, and measure \(\kappa\) against human labels.

Interview questions

How is VQA accuracy computed, and why the division by 3?basic
Each question has 10 human answers. An answer gets \(\min(1, \#\text{matching humans}/3)\): full credit if at least 3 humans gave it, partial credit otherwise, because people disagree on open answers. The official code averages this over the 10 subsets of 9 humans (leave one out), formula (192.1). If \(k\) humans gave the answer, the score is 0.3, 0.6, 0.9 for \(k = 1, 2, 3\) and 1.0 for \(k \ge 4\). Answers are normalized first (lower-case, punctuation, articles, number words to digits).
Your new VLM scores 20 points lower on VQAv2 than the old one, but it looks better in demos. What do you check?core
First the raw outputs: a chat-tuned model probably answers in sentences (“There are two dogs.”), which match no human answer and score 0. Check the prompt has the short-answer instruction, check the parse failure rate, and check normalization (the official script only normalizes when the human answers differ). Then compare on the same harness and image settings as the old model. Only after that compare real errors by reading a sample.
Why does DocVQA use ANLS instead of exact match? When is ANLS misleading?core
Answers are strings copied from the page, and reading has small errors (“2O21” for “2021”). Exact match scores such an answer 0, like a totally wrong one. ANLS gives \(1 - \mathrm{NL}\) when the normalized edit distance is below \(\tau = 0.5\), and 0 above, taking the best over accepted answers (192.3–192.4). It is misleading for values: “03/21/2021” instead of “03/12/2021” is a wrong date but scores 0.8; and a correct answer in a different format (“12 March 2021” vs “march 12, 2021”) can score 0. For product fields, parse and compare values exactly.
Explain CIDEr. Why TF-IDF, and what does the “D” in CIDEr-D add?core
CIDEr represents each caption as TF-IDF vectors of 1- to 4-grams, compares the candidate with each reference by cosine similarity, averages over references and over n (192.5–192.6). IDF is computed over the images of the test set, so n-grams that appear everywhere (“a”, “on the”) get weight near 0 and rare, image-specific n-grams dominate: it measures consensus on specific content. CIDEr-D adds clipping (the candidate’s n-gram weight cannot exceed the reference’s, so repetition does not help), a Gaussian length penalty with \(\sigma = 6\), and multiplies by 10 (192.7). Because IDF depends on the test set, scores are not comparable across test sets.
What does POPE measure, how is it built, and why report the yes ratio?core
POPE measures object hallucination with yes/no questions “Is there a X in the image?”: per image 3 objects that are present and 3 that are absent (500 COCO images, 3,000 questions per setting). Absent objects are sampled randomly, from the most popular objects, or adversarially from objects that co-occur with the ones present; scores drop from random to adversarial. It reports accuracy, precision, recall, F1 (main metric) and yes ratio. The yes ratio is needed because a model that always says yes gets recall 1, precision 0.5, F1 66.7; early models answered yes to about 99% of questions.
How do you know whether a benchmark actually tests vision?advanced
Run blind baselines: the VLM with the image removed, and its base LLM text-only. MMStar found the text-only Gemini Pro scored 42.9% on MMMU, and defined multimodal gain \(\mathrm{MG} = S_v - S_{wv}\) and leakage \(\mathrm{ML} = \max(0, S_{wv} - S_t)\) (192.9). Low MG means the image barely helps; high ML suggests the questions were in multimodal training data. Benchmarks can be built to avoid this: VQAv2 pairs each question with a second image with a different answer; Video-MME filtered out questions Gemini 1.5 Pro could answer from text (final text-only accuracy below 15%).
Model B beats model A by 3 points (83% vs 80%) on your 500-item test set. Do you ship B?advanced
Not on that number alone. Use a paired comparison on the same items. If A-only-right = 40 and B-only-right = 55, then \(\hat\Delta = 3.0\) points with SE \(= \frac{1}{500}\sqrt{95 - 225/500} = 1.94\) points, so the 95% interval is −0.8 to +6.8, and McNemar \(\chi^2 = 225/95 = 2.37 \lt 3.84\): not significant. A paired bootstrap agrees. Options: add items (the error shrinks as \(1/\sqrt{n}\)), read the 95 disagreements to see if B’s wins matter for the product, check per-slice results and cost/latency. If B wins on the important slice and is not worse elsewhere, it may still be the right choice, but say the evidence is weak.
Design the evaluation for a VLM that reads shipping labels for a logistics company.core
Build a domain test set from real label photos, stratified by carrier, label type, lighting, angle, damage; remove near-duplicates of training images. Fields (tracking number, postcode, weight, address) get clear guidelines with exact formats; two labelers on a subset, measure \(\kappa\). Metrics per field: exact match after format normalization for IDs and numbers (a one-digit error is a lost parcel), ANLS as a secondary “reading quality” metric, plus an “unreadable” class to test abstention. Size: about 900 items per slice for ±3 points at 70% (192.10). Run the blind baseline, report per-field, per-slice scores with intervals, and compare models with paired tests. Public benchmarks (DocVQA, OCRBench-style) only for a shortlist.
How would you use an LLM or VLM judge for detailed image descriptions, and what are its limits?advanced
Use a rubric with atomic checks (correct main objects, no objects that are absent, correct counts and colors, readable text copied correctly), give the judge the image (a VLM judge) or a detailed reference listing what is in the image, use temperature 0, a fixed judge version, and a short reason before the verdict. For comparisons, judge twice with the answer order swapped and count only consistent wins. Calibrate on 100–300 human-labeled items with Cohen’s kappa and correct for leniency. Limits: position bias (GPT-4 consistent on only 65% of swapped pairs in Zheng et al.), verbosity bias, self-preference, a text-only judge cannot see hallucinations not covered by the reference, and a VLM judge has the same perception errors as the model under test.
Why do multiple-choice video and image benchmarks still need care with answer extraction and option order?core
Models answer in many forms (“(B)”, “b”, “The answer is B because…”, or the option text), and a regex that fails counts as wrong, so a harness change can move scores. MVBench ends the prompt with “Best Option: (” so the next token is the letter, reaching 100% extraction. Models also prefer some positions; MMBench’s CircularEval asks each question with options rotated and requires all to be right, which makes chance \((1/4)^4 = 0.39\%\) for 4 options and removes the benefit of a letter bias. Also report the parse failure rate.
What changes when you evaluate video instead of images?advanced
The input itself is a setting: number of frames, frame resolution and sampling, plus subtitles and audio. On Video-MME, subtitles raised Gemini 1.5 Pro from 75.0% to 81.3%, mostly on long videos, and accuracy falls with video length (81.7% short vs 67.4% long without subtitles). So scores must be reported by length bucket and setting. Questions must need temporal information (MVBench’s tasks “cannot be effectively solved with a single frame”), so check a single-frame baseline as well as a no-video baseline. Temporal grounding needs IoU-based metrics, not text matching.

Exercises

  1. Ten humans answered “yes” six times and “no” four times. What VQA accuracy do the answers “yes” and “no” get? What does this say about the metric?
  2. Compute the ANLS score of the answer “$1,250” when the only reference is “$1,250.00”.
  3. On a POPE setting (1,500 present, 1,500 absent), a model says “yes” to 80% of all questions and has recall 0.95. Find TP, FP, TN, precision, accuracy and F1.
  4. You expect about 70% accuracy and want a 95% interval of \(\pm 3\) points. How many test items do you need?
  5. On 100 answers, judge and human both say correct 70 times, both wrong 10 times, judge correct and human wrong 15 times, judge wrong and human correct 5 times. Compute Cohen’s kappa.
  6. A model always answers “A” on a 4-option benchmark where the correct letter is uniform. What does it score with plain accuracy, and with CircularEval over the 4 rotations?
Answerscheck
1. Both get 1.0: \(k = 6\) and \(k = 4\) are both at least 4 (formula 192.2). The metric accepts any answer that enough humans gave, so on truly ambiguous questions two opposite answers are both “correct”. 2. Levenshtein distance 3 (add “.00”), longer length 9, \(\mathrm{NL} = 0.333 \lt 0.5\), score \(1 - 0.333 = 0.667\). 3. TP \(= 0.95 \times 1500 = 1425\); yes answers \(= 0.8 \times 3000 = 2400\), so FP \(= 975\), TN \(= 525\); precision \(= 1425/2400 = 0.594\), accuracy \(= (1425 + 525)/3000 = 0.65\), \(F_1 = 2 \cdot 0.594 \cdot 0.95/(0.594 + 0.95) = 0.731\). A high F1 with a yes ratio of 80%: heavy hallucination. 4. \(n = 1.96^2 \times 0.7 \times 0.3 / 0.03^2 = 896.4\), so 897 items. 5. \(p_o = 0.80\); judge correct rate 0.85, human 0.75; \(p_e = 0.85 \times 0.75 + 0.15 \times 0.25 = 0.675\); \(\kappa = (0.80 - 0.675)/(1 - 0.675) = 0.385\): weak agreement despite 80% raw agreement. 6. Plain accuracy 25%. Under CircularEval the correct option is at position A in only one of the 4 rotations, so the model is never right in all 4: score 0%.

Key takeaways

  • A VLM score is the result of a pipeline: prompt, model settings, parsing, metric and aggregation. Report all of them; differences between harnesses are often bigger than differences between models.
  • VQA accuracy (192.1) gives full credit when at least 3 of 10 humans agree, after normalization; short-answer prompts are part of the benchmark.
  • ANLS (192.4) forgives small reading errors below \(\tau = 0.5\), but treats near-miss numbers as almost right; use exact value checks for fields that must be exact.
  • CIDEr-D (192.7) rewards agreement on rare n-grams, with clipping and a length penalty; it depends on the test set and dislikes long, detailed captions.
  • POPE (192.8) probes object hallucination with balanced yes/no questions; read F1 together with the yes ratio. Blind baselines (192.9) tell you whether the image is needed.
  • Your own domain test set, sampled from real data with clear labels and enough items (192.10), decides; compare models with paired tests (192.11).
  • Judges score open answers, but have position, verbosity and self-preference biases and may not see the image; swap orders and calibrate with \(\kappa\) (192.12).

Key formulas

\[ \mathrm{Acc}(a) = \tfrac{1}{10}\textstyle\sum_i \min\!\left(1, \tfrac{\#\{j \ne i:\, h_j = a\}}{3}\right), \qquad s = 1 - \tfrac{\mathrm{lev}(a, o)}{\max(|a|, |o|)} \ \text{ if } \mathrm{NL} \lt 0.5 \]
\[ F_1 = \frac{2PR}{P + R}, \qquad \mathrm{SE} = \sqrt{\frac{p(1-p)}{n}}, \qquad \mathrm{SE}(\hat\Delta) = \frac{1}{n}\sqrt{b + c - \frac{(c-b)^2}{n}}, \qquad \kappa = \frac{p_o - p_e}{1 - p_e} \]

References for this session

  1. Yue et al. (2023). MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark
  2. Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images
  3. Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models
  4. Goyal et al. (2016). Making the V in VQA Matter
  5. Fu et al. (2024). Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
  6. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
  7. Goyal et al. (2016). Making the V in VQA Matter
  8. Agrawal et al. (VQA team) (code). Official VQA evaluation code (vqaEval.py)
  9. Biten, Tito, Mafla, Gomez, Rusiñol, Valveny, Jawahar & Karatzas (2019). Scene Text Visual Question Answering
  10. Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images
  11. Vedantam, Zitnick and Parikh (2015). CIDEr: Consensus-based Image Description Evaluation
  12. Chen et al. (coco-caption) (code). Microsoft COCO Caption Evaluation code (CIDEr-D scorer)
  13. Yue et al. (2023). MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark
  14. Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models
  15. Li et al. (2023). MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
  16. Fu et al. (2024). Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
  17. Chen et al. (2024). Are We on the Right Way for Evaluating Large Vision-Language Models? (MMStar)
  18. Liu et al. (2023). MMBench: Is Your Multi-modal Model an All-around Player?
  19. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena