Evaluating video understanding
A video score is only useful if the model needed the video to get it. This session shows how the three main kinds of video tests are scored (multiple-choice QA, temporal grounding and retrieval), how to check that a benchmark really needs the frames, and how to build and read a small test set for your own videos.
What you will learn
- How multiple-choice video QA is scored: accuracy, chance level, letter parsing and CircularEval, with the real designs of Video-MME and MVBench
- Why many video questions can be answered without the video, and the blind-baseline ladder (text only, one frame, shuffled frames, full video) that measures it, with published numbers
- Temporal grounding metrics: temporal IoU, R1@IoU=μ and mIoU, and how moment-length bias inflates them
- Retrieval metrics: R@k, median rank and mean rank from a similarity matrix, as reported on MSR-VTT
- How sure a score is: standard error, 95% intervals, sample size and paired tests between two models
- A step-by-step recipe for a domain video test set (sports, lectures, cameras) that resists the usual shortcuts
Why this matters
A Video-LLM gets 60% on a video benchmark. Is that good? It depends on a question most leaderboards do not answer: how much would it get without the video? On MVBench, Gemini 1.5 Pro scores 38.2% when it sees only the question and the options, and 60.5% with the video. Chance is 27.3%. So a large part of the score has nothing to do with seeing. [Certain] (numbers from Cores et al., Table 5)
In session 212 we built and trained video models. This session is about the last step before you trust one: measuring it. Video evaluation has three main families of tests, and each has its own metric:
- Multiple-choice video QA: the model reads a video, a question and 4 or 5 options, and picks a letter. The metric is accuracy. Video-MME and MVBench are the standard examples.
- Temporal grounding: the model gets a sentence (“the player shoots and scores”) and must return the start and end time of that moment. The metric is R1@IoU, the share of queries where the first answer overlaps the true moment enough.
- Retrieval: the model ranks many videos for a text query (or many captions for a video). The metric is R@\(k\), the share of queries where the right item is in the top \(k\).
Each family has its own trap. Multiple-choice questions can often be answered from the text alone. Grounding scores can be high when a model just predicts a “typical” moment. Retrieval numbers depend on the size of the gallery and the split. And all three are noisy when the test set is small. Then, at the end, you will want to know whether a model works on your videos: football matches, lectures, store cameras. No public benchmark answers that. You must build a small test set yourself, and do it so the shortcuts above do not fool you.
Multiple-choice video QA
Multiple choice is popular because scoring is simple and needs no human or LLM judge. The model sees the video, a question and \(K\) options, and must answer with one letter. Free-form answers (“describe what happens”) are harder to score: you need an LLM judge, and judges disagree with each other and change between versions. [Likely] For multiple choice, the metric is plain accuracy, and every benchmark also reports a chance level, the accuracy of random guessing:
- Read the symbols
- \(N\)
- number of questions in the test set
- \(\hat{y}_i, y_i\)
- the letter the model gave and the correct letter for question \(i\)
- \(\mathbb{1}[\cdot]\)
- 1 if the statement inside is true, else 0
- \(K_i\)
- number of options for question \(i\); chance for one question is \(1/K_i\)
With 4 options everywhere, chance is 25%. If the number of options varies, chance is the average of \(1/K_i\): with equal numbers of 3-, 4- and 5-option questions it is \((1/3 + 1/4 + 1/5)/3 = 26.1\%\). Always compare a score to chance, not to zero.
Video-MME and MVBench
The two benchmarks named in this session are built in very different ways, and they test different things.
| Video-MME (Fu et al., 2024) | MVBench (Li et al., 2023) | |
|---|---|---|
| Size | 900 videos, 254 hours, 2,700 questions (3 per video) | 20 tasks × 200 questions = 4,000 |
| Options | 4 (A to D) | 3 to 5, sampled per question; chance listed as 27.3% |
| Video length | short (under 2 min, mean 80.7 s), medium (4 to 15 min, mean 515.9 s), long (30 to 60 min, mean 2,466.7 s); 300 videos each | short clips; models are given 16 frames |
| How questions are made | written by human annotators across 6 domains and 30 sub-fields | generated automatically from existing video annotations, with tasks chosen so one frame should not be enough (“static-to-dynamic”) |
| Extra inputs | optional subtitles and audio | none |
| Answer parsing | prompt ends “Respond with only the letter (A, B, C, or D)”; the letter is taken with a regular expression | options in parentheses, answer prompt “Best Option: (” so the next token is the letter |
All values in the table are from the two papers. [Certain] Video-MME is the standard test for long video: it is the benchmark where frame count and token budget (session 206) matter. MVBench is a test of temporal skills on short clips: action order, counting, direction of motion, state change. The best result in the Video-MME paper is Gemini 1.5 Pro with 75.0% without subtitles and 81.3% with them. [Certain]
Getting a letter out of a chatty model
Accuracy looks trivial, but most errors in practice come from the plumbing. The model may write “The answer is (B) because…”, or “B. the man closes a door”, or refuse (“I cannot see the video”). Your parser decides what counts. The usual rules:
- Put the options in a fixed format and end the prompt with an instruction such as “Answer with the letter only.” or a prefix such as “Best Option: (”.
- Take the first standalone letter in the allowed set with a regular expression. If there is none, count the answer as wrong, never skip it. Skipping hides failures.
- Use greedy decoding (temperature 0) so the run is repeatable.
- Report the parse failure rate. A new model version that “gets worse” may only have changed its answer format.
Position bias and CircularEval
Models often prefer some letters: when unsure, they say “A”. If the correct answers in your set happen to be “A” more often than 1 in \(K\), that bias raises the score. MMBench introduced CircularEval to remove this effect. [Certain] Ask each question \(K\) times, each time with the options rotated by one position. The question counts as correct only if the model is right in all \(K\) rotations:
- Read the symbols
- \(r\)
- the rotation: the options are shifted by \(r\) positions, so the correct letter moves too
- \(\hat{y}_i^{(r)}, y_i^{(r)}\)
- the model’s letter and the correct letter in rotation \(r\)
- \(c_i\)
- 1 only if every rotation is answered correctly
A model that guesses at random now passes a 4-option question with probability \(0.25^4 = 0.4\%\), not 25%. CircularEval costs \(K\) times more inference, and its scores are lower than plain accuracy, so never compare a circular number with a plain one.
Three questions with 4 options. The model truly knows question 1. For the others it always answers “A”. The correct letters are B, A and C.
Plain accuracy: question 1 right, question 2 right by luck (the answer is A), question 3 wrong: \(2/3 = 66.7\%\).
CircularEval: question 1 stays right in all 4 rotations because the model finds the right option wherever it is. Question 2 is right only in the rotation where the answer is in position A, so it fails. Question 3 fails. Result: \(1/3 = 33.3\%\), which matches what the model really knows. The code later in this session prints exactly these two numbers.
- input: model, test set of (video, question, options, answer), frame budget \(F\), number of rotations \(R\) (1 for plain, \(K\) for circular)
- for each item do
- frames \(\leftarrow\) sample \(F\) frames uniformly // record \(F\) and the fps; scores depend on them
- for \(r = 0\) to \(R-1\) do
- options\(^{(r)}\) \(\leftarrow\) rotate options by \(r\); gold\(^{(r)}\) \(\leftarrow\) new letter of the answer
- text \(\leftarrow\) model(frames, question, options\(^{(r)}\)) with greedy decoding
- letter \(\leftarrow\) first allowed letter in text, else None // None counts as wrong
- if letter \(\ne\) gold\(^{(r)}\) then mark item wrong; break
- mark item right if no rotation failed
- return accuracy (213.1 or 213.2), its 95% interval (213.8), parse-failure rate, accuracy per duration and per task
Is the video needed? Blind baselines
The biggest problem in video evaluation is that many questions can be answered without watching. There are three reasons:
- The text gives it away. One option is longer, more specific, or the only one that makes sense with the question (“What does the chef do after cutting the onion?” with one cooking option).
- World knowledge is enough. “Which team wins this famous final?” The language model already knows.
- One frame is enough. “What color is the car?” needs vision, but not video: no motion, no order, no time.
Buch et al. showed this already in 2022: a model that looks at a single well-chosen frame (their “atemporal probe”) reached strong results on standard video QA and retrieval benchmarks. [Certain] Cores et al. (2024) measured it directly on MVBench with a ladder of conditions. Each step adds one kind of information:
- Read the symbols
- \(\text{Acc}_{\text{text}}\)
- accuracy when the model sees only the question and options (no frames). Also called the blind baseline
- \(\text{Acc}_{\text{shuffled}}\)
- accuracy when the same frames are given in random order
- \(\Delta_{\text{text}}\)
- how much the text alone gives above chance; it should be near 0 for a good benchmark
- \(\Delta_{\text{vision}}\)
- how much seeing adds
- \(\Delta_{\text{order}}\)
- how much the order of frames adds; a true temporal test needs this to be large
Figure 2 shows the published numbers. Choose a benchmark and a model.
How to read Figure 2
Look at the gap between the first two bars. On MVBench, text alone is 10.9 points above chance for Gemini: the questions leak answers. A single frame adds another 10.3 points, and shuffling the frames costs only 3.7 points against the full video, so most of MVBench is solvable without knowing the order of events. Now switch to TVBench: text alone is at chance (+0.3), one frame and shuffled frames are only about 3 points above chance, and reversed video falls below chance, likely because many options describe the same action in the opposite direction, so a reversed video points to the wrong one. [Likely] On TVBench almost the whole score comes from the video and its order. That is what a temporal benchmark should look like.
Long-video pitfalls
Long videos add more ways to get the right answer for the wrong reason:
- Subtitles carry the answer. In Video-MME, subtitles raise Gemini 1.5 Pro from 67.4% to 77.4% on long videos (+10.0 points), but only from 81.7% to 84.5% on short ones (+2.8). [Certain] For long videos, much of the information is in the speech, so a “long-video” score with subtitles is partly a reading test. Always report both settings.
- Too few frames to see anything. Several open models in the Video-MME paper were run with 8 frames. For a long video (mean 2,466.7 s) that is one frame every 308 seconds. If such a model still scores well above chance on long videos, the score likely comes from text and knowledge, not from the frames.
- Refusals look like low skill. Without frames, a model may refuse to answer. The Video-MME authors report that Gemini 1.5 Pro gets less than 15% in a text-only check [Certain], which is below the 25% chance level, so part of that number is refusals. When you run a blind baseline, force a letter (for example with the “Best Option: (” prefix), or the baseline looks better than the benchmark deserves.
- The video is famous. Videos from YouTube may be in the training data, with their titles, descriptions and comments. A model may “remember” the answer. This is data contamination, as in session 40.
Temporal grounding: R1@IoU
In temporal grounding (also called moment retrieval; see session 207), the model gets an untrimmed video and a sentence, and returns a time span \([\hat{s}, \hat{e}]\) in seconds. The true span \([s, e]\) was marked by an annotator. We compare the two spans with temporal intersection over union (tIoU), the 1-D version of the box IoU used in object detection:
- Read the symbols
- \(P = [\hat{s}, \hat{e}]\)
- the predicted span: predicted start and end time
- \(G = [s, e]\)
- the ground-truth span from the annotator
- \(|P \cap G|\)
- length of the overlap, 0 if the spans do not touch
- \(|P \cup G|\)
- length from the earliest start to the latest end (when the spans overlap this is the union; when they do not, the ratio is 0 anyway)
tIoU is 1 for a perfect span and 0 for no overlap. A test set has many queries, so we count how often the top-1 prediction is good enough:
- Read the symbols
- \(P_i^{(1)}\)
- the model’s first (highest-scored) span for query \(i\)
- \(\mu\)
- the IoU threshold; papers usually report \(\mu = 0.3, 0.5, 0.7\)
- R1@\(\mu\)
- “recall at 1 with IoU at least \(\mu\)”: the share of queries whose first span overlaps the truth enough. Written R1@0.5 or R@1, IoU=0.5
- mIoU
- mean tIoU over all queries, a softer number that has no threshold
Some systems return several spans; then R5@\(\mu\) checks whether any of the top 5 passes. This metric (“R@n, IoU=m”) was used with Charades-STA, introduced in the TALL paper by Gao et al. [Certain] QVHighlights reports R1@0.5 and R1@0.7 for moment retrieval as well. [Certain]
The true moment is \([10, 20]\) s; the model predicts \([12, 24]\) s. Overlap: \(\min(24, 20) - \max(12, 10) = 8\) s. Union: \(24 - 10 = 14\) s. tIoU \(= 8/14 = 0.571\). This counts as a hit at \(\mu = 0.5\), but a miss at \(\mu = 0.7\).
Now a set of six queries with tIoU values 0.571, 0.769, 0.438, 0, 0.914 and 0.538 (the defaults in Figure 3). R1@0.3 \(= 5/6 = 83.3\%\), R1@0.5 \(= 4/6 = 66.7\%\), R1@0.7 \(= 2/6 = 33.3\%\), and mIoU \(= 0.538\).
How to read Figure 3
Move the end of query 1 from 24 to 20: tIoU jumps to 0.8 and the query passes at 0.7. A shift of a few seconds changes the result at a strict threshold, which is why papers report several thresholds. Then turn on the blind prior and try a long span in the middle of the video, for example 8 to 26 seconds. Some queries still pass at 0.3, and mIoU is far from zero. A model that has learned only “moments are usually in the middle and about this long” can score this way. Yuan et al. showed that such biases let simple baselines without training reach strong results on the standard splits. [Certain]
Two more facts help read grounding papers. First, the threshold matters: R1@0.3 and R1@0.7 can differ by 30 points or more for the same model, so compare the same \(\mu\). Second, annotators themselves disagree about where a moment starts and ends, often by a few seconds. A tIoU of 1 is not achievable in practice, and on your own data you should measure the tIoU between two annotators as a ceiling. To fight the moment-length bias, Yuan et al. re-split Charades-STA and ActivityNet Captions so that moment positions differ between training and test [Certain], and proposed a discounted metric (dR@n,IoU@m) that lowers the credit of a hit when its start and end are far from the true ones. [Likely]
Retrieval: R@k and median rank
In text-to-video retrieval (session 208), the model embeds every caption and every video, and ranks the videos for each caption by cosine similarity. With \(N\) caption-video pairs we get an \(N \times N\) similarity matrix \(S\), where row \(i\) is caption \(i\) and the correct video is in column \(i\). The rank of the correct video is one plus the number of videos that score higher:
- Read the symbols
- \(S_{ij}\)
- similarity between caption \(i\) and video \(j\), for example the cosine of their embeddings
- \(S_{ii}\)
- similarity with the correct video (the diagonal)
- R@\(k\)
- recall at \(k\): share of queries whose correct video is in the top \(k\); higher is better
- MdR, MnR
- median and mean rank of the correct video; lower is better, and 1 is perfect
Video-to-text retrieval uses the same formulas on the columns of \(S\) (each video ranks the captions). The standard test is MSR-VTT: 10,000 clips with 200,000 captions, evaluated on the “1k-A” split of 1,000 clip-caption pairs. CLIP4Clip (mean pooling of CLIP frame features, 9k training videos) reports text-to-video R@1 = 43.1, R@5 = 70.4, R@10 = 80.8 and MdR = 2 there. [Certain] So at least half of the queries find the correct video among 1,000 in the top 2.
How to read Figure 4
For a query, every cell with a higher score than the diagonal cell pushes the correct item down one place. Caption 3 has one video above its own (0.56 against 0.51), so its rank is 2. Caption 6 is the worst: its correct video has the lowest score in the row, rank 6. Move \(k\) and watch R@\(k\) count the queries with rank at most \(k\). Switch to video-to-text: the same matrix gives different ranks, because now each column is ranked. Papers report both directions.
Text-to-video ranks are 1, 1, 2, 1, 2, 6. R@1 \(= 3/6 = 50\%\), R@2 \(= 5/6 = 83.3\%\), R@5 \(= 5/6 = 83.3\%\). Sorted ranks are 1, 1, 1, 2, 2, 6, so MdR \(= (1 + 2)/2 = 1.5\) and MnR \(= 13/6 = 2.17\). Video-to-text ranks are 1, 1, 1, 1, 1, 4: R@1 \(= 5/6 = 83.3\%\). The same model looks much better in one direction.
Three warnings. Gallery size: R@1 among 1,000 videos is much harder than among 100; numbers are comparable only on the same split. Many correct answers: MSR-VTT has 20 captions per clip and many clips look alike (“a man talks about cars”), so a “wrong” top-1 is sometimes a correct video with a different id. Test-time tricks: some methods normalize the whole test similarity matrix at once (for example a dual softmax over all queries), which uses information from other test queries and is not possible for a single live query. [Likely] Read the evaluation section before you compare.
How sure is a score?
A benchmark score is an estimate from a finite sample of questions. If each answer is right with probability \(p\), the accuracy on \(N\) independent questions has a standard error (SE), and an approximate 95% confidence interval:
Turn it around to find how many questions you need for a margin \(m\) (half the width of the interval):
- Read the symbols
- \(p, \hat{p}\)
- the true accuracy, and the accuracy you measured
- \(N\)
- number of independent questions
- 1.96
- the number of standard errors that covers 95% of a normal distribution
- \(m\)
- the margin you want, for example 0.03 for ±3 points
Questions about the same video are not independent (Video-MME has 3 per video), so the true uncertainty is somewhat larger. A bootstrap that resamples videos rather than questions handles this. [Likely]
When you compare two models on the same questions, use a paired test, which is much more sensitive. Count \(b\), the questions model A gets right and B gets wrong, and \(c\), the reverse. The questions both get right or both get wrong say nothing about the difference. McNemar’s test uses
and \(|z| \gt 1.96\) means the difference is significant at the 5% level (for \(b + c\) of about 25 or more; use an exact binomial test for fewer).
Video-MME has 2,700 questions. At \(p = 0.75\), SE \(= \sqrt{0.75 \cdot 0.25 / 2700} = 0.0083\), so the interval is ±1.63 points. One duration group has 900 questions: ±2.83 points. A domain test set of 200 questions at \(p = 0.6\): ±6.79 points.
Model A scores 62% and model B 65% on 900 questions. Treated as independent samples, the difference has SE \(\sqrt{(0.62 \cdot 0.38 + 0.65 \cdot 0.35)/900} = 0.0227\), so \(z = 0.03/0.0227 = 1.32\): not significant. You would need about 1,977 questions for this gap to reach \(z = 1.96\). But if the same 900 questions give \(b = 60\) and \(c = 35\) (so 25 more right for B), the paired test gives \(z = (60 - 35)/\sqrt{95} = 2.56\): significant. Same data, a better test.
For a margin of ±3 points at \(p = 0.7\), formula (213.9) asks for 897 questions; at ±5 points, 323.
How to read Figure 5
At the default (900 questions, 62% against 65%) the intervals overlap and \(z = 1.32\). Slide \(N\) to 2,000 and \(z\) passes 1.96. Note the shape: halving the interval needs four times more questions, because SE falls as \(1/\sqrt{N}\). Small domain test sets can rank models only when the gap is large, unless you use a paired test.
Building a domain video test set
Public benchmarks tell you about general skill. Your product needs a different answer: does this model find goals in our football broadcasts, or the right slide in our lectures? The only way to know is a test set made from your own videos. It does not need to be big; it needs to be honest. Here is a recipe that avoids the traps in this session.
- Define the task as the product sees it: input video length, question types, output (letter, span, ranking), and the metric (213.1, 213.5 or 213.6).
- Sample videos the way production will see them (cameras, lighting, length), and keep them out of all training and prompt-tuning data. Split by video, never by clip, or near-duplicates leak (see session 209).
- Choose \(N\) with formula (213.9): for ±5 points about 300 to 400 questions; for ±3 points about 900 to 1,100. Plan per subgroup if you will report per subgroup.
- Write questions that need time: order (“before or after”), counting, change, cause, and moments that are short. Add distractor options that are plausible from the text alone.
- Annotate twice. Two people answer each question (or mark each span). Measure agreement (accuracy between annotators, or tIoU between their spans). Fix or drop items where they disagree. Human agreement is your ceiling.
- Balance the answer positions so each letter is correct about \(1/K\) of the time, or use CircularEval (213.2).
- Run the blind ladder (213.3) with any strong model: text only, one frame, shuffled frames. Drop or rewrite items that text alone solves. For grounding, run the blind-prior span of Figure 3.
- Freeze the set, the prompt, the frame budget and the parser. Store a version id. Never tune on it; keep a separate development set for that.
- Report accuracy with its interval, the blind baselines, parse failures, and results per subgroup (duration, question type). Compare models with a paired test (213.10).
The code
All four metrics in one runnable, deterministic file: letter parsing with CircularEval, tIoU with R1@\(\mu\), retrieval ranks with R@\(k\) and median rank, and an interval checked by a bootstrap. The numbers it prints are the ones in the worked examples.
import re, torch torch.manual_seed(0) # 1) multiple-choice: parse the letter, score accuracy and CircularEval, formulas (213.1)-(213.2) def parse_letter(text, k): letters = "ABCDE"[:k] m = re.search(r"\(?\b([" + letters + r"])\b\)?", text.strip()) return m.group(1) if m else None # None counts as wrong def circular_eval(model, question, options, answer): k = len(options) for r in range(k): # rotate the options k times opts = options[r:] + options[:r] gold = "ABCDE"[opts.index(answer)] if parse_letter(model(question, opts), k) != gold: return 0 return 1 def biased_model(question, opts): # toy model: knows q0; otherwise always says "A" return "ABCD"[opts.index("red")] if question == "q0" else "(A)" qs = [("q0", ["blue", "red", "green", "black"], "red"), ("q1", ["sit", "run", "jump", "fall"], "sit"), ("q2", ["2", "3", "4", "5"], "4")] plain = sum(parse_letter(biased_model(q, o), 4) == "ABCD"[o.index(a)] for q, o, a in qs) / len(qs) circ = sum(circular_eval(biased_model, q, o, a) for q, o, a in qs) / len(qs) print(f"plain acc {plain:.3f} circular acc {circ:.3f}") # 2) temporal grounding: tIoU and R1@mu, formulas (213.4)-(213.5) def tiou(p, g): # p, g: (N, 2) tensors of [start, end] in seconds inter = (torch.minimum(p[:, 1], g[:, 1]) - torch.maximum(p[:, 0], g[:, 0])).clamp(min=0) union = torch.maximum(p[:, 1], g[:, 1]) - torch.minimum(p[:, 0], g[:, 0]) return inter / union gt = torch.tensor([[10, 20], [3, 9], [20, 31], [5, 12], [0, 6.4], [15, 22]]) pred = torch.tensor([[12, 24], [2.5, 8], [24, 36], [14, 18], [0, 7], [13, 26]]) iou = tiou(pred, gt) print("tIoU", [round(x, 3) for x in iou.tolist()]) for mu in (0.3, 0.5, 0.7): print(f"R1@{mu}: {(iou >= mu).float().mean().item():.3f}") print(f"mIoU: {iou.mean().item():.3f}") # 3) retrieval: R@k and median rank from a similarity matrix, formulas (213.6)-(213.7) S = torch.tensor([[0.82, 0.41, 0.30, 0.22, 0.35, 0.18], [0.37, 0.76, 0.29, 0.44, 0.21, 0.33], [0.48, 0.25, 0.51, 0.56, 0.31, 0.27], [0.20, 0.39, 0.34, 0.71, 0.28, 0.45], [0.43, 0.30, 0.22, 0.36, 0.47, 0.52], [0.40, 0.46, 0.38, 0.42, 0.44, 0.29]]) # row = text query, column = video def ranks(S): # rank of the correct item (the diagonal), 1 = best return 1 + (S > S.diag().unsqueeze(1)).sum(dim=1) r = ranks(S) print("t2v ranks", r.tolist(), "R@1", (r <= 1).float().mean().item(), "R@5", round((r <= 5).float().mean().item(), 3), "MdR", r.float().quantile(0.5).item()) # not .median(): see Common mistakes print("v2t ranks", ranks(S.T).tolist()) # 4) uncertainty: normal-approximation 95% interval (213.8) and a bootstrap check correct = (torch.rand(900) < 0.62).float() # 900 simulated answers p = correct.mean().item(); half = 1.96 * (p * (1 - p) / 900) ** 0.5 idx = torch.randint(0, 900, (2000, 900)) boot = correct[idx].mean(dim=1) lo, hi = boot.quantile(0.025).item(), boot.quantile(0.975).item() print(f"acc {p:.3f} +- {half:.3f} bootstrap [{lo:.3f}, {hi:.3f}]")
Output (PyTorch 2.8, CPU):
plain acc 0.667 circular acc 0.333 tIoU [0.571, 0.769, 0.438, 0.0, 0.914, 0.538] R1@0.3: 0.833 R1@0.5: 0.667 R1@0.7: 0.333 mIoU: 0.538 t2v ranks [1, 1, 2, 1, 2, 6] R@1 0.5 R@5 0.833 MdR 1.5 v2t ranks [1, 1, 1, 1, 1, 4] acc 0.626 +- 0.032 bootstrap [0.593, 0.658]
The normal formula (±0.032) and the bootstrap (0.593 to 0.658, about ±0.033) agree. Note that the rank code counts only strictly higher scores, so ties are resolved in the model’s favor. With real float embeddings ties are rare, but a model that outputs the same vector for every video would get rank 1 for all queries. Break ties pessimistically (count \(S_{ij} \ge S_{ii}\) for \(j \ne i\)) if that can happen.
Where this is used
| Setting | Test and metric | The trap to check |
|---|---|---|
| Choosing a Video-LLM for a chat or QA product | Video-MME (with and without subtitles), MVBench; accuracy per duration | Blind baseline; the frame budget each model was run with |
| Long-video QA and Video-RAG (session 210) | Video-MME long split, your own long-video set; accuracy | Subtitle-only baseline; answers from world knowledge |
| Key-moment detection in sports (goals, fouls) | Grounding: R1@0.5, R1@0.7, mIoU on your matches | Blind-prior span; annotator tIoU as the ceiling |
| Video search in a media archive | Text-to-video R@1, R@5, R@10, MdR on a held-out gallery of realistic size | Gallery size; several correct videos per query |
| Action recognition (session 203) | Top-1 and top-5 accuracy (the classification form of R@\(k\)) | One frame is often enough on scene-heavy datasets; check a shuffled-frame run |
| Near-duplicate detection and fingerprinting | Retrieval metrics (precision and recall at a threshold, average precision) | Train-test leakage of copies of the same video |
| Regression tests after quantizing or distilling a video model | Same benchmark before and after; paired test (213.10) | Small differences inside the interval; parse-format changes |
Common mistakes
- No blind baseline. Reporting 60% without the text-only score. On MVBench, text alone already gives 38.2% for Gemini 1.5 Pro.
- Comparing models run with different frame budgets. 8 frames and 64 frames are different experiments. Report frames and fps with every score.
- Dropping unparsable answers. Count them as wrong and report the rate; otherwise a model that refuses hard questions looks better.
- Mixing settings. Video-MME with subtitles and without subtitles, CircularEval and plain accuracy, R1@0.5 and R1@0.7: each pair is a different number.
- Using
torch.medianfor median rank. For an even number of values it returns the lower of the two middle values (1 instead of 1.5 in our example). Usequantile(0.5)or NumPy’s median. - Splitting a domain set by clip instead of by video. Clips from the same match land in both train and test, and the score is inflated.
- Declaring a winner inside the noise. A 2-point gain on 300 questions is well inside ±5 points. Use intervals and a paired test.
- Tuning prompts on the test set. After 20 prompt variants, the best one is partly luck. Tune on a development set; run the test set once.
Interview questions
How is a multiple-choice video benchmark like Video-MME scored, and what number must you always report next to it?basic
A model gets 70% on a video QA benchmark. How do you show that it actually uses the video?core
What is CircularEval and why is it used?core
Define R1@IoU=0.5 for temporal grounding. Compute tIoU for prediction [12, 24] s and truth [10, 20] s.basic
Why can a grounding model with no understanding of the query still score well on Charades-STA-like data?advanced
What do R@1, R@5 and MdR mean on MSR-VTT, and why is the split important?basic
How many questions do you need in a test set to measure accuracy within ±3 points?core
Two models: A is right and B wrong on 60 questions; B is right and A wrong on 35. Is the difference significant?core
Video-MME scores rise a lot with subtitles on long videos. What does that tell you?advanced
You must evaluate a Video-LLM for detecting key moments in your company’s football broadcasts. Describe the test set.advanced
Why is free-form video QA scored by an LLM judge less reliable than multiple choice?core
Exercises
- A benchmark has equal numbers of 3-, 4- and 5-option questions. What is the chance accuracy?
- Compute tIoU for prediction \([8, 22]\) s and truth \([10, 20]\) s. Does it pass at \(\mu = 0.7\)?
- How many questions do you need for ±2 points at \(p = 0.8\)?
- Five text queries have ranks 3, 1, 1, 7 and 2 for their correct video. Give R@1, R@5 and MdR.
- In Figure 2 (MVBench, Tarsier-34B), compute \(\Delta_{\text{text}}\), \(\Delta_{\text{vision}}\) and \(\Delta_{\text{order}}\). What do you conclude?
- Your domain set has 200 questions and model accuracy is 60%. What is the 95% interval? Is a new model at 64% clearly better?
Answerscheck
Key takeaways
- Video tests come in three families: multiple-choice QA (accuracy, 213.1), temporal grounding (tIoU and R1@\(\mu\), 213.4 and 213.5) and retrieval (R@\(k\) and median rank, 213.6 and 213.7).
- Video-MME tests long video (900 videos up to 1 hour, 2,700 four-option questions, optional subtitles); MVBench tests temporal skills on short clips (20 tasks × 200 questions).
- Always run blind baselines (213.3): on MVBench text alone gives 38.2% for Gemini 1.5 Pro against 27.3% chance, and shuffled frames lose only 3.7 points. TVBench was built so that only the ordered video helps.
- Parse letters strictly, count failures as wrong, and remove position bias with CircularEval (213.2). Report frames, fps and subtitle settings.
- Scores are noisy: ±1.63 points on 2,700 questions, ±6.8 on 200. Size test sets with (213.9) and compare models with a paired test (213.10).
- A domain test set needs held-out videos, double annotation, balanced answers, a blind-baseline filter, and a frozen protocol.
Key formulas
References for this session
- Fu et al. (2024). Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- Li et al. (2023). MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
- Fu et al. (2024). Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- Li et al. (2023). MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
- Cores, Dorkenwald, Mucientes, Snoek & Asano (2024). Lost in Time: A New Temporal Benchmark for VideoLLMs
- Buch, Eyzaguirre, Gaidon, Wu, Fei-Fei & Niebles (2022). Revisiting the "Video" in Video-Language Understanding
- Liu et al. (2023). MMBench: Is Your Multi-modal Model an All-around Player?
- Gao et al. (2017). TALL: Temporal Activity Localization via Language Query
- Yuan, Lan, Wang, Chen, Wang & Zhu (2021). A Closer Look at Temporal Sentence Grounding in Videos: Dataset and Metric
- Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
- Xu et al. (2016). MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models