LLM & VLM courseBlog
237Track 10: Capstone and interviews · session 4 of 6

capstone evaluation and failure analysis

This lab evaluates the sports key-moment search system from the last two sessions on a labeled test set you can run on a laptop. You compute tIoU, AP and recall stage by stage on real ranked lists, find out why each failure happens, discover that many “model errors” are label errors, decide with a bootstrap whether a fix is real, test whether a VLM judge can replace labels, and write a one-page report.

Time: 4 to 5 hours (mostly hands-on)Level: Advanced · hands-on labBefore this: Capstone design (235) and capstone build (236); evaluating video understanding; LLM-as-a-judge basics

What you will learn

  • How to build a labeled test set with exact moment spans, and why you split it by video into dev and test
  • How tIoU, matching, P@k, R@k, AP and mAP are computed on a ranked list of video spans, step by step
  • How to attribute every missed moment to a stage (retrieval, ranking, boundary) and read end-to-end recall as a product of stage rates
  • How to do error analysis: read each failure, tag a cause, count, and fix the largest cheap cause first
  • Why label errors look like model errors, and how to fix labels with a written rule applied to every video
  • How a paired bootstrap over queries tells you whether a gain is real, and why 12 queries are not enough
  • How to measure a VLM judge with Cohen’s kappa, TPR and FPR, why a system must not judge itself, and how to write the final report

Why this matters

In session 235 you designed a search system for sports video on paper. In session 236 you built it: an index of 2-s segments, a VLM re-ranker fused with the first-stage score, and a timestamp refiner, tested on a slideshow archive with exact labels. Now comes the question every reviewer, manager and interviewer will ask: does it work, and how do you know? This lab answers it with numbers. You will label a test set, measure retrieval and grounding stage by stage, find out why each failure happens, and write a one-page technical report.

Evaluation is where most real projects go wrong, and in quiet ways. A number can look good because the test set is too easy, too small, or labeled by the system itself. A fix can look large because the labels were wrong in the same place. In this lab every one of these traps happens to us on real data, and we show how to catch it:

  • The first test result is much worse than the development (dev) result: dev mAP 0.826, test mAP 0.554.
  • Error analysis shows that 13 of the 22 missed moments are label errors, not model errors.
  • After we fix the labels with a written rule, a tuning change that looked like a sure gain of +0.153 mAP becomes +0.062, with a 95% interval that includes zero.
  • A small VLM used as a judge agrees with the labels on 92% of frames, yet its agreement beyond chance is weak (\(\kappa = 0.29\)). The re-ranker judging its own top-10 says 66% of it is right; the labels say 42%.

The method is general. The same steps evaluate a RAG system (session 155), a VLM on a benchmark (session 192) or a video model (session 213).

Setup: a labeled test bed you can run on a laptop

A real evaluation of the design in session 235 needs hundreds of hours of football and editors who label moments. For a lab we need something smaller that keeps the important property: we know the exact true moments. So we build a small archive of “broadcasts” from a public photo dataset with human captions.

The archive. Flickr8k has about 8,000 photos, each with 5 captions written by different people [Certain]. We use its test and validation splits (2,000 photos). Each “video” is a 120-second slideshow: a sequence of photos, each held for 2 to 6 seconds like a camera shot. One frame per second is one photo, so a second of video is a photo. We made 48 videos (96 minutes, 1,483 shots).

The queries and the moments. We use 12 sports queries, for example “a surfer riding a wave” and “runners in a race”. A photo counts as showing a query if at least 3 of its 5 captions match a word pattern (for “runners in a race”: race, racing, marathon or runner). Into each video we place 1 or 2 moments: 1 to 4 consecutive shots of matching photos. That gives 92 labeled moments of 2 to 19 seconds (mean 9.4). Between moments we place background shots, chosen so that the task is not too easy: 13% of them are borderline photos (1 or 2 captions mention a query) and 40% are look-alike scenes (they share context words such as snow, water or field).

The split. Videos 0–15 are the dev split (31 moments): we may look at them as often as we like and tune on them. Videos 16–47 are the test split (61 moments): we run it only to report. We split by video, not by moment, so no shot or neighbor of a test moment is ever seen during tuning.

The system under test. We evaluate a smaller and partly different configuration than the one built in session 236. This lab was prepared in parallel with session 236, on its own archive, so that every intermediate output (the caption votes of every photo, the VLM score of every photo and query) is saved for error analysis, and it keeps the design’s threshold refiner (235.10) because its single knob \(\alpha\) makes the tuning step easy to follow. The embedding model and the small re-ranker are the same as in session 236; the archive, the segments, the fusion and the refiner differ. So the numbers on this page are not the numbers of session 236, but every method here applies unchanged to its outputs.

PartEvaluated in this labBuilt in session 236
Test archive48 slideshow videos of 120 s from Flickr8k (dev 16, test 32), 12 queries, 92 moments60 slideshow videos of 180 s from Flickr8k (dev 20, test 40), 14 queries, 84 test moments
Frame embeddingSigLIP-B/16 at 224 px, one frame per second, unit-length vectorssame
Segments and index8-s windows, 4-s stride, mean-pooled: 29 per video, 1,392 in total; exact search2-s segments, no overlap, flat FP16 index (formula 236.1)
Stage 1 searchcosine with the text vector of “a photo of query.”; top \(K = 50\) windows in the splitsame prompt; top 100 segments, touching hits grouped into moments, score = max (236.2)
VLM re-rankSmolVLM-500M-Instruct, “Does this image show query? Answer yes or no.”; log-odds \(\text{logit(Yes)} - \text{logit(No)}\) for every photo in the window, keep the maximum; final score = log-odds + z-scored cosineQwen2.5-VL-3B (or SmolVLM-500M); 3 keyframes per moment, mean log-odds over yes/Yes and no/No (236.4); \(M = 30\) candidates; fused as \(w \cdot\) z(cosine) + \((1-w) \cdot\) z(log-odds), \(w = 0.75\) (236.5)
Timestampsthreshold rule (235.10): grow from the peak second while the per-second cosine is at least \(\alpha \times\) peak, within \(\pm 6\) s of the window; \(\alpha = 0.5\) at firstcontrast search (236.6) within \(\pm 2\) s of the segment boundaries
Outputdrop a result that overlaps a better one in the same video at tIoU above 0.3; top \(N = 10\)drop overlaps at tIoU \(\ge 0.5\); top 10

On this page, “as built” means our configuration with its first settings (\(\alpha = 0.5\)), before any tuning; “tuned” is the same configuration after Step 4.

The VLM scored 2,285 distinct (photo, query) pairs: every photo inside a top-50 window of the dev or test split. All embeddings and VLM scores are computed once and saved, so every metric on this page is recomputed from files in seconds. That is the first engineering rule of evaluation: cache model outputs, recompute metrics.

Algorithm 1: build the labeled test bed
  1. input: photos with 5 captions each; 12 queries, each with a word pattern
  2. for each photo and query: votes \(\leftarrow\) number of captions that match the pattern
  3. positives(\(q\)) \(\leftarrow\) photos with votes \(\ge 3\); borderline \(\leftarrow\) photos with 1–2 votes for some query
  4. look-alike \(\leftarrow\) photos with 0 votes that share at least 2 context words with a query; easy \(\leftarrow\) the rest
  5. for each of 48 videos (seed 7): pick 1–2 queries and start times at least 40 s apart
  6.   fill 120 s with shots of 2–6 s: background shots from borderline (12%), look-alike (40%), easy (48%); at a start time insert 1–4 shots of positives(\(q\)) and record the moment \([s, e, q]\)
  7. never reuse a photo while unused ones remain; split videos 0–15 = dev, 16–47 = test
  8. return videos, moments, and the votes of every photo (we need them for error analysis)
shot of this query’s momentanother query’s momentborderline photolook-alike sceneunrelatedlabeled momentwindow and refined spancosine per second
Figure 1. One labeled moment of the test split and what the as-built system returns in that video. Top: the shots, colored by what their photo shows; the labeled moment with the caption rule (GT v1, solid) and after the label audit of Step 4 (GT v2, dashed). Middle: the SigLIP cosine between the query and each second. Bottom: for each top-10 window in this video, the refined span; the dashed line is the threshold \(\alpha \times\) peak. All values are real outputs of the lab.
How to read Figure 1

Start with “runners in a race”, moment 1 (video v16): there is no prediction at all, and the captions in the readout show why: a motorbike, two racing dogs and a rally car. The labels call this “runners in a race”. Then pick “surfer” and step to video v47: the moment has three shots, and the middle one (a weak photo, cosine 0.045) splits the predicted span in two. Move \(\alpha\) down to 0.2: the span grows over the whole moment and the tIoU jumps. This is the main knob we tune in Step 4.

Step 1: the metrics, computed by hand

A search result is a span of video: a video id, a start and an end. To decide if it is correct, we compare it with the labeled moments of the same query. Three ideas are needed: overlap, matching, and a score for a ranked list.

Overlap: temporal IoU

\[ \text{tIoU}(\hat m, m) = \frac{\max\!\big(0,\ \min(\hat b, b) - \max(\hat a, a)\big)}{\max(\hat b, b) - \min(\hat a, a)} \tag{237.1} \]

Here \(\hat m = [\hat a, \hat b]\) is a predicted span and \(m = [a, b]\) a labeled moment in the same video. It is the same formula as (235.2). tIoU is 1 for a perfect span and 0 for no overlap.

Matching: which result found which moment

A query has several labeled moments in the test videos (on average 5). We walk down the ranked list. Each result is matched to the unused moment in the same video with the highest tIoU; if that tIoU is at least a threshold \(\theta\), the result is a true positive (TP), and the moment is used up. Otherwise it is a false positive (FP). A second result on a moment that is already used is a FP too (a duplicate).

\[ \text{rel}_k = \begin{cases} 1 & \text{if } \max_{m \in \mathcal{G}_{\text{free}}} \text{tIoU}(\hat m_k, m) \ge \theta \\ 0 & \text{otherwise} \end{cases} \tag{237.2} \]

This is the rule used by object detection and by temporal-localization benchmarks such as ActivityNet [Likely]. Session 236 reports a stricter variant: precision interpolated over later ranks and mAP averaged over \(\theta = 0.5, 0.55, \dots, 0.95\) (236.3). Here we use one \(\theta\) at a time and no interpolation, so that every step can be checked by hand. We use \(\theta = 0.5\), the threshold in the design’s requirements, and also report \(\theta = 0.7\), which asks for tighter spans.

Scores for a ranked list

\[ P@k = \frac{1}{k}\sum_{i=1}^{k} \text{rel}_i, \qquad R@k = \frac{1}{G}\sum_{i=1}^{k} \text{rel}_i \tag{237.3} \]
\[ \text{AP@}N = \frac{1}{G} \sum_{k=1}^{N} P@k \cdot \text{rel}_k, \qquad \text{mAP} = \frac{1}{|Q|}\sum_{q \in Q} \text{AP}_q \tag{237.4} \]
Read the symbols
\(\text{rel}_k\)
1 if the result at rank \(k\) is a true positive, else 0
\(G\)
number of labeled moments for the query in the split (all of them, also those the system never returned)
\(N\)
length of the returned list; 10 here
\(P@k\), \(R@k\)
precision and recall of the first \(k\) results
\(Q\)
the set of queries; mAP gives every query the same weight

AP rewards putting hits early: each hit adds the precision at its rank. Dividing by \(G\), not by the number of hits, means that a moment the system never returns costs as much as a hit at a very low rank. Because hits after rank 10 are not seen, AP@10 is at most \(R@10\).

Worked example: AP for one query

A query has \(G = 3\) moments: v1 [20, 30], v2 [50, 56], v3 [8, 20]. The system returns: (1) v1 [21, 29], (2) v4 [60, 68], (3) v1 [22, 30], (4) v2 [51, 57], (5) v3 [10, 19].

Rank 1: tIoU with v1 [20, 30] is \(8/10 = 0.80\): TP. Rank 2: no moment in v4: FP. Rank 3: overlaps v1, but that moment is used: duplicate, FP. Rank 4: \(5/7 = 0.71\): TP. Rank 5: \(9/12 = 0.75\): TP.

\(\text{rel} = [1, 0, 0, 1, 1]\), \(P@5 = 3/5 = 0.6\), \(R@5 = 3/3 = 1.0\), and \(\text{AP} = \frac{1}{3}\big(\frac{1}{1} + \frac{2}{4} + \frac{3}{5}\big) = \frac{2.1}{3} = 0.70\). Move the duplicate below rank 5 and AP becomes \(\frac{1}{3}(1 + \frac{2}{3} + \frac{3}{4}) = 0.81\): the duplicate cost 0.11.

\(\theta\):
rankvideospan (s)best tIoUmatchP@kR@kAP term
Figure 2. Average precision of one query, computed step by step on the real top-10 list of the system on the test split. Each step reveals one rank: its best tIoU against the free labeled moments, TP or FP by (237.2), and the running \(P@k\), \(R@k\) and AP term (237.3, 237.4). The plot shows precision against recall; the shaded area is the AP.
How to read Figure 2

Pick “soccer” with the as-built system and labels v1 and press Play. Rank 1 is a hit, but rank 2 (v44, tIoU 0.25) and rank 4 (v29, tIoU 0.40) land on real moments with spans cut too short, so they count as FPs and the moments stay unfound. AP is 0.250. Switch to “Tuned”: the same moments are now hits and AP becomes 0.688. Change \(\theta\) to 0.3 and to 0.7 to see how much of the score depends on the threshold. Try “runners” with labels v1 and v2: the labels decide the score as much as the system does.

Step 2: first results, stage by stage

We run the as-built system on dev, then once on test. To see what each stage adds, we also run reduced versions of it (an ablation): stage 1 alone, stage 1 with refinement, and stage 1 with the VLM but without refinement.

System (labels v1)dev mAP@0.5test mAP@0.5test R@10test P@10test mAP@0.7
Stage 1 only (8-s windows)0.7470.6380.6890.3500.089
Stage 1 + VLM re-rank0.7720.6230.7210.3670.099
As built: VLM + refine (\(\alpha = 0.5\))0.8260.5540.6390.3250.477

Three things stand out. First, the as-built system is the best on dev and the worst on test. Second, the dev numbers are higher everywhere: with 16 videos, dev is small and, by chance, easier. Third, refinement helps a lot at \(\theta = 0.7\) (0.099 to 0.477): fixed 8-s windows almost never match a moment tightly. So refinement is necessary, but something in it fails on test. The end-to-end number cannot tell us what. The stages can.

Where is each moment lost?

The design session wrote end-to-end success as a product of stage success rates (235.13). With labels we can measure each factor. For every labeled moment we ask, in order: is it covered by one of the top-50 windows of stage 1 (covered means the overlap is at least half of the shorter of the two)? Is it covered by a window that made the final top 10? Is it matched with tIoU \(\ge \theta\)?

\[ R@10 = \underbrace{\frac{n_{50}}{G}}_{\text{stage 1: R@50}} \times \underbrace{\frac{n_{10}}{n_{50}}}_{\text{re-rank}} \times \underbrace{\frac{n_{\text{found}}}{n_{10}}}_{\text{boundaries}} \tag{237.5} \]
Read the symbols
\(n_{50}\)
moments covered by some top-50 stage-1 window
\(n_{10}\)
moments covered by the window of some final top-10 result
\(n_{\text{found}}\)
moments matched with tIoU \(\ge \theta\) (237.2)

Each missed moment gets exactly one label: retrieval (not in the top 50), ranking (in the top 50 but not the top 10), or boundary (in the top 10, but the span is wrong). For the as-built system on test: 39 found, 12 boundary, 2 ranking, 8 retrieval. So \(n_{50} = 61 - 8 = 53\), \(n_{10} = 53 - 2 = 51\) and \(n_{\text{found}} = 39\). The factors are \(53/61 = 0.869\), \(51/53 = 0.962\) and \(39/51 = 0.765\), and their product is \(39/61 = 0.639\), the R@10 in the table. The weakest factor is the boundaries.

The design asked for Hit@10 \(\ge 0.85\) (a moment is in the top 10) and tIoU \(\ge 0.5\) for 85% of the hits. As built, on test: Hit@10 = \(51/61 = 0.836\) and \(39/51 = 0.765\). Both requirements fail.

foundboundaryrankingretrieval
Figure 3. Stage attribution for five configurations. Each bar splits the labeled moments by the stage where they are lost; the right column gives mAP@0.5 and, on test, its 95% bootstrap interval over the 12 queries (Step 5). Click a row to see its factors in (237.5).
How to read Figure 3

On test with labels v1, compare “Stage 1 only” with “As built”: the red part (retrieval) does not change, because re-ranking and refinement never see moments outside the top 50. The orange part (boundary) grows from 4 to 12: refinement with \(\alpha = 0.5\) breaks spans. Switch to Dev: there are no retrieval or ranking misses at all, so dev could never have taught us about them. Then switch the labels to v2 and look at retrieval again: 8 becomes 2.

Step 3: error analysis: group the failures by cause

Stage attribution tells us where a moment is lost. Error analysis asks why. The method is simple and slow: open every failure, look at the frames and the labels, and write down a cause. Then count the causes. You fix the biggest group first. With 22 missed moments and 81 false positives on test, this takes about an hour, and it is the most useful hour of the project.

Missed moments

For each of the 22 missed moments we read the 5 captions of every photo in it. The causes fall into four groups:

CauseCountExampleWhose fault
Label too broad: the word rule labeled photos that do not show the query13“runners in a race”: a motorbike race, racing dogs, a rally car; “tennis player”: a dog with a tennis ballthe labels
Span too short: found, but the refined span covers only part of the moment6a 3-shot surf moment whose middle shot has a low score, so the span splitsrefinement (\(\alpha\))
Short moment diluted: a 2-s moment in an 8-s window2a skateboarder grinding a bench (cosine 0.109, high) is averaged with 6 s of other shotswindow size
Re-ranker said no: in the top 50, pushed out by the VLM1“a boy hitting a soccer ball with his chest”: VLM log-odds −4.8the VLM

More than half of the misses (13 of 22, 59%) are not errors of the system. The word pattern “race” matches dog races and car races; “tennis” matches tennis balls; “climb” matches ladders and trees. The system was right to skip these photos. This is common in practice: labels made by a rule, by a weaker model or in a hurry carry errors, and the errors sit exactly where the task is hard, so they look like model failures.

False positives

Not all false positives cost the same. AP only charges for a FP that is ranked above a hit; a FP below the last hit does not change AP (it changes P@10). Of the 81 FPs, 28 rank above some hit. We grouped them by a rule that uses the labels and captions of the evidence second (the second with the highest cosine in the span):

FP group (in this order)All 81The 28 that cost AP
boundary: overlaps an unfound moment, tIoU below \(\theta\)1310
duplicate: overlaps a moment that is already matched10
missing label: 3+ captions match, but no moment there21
another query’s moment: e.g. a skier for “skateboarder doing a trick”265
borderline: 1–2 of 5 captions mention the query156
look-alike scene: shares context words123
unrelated123

The largest group of costly FPs is again boundary: 10 of 28. A span with tIoU 0.4 is counted twice, as a FP and as a missed moment. So one cause, short spans from \(\alpha = 0.5\), explains 6 misses and 10 costly FPs. That is the fix to try first. The next group, borderline photos, is a label question again: if 2 of 5 people mention a race, is it one?

Algorithm 2: error analysis
  1. input: ranked results, matched labels, stage attribution, model scores per photo, captions
  2. for each missed moment: read its photos; tag one cause from a fixed list (label, span, short moment, re-ranker, …); if no tag fits, add a new one and re-tag all
  3. for each FP: tag by the first rule that fires: boundary, duplicate, missing label, another query, borderline, look-alike, unrelated
  4. mark FPs ranked above a hit as costly
  5. count tags; sort by count \(\times\) how cheap the fix is
  6. for label causes: write a stricter labeling rule and apply it to all videos, never only to failures
  7. for model causes: pick one knob per cause; tune it on dev only
  8. return a table of causes with counts and 2–3 examples each (this goes into the report)
shot of this query’s momentanother query’s momentborderlinelook-alikelabeled moment (v1)predicted spanevidence second of a FP
Figure 4. A failure browser for the as-built system on the test split (labels v1): all 22 missed moments and the 28 false positives that rank above a hit. Filter by cause with the chips; for each case you see 40 seconds of the video, the labels, the predictions, the captions, and the scores of every model.
How to read Figure 4

Click “Miss: label too broad” and step through: the captions show motorbikes, dogs and ladders. Then click “FP: boundary”: the predicted span (the outlined box) sits inside an orange labeled moment but covers too little of it. Under “FP: another query’s moment”, notice which sports get confused: three of the five costly cases are skateboarder results that land on skier moments. Across all 26 FPs of this group, 13 are team ball sports confused with each other (soccer, American football, basketball). Every case also shows the judge score that Step 6 uses.

Step 4: fix the labels, then tune on dev

Labels v2: a stricter rule, applied to everything

We rewrite the labeling rule. A photo still needs 3 of 5 matching captions, but a caption no longer counts if it also contains an exclusion word: for “runners in a race”: car, dog, motor, horse, bike; for “tennis player”: dog; for “climbing a rock wall”: ladder, stairs, tree, hill, skier, snow; for “swimming in a pool”: ocean, lake, river, sea, raft; for “surfer riding a wave”: wait, sit, line up, stand; for “skier going down”: up, uphill; for “cyclist”: motor; for “American football”: soccer. Each moment is then cut down to the runs of shots whose photos still pass.

We apply the rule to all 48 videos, dev and test. It rejects 31 of the 224 positive shots (14%). Of the 92 moments, 8 disappear and 13 get shorter. Test now has 53 moments; dev keeps its 31, though 4 of them are trimmed. Applying the rule everywhere matters: if you only relabel the failures, every change can only help the system, and the new score is biased upward.

Tune one knob on dev

Error analysis pointed at the refinement threshold \(\alpha\). We sweep it on dev with labels v2 and pick the value with the best dev mAP. Dev is flat at 0.878 from \(\alpha = 0.05\) to 0.30, so we take the middle: \(\alpha = 0.2\). Only then do we run test.

dev mAP@0.5 (labels v2)test mAP@0.5 (labels v2, hidden at first)
Figure 5. Tuning the refinement threshold on dev. Choose \(\alpha\) from the dev curve; then reveal the test curve, which in a real project you would see only for the chosen value. Both curves are real sweeps of the full system (VLM re-rank and refinement).
How to read Figure 5

The dev curve is a staircase: with only 31 moments, a small change of \(\alpha\) either flips one moment or nothing. Press “Show test”: the test curve is bumpy (0.765 at 0.2, 0.717 at 0.3, 0.758 at 0.35). If you picked \(\alpha\) by looking at test, you would pick a bump of noise and report it as a gain. At the chosen 0.2 the test result is 0.765, against 0.703 as built.

The final table

Test split, labels v2 (53 moments)mAP@0.595% intervalR@10mAP@0.7Hit@10\(\text{tIoU} \ge 0.5\) among hits
Stage 1 only0.669[0.555, 0.783]0.7740.0840.8680.891
Stage 1 + refine (\(\alpha = 0.2\))0.721[0.549, 0.863]0.8110.6080.8680.935
Stage 1 + VLM re-rank0.634[0.524, 0.752]0.7550.1030.8870.851
As built (\(\alpha = 0.5\))0.703[0.561, 0.836]0.8110.6230.9250.878
Tuned (\(\alpha = 0.2\))0.765[0.585, 0.912]0.8680.6700.9250.939

With the audited labels, both design requirements pass for the tuned system: Hit@10 = \(49/53 = 0.925\) and \(46/49 = 0.939\) of hits have tIoU \(\ge 0.5\). Notice that the label fix alone moved the as-built system from 0.554 to 0.703 mAP. Much of the “bad result” of Step 2 was the test set, not the system. The stage view of the tuned system (Figure 3, labels v2, test): 46 found, 3 boundary, 2 ranking, 2 retrieval, so \(0.962 \times 0.961 \times 0.939 = 0.868\).

Step 5: is the difference real?

Tuning raised test mAP from 0.703 to 0.765. Is that a real gain or luck in which 12 queries we have? mAP is an average over queries, and different queries vary a lot (per-query AP of the tuned system goes from 0.0 to 1.0). The question is how much the average would move if we had drawn another 12 queries of the same kind.

The bootstrap answers this without formulas for the distribution. Draw 12 queries with replacement from our 12, recompute the mean, repeat 10,000 times, and read the 2.5th and 97.5th percentiles. For a comparison of two systems, resample the per-query differences, so both systems are always scored on the same queries (a paired bootstrap). Paired tests like this are standard in information-retrieval evaluation [Likely].

\[ \Delta^{*(b)} = \frac{1}{n}\sum_{i=1}^{n} \big(\text{AP}^{A}_{j_i} - \text{AP}^{B}_{j_i}\big),\quad j_i \sim \text{Uniform}\{1..n\}, \qquad \text{CI}_{95\%} = \big[\Delta^{*}_{(2.5\%)},\ \Delta^{*}_{(97.5\%)}\big] \tag{237.6} \]
Read the symbols
\(n\)
number of queries (12)
\(j_i\)
the query drawn at position \(i\) of resample \(b\); the same query can be drawn several times
\(\Delta^{*(b)}\)
the mean difference of system A over system B in resample \(b\)
\(\Delta^{*}_{(2.5\%)}\)
the 2.5th percentile of the 10,000 resampled differences
Worked example: three comparisons, 10,000 resamples each (seed 0)

Tuned − as built, labels v1: +0.153, interval [+0.071, +0.242]. No resample is \(\le 0\). With the old labels this looks like a certain gain.

Tuned − as built, labels v2: +0.062, interval [−0.083, +0.202]; 19.6% of resamples are \(\le 0\). Much of the old gain came from boundary “errors” on moments whose labels were wrong. With 12 queries we cannot say that the tuned system is better.

Tuned − stage 1 + refine, labels v2 (the value of the VLM re-ranker): +0.044, interval [−0.009, +0.086]. Also not clearly above zero.

query where tuned is betterquery where it is worsehow often a query is drawn / resampled means95% interval
Figure 6. The paired bootstrap (237.6) for tuned against as built. Top: the per-query difference in AP, and, after a draw, how many times each query was picked. Bottom: the histogram of resampled mean differences, the observed mean (green line), zero (thin line), and the 95% interval once there are at least 200 draws. The page uses its own seeded random generator, so its interval differs a little from the Python numbers in the text.
How to read Figure 6

Draw one resample at a time first: some queries appear 2 or 3 times and others not at all, and the mean moves with them. With labels v2, one query (“runners”) has a large negative difference: the low threshold makes the span spill from a real race into the next shot, a dog race, which the new labels reject. Then draw 1,000 several times. With labels v1 the whole histogram sits right of zero; with v2 it straddles zero. The width of the interval is set mostly by the number of queries: 12 is a smoke test, which is why the design asked for 500.

What do we do with an interval that includes zero? We report it as it is: “a gain of +0.062 mAP, not significant with 12 queries”. The other metrics point the same way (mAP@0.7 0.623 to 0.670, tIoU-among-hits 0.878 to 0.939), and the change is free (one constant), so we keep \(\alpha = 0.2\) and list “more queries” as the first next step. What we must not do is change the test set, the metric or the seed until the interval looks better.

Step 6: can a VLM judge replace the labels?

Labels are expensive. A popular shortcut is LLM-as-a-judge: ask a strong model whether each result is correct. Zheng et al. found that strong LLM judges agree with human preferences more than 80% of the time, about as often as humans agree with each other, but they also documented position, verbosity and self-enhancement biases [Certain]. Before you trust a judge on your task, you measure it against labels, like any other model.

We use two judges on the 2,285 (photo, query) pairs, with labels from the audited rule (187 pairs where the photo shows the query, 1,992 where no caption mentions it; 106 borderline pairs left out): an independent judge, SmolVLM-256M with a different prompt, and the re-ranker itself, SmolVLM-500M, as a self-judge. Accuracy is a poor measure here: 91.4% of the pairs are negatives, so a judge that always says “no” is 91.4% accurate. We use Cohen’s kappa, which subtracts the agreement expected by chance:

\[ \kappa = \frac{p_o - p_e}{1 - p_e}, \qquad p_e = \frac{(\text{TP}+\text{FP})(\text{TP}+\text{FN}) + (\text{FN}+\text{TN})(\text{FP}+\text{TN})}{n^2} \tag{237.7} \]
Read the symbols
\(p_o\)
observed agreement: the share of pairs where judge and label agree (accuracy)
\(p_e\)
agreement expected if the judge said “yes” at its own rate but at random
\(\kappa\)
1 is perfect agreement, 0 is chance level; values below about 0.4 are usually called weak [Likely]

A judge can also estimate a precision without labels: the share of top-10 results it calls correct. If we know its true-positive rate (TPR) and false-positive rate (FPR) from a small labeled sample, we can correct that share:

\[ \hat p = \frac{q - \text{FPR}}{\text{TPR} - \text{FPR}} \tag{237.8} \]

Here \(q\) is the share of results the judge accepts and \(\hat p\) the corrected estimate of the true share. This is the classic correction for an imperfect test [Likely]; it is only as good as the TPR and FPR estimates.

Worked example: two judges at threshold 0 (log-odds above 0 means “yes”)

Independent judge (SmolVLM-256M). TP 42, FN 145, FP 33, TN 1,959, so \(n = 2{,}179\). Accuracy \(p_o = 2{,}001/2{,}179 = 0.918\). Chance agreement \(p_e = (75 \cdot 187 + 2{,}104 \cdot 1{,}992)/2{,}179^2 = 0.886\). So \(\kappa = (0.918 - 0.886)/(1 - 0.886) = 0.286\): weak. It says “yes” to only 42 of the 187 frames that show the query (TPR 0.225, FPR 0.017).

Self-judge (the re-ranker). TP 139, FN 48, FP 70, TN 1,922: accuracy 0.946, \(\kappa = 0.672\), TPR 0.743, FPR 0.035. It looks much better, but these are partly its own decisions.

Judging the final top-10. For 118 of the 120 test results, both judges scored the evidence frame. The labels say 50 of them (0.424) show the query. The independent judge accepts 22 (0.186): too strict. The self-judge accepts 78 (0.661): it grades its own picks far too kindly. Correcting the independent judge with (237.8) and its rates on all pairs gives \((0.186 - 0.017)/(0.225 - 0.017) = 0.817\): far off, because the negatives in a top-10 list are the hard ones the system chose, and the judge’s FPR on them is higher (0.056 on the dev top-10) than on random pairs. Calibrating TPR and FPR on the dev top-10 instead (120 results, 31 positive) gives 0.556: closer, but 120 labeled items are still few.

Figure 7. Judge against labels on single frames. Top: the judge’s log-odds for frames that show the query (green, up) and frames that do not (red, down); bar heights use a square-root scale so the small positive class stays visible. The table is the confusion matrix at the threshold; the readout gives accuracy, chance agreement, kappa (237.7), and the judge’s estimate of the frame precision of the final top-10 list, raw and corrected (237.8).
How to read Figure 7

Start with the independent judge at threshold 0: most green bars sit left of the line, so the judge misses most true frames, while accuracy is still 0.918 because negatives dominate. Move the threshold down to about −3: TPR rises and kappa improves, at the cost of more false yes answers. Switch to the self-judge: the classes separate better, but its estimate of the final top-10 precision is far above the labels. Watch the “corrected” value: with rates measured on all pairs it overshoots, because hard negatives in a top-10 list fool the judge more often than average negatives.

The lesson is not “judges are useless”. It is: a judge is a model, so evaluate it. Use a judge that is independent of the system (a different model, ideally a stronger one), calibrate its threshold and its TPR and FPR on a labeled sample, report kappa and not only accuracy, and never let the system grade itself.

Step 7: write the technical report

The report is the product of this lab. Readers are busy: put the answer first, numbers with intervals, then the limits and what to do next. One page is enough. Here is ours.

Key-moment search: evaluation of the lab configuration (labels v2)

Summary. On 32 held-out videos (53 moments, 12 queries), the tuned system reaches \(\text{mAP@0.5} = 0.765\) (95% bootstrap interval over queries [0.585, 0.912]), \(R@10 = 0.868\) and \(\text{Hit@10} = 0.925\); 93.9% of hits have \(\text{tIoU} \ge 0.5\). Both design requirements (\(\text{Hit@10} \ge 0.85\); \(\text{tIoU} \ge 0.5\) for 85% of hits) are met on this set. The set is small; the intervals are wide.

Setup

Slideshow videos from Flickr8k test and validation photos, 1 frame per second; dev = 16 videos, test = 32 videos, split by video. Labels from 5 captions per photo with a written rule (v2). SigLIP-B/16 windows of 8 s (stride 4 s), top 50; SmolVLM-500M yes/no re-rank; threshold refinement (\(\alpha\) tuned on dev). This is a smaller configuration than the session-236 build (2-s segments, fused Qwen2.5-VL-3B, contrast refiner); its results do not transfer to that build without re-running.

Results by stage (test)

Stage-1 \(R@50 = 0.962\); re-rank keeps 0.961 of those in the top 10; boundaries are right for 0.939 of those. The VLM re-ranker adds +0.044 mAP over stage 1 + refinement (interval [−0.009, +0.086]); refinement with \(\alpha = 0.2\) adds +0.062 over \(\alpha = 0.5\) ([−0.083, +0.202]).

Failure analysis

Of the 22 misses of the first run, 13 were label errors (a word rule that matched dog and car races, tennis balls, ladders), 6 were spans cut short by \(\alpha = 0.5\), 2 were 2-s moments diluted in 8-s windows, 1 was a re-ranker error. The costly false positives were mostly boundary errors (10 of 28), then borderline photos (6) and other sports with similar scenes (5).

Limits

  • 12 queries and 53 test moments: differences under about 0.15 mAP are not resolvable.
  • Photos are not video: no motion, no replays, no commentary; real broadcast results will differ.
  • Labels come from captions and a rule, not from annotators who watched the video; 14% of the first labels were wrong.
  • The re-ranker (500M) is much smaller than the 3B model of the design and of the session-236 build; its frame-level AUC (0.945) is below SigLIP’s (0.971). No latency was measured on an L4.

Next steps

  1. Grow the test set to at least 100 queries, labeled by two people with measured agreement.
  2. Add 4-s windows next to 8-s windows (both short-moment misses).
  3. Run this evaluation on the configuration of session 236 (2-s segments with grouping, which also avoids the short-moment misses; fused Qwen2.5-VL-3B; contrast refiner), with the same label audit and bootstrap; measure p95 latency on an L4 there.
  4. Compare the threshold refiner with the contrast search of session 236 on the same labels, then try a learned boundary model once labels exist (session 207).

The full program

Algorithm 3 is the whole evaluation loop. The PyTorch program after it contains the metric code used on this page, the paired bootstrap on our real per-query APs, and kappa and the judge correction on our real confusion counts. It runs in under a second, with no downloads, and is deterministic; the printed values are in the comments.

Algorithm 3: evaluate, analyze, report
  1. input: system \(S\) with knobs, labeled videos split into dev and test by video, queries
  2. run every stage of \(S\) once on dev and test; cache embeddings, candidates, VLM scores
  3. on dev: compute mAP, R@10, stage attribution (237.5); run the ablations
  4. run test once; compute the same numbers and bootstrap intervals (237.6)
  5. error analysis (Algorithm 2) on dev and test failures \(\rightarrow\) table of causes
  6. label causes \(\rightarrow\) new labeling rule on all videos; re-score every system
  7. model causes \(\rightarrow\) choose one knob per cause; tune on dev only; run test once
  8. if a judge is used: measure kappa, TPR, FPR on labeled pairs (237.7, 237.8)
  9. return a one-page report: summary with intervals, setup, stage table, causes, limits, next steps
import torch

def tiou(p, g):                                    # (237.1) spans are (start, end) in seconds
    inter = max(0.0, min(p[1], g[1]) - max(p[0], g[0]))
    union = max(p[1], g[1]) - min(p[0], g[0])
    return inter / union if union > 0 else 0.0

def match(preds, gts, theta=0.5):                  # (237.2) greedy, in rank order
    used, rel = set(), []
    for vid, span in preds:
        cand = [(tiou(span, g[1]), j) for j, g in enumerate(gts) if g[0] == vid and j not in used]
        best, j = max(cand, default=(0.0, -1))
        if best >= theta:
            used.add(j); rel.append(1)
        else:
            rel.append(0)
    return rel

def average_precision(rel, n_gt):                  # (237.4) divide by ALL labeled moments
    hits, s = 0, 0.0
    for k, r in enumerate(rel, start=1):
        if r:
            hits += 1; s += hits / k
    return s / n_gt if n_gt else 0.0

# 1) the worked example: one query, three labeled moments
gts = [("v1", (20, 30)), ("v2", (50, 56)), ("v3", (8, 20))]
preds = [("v1", (21, 29)), ("v4", (60, 68)), ("v1", (22, 30)), ("v2", (51, 57)), ("v3", (10, 19))]
rel = match(preds, gts)
print("relevance:", rel, " P@5:", sum(rel) / 5, " R@5:", sum(rel) / len(gts))   # [1, 0, 0, 1, 1] 0.6 1.0
print("AP:", round(average_precision(rel, len(gts)), 3))       # 0.7

# 2) stage attribution (237.5): counts from the tuned system on test, labels v2
G, n50, n10, found = 53, 51, 49, 46
print("factors:", round(n50 / G, 3), round(n10 / n50, 3), round(found / n10, 3), "R@10:", round(found / G, 3))
# factors: 0.962 0.961 0.939 R@10: 0.868

# 3) paired bootstrap over queries (237.6): real per-query AP on test, labels v2
ap_tuned = torch.tensor([0.688, 1.0, 1.0, 0.976, 0.717, 1.0, 0.861, 0.786, 0.804, 0.0, 0.35, 1.0])
ap_built = torch.tensor([0.25, 1.0, 1.0, 0.786, 0.717, 0.65, 0.861, 0.786, 0.358, 0.5, 0.525, 1.0])
d = ap_tuned - ap_built
gen = torch.Generator().manual_seed(0)
idx = torch.randint(0, len(d), (10_000, len(d)), generator=gen)     # 10,000 resamples of 12 queries
means = d[idx].mean(dim=1)
lo, hi = torch.quantile(means, torch.tensor([0.025, 0.975]))
print("mAP tuned %.3f built %.3f diff %+.3f" % (ap_tuned.mean(), ap_built.mean(), d.mean()))   # 0.765 0.703 +0.062
print("95%% interval [%+.3f, %+.3f], share <= 0: %.3f" % (lo, hi, (means <= 0).float().mean()))
# 95% interval [-0.084, +0.206], share <= 0: 0.196  (Python random with seed 0 gave [-0.083, +0.202])

# 4) a judge against labels: Cohen's kappa (237.7) and the corrected precision (237.8)
def kappa(tp, fn, fp, tn):
    n = tp + fn + fp + tn
    po = (tp + tn) / n
    pe = ((tp + fp) * (tp + fn) + (fn + tn) * (fp + tn)) / n ** 2
    return po, pe, (po - pe) / (1 - pe)
for name, c in [("independent 256M", (42, 145, 33, 1959)), ("self 500M", (139, 48, 70, 1922))]:
    po, pe, k = kappa(*c)
    tpr, fpr = c[0] / (c[0] + c[1]), c[2] / (c[2] + c[3])
    print("%-17s acc %.3f  p_e %.3f  kappa %.3f  TPR %.3f  FPR %.3f" % (name, po, pe, k, tpr, fpr))
# independent 256M  acc 0.918  p_e 0.886  kappa 0.286  TPR 0.225  FPR 0.017
# self 500M         acc 0.946  p_e 0.835  kappa 0.672  TPR 0.743  FPR 0.035
q, tpr, fpr = 22 / 118, 42 / 187, 33 / 1992       # the judge accepts 22 of 118 top-10 results
print("judge says %.3f, corrected %.3f, labels say %.3f" % (q, (q - fpr) / (tpr - fpr), 50 / 118))
# judge says 0.186, corrected 0.817, labels say 0.424

Where this is used

ApplicationWhat you evaluateWhat is special
Sports key-moment search (this capstone)R@K of stage 1, mAP@tIoU, Hit@10, tIoU among hitsreplays and duplicates; game clock vs video time
Temporal grounding benchmarks (QVHighlights, Charades-STA)R1@IoU=0.5 and 0.7, mAP over several IoU thresholdsone query per video; the metrics in (237.1)–(237.4) are the same ideas [Likely]
Text-to-video retrieval (MSR-VTT style)R@1, R@5, R@10, median rankone correct video per query, so AP and R@K are simpler
Video-RAG and question answering over videoretrieval recall of the evidence clips, then answer accuracyan answer can be right with wrong evidence: score both stages
RAG over documentsrecall@K of passages, faithfulness, answer correctnessLLM judges are common; measure them against human labels first
Content moderation and fingerprintingprecision at a fixed recall, false-positive rate per hour of videothe costly error type decides the threshold
A/B tests of a deployed searchclick-through, reformulations, time to a used clipoffline gains must be confirmed online; intervals over users, not queries

Common mistakes

  • Tuning on the test set. Every look at test while choosing a knob leaks it. Figure 5 shows how bumpy a small test curve is: picking its maximum reports noise as a gain.
  • Splitting by moment instead of by video. Neighboring shots of the same video then sit in both splits, and the dev score predicts nothing.
  • Trusting the labels. In our first run, 13 of 22 misses were label errors. Read the failures before you change the model.
  • Fixing only the labels of failures. It can only raise the score. Write a rule and apply it to every video.
  • Reporting one end-to-end number. Without stage attribution (237.5) you cannot tell a retrieval problem from a boundary problem; they need opposite fixes.
  • Ignoring the interval. +0.062 mAP on 12 queries has an interval from −0.083 to +0.202. Report it, and grow the set.
  • Accuracy for a judge on imbalanced data. A judge that says “no” to everything is 91.4% accurate here. Use kappa, TPR and FPR.
  • Letting the system grade itself. The re-ranker as judge says its own top-10 is much better than it is, because it chose those results.

Interview questions

How do you evaluate a video moment-retrieval system end to end?basic
Build a labeled set of queries with moment spans, split by video into dev and test. Measure stage 1 recall at the candidate depth (R@K), the re-ranked list (mAP and R@10 with a tIoU threshold, often 0.5 and 0.7), and the boundaries (tIoU among hits). Attribute every missed moment to the stage where it was lost. Add bootstrap intervals over queries. Tune only on dev; run test once. Then measure latency and cost under load, and confirm online with user metrics.
Define tIoU, and explain how a ranked list of spans becomes AP.basic
tIoU is the length of the overlap of a predicted span and a labeled span divided by the length of their union (237.1). Walk down the ranked list; match each result to the unused labeled moment in the same video with the highest tIoU; it is a hit if that tIoU is at least \(\theta\) (237.2). AP is the sum of the precision at the rank of each hit, divided by the number of labeled moments \(G\) (237.4). In the worked example, hits at ranks 1, 4 and 5 with \(G = 3\) give \((1 + 0.5 + 0.6)/3 = 0.70\).
Why divide AP by the number of labeled moments rather than by the number of hits?core
Dividing by the hits would ignore moments the system never returns: a system that returns one perfect result and misses nine moments would score 1.0. Dividing by G charges every missed moment, so AP@N is at most R@N. It is the convention of detection-style AP.
Your dev score is 0.83 and your test score is 0.55. What do you check?core
First, whether dev is representative: how big it is, and whether its failure types appear at all. In our case dev had no retrieval or ranking misses, so it could not show those problems. Second, the stage attribution on test, to see which stage is responsible. Third, the labels on the test failures: here 13 of 22 misses were label errors. Fourth, whether anything leaked or was tuned on noise. Only then change the model.
Why can a boundary error hurt mAP twice?core
A result with tIoU below \(\theta\) on a real moment is a false positive, and the moment stays unmatched, so it is also a miss. If the FP ranks above other hits it lowers their precision too. In our first run, short spans explained 6 misses and 10 of the 28 costly false positives.
How do you know that one system is better than another on 12 queries?core
Compare them on the same queries and resample the per-query differences: a paired bootstrap (237.6), or a paired randomization test. Report the mean difference and its 95% interval. With 12 queries, our +0.062 mAP had an interval of [−0.083, +0.202], so it is not resolved. The interval narrows roughly as one over the square root of the number of queries, so ~4 times more queries halve it [Likely].
You discover label errors in the test set. What is the right procedure?core
Write down why the labels are wrong as a rule or a guideline, apply it to all videos in both splits (not only to failures), version the labels (v1, v2), re-score every system on the new labels, and report both versions. If the rule needs judgment, have two people relabel and measure their agreement with kappa. Fixing only failures can only raise the score.
Would you use an LLM or VLM as a judge instead of human labels?advanced
As a cheap first pass, yes, after measuring it. Label a few hundred items, compute the judge’s kappa, TPR and FPR, pick its threshold there, and correct its estimates with (237.8). Use a model that is independent of the system and at least as strong. Known biases include position, verbosity and self-enhancement. In our lab the independent 256M judge had \(\kappa = 0.29\) and the re-ranker judging itself estimated top-10 frame precision at 0.66 against a true 0.42.
Why did the self-judge overestimate precision so much?advanced
Selection. The top-10 list contains exactly the photos that the re-ranker scored highest, including its own false positives. Asking the same model again repeats the same mistakes, so it says “yes” to almost everything it picked. An independent judge makes different mistakes, and its errors can be corrected with its TPR and FPR. This is the same reason you do not evaluate a model on its own training data.
How do you choose what to fix first after error analysis?advanced
Rank causes by how many failures (or how much AP) they cost, times how cheap the fix is, and check that the fix is testable on dev. In our lab: labels first (13 misses, fixed by a rule), then the refinement threshold (6 misses and 10 costly FPs, a one-constant change), then multi-scale windows (2 misses, a re-index), then the re-ranker (1 miss, a bigger model and more cost).
What belongs in a one-page technical report on the system?basic
A summary with the main numbers and their intervals and whether the requirements are met; the setup (data, splits, labels, system version); a stage-by-stage table with ablations; the failure causes with counts and examples; the limits (size of the test set, label quality, what was not measured, such as latency on the target GPU); and concrete next steps in order.

Exercises

  1. A query has \(G = 4\) moments. The top 6 results have rel = [0, 1, 1, 0, 0, 1]. Compute \(P@6\), \(R@6\) and AP@6.
  2. Predicted span [12, 20], labeled moment [14, 26]. What is the tIoU? Is it a hit at \(\theta = 0.5\)?
  3. Stage 1 covers 51 of 53 moments, 49 survive the re-ranker, 46 are matched. Write the three factors of (237.5) and check the product.
  4. With the as-built system on labels v1, why did lowering \(\alpha\) also reduce the number of false positives?
  5. A judge has TPR 0.8 and FPR 0.1 and accepts 50% of the results. Estimate the true precision with (237.8).
  6. Use Figure 6 with labels v1 and v2. Explain in two sentences why the interval moves.
  7. Write a stricter labeling rule for “a skateboarder doing a trick”. Which photos would it remove? How would you check that it does not remove good ones?
Answerscheck
1. Hits at ranks 2, 3 and 6: \(P@6 = 3/6 = 0.5\), \(R@6 = 3/4 = 0.75\), AP \(= \frac{1}{4}(\frac{1}{2} + \frac{2}{3} + \frac{3}{6}) = \frac{1.667}{4} = 0.417\). 2. Overlap \(20 - 14 = 6\), union \(26 - 12 = 14\), tIoU \(= 0.43\): not a hit; it is a FP and the moment stays unfound. 3. \(51/53 = 0.962\), \(49/51 = 0.961\), \(46/49 = 0.939\); the product is \(46/53 = 0.868\). 4. Many FPs were boundary errors: a short span on a real moment. When the span grows over the whole moment it becomes a hit, so one FP and one miss turn into one TP. 5. \((0.5 - 0.1)/(0.8 - 0.1) = 0.571\). 6. With v1, the tuned system fixes many boundary “errors” on all queries, so nearly all per-query differences are positive. With v2, several of those moments were removed or trimmed and “runners” gets worse, so the differences have both signs and the interval crosses zero. 7. For example, exclude captions with scooter, sled, snowboard and require one of trick, jump, grind, ramp, air in at least 2 captions. To check it, read a random sample of the photos it keeps and removes (say 30 each) and count mistakes, or have a second person label the same sample and compute kappa.

Key takeaways

  • Label a test set with exact spans, split it by video into dev and test, cache all model outputs, and tune only on dev.
  • Measure each stage: stage-1 R@K, the re-ranker’s top 10, and the boundaries; their product is the end-to-end recall (237.5).
  • Error analysis means reading every failure and counting causes. In our first run 13 of 22 misses were label errors, and one knob (\(\alpha\)) explained most of the rest.
  • Fix labels by a rule applied to every video, version them, and re-score everything.
  • Report intervals: with 12 queries, a +0.062 mAP gain has a 95% interval of [−0.083, +0.202].
  • A VLM judge is a model: measure its kappa, TPR and FPR against labels, keep it independent of the system, and correct its estimates.

Key formulas

\[ \text{tIoU} = \frac{|\hat m \cap m|}{|\hat m \cup m|}, \quad \text{AP} = \frac{1}{G}\sum_k P@k\,\text{rel}_k, \quad R@10 = \frac{n_{50}}{G}\cdot\frac{n_{10}}{n_{50}}\cdot\frac{n_{\text{found}}}{n_{10}}, \quad \kappa = \frac{p_o - p_e}{1 - p_e}, \quad \hat p = \frac{q - \text{FPR}}{\text{TPR} - \text{FPR}} \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  2. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  4. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
  5. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  6. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
  7. Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
  8. Gao et al. (2017). TALL: Temporal Activity Localization via Language Query
  9. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  10. Hodosh, Young & Hockenmaier (2013). Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
  11. Cohen (1960). A Coefficient of Agreement for Nominal Scales
  12. Efron & Tibshirani (1993). An Introduction to the Bootstrap
  13. Smucker, Allan & Carterette (2007). A Comparison of Statistical Significance Tests for Information Retrieval Evaluation
  14. Caba Heilbron, Escorcia, Ghanem & Niebles (2015). ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding