capstone evaluation and failure analysis
This lab evaluates the sports key-moment search system from the last two sessions on a labeled test set you can run on a laptop. You compute tIoU, AP and recall stage by stage on real ranked lists, find out why each failure happens, discover that many “model errors” are label errors, decide with a bootstrap whether a fix is real, test whether a VLM judge can replace labels, and write a one-page report.
What you will learn
- How to build a labeled test set with exact moment spans, and why you split it by video into dev and test
- How tIoU, matching, P@k, R@k, AP and mAP are computed on a ranked list of video spans, step by step
- How to attribute every missed moment to a stage (retrieval, ranking, boundary) and read end-to-end recall as a product of stage rates
- How to do error analysis: read each failure, tag a cause, count, and fix the largest cheap cause first
- Why label errors look like model errors, and how to fix labels with a written rule applied to every video
- How a paired bootstrap over queries tells you whether a gain is real, and why 12 queries are not enough
- How to measure a VLM judge with Cohen’s kappa, TPR and FPR, why a system must not judge itself, and how to write the final report
Why this matters
In session 235 you designed a search system for sports video on paper. In session 236 you built it: an index of 2-s segments, a VLM re-ranker fused with the first-stage score, and a timestamp refiner, tested on a slideshow archive with exact labels. Now comes the question every reviewer, manager and interviewer will ask: does it work, and how do you know? This lab answers it with numbers. You will label a test set, measure retrieval and grounding stage by stage, find out why each failure happens, and write a one-page technical report.
Evaluation is where most real projects go wrong, and in quiet ways. A number can look good because the test set is too easy, too small, or labeled by the system itself. A fix can look large because the labels were wrong in the same place. In this lab every one of these traps happens to us on real data, and we show how to catch it:
- The first test result is much worse than the development (dev) result: dev mAP 0.826, test mAP 0.554.
- Error analysis shows that 13 of the 22 missed moments are label errors, not model errors.
- After we fix the labels with a written rule, a tuning change that looked like a sure gain of +0.153 mAP becomes +0.062, with a 95% interval that includes zero.
- A small VLM used as a judge agrees with the labels on 92% of frames, yet its agreement beyond chance is weak (\(\kappa = 0.29\)). The re-ranker judging its own top-10 says 66% of it is right; the labels say 42%.
The method is general. The same steps evaluate a RAG system (session 155), a VLM on a benchmark (session 192) or a video model (session 213).
Setup: a labeled test bed you can run on a laptop
A real evaluation of the design in session 235 needs hundreds of hours of football and editors who label moments. For a lab we need something smaller that keeps the important property: we know the exact true moments. So we build a small archive of “broadcasts” from a public photo dataset with human captions.
The archive. Flickr8k has about 8,000 photos, each with 5 captions written by different people [Certain]. We use its test and validation splits (2,000 photos). Each “video” is a 120-second slideshow: a sequence of photos, each held for 2 to 6 seconds like a camera shot. One frame per second is one photo, so a second of video is a photo. We made 48 videos (96 minutes, 1,483 shots).
The queries and the moments. We use 12 sports queries, for example “a surfer riding a wave” and “runners in a race”. A photo counts as showing a query if at least 3 of its 5 captions match a word pattern (for “runners in a race”: race, racing, marathon or runner). Into each video we place 1 or 2 moments: 1 to 4 consecutive shots of matching photos. That gives 92 labeled moments of 2 to 19 seconds (mean 9.4). Between moments we place background shots, chosen so that the task is not too easy: 13% of them are borderline photos (1 or 2 captions mention a query) and 40% are look-alike scenes (they share context words such as snow, water or field).
The split. Videos 0–15 are the dev split (31 moments): we may look at them as often as we like and tune on them. Videos 16–47 are the test split (61 moments): we run it only to report. We split by video, not by moment, so no shot or neighbor of a test moment is ever seen during tuning.
The system under test. We evaluate a smaller and partly different configuration than the one built in session 236. This lab was prepared in parallel with session 236, on its own archive, so that every intermediate output (the caption votes of every photo, the VLM score of every photo and query) is saved for error analysis, and it keeps the design’s threshold refiner (235.10) because its single knob \(\alpha\) makes the tuning step easy to follow. The embedding model and the small re-ranker are the same as in session 236; the archive, the segments, the fusion and the refiner differ. So the numbers on this page are not the numbers of session 236, but every method here applies unchanged to its outputs.
| Part | Evaluated in this lab | Built in session 236 |
|---|---|---|
| Test archive | 48 slideshow videos of 120 s from Flickr8k (dev 16, test 32), 12 queries, 92 moments | 60 slideshow videos of 180 s from Flickr8k (dev 20, test 40), 14 queries, 84 test moments |
| Frame embedding | SigLIP-B/16 at 224 px, one frame per second, unit-length vectors | same |
| Segments and index | 8-s windows, 4-s stride, mean-pooled: 29 per video, 1,392 in total; exact search | 2-s segments, no overlap, flat FP16 index (formula 236.1) |
| Stage 1 search | cosine with the text vector of “a photo of query.”; top \(K = 50\) windows in the split | same prompt; top 100 segments, touching hits grouped into moments, score = max (236.2) |
| VLM re-rank | SmolVLM-500M-Instruct, “Does this image show query? Answer yes or no.”; log-odds \(\text{logit(Yes)} - \text{logit(No)}\) for every photo in the window, keep the maximum; final score = log-odds + z-scored cosine | Qwen2.5-VL-3B (or SmolVLM-500M); 3 keyframes per moment, mean log-odds over yes/Yes and no/No (236.4); \(M = 30\) candidates; fused as \(w \cdot\) z(cosine) + \((1-w) \cdot\) z(log-odds), \(w = 0.75\) (236.5) |
| Timestamps | threshold rule (235.10): grow from the peak second while the per-second cosine is at least \(\alpha \times\) peak, within \(\pm 6\) s of the window; \(\alpha = 0.5\) at first | contrast search (236.6) within \(\pm 2\) s of the segment boundaries |
| Output | drop a result that overlaps a better one in the same video at tIoU above 0.3; top \(N = 10\) | drop overlaps at tIoU \(\ge 0.5\); top 10 |
On this page, “as built” means our configuration with its first settings (\(\alpha = 0.5\)), before any tuning; “tuned” is the same configuration after Step 4.
The VLM scored 2,285 distinct (photo, query) pairs: every photo inside a top-50 window of the dev or test split. All embeddings and VLM scores are computed once and saved, so every metric on this page is recomputed from files in seconds. That is the first engineering rule of evaluation: cache model outputs, recompute metrics.
- input: photos with 5 captions each; 12 queries, each with a word pattern
- for each photo and query: votes \(\leftarrow\) number of captions that match the pattern
- positives(\(q\)) \(\leftarrow\) photos with votes \(\ge 3\); borderline \(\leftarrow\) photos with 1–2 votes for some query
- look-alike \(\leftarrow\) photos with 0 votes that share at least 2 context words with a query; easy \(\leftarrow\) the rest
- for each of 48 videos (seed 7): pick 1–2 queries and start times at least 40 s apart
- fill 120 s with shots of 2–6 s: background shots from borderline (12%), look-alike (40%), easy (48%); at a start time insert 1–4 shots of positives(\(q\)) and record the moment \([s, e, q]\)
- never reuse a photo while unused ones remain; split videos 0–15 = dev, 16–47 = test
- return videos, moments, and the votes of every photo (we need them for error analysis)
How to read Figure 1
Start with “runners in a race”, moment 1 (video v16): there is no prediction at all, and the captions in the readout show why: a motorbike, two racing dogs and a rally car. The labels call this “runners in a race”. Then pick “surfer” and step to video v47: the moment has three shots, and the middle one (a weak photo, cosine 0.045) splits the predicted span in two. Move \(\alpha\) down to 0.2: the span grows over the whole moment and the tIoU jumps. This is the main knob we tune in Step 4.
Step 1: the metrics, computed by hand
A search result is a span of video: a video id, a start and an end. To decide if it is correct, we compare it with the labeled moments of the same query. Three ideas are needed: overlap, matching, and a score for a ranked list.
Overlap: temporal IoU
Here \(\hat m = [\hat a, \hat b]\) is a predicted span and \(m = [a, b]\) a labeled moment in the same video. It is the same formula as (235.2). tIoU is 1 for a perfect span and 0 for no overlap.
Matching: which result found which moment
A query has several labeled moments in the test videos (on average 5). We walk down the ranked list. Each result is matched to the unused moment in the same video with the highest tIoU; if that tIoU is at least a threshold \(\theta\), the result is a true positive (TP), and the moment is used up. Otherwise it is a false positive (FP). A second result on a moment that is already used is a FP too (a duplicate).
This is the rule used by object detection and by temporal-localization benchmarks such as ActivityNet [Likely]. Session 236 reports a stricter variant: precision interpolated over later ranks and mAP averaged over \(\theta = 0.5, 0.55, \dots, 0.95\) (236.3). Here we use one \(\theta\) at a time and no interpolation, so that every step can be checked by hand. We use \(\theta = 0.5\), the threshold in the design’s requirements, and also report \(\theta = 0.7\), which asks for tighter spans.
Scores for a ranked list
- Read the symbols
- \(\text{rel}_k\)
- 1 if the result at rank \(k\) is a true positive, else 0
- \(G\)
- number of labeled moments for the query in the split (all of them, also those the system never returned)
- \(N\)
- length of the returned list; 10 here
- \(P@k\), \(R@k\)
- precision and recall of the first \(k\) results
- \(Q\)
- the set of queries; mAP gives every query the same weight
AP rewards putting hits early: each hit adds the precision at its rank. Dividing by \(G\), not by the number of hits, means that a moment the system never returns costs as much as a hit at a very low rank. Because hits after rank 10 are not seen, AP@10 is at most \(R@10\).
A query has \(G = 3\) moments: v1 [20, 30], v2 [50, 56], v3 [8, 20]. The system returns: (1) v1 [21, 29], (2) v4 [60, 68], (3) v1 [22, 30], (4) v2 [51, 57], (5) v3 [10, 19].
Rank 1: tIoU with v1 [20, 30] is \(8/10 = 0.80\): TP. Rank 2: no moment in v4: FP. Rank 3: overlaps v1, but that moment is used: duplicate, FP. Rank 4: \(5/7 = 0.71\): TP. Rank 5: \(9/12 = 0.75\): TP.
\(\text{rel} = [1, 0, 0, 1, 1]\), \(P@5 = 3/5 = 0.6\), \(R@5 = 3/3 = 1.0\), and \(\text{AP} = \frac{1}{3}\big(\frac{1}{1} + \frac{2}{4} + \frac{3}{5}\big) = \frac{2.1}{3} = 0.70\). Move the duplicate below rank 5 and AP becomes \(\frac{1}{3}(1 + \frac{2}{3} + \frac{3}{4}) = 0.81\): the duplicate cost 0.11.
| rank | video | span (s) | best tIoU | match | P@k | R@k | AP term |
|---|
How to read Figure 2
Pick “soccer” with the as-built system and labels v1 and press Play. Rank 1 is a hit, but rank 2 (v44, tIoU 0.25) and rank 4 (v29, tIoU 0.40) land on real moments with spans cut too short, so they count as FPs and the moments stay unfound. AP is 0.250. Switch to “Tuned”: the same moments are now hits and AP becomes 0.688. Change \(\theta\) to 0.3 and to 0.7 to see how much of the score depends on the threshold. Try “runners” with labels v1 and v2: the labels decide the score as much as the system does.
Step 2: first results, stage by stage
We run the as-built system on dev, then once on test. To see what each stage adds, we also run reduced versions of it (an ablation): stage 1 alone, stage 1 with refinement, and stage 1 with the VLM but without refinement.
| System (labels v1) | dev mAP@0.5 | test mAP@0.5 | test R@10 | test P@10 | test mAP@0.7 |
|---|---|---|---|---|---|
| Stage 1 only (8-s windows) | 0.747 | 0.638 | 0.689 | 0.350 | 0.089 |
| Stage 1 + VLM re-rank | 0.772 | 0.623 | 0.721 | 0.367 | 0.099 |
| As built: VLM + refine (\(\alpha = 0.5\)) | 0.826 | 0.554 | 0.639 | 0.325 | 0.477 |
Three things stand out. First, the as-built system is the best on dev and the worst on test. Second, the dev numbers are higher everywhere: with 16 videos, dev is small and, by chance, easier. Third, refinement helps a lot at \(\theta = 0.7\) (0.099 to 0.477): fixed 8-s windows almost never match a moment tightly. So refinement is necessary, but something in it fails on test. The end-to-end number cannot tell us what. The stages can.
Where is each moment lost?
The design session wrote end-to-end success as a product of stage success rates (235.13). With labels we can measure each factor. For every labeled moment we ask, in order: is it covered by one of the top-50 windows of stage 1 (covered means the overlap is at least half of the shorter of the two)? Is it covered by a window that made the final top 10? Is it matched with tIoU \(\ge \theta\)?
- Read the symbols
- \(n_{50}\)
- moments covered by some top-50 stage-1 window
- \(n_{10}\)
- moments covered by the window of some final top-10 result
- \(n_{\text{found}}\)
- moments matched with tIoU \(\ge \theta\) (237.2)
Each missed moment gets exactly one label: retrieval (not in the top 50), ranking (in the top 50 but not the top 10), or boundary (in the top 10, but the span is wrong). For the as-built system on test: 39 found, 12 boundary, 2 ranking, 8 retrieval. So \(n_{50} = 61 - 8 = 53\), \(n_{10} = 53 - 2 = 51\) and \(n_{\text{found}} = 39\). The factors are \(53/61 = 0.869\), \(51/53 = 0.962\) and \(39/51 = 0.765\), and their product is \(39/61 = 0.639\), the R@10 in the table. The weakest factor is the boundaries.
The design asked for Hit@10 \(\ge 0.85\) (a moment is in the top 10) and tIoU \(\ge 0.5\) for 85% of the hits. As built, on test: Hit@10 = \(51/61 = 0.836\) and \(39/51 = 0.765\). Both requirements fail.
How to read Figure 3
On test with labels v1, compare “Stage 1 only” with “As built”: the red part (retrieval) does not change, because re-ranking and refinement never see moments outside the top 50. The orange part (boundary) grows from 4 to 12: refinement with \(\alpha = 0.5\) breaks spans. Switch to Dev: there are no retrieval or ranking misses at all, so dev could never have taught us about them. Then switch the labels to v2 and look at retrieval again: 8 becomes 2.
Step 3: error analysis: group the failures by cause
Stage attribution tells us where a moment is lost. Error analysis asks why. The method is simple and slow: open every failure, look at the frames and the labels, and write down a cause. Then count the causes. You fix the biggest group first. With 22 missed moments and 81 false positives on test, this takes about an hour, and it is the most useful hour of the project.
Missed moments
For each of the 22 missed moments we read the 5 captions of every photo in it. The causes fall into four groups:
| Cause | Count | Example | Whose fault |
|---|---|---|---|
| Label too broad: the word rule labeled photos that do not show the query | 13 | “runners in a race”: a motorbike race, racing dogs, a rally car; “tennis player”: a dog with a tennis ball | the labels |
| Span too short: found, but the refined span covers only part of the moment | 6 | a 3-shot surf moment whose middle shot has a low score, so the span splits | refinement (\(\alpha\)) |
| Short moment diluted: a 2-s moment in an 8-s window | 2 | a skateboarder grinding a bench (cosine 0.109, high) is averaged with 6 s of other shots | window size |
| Re-ranker said no: in the top 50, pushed out by the VLM | 1 | “a boy hitting a soccer ball with his chest”: VLM log-odds −4.8 | the VLM |
More than half of the misses (13 of 22, 59%) are not errors of the system. The word pattern “race” matches dog races and car races; “tennis” matches tennis balls; “climb” matches ladders and trees. The system was right to skip these photos. This is common in practice: labels made by a rule, by a weaker model or in a hurry carry errors, and the errors sit exactly where the task is hard, so they look like model failures.
False positives
Not all false positives cost the same. AP only charges for a FP that is ranked above a hit; a FP below the last hit does not change AP (it changes P@10). Of the 81 FPs, 28 rank above some hit. We grouped them by a rule that uses the labels and captions of the evidence second (the second with the highest cosine in the span):
| FP group (in this order) | All 81 | The 28 that cost AP |
|---|---|---|
| boundary: overlaps an unfound moment, tIoU below \(\theta\) | 13 | 10 |
| duplicate: overlaps a moment that is already matched | 1 | 0 |
| missing label: 3+ captions match, but no moment there | 2 | 1 |
| another query’s moment: e.g. a skier for “skateboarder doing a trick” | 26 | 5 |
| borderline: 1–2 of 5 captions mention the query | 15 | 6 |
| look-alike scene: shares context words | 12 | 3 |
| unrelated | 12 | 3 |
The largest group of costly FPs is again boundary: 10 of 28. A span with tIoU 0.4 is counted twice, as a FP and as a missed moment. So one cause, short spans from \(\alpha = 0.5\), explains 6 misses and 10 costly FPs. That is the fix to try first. The next group, borderline photos, is a label question again: if 2 of 5 people mention a race, is it one?
- input: ranked results, matched labels, stage attribution, model scores per photo, captions
- for each missed moment: read its photos; tag one cause from a fixed list (label, span, short moment, re-ranker, …); if no tag fits, add a new one and re-tag all
- for each FP: tag by the first rule that fires: boundary, duplicate, missing label, another query, borderline, look-alike, unrelated
- mark FPs ranked above a hit as costly
- count tags; sort by count \(\times\) how cheap the fix is
- for label causes: write a stricter labeling rule and apply it to all videos, never only to failures
- for model causes: pick one knob per cause; tune it on dev only
- return a table of causes with counts and 2–3 examples each (this goes into the report)
How to read Figure 4
Click “Miss: label too broad” and step through: the captions show motorbikes, dogs and ladders. Then click “FP: boundary”: the predicted span (the outlined box) sits inside an orange labeled moment but covers too little of it. Under “FP: another query’s moment”, notice which sports get confused: three of the five costly cases are skateboarder results that land on skier moments. Across all 26 FPs of this group, 13 are team ball sports confused with each other (soccer, American football, basketball). Every case also shows the judge score that Step 6 uses.
Step 4: fix the labels, then tune on dev
Labels v2: a stricter rule, applied to everything
We rewrite the labeling rule. A photo still needs 3 of 5 matching captions, but a caption no longer counts if it also contains an exclusion word: for “runners in a race”: car, dog, motor, horse, bike; for “tennis player”: dog; for “climbing a rock wall”: ladder, stairs, tree, hill, skier, snow; for “swimming in a pool”: ocean, lake, river, sea, raft; for “surfer riding a wave”: wait, sit, line up, stand; for “skier going down”: up, uphill; for “cyclist”: motor; for “American football”: soccer. Each moment is then cut down to the runs of shots whose photos still pass.
We apply the rule to all 48 videos, dev and test. It rejects 31 of the 224 positive shots (14%). Of the 92 moments, 8 disappear and 13 get shorter. Test now has 53 moments; dev keeps its 31, though 4 of them are trimmed. Applying the rule everywhere matters: if you only relabel the failures, every change can only help the system, and the new score is biased upward.
Tune one knob on dev
Error analysis pointed at the refinement threshold \(\alpha\). We sweep it on dev with labels v2 and pick the value with the best dev mAP. Dev is flat at 0.878 from \(\alpha = 0.05\) to 0.30, so we take the middle: \(\alpha = 0.2\). Only then do we run test.
How to read Figure 5
The dev curve is a staircase: with only 31 moments, a small change of \(\alpha\) either flips one moment or nothing. Press “Show test”: the test curve is bumpy (0.765 at 0.2, 0.717 at 0.3, 0.758 at 0.35). If you picked \(\alpha\) by looking at test, you would pick a bump of noise and report it as a gain. At the chosen 0.2 the test result is 0.765, against 0.703 as built.
The final table
| Test split, labels v2 (53 moments) | mAP@0.5 | 95% interval | R@10 | mAP@0.7 | Hit@10 | \(\text{tIoU} \ge 0.5\) among hits |
|---|---|---|---|---|---|---|
| Stage 1 only | 0.669 | [0.555, 0.783] | 0.774 | 0.084 | 0.868 | 0.891 |
| Stage 1 + refine (\(\alpha = 0.2\)) | 0.721 | [0.549, 0.863] | 0.811 | 0.608 | 0.868 | 0.935 |
| Stage 1 + VLM re-rank | 0.634 | [0.524, 0.752] | 0.755 | 0.103 | 0.887 | 0.851 |
| As built (\(\alpha = 0.5\)) | 0.703 | [0.561, 0.836] | 0.811 | 0.623 | 0.925 | 0.878 |
| Tuned (\(\alpha = 0.2\)) | 0.765 | [0.585, 0.912] | 0.868 | 0.670 | 0.925 | 0.939 |
With the audited labels, both design requirements pass for the tuned system: Hit@10 = \(49/53 = 0.925\) and \(46/49 = 0.939\) of hits have tIoU \(\ge 0.5\). Notice that the label fix alone moved the as-built system from 0.554 to 0.703 mAP. Much of the “bad result” of Step 2 was the test set, not the system. The stage view of the tuned system (Figure 3, labels v2, test): 46 found, 3 boundary, 2 ranking, 2 retrieval, so \(0.962 \times 0.961 \times 0.939 = 0.868\).
Step 5: is the difference real?
Tuning raised test mAP from 0.703 to 0.765. Is that a real gain or luck in which 12 queries we have? mAP is an average over queries, and different queries vary a lot (per-query AP of the tuned system goes from 0.0 to 1.0). The question is how much the average would move if we had drawn another 12 queries of the same kind.
The bootstrap answers this without formulas for the distribution. Draw 12 queries with replacement from our 12, recompute the mean, repeat 10,000 times, and read the 2.5th and 97.5th percentiles. For a comparison of two systems, resample the per-query differences, so both systems are always scored on the same queries (a paired bootstrap). Paired tests like this are standard in information-retrieval evaluation [Likely].
- Read the symbols
- \(n\)
- number of queries (12)
- \(j_i\)
- the query drawn at position \(i\) of resample \(b\); the same query can be drawn several times
- \(\Delta^{*(b)}\)
- the mean difference of system A over system B in resample \(b\)
- \(\Delta^{*}_{(2.5\%)}\)
- the 2.5th percentile of the 10,000 resampled differences
Tuned − as built, labels v1: +0.153, interval [+0.071, +0.242]. No resample is \(\le 0\). With the old labels this looks like a certain gain.
Tuned − as built, labels v2: +0.062, interval [−0.083, +0.202]; 19.6% of resamples are \(\le 0\). Much of the old gain came from boundary “errors” on moments whose labels were wrong. With 12 queries we cannot say that the tuned system is better.
Tuned − stage 1 + refine, labels v2 (the value of the VLM re-ranker): +0.044, interval [−0.009, +0.086]. Also not clearly above zero.
How to read Figure 6
Draw one resample at a time first: some queries appear 2 or 3 times and others not at all, and the mean moves with them. With labels v2, one query (“runners”) has a large negative difference: the low threshold makes the span spill from a real race into the next shot, a dog race, which the new labels reject. Then draw 1,000 several times. With labels v1 the whole histogram sits right of zero; with v2 it straddles zero. The width of the interval is set mostly by the number of queries: 12 is a smoke test, which is why the design asked for 500.
What do we do with an interval that includes zero? We report it as it is: “a gain of +0.062 mAP, not significant with 12 queries”. The other metrics point the same way (mAP@0.7 0.623 to 0.670, tIoU-among-hits 0.878 to 0.939), and the change is free (one constant), so we keep \(\alpha = 0.2\) and list “more queries” as the first next step. What we must not do is change the test set, the metric or the seed until the interval looks better.
Step 6: can a VLM judge replace the labels?
Labels are expensive. A popular shortcut is LLM-as-a-judge: ask a strong model whether each result is correct. Zheng et al. found that strong LLM judges agree with human preferences more than 80% of the time, about as often as humans agree with each other, but they also documented position, verbosity and self-enhancement biases [Certain]. Before you trust a judge on your task, you measure it against labels, like any other model.
We use two judges on the 2,285 (photo, query) pairs, with labels from the audited rule (187 pairs where the photo shows the query, 1,992 where no caption mentions it; 106 borderline pairs left out): an independent judge, SmolVLM-256M with a different prompt, and the re-ranker itself, SmolVLM-500M, as a self-judge. Accuracy is a poor measure here: 91.4% of the pairs are negatives, so a judge that always says “no” is 91.4% accurate. We use Cohen’s kappa, which subtracts the agreement expected by chance:
- Read the symbols
- \(p_o\)
- observed agreement: the share of pairs where judge and label agree (accuracy)
- \(p_e\)
- agreement expected if the judge said “yes” at its own rate but at random
- \(\kappa\)
- 1 is perfect agreement, 0 is chance level; values below about 0.4 are usually called weak [Likely]
A judge can also estimate a precision without labels: the share of top-10 results it calls correct. If we know its true-positive rate (TPR) and false-positive rate (FPR) from a small labeled sample, we can correct that share:
Here \(q\) is the share of results the judge accepts and \(\hat p\) the corrected estimate of the true share. This is the classic correction for an imperfect test [Likely]; it is only as good as the TPR and FPR estimates.
Independent judge (SmolVLM-256M). TP 42, FN 145, FP 33, TN 1,959, so \(n = 2{,}179\). Accuracy \(p_o = 2{,}001/2{,}179 = 0.918\). Chance agreement \(p_e = (75 \cdot 187 + 2{,}104 \cdot 1{,}992)/2{,}179^2 = 0.886\). So \(\kappa = (0.918 - 0.886)/(1 - 0.886) = 0.286\): weak. It says “yes” to only 42 of the 187 frames that show the query (TPR 0.225, FPR 0.017).
Self-judge (the re-ranker). TP 139, FN 48, FP 70, TN 1,922: accuracy 0.946, \(\kappa = 0.672\), TPR 0.743, FPR 0.035. It looks much better, but these are partly its own decisions.
Judging the final top-10. For 118 of the 120 test results, both judges scored the evidence frame. The labels say 50 of them (0.424) show the query. The independent judge accepts 22 (0.186): too strict. The self-judge accepts 78 (0.661): it grades its own picks far too kindly. Correcting the independent judge with (237.8) and its rates on all pairs gives \((0.186 - 0.017)/(0.225 - 0.017) = 0.817\): far off, because the negatives in a top-10 list are the hard ones the system chose, and the judge’s FPR on them is higher (0.056 on the dev top-10) than on random pairs. Calibrating TPR and FPR on the dev top-10 instead (120 results, 31 positive) gives 0.556: closer, but 120 labeled items are still few.
How to read Figure 7
Start with the independent judge at threshold 0: most green bars sit left of the line, so the judge misses most true frames, while accuracy is still 0.918 because negatives dominate. Move the threshold down to about −3: TPR rises and kappa improves, at the cost of more false yes answers. Switch to the self-judge: the classes separate better, but its estimate of the final top-10 precision is far above the labels. Watch the “corrected” value: with rates measured on all pairs it overshoots, because hard negatives in a top-10 list fool the judge more often than average negatives.
The lesson is not “judges are useless”. It is: a judge is a model, so evaluate it. Use a judge that is independent of the system (a different model, ideally a stronger one), calibrate its threshold and its TPR and FPR on a labeled sample, report kappa and not only accuracy, and never let the system grade itself.
Step 7: write the technical report
The report is the product of this lab. Readers are busy: put the answer first, numbers with intervals, then the limits and what to do next. One page is enough. Here is ours.
Key-moment search: evaluation of the lab configuration (labels v2)
Summary. On 32 held-out videos (53 moments, 12 queries), the tuned system reaches \(\text{mAP@0.5} = 0.765\) (95% bootstrap interval over queries [0.585, 0.912]), \(R@10 = 0.868\) and \(\text{Hit@10} = 0.925\); 93.9% of hits have \(\text{tIoU} \ge 0.5\). Both design requirements (\(\text{Hit@10} \ge 0.85\); \(\text{tIoU} \ge 0.5\) for 85% of hits) are met on this set. The set is small; the intervals are wide.
Setup
Slideshow videos from Flickr8k test and validation photos, 1 frame per second; dev = 16 videos, test = 32 videos, split by video. Labels from 5 captions per photo with a written rule (v2). SigLIP-B/16 windows of 8 s (stride 4 s), top 50; SmolVLM-500M yes/no re-rank; threshold refinement (\(\alpha\) tuned on dev). This is a smaller configuration than the session-236 build (2-s segments, fused Qwen2.5-VL-3B, contrast refiner); its results do not transfer to that build without re-running.
Results by stage (test)
Stage-1 \(R@50 = 0.962\); re-rank keeps 0.961 of those in the top 10; boundaries are right for 0.939 of those. The VLM re-ranker adds +0.044 mAP over stage 1 + refinement (interval [−0.009, +0.086]); refinement with \(\alpha = 0.2\) adds +0.062 over \(\alpha = 0.5\) ([−0.083, +0.202]).
Failure analysis
Of the 22 misses of the first run, 13 were label errors (a word rule that matched dog and car races, tennis balls, ladders), 6 were spans cut short by \(\alpha = 0.5\), 2 were 2-s moments diluted in 8-s windows, 1 was a re-ranker error. The costly false positives were mostly boundary errors (10 of 28), then borderline photos (6) and other sports with similar scenes (5).
Limits
- 12 queries and 53 test moments: differences under about 0.15 mAP are not resolvable.
- Photos are not video: no motion, no replays, no commentary; real broadcast results will differ.
- Labels come from captions and a rule, not from annotators who watched the video; 14% of the first labels were wrong.
- The re-ranker (500M) is much smaller than the 3B model of the design and of the session-236 build; its frame-level AUC (0.945) is below SigLIP’s (0.971). No latency was measured on an L4.
Next steps
- Grow the test set to at least 100 queries, labeled by two people with measured agreement.
- Add 4-s windows next to 8-s windows (both short-moment misses).
- Run this evaluation on the configuration of session 236 (2-s segments with grouping, which also avoids the short-moment misses; fused Qwen2.5-VL-3B; contrast refiner), with the same label audit and bootstrap; measure p95 latency on an L4 there.
- Compare the threshold refiner with the contrast search of session 236 on the same labels, then try a learned boundary model once labels exist (session 207).
The full program
Algorithm 3 is the whole evaluation loop. The PyTorch program after it contains the metric code used on this page, the paired bootstrap on our real per-query APs, and kappa and the judge correction on our real confusion counts. It runs in under a second, with no downloads, and is deterministic; the printed values are in the comments.
- input: system \(S\) with knobs, labeled videos split into dev and test by video, queries
- run every stage of \(S\) once on dev and test; cache embeddings, candidates, VLM scores
- on dev: compute mAP, R@10, stage attribution (237.5); run the ablations
- run test once; compute the same numbers and bootstrap intervals (237.6)
- error analysis (Algorithm 2) on dev and test failures \(\rightarrow\) table of causes
- label causes \(\rightarrow\) new labeling rule on all videos; re-score every system
- model causes \(\rightarrow\) choose one knob per cause; tune on dev only; run test once
- if a judge is used: measure kappa, TPR, FPR on labeled pairs (237.7, 237.8)
- return a one-page report: summary with intervals, setup, stage table, causes, limits, next steps
import torch def tiou(p, g): # (237.1) spans are (start, end) in seconds inter = max(0.0, min(p[1], g[1]) - max(p[0], g[0])) union = max(p[1], g[1]) - min(p[0], g[0]) return inter / union if union > 0 else 0.0 def match(preds, gts, theta=0.5): # (237.2) greedy, in rank order used, rel = set(), [] for vid, span in preds: cand = [(tiou(span, g[1]), j) for j, g in enumerate(gts) if g[0] == vid and j not in used] best, j = max(cand, default=(0.0, -1)) if best >= theta: used.add(j); rel.append(1) else: rel.append(0) return rel def average_precision(rel, n_gt): # (237.4) divide by ALL labeled moments hits, s = 0, 0.0 for k, r in enumerate(rel, start=1): if r: hits += 1; s += hits / k return s / n_gt if n_gt else 0.0 # 1) the worked example: one query, three labeled moments gts = [("v1", (20, 30)), ("v2", (50, 56)), ("v3", (8, 20))] preds = [("v1", (21, 29)), ("v4", (60, 68)), ("v1", (22, 30)), ("v2", (51, 57)), ("v3", (10, 19))] rel = match(preds, gts) print("relevance:", rel, " P@5:", sum(rel) / 5, " R@5:", sum(rel) / len(gts)) # [1, 0, 0, 1, 1] 0.6 1.0 print("AP:", round(average_precision(rel, len(gts)), 3)) # 0.7 # 2) stage attribution (237.5): counts from the tuned system on test, labels v2 G, n50, n10, found = 53, 51, 49, 46 print("factors:", round(n50 / G, 3), round(n10 / n50, 3), round(found / n10, 3), "R@10:", round(found / G, 3)) # factors: 0.962 0.961 0.939 R@10: 0.868 # 3) paired bootstrap over queries (237.6): real per-query AP on test, labels v2 ap_tuned = torch.tensor([0.688, 1.0, 1.0, 0.976, 0.717, 1.0, 0.861, 0.786, 0.804, 0.0, 0.35, 1.0]) ap_built = torch.tensor([0.25, 1.0, 1.0, 0.786, 0.717, 0.65, 0.861, 0.786, 0.358, 0.5, 0.525, 1.0]) d = ap_tuned - ap_built gen = torch.Generator().manual_seed(0) idx = torch.randint(0, len(d), (10_000, len(d)), generator=gen) # 10,000 resamples of 12 queries means = d[idx].mean(dim=1) lo, hi = torch.quantile(means, torch.tensor([0.025, 0.975])) print("mAP tuned %.3f built %.3f diff %+.3f" % (ap_tuned.mean(), ap_built.mean(), d.mean())) # 0.765 0.703 +0.062 print("95%% interval [%+.3f, %+.3f], share <= 0: %.3f" % (lo, hi, (means <= 0).float().mean())) # 95% interval [-0.084, +0.206], share <= 0: 0.196 (Python random with seed 0 gave [-0.083, +0.202]) # 4) a judge against labels: Cohen's kappa (237.7) and the corrected precision (237.8) def kappa(tp, fn, fp, tn): n = tp + fn + fp + tn po = (tp + tn) / n pe = ((tp + fp) * (tp + fn) + (fn + tn) * (fp + tn)) / n ** 2 return po, pe, (po - pe) / (1 - pe) for name, c in [("independent 256M", (42, 145, 33, 1959)), ("self 500M", (139, 48, 70, 1922))]: po, pe, k = kappa(*c) tpr, fpr = c[0] / (c[0] + c[1]), c[2] / (c[2] + c[3]) print("%-17s acc %.3f p_e %.3f kappa %.3f TPR %.3f FPR %.3f" % (name, po, pe, k, tpr, fpr)) # independent 256M acc 0.918 p_e 0.886 kappa 0.286 TPR 0.225 FPR 0.017 # self 500M acc 0.946 p_e 0.835 kappa 0.672 TPR 0.743 FPR 0.035 q, tpr, fpr = 22 / 118, 42 / 187, 33 / 1992 # the judge accepts 22 of 118 top-10 results print("judge says %.3f, corrected %.3f, labels say %.3f" % (q, (q - fpr) / (tpr - fpr), 50 / 118)) # judge says 0.186, corrected 0.817, labels say 0.424
Where this is used
| Application | What you evaluate | What is special |
|---|---|---|
| Sports key-moment search (this capstone) | R@K of stage 1, mAP@tIoU, Hit@10, tIoU among hits | replays and duplicates; game clock vs video time |
| Temporal grounding benchmarks (QVHighlights, Charades-STA) | R1@IoU=0.5 and 0.7, mAP over several IoU thresholds | one query per video; the metrics in (237.1)–(237.4) are the same ideas [Likely] |
| Text-to-video retrieval (MSR-VTT style) | R@1, R@5, R@10, median rank | one correct video per query, so AP and R@K are simpler |
| Video-RAG and question answering over video | retrieval recall of the evidence clips, then answer accuracy | an answer can be right with wrong evidence: score both stages |
| RAG over documents | recall@K of passages, faithfulness, answer correctness | LLM judges are common; measure them against human labels first |
| Content moderation and fingerprinting | precision at a fixed recall, false-positive rate per hour of video | the costly error type decides the threshold |
| A/B tests of a deployed search | click-through, reformulations, time to a used clip | offline gains must be confirmed online; intervals over users, not queries |
Common mistakes
- Tuning on the test set. Every look at test while choosing a knob leaks it. Figure 5 shows how bumpy a small test curve is: picking its maximum reports noise as a gain.
- Splitting by moment instead of by video. Neighboring shots of the same video then sit in both splits, and the dev score predicts nothing.
- Trusting the labels. In our first run, 13 of 22 misses were label errors. Read the failures before you change the model.
- Fixing only the labels of failures. It can only raise the score. Write a rule and apply it to every video.
- Reporting one end-to-end number. Without stage attribution (237.5) you cannot tell a retrieval problem from a boundary problem; they need opposite fixes.
- Ignoring the interval. +0.062 mAP on 12 queries has an interval from −0.083 to +0.202. Report it, and grow the set.
- Accuracy for a judge on imbalanced data. A judge that says “no” to everything is 91.4% accurate here. Use kappa, TPR and FPR.
- Letting the system grade itself. The re-ranker as judge says its own top-10 is much better than it is, because it chose those results.
Interview questions
How do you evaluate a video moment-retrieval system end to end?basic
Define tIoU, and explain how a ranked list of spans becomes AP.basic
Why divide AP by the number of labeled moments rather than by the number of hits?core
Your dev score is 0.83 and your test score is 0.55. What do you check?core
Why can a boundary error hurt mAP twice?core
How do you know that one system is better than another on 12 queries?core
You discover label errors in the test set. What is the right procedure?core
Would you use an LLM or VLM as a judge instead of human labels?advanced
Why did the self-judge overestimate precision so much?advanced
How do you choose what to fix first after error analysis?advanced
What belongs in a one-page technical report on the system?basic
Exercises
- A query has \(G = 4\) moments. The top 6 results have rel = [0, 1, 1, 0, 0, 1]. Compute \(P@6\), \(R@6\) and AP@6.
- Predicted span [12, 20], labeled moment [14, 26]. What is the tIoU? Is it a hit at \(\theta = 0.5\)?
- Stage 1 covers 51 of 53 moments, 49 survive the re-ranker, 46 are matched. Write the three factors of (237.5) and check the product.
- With the as-built system on labels v1, why did lowering \(\alpha\) also reduce the number of false positives?
- A judge has TPR 0.8 and FPR 0.1 and accepts 50% of the results. Estimate the true precision with (237.8).
- Use Figure 6 with labels v1 and v2. Explain in two sentences why the interval moves.
- Write a stricter labeling rule for “a skateboarder doing a trick”. Which photos would it remove? How would you check that it does not remove good ones?
Answerscheck
Key takeaways
- Label a test set with exact spans, split it by video into dev and test, cache all model outputs, and tune only on dev.
- Measure each stage: stage-1 R@K, the re-ranker’s top 10, and the boundaries; their product is the end-to-end recall (237.5).
- Error analysis means reading every failure and counting causes. In our first run 13 of 22 misses were label errors, and one knob (\(\alpha\)) explained most of the rest.
- Fix labels by a rule applied to every video, version them, and re-score everything.
- Report intervals: with 12 queries, a +0.062 mAP gain has a 95% interval of [−0.083, +0.202].
- A VLM judge is a model: measure its kappa, TPR and FPR against labels, keep it independent of the system, and correct its estimates.
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
- Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
- Gao et al. (2017). TALL: Temporal Activity Localization via Language Query
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Hodosh, Young & Hockenmaier (2013). Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
- Cohen (1960). A Coefficient of Agreement for Nominal Scales
- Efron & Tibshirani (1993). An Introduction to the Bootstrap
- Smucker, Allan & Carterette (2007). A Comparison of Statistical Significance Tests for Information Retrieval Evaluation
- Caba Heilbron, Escorcia, Ghanem & Niebles (2015). ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding