key-moment detection
In this lab you build a training-free key-moment detector and measure it honestly. You score every second of a video in two ways, by SigLIP similarity to an event prompt and by asking a small VLM (SmolVLM-500M) a yes/no question, fuse the two scores, turn the score curve into moments with sliding windows and temporal non-maximum suppression, and evaluate against labeled moments with R1@IoU and mAP, exactly as QVHighlights does. Every number comes from our own run on 30 test videos built from real photos.
What you will learn
- How to build a small moment benchmark with exact labels, with hard negatives, and what its labels can and cannot tell you
- How to score each second of a video by cosine similarity to an event prompt, and by the log-odds of “Yes” from a VLM
- Why you must normalize two scores per video before you add them, and how to choose the fusion weight without fooling yourself
- How to turn a score curve into moments: multi-scale sliding windows, a contrast score against the neighbours, and greedy temporal NMS
- How to compute temporal IoU, R1@0.5, R1@0.7 and mAP over ten IoU thresholds, the same way as the QVHighlights evaluation code
- How to read the failures: near-miss scenes (rugby for soccer, motorcycles for bikes), labels made from words, and windows that are too short
- How a cascade that asks the VLM only about promising shots cuts its cost by about four times at the same mAP
Why this matters
In session 207 you learned what key-moment detection is, how it is measured, and how trained models such as Moment-DETR and UniVTG solve it. In this lab you build a detector that needs no training at all, from two models you can download today, and you measure it with the same metrics as the QVHighlights benchmark. A key moment here is a time span \([s, e]\) of a video where a described event happens, for example “a person riding a bike” from second 63 to second 76.
This training-free design is what many teams build first, because it works for any new event that someone can describe in words. It has two parts. A cheap scorer (SigLIP) compares every frame with a text prompt. An expensive scorer (a small VLM) answers a yes/no question about every frame. You then fuse the two scores, turn the score curve into moments, and remove duplicates with non-maximum suppression in time (temporal NMS). Every number on this page comes from our own run of the code below, unless it is tagged as coming from a paper.
What you will find, in short: on our 30 test videos, the fused detector reaches R1@0.5 = 0.800 and mAP = 0.738, against 0.700 and 0.613 for SigLIP alone and 0.767 and 0.715 for the VLM alone. The way you score a window matters even more than the models: ranking windows by their plain mean score gives mAP = 0.255, because short windows win. And the remaining errors are instructive: rugby taken for soccer, motorcycles taken for bikes, and labels that were made from words rather than from what the picture shows.
Setup
You need Python 3.10 or newer, about 2 GB of disk for the two models and the photos, and ideally a GPU (an Apple-silicon laptop GPU is enough). We used torch 2.15 and transformers 5.23; any recent versions should work.
python -m venv venv && source venv/bin/activate pip install torch transformers pandas pyarrow pillow sentencepiece protobuf # the photos: the test split of the jxie/flickr8k dataset on Hugging Face, saved as flickr8k/test.parquet
The two scorers, with facts read from the loaded checkpoints and from our run [Certain]:
| Cheap scorer: SigLIP B/16 | Expensive scorer: SmolVLM-500M | |
|---|---|---|
| Hugging Face id | google/siglip-base-patch16-224 | HuggingFaceTB/SmolVLM-500M-Instruct |
| Parameters | 203.2 million (both towers) | 507.5 million |
| What it returns per frame | a cosine with the event prompt | logits for the next token after a question |
| Image input | one \(224 \times 224\) image, 196 patches | 13 tiles of \(512 \times 512\) (12 crops plus a global view); 885 tokens per question in total |
| Work per frame and query | image embedded once, then one dot product per query | one full forward pass per (frame, query) pair |
| Our cost | 567 photos embedded in seconds | 793 pairs in 27 minutes, about 2.05 s per pair on a shared laptop GPU |
The last row is the whole economic story of this lab. The image embedding of SigLIP does not depend on the query, so you compute it once per frame and then any new query costs only a dot product. The VLM must read the frame and the question together, so every new query means a new forward pass over every frame. Step 8 shows how to use the cheap scorer to decide where the expensive one is worth running.
The code is four small files, run in order: bench.py builds the test videos, sim_score.py and vlm_score.py compute the two scores, and detect.py turns the scores into moments and evaluates them.
Step 1: a benchmark with exact labels
Real moment benchmarks such as QVHighlights need you to download many YouTube videos, and their labels are noisy: people disagree about where a moment starts and ends (session 207). For a first lab we want something smaller whose labels we fully control. So we build slideshow videos from real photos. Each photo is a shot that stays on screen for 2 to 5 seconds. A moment is 2 to 4 consecutive shots of the event. Everything else is background.
The photos come from the 1,000-image test split of Flickr8k, where every photo has 5 captions written by different people. A photo counts as a clear positive for an event if at least 3 of its 5 captions mention it (for example, both “dog” and “snow” for “a dog playing in the snow”). A photo that 1 or 2 captions mention is unclear, and we never use it. Half of the background shots are hard negatives: photos whose captions share a word with the event (dogs on grass, motorcycles, rugby on a field), because in a real video most of the non-event time looks similar to the event. The other half are random photos.
# bench.py: build 30 "slideshow videos" from Flickr8k test photos, with exact moment labels import pandas as pd, re, json, random d = pd.read_parquet("flickr8k/test.parquet") # jxie/flickr8k test split: 1,000 photos, 5 captions each caps = [[d.iloc[i][f"caption_{j}"].lower() for j in range(5)] for i in range(len(d))] def votes(p): return [sum(bool(re.search(p, c)) for c in cs) for cs in caps] # captions that match EVENTS = { # query: (patterns that must ALL match >= 3 of 5 captions, hard-negative pattern) "a dog playing in the snow": ([r"\bdogs?\b", r"\bsnow"], r"\bdogs?\b|\bsnow"), "a person riding a bike": ([r"\b(bike|bicycl|biker|cyclist)"], r"\b(rid(e|es|ing)|motor\w*)\b"), "people on a beach by the ocean": ([r"\b(beach|ocean|surf|waves?|sea)\b"], r"\b(water|lake|pool|river|swim\w*)\b"), "a person climbing a rock wall": ([r"\bclimb"], r"\b(rock|rocks|mountain|cliff|wall)\b"), "a person on a skateboard": ([r"skate"], r"\b(ramp|snowboard\w*|scooter|roller\w*|jump\w*)\b"), "people playing soccer": ([r"football|soccer"], r"\b(ball|field|rugby|basketball|grass)\b"), } rng = random.Random(0); T = 120; V = 5; videos = [] for qi, (q, (pats, hard)) in enumerate(EVENTS.items()): vs = [votes(p) for p in pats]; hv = votes(hard) pos = [i for i in range(len(d)) if all(v[i] >= 3 for v in vs)] # clear positives amb = [i for i in range(len(d)) if i not in pos and all(v[i] >= 1 for v in vs)] # unclear: never used neg = [i for i in range(len(d)) if i not in pos and i not in amb] hneg = [i for i in neg if hv[i] >= 2] # hard negatives for v in range(V): n_mom = rng.choice([1, 2]) while True: # moment start times, >= 30 s apart starts = sorted(rng.sample(range(5, T - 25), n_mom)) if n_mom == 1 or starts[1] - starts[0] >= 30: break used = set(); pp = pos[:]; rng.shuffle(pp); nn = neg[:]; rng.shuffle(nn); hh = hneg[:]; rng.shuffle(hh) def take(pool): # next photo not yet used in this video while True: x = pool.pop() if x not in used: used.add(x); return x shots = []; moments = []; t = 0; k = 0 while t < T: if k < n_mom and t >= starts[k]: # a moment: 2 to 4 event shots in a row s = t for _ in range(rng.randint(2, 4)): L = rng.randint(2, 5); shots.append([take(pp), t, min(T, t + L), 1]); t += L moments.append([s, min(T, t)]); k += 1 L = rng.randint(2, 5) # background shot: half hard, half random shots.append([take(hh if rng.random() < 0.5 else nn), t, min(T, t + L), 0]); t += L videos.append({"vid": f"q{qi}v{v}", "query": q, "T": T, "shots": shots, "moments": moments}) json.dump({"events": {q: p for q, (p, h) in EVENTS.items()}, "videos": videos}, open("bench.json", "w"))
What it builds [Certain, from our run]:
| Property | Value |
|---|---|
| Videos | 30 (6 events × 5 videos), each 120 s long, 1 frame per second |
| Shots | 1,043 in total, 111 of them event shots; 567 different photos |
| Labeled moments | 41 (19 videos with one moment, 11 with two); 5 to 18 s long, 9.9 s on average |
| Time inside moments | 11.3% of all seconds |
| Clear positives available per event | dog in snow 22, bike 41, beach 55, climbing 24, skateboard 18, soccer 22 |
Be honest about what this benchmark does not test. Real moments have no hard cuts: a goal starts somewhere in a continuous shot, and people disagree about where. Real videos have motion that a single frame cannot show (“a person starts running”). And our labels come from words in captions, which, as Step 7 shows, is itself a source of errors. The benchmark is good for comparing scorers, fusion rules and window rules. It is not a promise about accuracy on football broadcasts.
Step 2: score each second by similarity to an event prompt
The cheap scorer is a dual encoder (session 167). We embed every frame once, embed the prompt “a photo of {query}.” once, and use their cosine as the score of that second:
- Read the symbols
- \(t\)
- the second (the segment) of the video, \(t = 0, \dots, T-1\), with \(T = 120\)
- \(\mathbf{u}_t\)
- unit image embedding of the frame at second \(t\)
- \(\mathbf{w}_q\)
- unit text embedding of the prompt for query \(q\)
- \(a_t\)
- the similarity score of second \(t\); for SigLIP B/16 in our data, event shots have cosines of about 0.01 to 0.12 and background shots about −0.07 to 0.05 (5th to 95th percentile)
Here a segment is one second, sampled as one frame. With real video you would sample 1 or 2 frames per second and average their embeddings over a 1- or 2-second segment (QVHighlights uses 2-second clips [Certain]). Note the SigLIP detail from the earlier lab: its texts must be padded to 64 tokens, as in training.
# sim_score.py: cosine of every benchmark photo with every event prompt (SigLIP and CLIP) import torch, json, io, pandas as pd, torch.nn.functional as F from PIL import Image from transformers import AutoModel, AutoProcessor torch.manual_seed(0); torch.set_grad_enabled(False) PQ = "flickr8k/test.parquet" # the jxie/flickr8k test split (1,000 photos, 5 captions each) d = pd.read_parquet(PQ); B = json.load(open("bench.json")) imgs = sorted({s[0] for v in B["videos"] for s in v["shots"]}); Q = list(B["events"]) res = {"imgs": imgs, "queries": Q} for k, mid in [("siglip", "google/siglip-base-patch16-224"), ("clip", "openai/clip-vit-base-patch32")]: m = AutoModel.from_pretrained(mid).eval(); p = AutoProcessor.from_pretrained(mid) vec = lambda o: o if isinstance(o, torch.Tensor) else o.pooler_output U = [] for a in range(0, len(imgs), 64): ims = [Image.open(io.BytesIO(d.iloc[i]["image"]["bytes"])).convert("RGB") for i in imgs[a:a + 64]] x = p(images=ims, return_tensors="pt")["pixel_values"] U.append(F.normalize(vec(m.get_image_features(pixel_values=x)), dim=-1)) U = torch.cat(U) # (567, d) unit vectors texts = [f"a photo of {q}." for q in Q] # one prompt per event kw = dict(padding="max_length", max_length=64) if k == "siglip" else dict(padding=True) x = p(text=texts, truncation=True, return_tensors="pt", **kw) W = F.normalize(vec(m.get_text_features(**{n: x[n] for n in ("input_ids", "attention_mask") if n in x})), dim=-1) res[k] = (U @ W.T).tolist() # (567, 6) cosines json.dump(res, open("sim.json", "w"))
The table in Step 3 shows how well this cosine separates event photos from background photos. In short: SigLIP separates them well, CLIP B/32 also does, except for soccer, where CLIP is close to a coin flip.
Step 3: score each second with a VLM yes/no question
The expensive scorer asks a VLM about each frame: “Does this image show {query}? Answer yes or no.” We do not let the model generate text. We run one forward pass and read the logits of the next token, the first token of the answer. The difference of the two logits is the log-odds of “Yes” against “No”:
- Read the symbols
- \(\ell_t(\cdot)\)
- the logit the VLM gives to a token at the first answer position
- \(b_t\)
- the VLM score of second \(t\); positive means “Yes” is more likely than “No”
- \(\sigma\)
- the logistic sigmoid; \(\sigma(b_t)\) is the probability of “Yes” if the model had to choose between only “Yes” and “No”
Why log-odds and not the probability? The probability saturates. In our data, about half of the event shots get \(P(\text{Yes})\) above 0.99 and 84% of the background shots get less than 0.01, so the differences you care about (0.9991 against 0.9999) are tiny in probability but clear in log-odds. Log-odds also behave like the cosine: roughly symmetric, no hard floor or ceiling.
- input: frames \(x_0, \dots, x_{T-1}\) (one per segment), query text \(q\), a dual encoder, a VLM
- \(\mathbf{w}_q \leftarrow\) normalize(text_encoder(“a photo of \(q\).”)) // once per query
- for \(t = 0, \dots, T-1\):
- \(\mathbf{u}_t \leftarrow\) normalize(image_encoder(\(x_t\))) // once per frame, cache it: it does not depend on \(q\)
- \(a_t \leftarrow \mathbf{u}_t^\top \mathbf{w}_q\) // formula (215.1)
- \(\ell \leftarrow\) VLM logits at the first answer position for (\(x_t\), “Does this image show \(q\)? Answer yes or no.”)
- \(b_t \leftarrow \ell[\text{“ Yes”}] - \ell[\text{“ No”}]\) // formula (215.2); one forward pass, no generation
- return the two curves \(a_{0..T-1}\) and \(b_{0..T-1}\)
Two practical details decide whether this works. First, find the right token ids. SmolVLM’s chat template ends with “Assistant:”, so the answer token is “ Yes” with a leading space. We checked that the top-1 next tokens for our photos are indeed “ Yes” and “ No”; if you read the id of “Yes” without the space, you read a token the model almost never puts there. Second, keep the question fixed. Changing “Answer yes or no.” to another wording changes the logits, so tune the wording on a validation set and then freeze it.
# vlm_score.py: P(Yes) from SmolVLM-500M for every (photo, query) pair in the benchmark import torch, json, io, pandas as pd from PIL import Image from transformers import AutoProcessor, AutoModelForImageTextToText torch.manual_seed(0); torch.set_grad_enabled(False) MID = "HuggingFaceTB/SmolVLM-500M-Instruct" dev = "mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu") d = pd.read_parquet("flickr8k/test.parquet"); B = json.load(open("bench.json")) proc = AutoProcessor.from_pretrained(MID) model = AutoModelForImageTextToText.from_pretrained(MID, dtype=torch.float32).to(dev).eval() tok = proc.tokenizer YES, = tok.encode(" Yes", add_special_tokens=False) # the answer starts after "Assistant:" NO, = tok.encode(" No", add_special_tokens=False) # so the tokens carry a leading space pairs = sorted({(s[0], v["query"]) for v in B["videos"] for s in v["shots"]}) # 793 pairs out = {} for i, q in pairs: img = Image.open(io.BytesIO(d.iloc[i]["image"]["bytes"])).convert("RGB") msgs = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": f"Does this image show {q}? Answer yes or no."}]}] x = proc(text=proc.apply_chat_template(msgs, add_generation_prompt=True), images=[img], return_tensors="pt").to(dev) lg = model(**x).logits[0, -1].float().cpu() # one forward pass, no generation two = torch.stack([lg[YES], lg[NO]]) out[f"{i}|{q}"] = {"p_yes": torch.softmax(two, 0)[0].item(), "lyes": lg[YES].item(), "lno": lg[NO].item()} json.dump(out, open("vlm.json", "w"))
Now compare the two scorers on single photos. For each event we take the photos used in its 5 videos and compute the AUC: the probability that a random event photo gets a higher score than a random background photo (1.0 is perfect, 0.5 is a coin flip). The last two columns give how often the VLM says “Yes” (\(b_t \gt 0\)) for event photos and for background photos.
| Event (event / background photos) | AUC SigLIP | AUC CLIP B/32 | AUC VLM | VLM “Yes” on events | VLM “Yes” on background |
|---|---|---|---|---|---|
| a dog playing in the snow (13 / 140) | 0.988 | 0.995 | 1.000 | 0.77 | 0.00 |
| a person riding a bike (14 / 95) | 0.983 | 0.980 | 0.978 | 0.93 | 0.08 |
| people on a beach by the ocean (17 / 126) | 0.919 | 0.947 | 0.937 | 0.76 | 0.06 |
| a person climbing a rock wall (12 / 107) | 0.970 | 0.975 | 0.995 | 1.00 | 0.09 |
| a person on a skateboard (10 / 118) | 0.995 | 0.981 | 0.993 | 1.00 | 0.04 |
| people playing soccer (14 / 127) | 0.939 | 0.547 | 0.972 | 0.57 | 0.04 |
| mean | 0.966 | 0.904 | 0.979 |
Both scorers are good on single photos, and the VLM is a little better on average. But per-photo AUC is not the task. A detector must find a span, and a single background photo that scores very high (a motorcycle for “bike”) can win the top rank. The two scorers make different mistakes, which is why fusing them helps.
Step 4: combine the two scores
You cannot simply add \(a_t\) and \(b_t\). In our data the cosines span about 0.26 (from −0.13 to 0.13), the log-odds about 20 (from −11.4 to 8.8). Their sum would be just the VLM. So we first put both on the same scale with a z-score per video (subtract the video’s mean, divide by its standard deviation), and then take a weighted sum:
- Read the symbols
- \(\bar{x}, \operatorname{std}(x)\)
- mean and standard deviation of the curve over the \(T\) seconds of one video
- \(w\)
- the fusion weight: \(w = 1\) is SigLIP only, \(w = 0\) is the VLM only; we use \(w = 0.5\)
- \(f_t\)
- the fused score of second \(t\); a value of 2 means “two standard deviations above this video’s average”
A z-score per video is a choice with a cost. It makes the scores of different videos comparable in scale, and it removes a video-wide offset (a dark video where all cosines are low). But it assumes every video contains something: in a video with no event at all, the best background second still gets a high z-score. For a product that must also say “not in this video”, you need an absolute threshold on the raw scores as well. In our benchmark every video has at least one moment, so z-scores are fine.
In this video the SigLIP curve has mean 0.0164 and standard deviation 0.0316; the VLM curve has mean −4.1415 and standard deviation 5.7532. Some shots (cos = SigLIP cosine, lo = VLM log-odds):
| Seconds | Caption (first of 5) | cos | \(z(a)\) | lo | \(z(b)\) | \(f\) at \(w = 0.5\) |
|---|---|---|---|---|---|---|
| 33–38 | A man racing an orange motorcycle | 0.0721 | 1.76 | 7.39 | 2.00 | 1.88 |
| 63–66 ✓ | A bike rider is doing a stunt on a bicycle over a forest | 0.0675 | 1.62 | 6.74 | 1.89 | 1.75 |
| 66–71 ✓ | A dirt bike rider is in the air pulling off a trick | 0.0331 | 0.53 | 6.40 | 1.83 | 1.18 |
| 71–74 ✓ | A dirt biker leaps through the air | 0.0661 | 1.57 | 6.23 | 1.80 | 1.69 |
| 74–76 ✓ | a boy with a blue helmet is riding a bike | 0.0917 | 2.38 | 6.95 | 1.93 | 2.16 |
| 80–84 | a man doing tricks on a motorcycle | 0.0527 | 1.15 | 5.28 | 1.64 | 1.39 |
Check one value: for the last event shot, \(z(a) = (0.0917 - 0.0164)/0.0316 = 2.38\) and \(z(b) = (6.95 - (-4.1415))/5.7532 = 1.93\), so \(f = 0.5 \times 2.38 + 0.5 \times 1.93 = 2.16\) (rounded). Both scorers like the motorcycles almost as much as the bikes. SigLIP alone puts its top moment on the motorcycle tricks at [80, 84] (tIoU 0), the VLM alone on the motorcycle race at [33, 39] (tIoU 0). The fused curve is high on all four event shots in a row, and its top moment is [63, 75], with tIoU 0.923. Neither scorer alone found this moment; together they did. Notice also that two of the four “bike” shots are dirt bikes, which are motorcycles: the label rule counted the word “bike”. We return to this in Step 7.
How to read Figure 2
Start with q1v4 (“a person riding a bike”) at \(w = 0.5\): the top-1 moment is [63, 75] with tIoU 0.923. Click the shot at 33–38 s: a motorcycle race with VLM log-odds 7.39. Now drag \(w\) to 1 (SigLIP only): the top-1 jumps to [80, 84], the motorcycle tricks, tIoU 0. Drag \(w\) to 0 (VLM only): the top-1 is [33, 39], the motorcycle race. Then look at q5v4 (soccer): at every \(w\) the top moment is a rugby shot at 84–88 s, which both models score as soccer. Finally pick q4v0 (skateboard): an easy video where all three curves agree.
Step 5: from a score curve to moments
A curve of 120 numbers is not yet an answer. The output must be a short ranked list of moments \([s, e]\) with scores. We use the simplest method that works: multi-scale sliding windows. Every window of length 4, 6, 8, 10, 12, 14, 16 or 20 seconds, at every start second, is a candidate (878 per video). Then we give each window a score, sort, and remove duplicates with NMS.
The obvious window score is the mean of \(f_t\) inside the window. It has a hidden bias: short windows win. The best 4-second window inside a moment always has a mean at least as high as the whole moment, because the whole moment includes its weaker shots. So the plain mean returns a short piece of the moment, and the tIoU is low. The fix is a contrast score: the mean inside minus the mean of the two neighbouring flanks. A window that cuts a moment in half has the other half in its flank, which lowers its score. A window that covers the moment exactly has background on both sides, which raises it:
- Read the symbols
- \([s, e)\)
- a candidate window from second \(s\) up to (not including) second \(e\); its length is \(e - s\)
- \(N(s, e)\)
- the two flanks: \(h\) seconds before and \(h\) seconds after the window, clipped at the video’s ends
- \(h\)
- flank width, half the window length but at least 2 seconds
- \(S(s, e)\)
- the window score: high when the inside is high and the surroundings are low
This is the same idea as the boundary-aware scores in trained detectors, done by hand. UniVTG, for example, lets every clip predict whether it is inside the moment and how far the boundaries are (session 207); our flank term is a crude, training-free way to reward clean boundaries.
- input: fused curve \(f_{0..T-1}\), window lengths \(\mathcal{L}\), NMS threshold \(\tau\), number of outputs \(K\)
- \(c \leftarrow\) prefix sums of \(f\) // each window mean is then one subtraction
- for each \(L \in \mathcal{L}\) and each start \(s = 0, \dots, T-L\): add \((S(s, s+L), s, s+L)\) to \(P\) // formula (215.4)
- sort \(P\) by score, high first (ties: earlier start, then earlier end)
- keep \(\leftarrow\) [ ]
- for each \(p \in P\) in order: if \(\operatorname{tIoU}(p, k) \lt \tau\) for every \(k \in\) keep: append \(p\) to keep
- stop when keep has \(K\) moments // we use \(\tau = 0.5\), \(K = 10\)
- return keep, a ranked list of \(K\) moments
Why NMS? Sliding windows produce many near-copies of every good moment: [63, 75], [64, 74], [63, 73] and so on all score high. Without NMS the top 10 would be ten versions of one moment, and the second true moment of the video would never appear in the list. NMS keeps a candidate only if it overlaps every already kept moment by less than \(\tau\). Detection transformers such as Moment-DETR avoid this step because their training loss already punishes duplicates [Certain]; a hand-made proposal method needs it.
Window [63, 75) (12 s, so \(h = 6\)): the mean of \(f\) inside is 1.5313; the flanks [57, 63) and [75, 81) have mean −0.3263; so \(S = 1.5313 - (-0.3263) = 1.8577\). Its tIoU with the truth [63, 76] is \(12/13 = 0.923\).
Window [72, 76) (4 s, \(h = 2\)): the inside mean is higher, 1.9213, because it covers only the two strongest event shots. But its flanks [70, 72) and [76, 78) include event seconds and have mean 0.3321, so \(S = 1.9213 - 0.3321 = 1.5892\). Its tIoU is only \(4/13 = 0.308\).
With the plain mean, [72, 76) is the top-1 moment (1.9213 against 1.5313). With the contrast score, [63, 75) is (1.8577 against 1.5892). The perfect window [63, 76) would score even higher, 1.9693, but a 13-second window is not in our list of lengths, so 0.923 is the best tIoU this grid can give here.
How to read Figure 3
With the contrast score and \(\tau = 0.5\), step 1 keeps [63, 75] (tIoU 0.923 with the truth). The next candidates are near-copies such as [64, 76] and are suppressed. Press Play: 24 candidates are examined to keep 5 moments, and three of the five sit on motorcycle shots. Note that [72, 76] is kept even though it lies inside [63, 75]: its tIoU with [63, 75] is only \(3/13 = 0.23\), below \(\tau\). A short window inside a long one has a small tIoU, so NMS does not remove it. Now switch to the plain mean: the top-1 becomes [72, 76] (tIoU 0.308), and the list fills up with short pieces of moments.
Step 6: evaluate with tIoU-based metrics
We use the metrics of session 207 and compute them the way the QVHighlights evaluation code does [Certain, from reading standalone_eval]. Temporal IoU compares a predicted moment \(\hat{M} = [\hat{s}, \hat{e}]\) with a true moment \(M = [s, e]\):
R1@\(m\) looks only at the top-1 prediction. When a video has two true moments, the top-1 is compared with the one it overlaps most. mIoU is the average of that best tIoU:
mAP looks at the whole ranked list of \(K = 10\) predictions. Walk down the list; a prediction is a true positive if its tIoU with a still-unmatched true moment is at least \(\theta\); each true moment can be matched once. Average precision is the area under the precision–recall curve, with precision made non-increasing (the VOC style), and mAP averages over videos and over ten thresholds:
- Read the symbols
- \(N\)
- the number of (video, query) pairs, 30 here
- \(\hat{M}_{n,1}\)
- the top-ranked prediction for pair \(n\); \(M_{n,j}\) its \(j\)-th true moment
- \(G_n\)
- the number of true moments of pair \(n\) (1 or 2 here)
- \(\text{prec}_j\)
- precision after the first \(j\) predictions: true positives so far divided by \(j\)
- \(\theta\)
- the tIoU threshold that makes a prediction count as a true positive
- input: ranked predictions \(p_1, \dots, p_K\), true moments \(M_1, \dots, M_G\), threshold \(\theta\)
- used \(\leftarrow\) [false] \(\times\, G\)
- for \(i = 1, \dots, K\): \(j^* \leftarrow \argmax_j \operatorname{tIoU}(p_i, M_j)\) over unused \(j\)
- if \(\operatorname{tIoU}(p_i, M_{j^*}) \ge \theta\): \(\text{TP}_i \leftarrow 1\), used[\(j^*\)] \(\leftarrow\) true else \(\text{TP}_i \leftarrow 0\)
- \(\text{prec}_i \leftarrow (\text{TP}_1 + \dots + \text{TP}_i) / i\)
- return \(\frac{1}{G} \sum_{i:\,\text{TP}_i = 1} \max_{j \ge i} \text{prec}_j\) // formula (215.7); unmatched true moments add 0
Here is the whole detector and evaluator. It is plain PyTorch and Python, it runs in a few seconds, and it is deterministic: there is no randomness, and ties are broken by start and end time.
# detect.py: fuse per-second scores, propose windows, temporal NMS, tIoU metrics import json, torch B = json.load(open("bench.json")); S = json.load(open("sim.json")); VL = json.load(open("vlm.json")) Q = S["queries"]; row = {im: n for n, im in enumerate(S["imgs"])} def curves(v, enc="siglip"): # one value per second: shots are held still q = Q.index(v["query"]); a = torch.zeros(v["T"], dtype=torch.float64); b = torch.zeros_like(a) for im, s, e, _ in v["shots"]: a[s:e] = S[enc][row[im]][q] # cosine with the event prompt r = VL[f"{im}|{v['query']}"]; b[s:e] = r["lyes"] - r["lno"] # VLM log-odds of "Yes" return a, b def z(x): return (x - x.mean()) / (x.std(unbiased=False) + 1e-8) # per-video z-score def tiou(p, g): inter = max(0.0, min(p[1], g[1]) - max(p[0], g[0])) return inter / (max(p[1], g[1]) - min(p[0], g[0])) LENS = (4, 6, 8, 10, 12, 14, 16, 20) # window lengths in seconds, stride 1 s def windows(f, mode="contrast"): T = len(f); c = [0.0] + f.cumsum(0).tolist(); out = [] for L in LENS: for s in range(T - L + 1): e = s + L; sc = (c[e] - c[s]) / L # mean score inside if mode == "contrast": # minus mean of the flanks fl = max(2, L // 2); a0, b1 = max(0, s - fl), min(T, e + fl) n = (s - a0) + (b1 - e) sc -= ((c[s] - c[a0]) + (c[b1] - c[e])) / n if n else 0.0 out.append((round(sc, 9), s, e)) # round: stable ties return sorted(out, key=lambda p: (-p[0], p[1], p[2])) # deterministic tie-break def nms(props, thr=0.5, k=10): # greedy temporal non-maximum suppression keep = [] for p in props: if all(tiou(p[1:], q[1:]) < thr for q in keep): keep.append(p) if len(keep) == k: break return keep def average_precision(preds, gts, th): # interpolated AP for one query used = [False] * len(gts); tp = [] for p in preds: ious = [-1 if used[j] else tiou(p[1:], g) for j, g in enumerate(gts)] j = max(range(len(gts)), key=lambda j: ious[j]) hit = ious[j] >= th; tp.append(hit) if hit: used[j] = True ap, best, hits = 0.0, 0.0, 0 prec = [sum(tp[:i + 1]) / (i + 1) for i in range(len(tp))] for i in range(len(tp)): # area under the monotone PR curve if tp[i]: ap += max(prec[i:]) / len(gts) return ap TH = [round(0.5 + 0.05 * i, 2) for i in range(10)] def evaluate(w=0.5, mode="contrast", thr=0.5, k=10, enc="siglip"): n = len(B["videos"]); r5 = r7 = miou = 0.0; ap = {t: 0.0 for t in TH} for v in B["videos"]: a, b = curves(v, enc); f = w * z(a) + (1 - w) * z(b) P = nms(windows(f, mode), thr, k) if thr < 1 else windows(f, mode)[:k] g = v["moments"]; best = max(tiou(P[0][1:], gg) for gg in g) r5 += best >= 0.5; r7 += best >= 0.7; miou += best for t in TH: ap[t] += average_precision(P, g, t) m = {t: x / n for t, x in ap.items()} return {"R1@0.5": r5 / n, "R1@0.7": r7 / n, "mIoU": miou / n, "mAP@0.5": m[0.5], "mAP@0.75": m[0.75], "mAP": sum(m.values()) / len(TH)} if __name__ == "__main__": torch.manual_seed(0) for name, w in [("similarity only", 1.0), ("VLM only", 0.0), ("fused 50/50", 0.5)]: r = evaluate(w); print(f"{name:16s}", " ".join(f"{k} {x:.3f}" for k, x in r.items()))
Our results on the 30 videos (\(w = 0.5\), contrast score, \(\tau = 0.5\), \(K = 10\), unless stated):
| Detector | R1@0.5 | R1@0.7 | mIoU | mAP@0.5 | mAP@0.75 | mAP |
|---|---|---|---|---|---|---|
| CLIP B/32 only | 0.600 | 0.500 | 0.552 | 0.657 | 0.503 | 0.499 |
| SigLIP only (\(w = 1\)) | 0.700 | 0.633 | 0.651 | 0.773 | 0.633 | 0.613 |
| VLM only (\(w = 0\)) | 0.767 | 0.733 | 0.734 | 0.838 | 0.764 | 0.715 |
| CLIP + VLM | 0.767 | 0.767 | 0.728 | 0.836 | 0.697 | 0.706 |
| SigLIP + VLM | 0.800 | 0.767 | 0.746 | 0.867 | 0.746 | 0.738 |
| SigLIP + VLM, plain mean score | 0.400 | 0.067 | 0.387 | 0.562 | 0.232 | 0.255 |
| SigLIP + VLM, no NMS | 0.800 | 0.767 | 0.746 | 0.787 | 0.732 | 0.697 |
Four lessons. (1) Fusion helps: SigLIP + VLM beats each scorer alone on R1 and on mAP, and CLIP + VLM beats CLIP alone by a lot. (2) The window score matters most: with the plain mean, R1@0.7 falls to 0.067, because the top window is a short piece of the moment. (3) NMS does not change R1 at all, since the top-1 candidate is always kept, but it raises mAP@0.5 from 0.787 to 0.867, because the second true moment can now enter the top 10. (4) A good per-photo AUC (0.966 for SigLIP) does not mean a good detector (R1@0.5 = 0.700): one high-scoring wrong shot is enough to take the top rank.
How sure are these differences? With 30 videos, one video is 3.3 points of R1. We resampled the 30 videos with replacement 10,000 times (a bootstrap). The 95% interval of the fused mAP is [0.632, 0.834]. The paired gain of fusion over SigLIP alone is +0.125, interval [0.046, 0.211]: fusion is better on 13 videos, worse on 2, equal on 15. The gain over the VLM alone is +0.023, interval [−0.048, 0.094]: better on 8 videos, worse on 6. So “fusion beats SigLIP” is solid here; “fusion beats the VLM” is not proven by 30 videos.
detect.py, rerun live in your browser on the measured scores of all 30 videos. The curves show three metrics for every \(w\) from 0 to 1 in steps of 0.05; the vertical line is the current \(w\). Change the window score or the NMS threshold to rerun everything.How to read Figure 4
At the default (\(w = 0.5\), contrast, \(\tau = 0.5\)) the readout gives R1@0.5 = 0.800, R1@0.7 = 0.767, mAP@0.5 = 0.867 and mAP = 0.738, the bold row of the table. The left end of the curves is the VLM alone (mAP 0.715), the right end SigLIP alone (0.613). The best mAP is 0.750 at \(w = 0.4\), but the curve is bumpy: on 30 videos, picking the best \(w\) on the same videos you report is overfitting. Click “no NMS”: R1 does not move, mAP@0.5 drops to 0.787. Click “plain mean”: every curve collapses.
A note on highlight detection, the second task of QVHighlights: there, the output is a score per clip, and HIT@1 asks whether the single highest-scoring clip is a true highlight. The per-second curves of this lab give that for free. The top second falls inside a true moment in 25 of 30 videos (0.833) for SigLIP alone and in 26 of 30 (0.867) for the VLM alone and for the fused score.
Step 7: study the failures
Averages hide the reasons. Six videos have a fused top-1 below tIoU 0.5. Looking at them one by one is the most useful hour of this lab:
| Video, query | Top-1 moment | What is on screen (first caption) | Why |
|---|---|---|---|
| q5v4 and q5v2, “people playing soccer” | [84, 88] and [60, 64] | A bunch of men play rugby on a muddy field | Near-miss scene: SigLIP cosine 0.106, VLM log-odds 7.13 (P(Yes) = 0.999). Both models see “team sport on grass”. |
| q1v2, “a person riding a bike” | [109, 113] | A man racing an orange motorcycle | Near-miss object; the VLM says Yes with log-odds 7.39. Is a motorcycle a “bike”? English says “sometimes”. |
| q2v4, “people on a beach by the ocean” | [94, 106] | A man and a woman at the waterfront; a child on a wooden dock | Near-miss scene: water but no beach; the VLM says Yes to both (\(P = 0.91\) and 0.90). |
| q2v0, “people on a beach by the ocean” | [22, 26] | Two men sitting on the bank of a lake with an ice chest | Same family: SigLIP likes the lake (cosine 0.065) while the VLM leans No (log-odds −1.77). The VLM alone gets this video right; here fusion hurts. |
| q0v3, “a dog playing in the snow” | [82, 86], tIoU 0.29 | A dog sled team seen from behind (labeled positive, at 79–82) | Label versus question: the captions say dog and snow, but a sled team is not “playing”. The VLM says No (log-odds −5.48), so the detector cuts the moment in half. |
There are three kinds of error here, and each has a different fix.
- Near-miss scenes and objects (rugby, motorcycles, lakes). Both scorers make the same mistake, so fusion cannot fix it. Fixes: a more specific query (“a person pedaling a bicycle”), a negative prompt (subtract the similarity to “a photo of rugby”), or a stronger VLM as the verifier for the top candidates only.
- Labels that disagree with the question. Our labels come from words in captions. “A boy dressed in soccer attire … getting out of a car” is labeled as soccer (the VLM says No, log-odds −3.70, and it is right). Dirt bikes are labeled as bikes. A real benchmark has the same problem in another form: the annotator’s idea of the event is not the user’s. Read your false positives and false negatives before you trust a 3-point change.
- Boundaries. When one shot inside a moment scores low (the dog sled), the best-scoring window covers only part of the moment. Fixes: smooth the curve before windowing, add a gap-tolerant grouping step, or train a small boundary regressor as in TALL or UniVTG (session 207).
Step 8: make it cheaper with a cascade
The VLM cost us 27 minutes for 793 photo–query pairs; SigLIP cost seconds. In production, the VLM is the bill. A cascade runs the expensive model only where the cheap model says “maybe”. We ask the VLM only about shots whose SigLIP z-score is at least a threshold \(\tau_c\). The other shots get the lowest VLM score of the video, as if the VLM had said a clear No.
| Cascade threshold \(\tau_c\) on z(a) | Shots sent to the VLM | R1@0.5 | mAP |
|---|---|---|---|
| none (all shots) | 100% | 0.800 | 0.738 |
| 0.0 | 44.8% | 0.733 | 0.711 |
| 0.5 | 26.3% | 0.767 | 0.739 |
| 1.0 | 15.6% | 0.667 | 0.624 |
| 1.5 | 9.9% | 0.533 | 0.505 |
At \(\tau_c = 0.5\) the VLM sees about a quarter of the shots and the mAP is unchanged (0.739 against 0.738). Going further hurts quickly, because event shots that SigLIP scores low never reach the VLM (at \(\tau_c = 1.0\), the dirt-bike shot of q1v4 with z(a) = 0.53 is never asked). The numbers are not monotone in \(\tau_c\) (0.0 is worse than 0.5): filling unasked shots with the minimum changes the z-scores of the VLM curve, and 30 videos are few. Treat the shape, not the exact values, as the result. The same idea runs at scale in session 211 pipelines: cheap embeddings for everything, expensive models only on candidates.
Where this is used
| Application | What from this lab carries over |
|---|---|
| Sports highlights (goals, dunks, crashes) | Per-segment scores against event prompts, a VLM verifier for top candidates, temporal NMS so one goal gives one clip; near-miss scenes (rugby for soccer) are the main error |
| Video chapters and “key moments” in search results | Windows with a contrast score (215.4) give clean boundaries; mAP over tIoU thresholds measures them |
| Content moderation of long videos | The VLM yes/no log-odds (215.2) as a policy check per segment, with a cascade (Step 8) to control cost; absolute thresholds instead of per-video z-scores, since most videos contain nothing |
| Security and body-camera review | Text queries for rare events over hours of footage; frame embeddings computed once and reused for every new query |
| Video-RAG | The same scores retrieve the moments a VLM should watch before it answers a question (session 210) |
| Text-to-video retrieval | Averaging segment embeddings over a moment gives clip vectors for a search index (session 208) |
| Training data for trained detectors | Training-free detections as pseudo-labels, the idea behind the large-scale pseudo supervision of UniVTG [Likely] |
Common mistakes
- Adding raw scores from different models. A cosine (span about 0.26) plus log-odds (span about 20) is just the log-odds. Normalize each score first, per video or with statistics from a validation set.
- Ranking windows by their mean score. Short windows inside a moment always win. In our run this alone took R1@0.7 from 0.767 to 0.067. Use a contrast score, a length prior, or a boundary model.
- Forgetting NMS, or believing NMS removes all duplicates. Without NMS mAP@0.5 fell from 0.867 to 0.787. With NMS, a short window inside a long one still survives, because its tIoU with the long one is small.
- Reading the wrong token for “Yes”. After “Assistant:” the answer token has a leading space (“ Yes”). Check the top-1 next tokens on a few examples before you trust \(P(\text{Yes})\).
- Using \(P(\text{Yes})\) instead of log-odds. Probabilities saturate near 0 and 1, which flattens exactly the differences between strong candidates.
- Tuning the fusion weight on the test videos. The best \(w\) on our 30 videos (0.4) is partly noise. Choose \(w\) on separate validation videos and report the test set once.
- Reporting R1 or mAP without the protocol. State the tIoU thresholds, \(\ge\) or \(\gt\), how many predictions per query enter mAP (10 here), and how a top-1 is matched when there are several true moments.
- Trusting labels you made from text. Caption words label a dog sled as “a dog playing” and a dirt bike as a “bike”. Look at errors before you decide that a model is wrong.
Interview questions
Design a key-moment detector for a new event type with no labeled training data.core
Why normalize the two scores before fusing them, and what is the risk of a per-video z-score?core
Why does ranking windows by their mean score give poor tIoU, and how do you fix it?core
What does temporal NMS do, and why did it change mAP but not R1?basic
How do you turn a generative VLM into a per-frame scorer, and what can go wrong?core
The VLM is far more expensive per frame than the embedding scorer. How do you cut its cost?advanced
Your detector reports R1@0.5 = 0.80 on 30 videos. A change brings it to 0.83. Ship it?core
Moment-DETR needs no NMS. Why does your pipeline need it?core
What is Soft-NMS, and when would it help here?advanced
Your training-free detector works on the slideshow benchmark. What changes on real video?advanced
Exercises
- A prediction [63, 75] is compared with the truth [63, 76], and a prediction [72, 76] with the same truth. Compute both tIoUs. Which R1 thresholds (0.5, 0.7) does each pass?
- Window A has inside mean 1.5313 and flank mean −0.3263; window B has inside mean 1.9213 and flank mean 0.3321. Which wins with the plain mean, and which with the contrast score (215.4)?
- A video has 2 true moments. The ranked predictions are TP, FP, FP, TP at some threshold \(\theta\). Compute the precisions and the AP of formula (215.7).
- The VLM gives log-odds 7.39 for one shot and −7.44 for another. What is \(P(\text{Yes})\) for each? Why is 0.9994 against 0.9999 a bad scale to compare two strong candidates?
- Our VLM run took 27 minutes for 793 pairs. Estimate the time for 1 hour of video at 1 frame per second and 5 queries, without and with the cascade at \(\tau_c = 0.5\) (26.3% of shots).
- Our window lengths are all even, so the true moment [63, 76] (13 s) cannot be matched exactly. Name two ways to get better boundaries without training a model.
Answerscheck
Key takeaways
- A training-free key-moment detector has five parts: a cheap per-segment scorer (215.1), an expensive verifier (215.2), a fusion rule (215.3), windows with a boundary-aware score plus temporal NMS (215.4), and tIoU-based evaluation (215.5 to 215.7).
- On our 30 videos, SigLIP + SmolVLM-500M reached R1@0.5 = 0.800 and mAP = 0.738, against 0.700 / 0.613 for SigLIP alone and 0.767 / 0.715 for the VLM alone. The gain over SigLIP is solid (bootstrap interval [0.046, 0.211]); the gain over the VLM is not proven by 30 videos.
- The window score mattered more than either model: the plain mean gave mAP 0.255 and R1@0.7 0.067, because short windows win.
- NMS never changes R1 but raised mAP@0.5 from 0.787 to 0.867; a short window inside a long one survives NMS because their tIoU is small.
- Read the VLM as log-odds of the “ Yes” token, with the leading space. Normalize scores before you add them.
- Most remaining errors are near-miss scenes (rugby for soccer, motorcycles for bikes, lakes for beaches) and labels made from words. A cascade sent only 26.3% of shots to the VLM at the same mAP.
Key formulas
References for this session
- Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
- Lin et al. (2023). UniVTG: Towards Unified Video-Language Temporal Grounding
- Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
- Lin et al. (2023). UniVTG: Towards Unified Video-Language Temporal Grounding
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
- Hodosh, Young & Hockenmaier (2013). Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (Journal of Artificial Intelligence Research)
- Lei et al. (code). moment_detr repository: standalone_eval/eval.py and utils.py (QVHighlights evaluation)
- Bodla et al. (2017). Soft-NMS: Improving Object Detection With One Line of Code
- Gao et al. (2017). TALL: Temporal Activity Localization via Language Query
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Hugging Face (model card). HuggingFaceTB/SmolVLM-500M-Instruct
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models