LLM & VLM courseBlog
215Track 8: Video and multimodal systems · session 15 of 21

key-moment detection

In this lab you build a training-free key-moment detector and measure it honestly. You score every second of a video in two ways, by SigLIP similarity to an event prompt and by asking a small VLM (SmolVLM-500M) a yes/no question, fuse the two scores, turn the score curve into moments with sliding windows and temporal non-maximum suppression, and evaluate against labeled moments with R1@IoU and mAP, exactly as QVHighlights does. Every number comes from our own run on 30 test videos built from real photos.

Time: 3 to 4 hours (the VLM scoring run takes about 30 minutes on a laptop GPU)Level: Beginner · hands-on labBefore this: Session 207 (temporal grounding and key-moment detection) and Session 167 (SigLIP); Session 205 (Video-LLMs) helps; Python and basic PyTorch

What you will learn

  • How to build a small moment benchmark with exact labels, with hard negatives, and what its labels can and cannot tell you
  • How to score each second of a video by cosine similarity to an event prompt, and by the log-odds of “Yes” from a VLM
  • Why you must normalize two scores per video before you add them, and how to choose the fusion weight without fooling yourself
  • How to turn a score curve into moments: multi-scale sliding windows, a contrast score against the neighbours, and greedy temporal NMS
  • How to compute temporal IoU, R1@0.5, R1@0.7 and mAP over ten IoU thresholds, the same way as the QVHighlights evaluation code
  • How to read the failures: near-miss scenes (rugby for soccer, motorcycles for bikes), labels made from words, and windows that are too short
  • How a cascade that asks the VLM only about promising shots cuts its cost by about four times at the same mAP

Why this matters

In session 207 you learned what key-moment detection is, how it is measured, and how trained models such as Moment-DETR and UniVTG solve it. In this lab you build a detector that needs no training at all, from two models you can download today, and you measure it with the same metrics as the QVHighlights benchmark. A key moment here is a time span \([s, e]\) of a video where a described event happens, for example “a person riding a bike” from second 63 to second 76.

This training-free design is what many teams build first, because it works for any new event that someone can describe in words. It has two parts. A cheap scorer (SigLIP) compares every frame with a text prompt. An expensive scorer (a small VLM) answers a yes/no question about every frame. You then fuse the two scores, turn the score curve into moments, and remove duplicates with non-maximum suppression in time (temporal NMS). Every number on this page comes from our own run of the code below, unless it is tagged as coming from a paper.

What you will find, in short: on our 30 test videos, the fused detector reaches R1@0.5 = 0.800 and mAP = 0.738, against 0.700 and 0.613 for SigLIP alone and 0.767 and 0.715 for the VLM alone. The way you score a window matters even more than the models: ranking windows by their plain mean score gives mAP = 0.255, because short windows win. And the remaining errors are instructive: rugby taken for soccer, motorcycles taken for bikes, and labels that were made from words rather than from what the picture shows.

Setup

You need Python 3.10 or newer, about 2 GB of disk for the two models and the photos, and ideally a GPU (an Apple-silicon laptop GPU is enough). We used torch 2.15 and transformers 5.23; any recent versions should work.

python -m venv venv && source venv/bin/activate
pip install torch transformers pandas pyarrow pillow sentencepiece protobuf
# the photos: the test split of the jxie/flickr8k dataset on Hugging Face, saved as flickr8k/test.parquet

The two scorers, with facts read from the loaded checkpoints and from our run [Certain]:

Cheap scorer: SigLIP B/16Expensive scorer: SmolVLM-500M
Hugging Face idgoogle/siglip-base-patch16-224HuggingFaceTB/SmolVLM-500M-Instruct
Parameters203.2 million (both towers)507.5 million
What it returns per framea cosine with the event promptlogits for the next token after a question
Image inputone \(224 \times 224\) image, 196 patches13 tiles of \(512 \times 512\) (12 crops plus a global view); 885 tokens per question in total
Work per frame and queryimage embedded once, then one dot product per queryone full forward pass per (frame, query) pair
Our cost567 photos embedded in seconds793 pairs in 27 minutes, about 2.05 s per pair on a shared laptop GPU

The last row is the whole economic story of this lab. The image embedding of SigLIP does not depend on the query, so you compute it once per frame and then any new query costs only a dot product. The VLM must read the frame and the question together, so every new query means a new forward pass over every frame. Step 8 shows how to use the cheap scorer to decide where the expensive one is worth running.

video1 frame per s+ query text SigLIP: cosineframe vs “a photo of {q}.” VLM: yes or no?log P(Yes) − log P(No) cheap: one dot productexpensive: one forward pass z-score, fusew·z(a) + (1−w)·z(b) windows4–20 s, contrast temporal NMSkeep top 10 evaluateR1@0.5, R1@0.7mAP@0.5:0.95 labelstrue moments
Figure 1. The pipeline of this lab. The two scorers see the same frames. SigLIP needs one image embedding per frame, reused for every query; the VLM needs one forward pass per frame and query. Everything after the fusion box is cheap arithmetic on a curve of 120 numbers per video.

The code is four small files, run in order: bench.py builds the test videos, sim_score.py and vlm_score.py compute the two scores, and detect.py turns the scores into moments and evaluates them.

Step 1: a benchmark with exact labels

Real moment benchmarks such as QVHighlights need you to download many YouTube videos, and their labels are noisy: people disagree about where a moment starts and ends (session 207). For a first lab we want something smaller whose labels we fully control. So we build slideshow videos from real photos. Each photo is a shot that stays on screen for 2 to 5 seconds. A moment is 2 to 4 consecutive shots of the event. Everything else is background.

The photos come from the 1,000-image test split of Flickr8k, where every photo has 5 captions written by different people. A photo counts as a clear positive for an event if at least 3 of its 5 captions mention it (for example, both “dog” and “snow” for “a dog playing in the snow”). A photo that 1 or 2 captions mention is unclear, and we never use it. Half of the background shots are hard negatives: photos whose captions share a word with the event (dogs on grass, motorcycles, rugby on a field), because in a real video most of the non-event time looks similar to the event. The other half are random photos.

# bench.py: build 30 "slideshow videos" from Flickr8k test photos, with exact moment labels
import pandas as pd, re, json, random
d = pd.read_parquet("flickr8k/test.parquet")   # jxie/flickr8k test split: 1,000 photos, 5 captions each
caps = [[d.iloc[i][f"caption_{j}"].lower() for j in range(5)] for i in range(len(d))]
def votes(p): return [sum(bool(re.search(p, c)) for c in cs) for cs in caps]   # captions that match
EVENTS = {  # query: (patterns that must ALL match >= 3 of 5 captions, hard-negative pattern)
 "a dog playing in the snow":      ([r"\bdogs?\b", r"\bsnow"], r"\bdogs?\b|\bsnow"),
 "a person riding a bike":         ([r"\b(bike|bicycl|biker|cyclist)"], r"\b(rid(e|es|ing)|motor\w*)\b"),
 "people on a beach by the ocean": ([r"\b(beach|ocean|surf|waves?|sea)\b"], r"\b(water|lake|pool|river|swim\w*)\b"),
 "a person climbing a rock wall":  ([r"\bclimb"], r"\b(rock|rocks|mountain|cliff|wall)\b"),
 "a person on a skateboard":       ([r"skate"], r"\b(ramp|snowboard\w*|scooter|roller\w*|jump\w*)\b"),
 "people playing soccer":          ([r"football|soccer"], r"\b(ball|field|rugby|basketball|grass)\b"),
}
rng = random.Random(0); T = 120; V = 5; videos = []
for qi, (q, (pats, hard)) in enumerate(EVENTS.items()):
    vs = [votes(p) for p in pats]; hv = votes(hard)
    pos = [i for i in range(len(d)) if all(v[i] >= 3 for v in vs)]                  # clear positives
    amb = [i for i in range(len(d)) if i not in pos and all(v[i] >= 1 for v in vs)]  # unclear: never used
    neg = [i for i in range(len(d)) if i not in pos and i not in amb]
    hneg = [i for i in neg if hv[i] >= 2]                                           # hard negatives
    for v in range(V):
        n_mom = rng.choice([1, 2])
        while True:                                   # moment start times, >= 30 s apart
            starts = sorted(rng.sample(range(5, T - 25), n_mom))
            if n_mom == 1 or starts[1] - starts[0] >= 30: break
        used = set(); pp = pos[:]; rng.shuffle(pp); nn = neg[:]; rng.shuffle(nn); hh = hneg[:]; rng.shuffle(hh)
        def take(pool):                               # next photo not yet used in this video
            while True:
                x = pool.pop()
                if x not in used: used.add(x); return x
        shots = []; moments = []; t = 0; k = 0
        while t < T:
            if k < n_mom and t >= starts[k]:          # a moment: 2 to 4 event shots in a row
                s = t
                for _ in range(rng.randint(2, 4)):
                    L = rng.randint(2, 5); shots.append([take(pp), t, min(T, t + L), 1]); t += L
                moments.append([s, min(T, t)]); k += 1
            L = rng.randint(2, 5)                     # background shot: half hard, half random
            shots.append([take(hh if rng.random() < 0.5 else nn), t, min(T, t + L), 0]); t += L
        videos.append({"vid": f"q{qi}v{v}", "query": q, "T": T, "shots": shots, "moments": moments})
json.dump({"events": {q: p for q, (p, h) in EVENTS.items()}, "videos": videos}, open("bench.json", "w"))

What it builds [Certain, from our run]:

PropertyValue
Videos30 (6 events × 5 videos), each 120 s long, 1 frame per second
Shots1,043 in total, 111 of them event shots; 567 different photos
Labeled moments41 (19 videos with one moment, 11 with two); 5 to 18 s long, 9.9 s on average
Time inside moments11.3% of all seconds
Clear positives available per eventdog in snow 22, bike 41, beach 55, climbing 24, skateboard 18, soccer 22

Be honest about what this benchmark does not test. Real moments have no hard cuts: a goal starts somewhere in a continuous shot, and people disagree about where. Real videos have motion that a single frame cannot show (“a person starts running”). And our labels come from words in captions, which, as Step 7 shows, is itself a source of errors. The benchmark is good for comparing scorers, fusion rules and window rules. It is not a promise about accuracy on football broadcasts.

Step 2: score each second by similarity to an event prompt

The cheap scorer is a dual encoder (session 167). We embed every frame once, embed the prompt “a photo of {query}.” once, and use their cosine as the score of that second:

\[ a_t = \mathbf{u}_t^\top \mathbf{w}_q, \qquad \lVert \mathbf{u}_t \rVert = \lVert \mathbf{w}_q \rVert = 1 \tag{215.1} \]
Read the symbols
\(t\)
the second (the segment) of the video, \(t = 0, \dots, T-1\), with \(T = 120\)
\(\mathbf{u}_t\)
unit image embedding of the frame at second \(t\)
\(\mathbf{w}_q\)
unit text embedding of the prompt for query \(q\)
\(a_t\)
the similarity score of second \(t\); for SigLIP B/16 in our data, event shots have cosines of about 0.01 to 0.12 and background shots about −0.07 to 0.05 (5th to 95th percentile)

Here a segment is one second, sampled as one frame. With real video you would sample 1 or 2 frames per second and average their embeddings over a 1- or 2-second segment (QVHighlights uses 2-second clips [Certain]). Note the SigLIP detail from the earlier lab: its texts must be padded to 64 tokens, as in training.

# sim_score.py: cosine of every benchmark photo with every event prompt (SigLIP and CLIP)
import torch, json, io, pandas as pd, torch.nn.functional as F
from PIL import Image
from transformers import AutoModel, AutoProcessor
torch.manual_seed(0); torch.set_grad_enabled(False)
PQ = "flickr8k/test.parquet"          # the jxie/flickr8k test split (1,000 photos, 5 captions each)
d = pd.read_parquet(PQ); B = json.load(open("bench.json"))
imgs = sorted({s[0] for v in B["videos"] for s in v["shots"]}); Q = list(B["events"])
res = {"imgs": imgs, "queries": Q}
for k, mid in [("siglip", "google/siglip-base-patch16-224"), ("clip", "openai/clip-vit-base-patch32")]:
    m = AutoModel.from_pretrained(mid).eval(); p = AutoProcessor.from_pretrained(mid)
    vec = lambda o: o if isinstance(o, torch.Tensor) else o.pooler_output
    U = []
    for a in range(0, len(imgs), 64):
        ims = [Image.open(io.BytesIO(d.iloc[i]["image"]["bytes"])).convert("RGB") for i in imgs[a:a + 64]]
        x = p(images=ims, return_tensors="pt")["pixel_values"]
        U.append(F.normalize(vec(m.get_image_features(pixel_values=x)), dim=-1))
    U = torch.cat(U)                                         # (567, d) unit vectors
    texts = [f"a photo of {q}." for q in Q]                  # one prompt per event
    kw = dict(padding="max_length", max_length=64) if k == "siglip" else dict(padding=True)
    x = p(text=texts, truncation=True, return_tensors="pt", **kw)
    W = F.normalize(vec(m.get_text_features(**{n: x[n] for n in ("input_ids", "attention_mask") if n in x})), dim=-1)
    res[k] = (U @ W.T).tolist()                              # (567, 6) cosines
json.dump(res, open("sim.json", "w"))

The table in Step 3 shows how well this cosine separates event photos from background photos. In short: SigLIP separates them well, CLIP B/32 also does, except for soccer, where CLIP is close to a coin flip.

Step 3: score each second with a VLM yes/no question

The expensive scorer asks a VLM about each frame: “Does this image show {query}? Answer yes or no.” We do not let the model generate text. We run one forward pass and read the logits of the next token, the first token of the answer. The difference of the two logits is the log-odds of “Yes” against “No”:

\[ b_t = \ell_t(\text{Yes}) - \ell_t(\text{No}) = \log \frac{P(\text{Yes} \mid \text{frame}_t, q)}{P(\text{No} \mid \text{frame}_t, q)}, \qquad P(\text{Yes} \mid \text{Yes or No}) = \sigma(b_t) \tag{215.2} \]
Read the symbols
\(\ell_t(\cdot)\)
the logit the VLM gives to a token at the first answer position
\(b_t\)
the VLM score of second \(t\); positive means “Yes” is more likely than “No”
\(\sigma\)
the logistic sigmoid; \(\sigma(b_t)\) is the probability of “Yes” if the model had to choose between only “Yes” and “No”

Why log-odds and not the probability? The probability saturates. In our data, about half of the event shots get \(P(\text{Yes})\) above 0.99 and 84% of the background shots get less than 0.01, so the differences you care about (0.9991 against 0.9999) are tiny in probability but clear in log-odds. Log-odds also behave like the cosine: roughly symmetric, no hard floor or ceiling.

Algorithm 1: score every segment of one video for one query
  1. input: frames \(x_0, \dots, x_{T-1}\) (one per segment), query text \(q\), a dual encoder, a VLM
  2. \(\mathbf{w}_q \leftarrow\) normalize(text_encoder(“a photo of \(q\).”)) // once per query
  3. for \(t = 0, \dots, T-1\):
  4.   \(\mathbf{u}_t \leftarrow\) normalize(image_encoder(\(x_t\))) // once per frame, cache it: it does not depend on \(q\)
  5.   \(a_t \leftarrow \mathbf{u}_t^\top \mathbf{w}_q\) // formula (215.1)
  6.   \(\ell \leftarrow\) VLM logits at the first answer position for (\(x_t\), “Does this image show \(q\)? Answer yes or no.”)
  7.   \(b_t \leftarrow \ell[\text{“ Yes”}] - \ell[\text{“ No”}]\) // formula (215.2); one forward pass, no generation
  8. return the two curves \(a_{0..T-1}\) and \(b_{0..T-1}\)

Two practical details decide whether this works. First, find the right token ids. SmolVLM’s chat template ends with “Assistant:”, so the answer token is “ Yes” with a leading space. We checked that the top-1 next tokens for our photos are indeed “ Yes” and “ No”; if you read the id of “Yes” without the space, you read a token the model almost never puts there. Second, keep the question fixed. Changing “Answer yes or no.” to another wording changes the logits, so tune the wording on a validation set and then freeze it.

# vlm_score.py: P(Yes) from SmolVLM-500M for every (photo, query) pair in the benchmark
import torch, json, io, pandas as pd
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText
torch.manual_seed(0); torch.set_grad_enabled(False)
MID = "HuggingFaceTB/SmolVLM-500M-Instruct"
dev = "mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu")
d = pd.read_parquet("flickr8k/test.parquet"); B = json.load(open("bench.json"))
proc = AutoProcessor.from_pretrained(MID)
model = AutoModelForImageTextToText.from_pretrained(MID, dtype=torch.float32).to(dev).eval()
tok = proc.tokenizer
YES, = tok.encode(" Yes", add_special_tokens=False)          # the answer starts after "Assistant:"
NO, = tok.encode(" No", add_special_tokens=False)            # so the tokens carry a leading space
pairs = sorted({(s[0], v["query"]) for v in B["videos"] for s in v["shots"]})   # 793 pairs
out = {}
for i, q in pairs:
    img = Image.open(io.BytesIO(d.iloc[i]["image"]["bytes"])).convert("RGB")
    msgs = [{"role": "user", "content": [{"type": "image"},
             {"type": "text", "text": f"Does this image show {q}? Answer yes or no."}]}]
    x = proc(text=proc.apply_chat_template(msgs, add_generation_prompt=True),
             images=[img], return_tensors="pt").to(dev)
    lg = model(**x).logits[0, -1].float().cpu()                # one forward pass, no generation
    two = torch.stack([lg[YES], lg[NO]])
    out[f"{i}|{q}"] = {"p_yes": torch.softmax(two, 0)[0].item(),
                       "lyes": lg[YES].item(), "lno": lg[NO].item()}
json.dump(out, open("vlm.json", "w"))

Now compare the two scorers on single photos. For each event we take the photos used in its 5 videos and compute the AUC: the probability that a random event photo gets a higher score than a random background photo (1.0 is perfect, 0.5 is a coin flip). The last two columns give how often the VLM says “Yes” (\(b_t \gt 0\)) for event photos and for background photos.

Event (event / background photos)AUC SigLIPAUC CLIP B/32AUC VLMVLM “Yes” on eventsVLM “Yes” on background
a dog playing in the snow (13 / 140)0.9880.9951.0000.770.00
a person riding a bike (14 / 95)0.9830.9800.9780.930.08
people on a beach by the ocean (17 / 126)0.9190.9470.9370.760.06
a person climbing a rock wall (12 / 107)0.9700.9750.9951.000.09
a person on a skateboard (10 / 118)0.9950.9810.9931.000.04
people playing soccer (14 / 127)0.9390.5470.9720.570.04
mean0.9660.9040.979

Both scorers are good on single photos, and the VLM is a little better on average. But per-photo AUC is not the task. A detector must find a span, and a single background photo that scores very high (a motorcycle for “bike”) can win the top rank. The two scorers make different mistakes, which is why fusing them helps.

Step 4: combine the two scores

You cannot simply add \(a_t\) and \(b_t\). In our data the cosines span about 0.26 (from −0.13 to 0.13), the log-odds about 20 (from −11.4 to 8.8). Their sum would be just the VLM. So we first put both on the same scale with a z-score per video (subtract the video’s mean, divide by its standard deviation), and then take a weighted sum:

\[ z(x)_t = \frac{x_t - \bar{x}}{\operatorname{std}(x)}, \qquad f_t = w \, z(a)_t + (1 - w) \, z(b)_t \tag{215.3} \]
Read the symbols
\(\bar{x}, \operatorname{std}(x)\)
mean and standard deviation of the curve over the \(T\) seconds of one video
\(w\)
the fusion weight: \(w = 1\) is SigLIP only, \(w = 0\) is the VLM only; we use \(w = 0.5\)
\(f_t\)
the fused score of second \(t\); a value of 2 means “two standard deviations above this video’s average”

A z-score per video is a choice with a cost. It makes the scores of different videos comparable in scale, and it removes a video-wide offset (a dark video where all cosines are low). But it assumes every video contains something: in a video with no event at all, the best background second still gets a high z-score. For a product that must also say “not in this video”, you need an absolute threshold on the raw scores as well. In our benchmark every video has at least one moment, so z-scores are fine.

Worked example: video q1v4, “a person riding a bike”, true moment [63, 76] s

In this video the SigLIP curve has mean 0.0164 and standard deviation 0.0316; the VLM curve has mean −4.1415 and standard deviation 5.7532. Some shots (cos = SigLIP cosine, lo = VLM log-odds):

SecondsCaption (first of 5)cos\(z(a)\)lo\(z(b)\)\(f\) at \(w = 0.5\)
33–38A man racing an orange motorcycle0.07211.767.392.001.88
63–66 ✓A bike rider is doing a stunt on a bicycle over a forest0.06751.626.741.891.75
66–71 ✓A dirt bike rider is in the air pulling off a trick0.03310.536.401.831.18
71–74 ✓A dirt biker leaps through the air0.06611.576.231.801.69
74–76 ✓a boy with a blue helmet is riding a bike0.09172.386.951.932.16
80–84a man doing tricks on a motorcycle0.05271.155.281.641.39

Check one value: for the last event shot, \(z(a) = (0.0917 - 0.0164)/0.0316 = 2.38\) and \(z(b) = (6.95 - (-4.1415))/5.7532 = 1.93\), so \(f = 0.5 \times 2.38 + 0.5 \times 1.93 = 2.16\) (rounded). Both scorers like the motorcycles almost as much as the bikes. SigLIP alone puts its top moment on the motorcycle tricks at [80, 84] (tIoU 0), the VLM alone on the motorcycle race at [33, 39] (tIoU 0). The fused curve is high on all four event shots in a row, and its top moment is [63, 75], with tIoU 0.923. Neither scorer alone found this moment; together they did. Notice also that two of the four “bike” shots are dirt bikes, which are motorcycles: the label rule counted the word “bike”. We return to this in Step 7.

SigLIP \(z(a)\)VLM \(z(b)\)fused \(f\)true moment
Figure 2. Measured scores for all 30 videos. Each step of a curve is one shot (one photo held for 2 to 5 seconds). The bars under the fused curve are the top three moments after windows and NMS (Step 5), labeled with their rank and their tIoU with the best-matching true moment. Click any shot to see its caption and its two raw scores.
How to read Figure 2

Start with q1v4 (“a person riding a bike”) at \(w = 0.5\): the top-1 moment is [63, 75] with tIoU 0.923. Click the shot at 33–38 s: a motorcycle race with VLM log-odds 7.39. Now drag \(w\) to 1 (SigLIP only): the top-1 jumps to [80, 84], the motorcycle tricks, tIoU 0. Drag \(w\) to 0 (VLM only): the top-1 is [33, 39], the motorcycle race. Then look at q5v4 (soccer): at every \(w\) the top moment is a rugby shot at 84–88 s, which both models score as soccer. Finally pick q4v0 (skateboard): an easy video where all three curves agree.

Step 5: from a score curve to moments

A curve of 120 numbers is not yet an answer. The output must be a short ranked list of moments \([s, e]\) with scores. We use the simplest method that works: multi-scale sliding windows. Every window of length 4, 6, 8, 10, 12, 14, 16 or 20 seconds, at every start second, is a candidate (878 per video). Then we give each window a score, sort, and remove duplicates with NMS.

The obvious window score is the mean of \(f_t\) inside the window. It has a hidden bias: short windows win. The best 4-second window inside a moment always has a mean at least as high as the whole moment, because the whole moment includes its weaker shots. So the plain mean returns a short piece of the moment, and the tIoU is low. The fix is a contrast score: the mean inside minus the mean of the two neighbouring flanks. A window that cuts a moment in half has the other half in its flank, which lowers its score. A window that covers the moment exactly has background on both sides, which raises it:

\[ S(s, e) = \frac{1}{e - s} \sum_{t=s}^{e-1} f_t \;-\; \frac{1}{|N(s,e)|} \sum_{t \in N(s,e)} f_t, \qquad N(s,e) = [s - h, s) \cup [e, e + h), \; h = \max\!\big(2, \lfloor (e-s)/2 \rfloor\big) \tag{215.4} \]
Read the symbols
\([s, e)\)
a candidate window from second \(s\) up to (not including) second \(e\); its length is \(e - s\)
\(N(s, e)\)
the two flanks: \(h\) seconds before and \(h\) seconds after the window, clipped at the video’s ends
\(h\)
flank width, half the window length but at least 2 seconds
\(S(s, e)\)
the window score: high when the inside is high and the surroundings are low

This is the same idea as the boundary-aware scores in trained detectors, done by hand. UniVTG, for example, lets every clip predict whether it is inside the moment and how far the boundaries are (session 207); our flank term is a crude, training-free way to reward clean boundaries.

Algorithm 2: proposals and greedy temporal NMS
  1. input: fused curve \(f_{0..T-1}\), window lengths \(\mathcal{L}\), NMS threshold \(\tau\), number of outputs \(K\)
  2. \(c \leftarrow\) prefix sums of \(f\) // each window mean is then one subtraction
  3. for each \(L \in \mathcal{L}\) and each start \(s = 0, \dots, T-L\): add \((S(s, s+L), s, s+L)\) to \(P\) // formula (215.4)
  4. sort \(P\) by score, high first (ties: earlier start, then earlier end)
  5. keep \(\leftarrow\) [ ]
  6. for each \(p \in P\) in order: if \(\operatorname{tIoU}(p, k) \lt \tau\) for every \(k \in\) keep: append \(p\) to keep
  7.   stop when keep has \(K\) moments // we use \(\tau = 0.5\), \(K = 10\)
  8. return keep, a ranked list of \(K\) moments

Why NMS? Sliding windows produce many near-copies of every good moment: [63, 75], [64, 74], [63, 73] and so on all score high. Without NMS the top 10 would be ten versions of one moment, and the second true moment of the video would never appear in the list. NMS keeps a candidate only if it overlaps every already kept moment by less than \(\tau\). Detection transformers such as Moment-DETR avoid this step because their training loss already punishes duplicates [Certain]; a hand-made proposal method needs it.

Worked example: mean score against contrast score on video q1v4

Window [63, 75) (12 s, so \(h = 6\)): the mean of \(f\) inside is 1.5313; the flanks [57, 63) and [75, 81) have mean −0.3263; so \(S = 1.5313 - (-0.3263) = 1.8577\). Its tIoU with the truth [63, 76] is \(12/13 = 0.923\).

Window [72, 76) (4 s, \(h = 2\)): the inside mean is higher, 1.9213, because it covers only the two strongest event shots. But its flanks [70, 72) and [76, 78) include event seconds and have mean 0.3321, so \(S = 1.9213 - 0.3321 = 1.5892\). Its tIoU is only \(4/13 = 0.308\).

With the plain mean, [72, 76) is the top-1 moment (1.9213 against 1.5313). With the contrast score, [63, 75) is (1.8577 against 1.5892). The perfect window [63, 76) would score even higher, 1.9693, but a 13-second window is not in our list of lengths, so 0.923 is the best tIoU this grid can give here.

Figure 3. Greedy temporal NMS (Algorithm 2) on video q1v4 at \(w = 0.5\), step by step, until 5 moments are kept. Each step takes the next candidate in score order and keeps it or suppresses it. Switch the window score to see the bias toward short windows.
How to read Figure 3

With the contrast score and \(\tau = 0.5\), step 1 keeps [63, 75] (tIoU 0.923 with the truth). The next candidates are near-copies such as [64, 76] and are suppressed. Press Play: 24 candidates are examined to keep 5 moments, and three of the five sit on motorcycle shots. Note that [72, 76] is kept even though it lies inside [63, 75]: its tIoU with [63, 75] is only \(3/13 = 0.23\), below \(\tau\). A short window inside a long one has a small tIoU, so NMS does not remove it. Now switch to the plain mean: the top-1 becomes [72, 76] (tIoU 0.308), and the list fills up with short pieces of moments.

Step 6: evaluate with tIoU-based metrics

We use the metrics of session 207 and compute them the way the QVHighlights evaluation code does [Certain, from reading standalone_eval]. Temporal IoU compares a predicted moment \(\hat{M} = [\hat{s}, \hat{e}]\) with a true moment \(M = [s, e]\):

\[ \operatorname{tIoU}(\hat{M}, M) = \frac{\max\!\big(0,\ \min(\hat{e}, e) - \max(\hat{s}, s)\big)}{\max(\hat{e}, e) - \min(\hat{s}, s)} \tag{215.5} \]

R1@\(m\) looks only at the top-1 prediction. When a video has two true moments, the top-1 is compared with the one it overlaps most. mIoU is the average of that best tIoU:

\[ \text{R1@}m = \frac{1}{N}\sum_{n=1}^{N} \mathbb{1}\Big[\max_{j} \operatorname{tIoU}\big(\hat{M}_{n,1}, M_{n,j}\big) \ge m\Big] \tag{215.6} \]

mAP looks at the whole ranked list of \(K = 10\) predictions. Walk down the list; a prediction is a true positive if its tIoU with a still-unmatched true moment is at least \(\theta\); each true moment can be matched once. Average precision is the area under the precision–recall curve, with precision made non-increasing (the VOC style), and mAP averages over videos and over ten thresholds:

\[ \text{AP}_n(\theta) = \frac{1}{G_n} \sum_{i:\ \text{TP}_i} \max_{j \ge i} \text{prec}_j, \qquad \text{mAP} = \frac{1}{10}\sum_{\theta \in \{0.5, 0.55, \dots, 0.95\}} \frac{1}{N}\sum_{n=1}^{N} \text{AP}_n(\theta) \tag{215.7} \]
Read the symbols
\(N\)
the number of (video, query) pairs, 30 here
\(\hat{M}_{n,1}\)
the top-ranked prediction for pair \(n\); \(M_{n,j}\) its \(j\)-th true moment
\(G_n\)
the number of true moments of pair \(n\) (1 or 2 here)
\(\text{prec}_j\)
precision after the first \(j\) predictions: true positives so far divided by \(j\)
\(\theta\)
the tIoU threshold that makes a prediction count as a true positive
Algorithm 3: AP for one video at threshold \(\theta\)
  1. input: ranked predictions \(p_1, \dots, p_K\), true moments \(M_1, \dots, M_G\), threshold \(\theta\)
  2. used \(\leftarrow\) [false] \(\times\, G\)
  3. for \(i = 1, \dots, K\): \(j^* \leftarrow \argmax_j \operatorname{tIoU}(p_i, M_j)\) over unused \(j\)
  4.   if \(\operatorname{tIoU}(p_i, M_{j^*}) \ge \theta\): \(\text{TP}_i \leftarrow 1\), used[\(j^*\)] \(\leftarrow\) true else \(\text{TP}_i \leftarrow 0\)
  5. \(\text{prec}_i \leftarrow (\text{TP}_1 + \dots + \text{TP}_i) / i\)
  6. return \(\frac{1}{G} \sum_{i:\,\text{TP}_i = 1} \max_{j \ge i} \text{prec}_j\) // formula (215.7); unmatched true moments add 0

Here is the whole detector and evaluator. It is plain PyTorch and Python, it runs in a few seconds, and it is deterministic: there is no randomness, and ties are broken by start and end time.

# detect.py: fuse per-second scores, propose windows, temporal NMS, tIoU metrics
import json, torch
B = json.load(open("bench.json")); S = json.load(open("sim.json")); VL = json.load(open("vlm.json"))
Q = S["queries"]; row = {im: n for n, im in enumerate(S["imgs"])}

def curves(v, enc="siglip"):                   # one value per second: shots are held still
    q = Q.index(v["query"]); a = torch.zeros(v["T"], dtype=torch.float64); b = torch.zeros_like(a)
    for im, s, e, _ in v["shots"]:
        a[s:e] = S[enc][row[im]][q]                          # cosine with the event prompt
        r = VL[f"{im}|{v['query']}"]; b[s:e] = r["lyes"] - r["lno"]   # VLM log-odds of "Yes"
    return a, b

def z(x): return (x - x.mean()) / (x.std(unbiased=False) + 1e-8)   # per-video z-score

def tiou(p, g):
    inter = max(0.0, min(p[1], g[1]) - max(p[0], g[0]))
    return inter / (max(p[1], g[1]) - min(p[0], g[0]))

LENS = (4, 6, 8, 10, 12, 14, 16, 20)            # window lengths in seconds, stride 1 s
def windows(f, mode="contrast"):
    T = len(f); c = [0.0] + f.cumsum(0).tolist(); out = []
    for L in LENS:
        for s in range(T - L + 1):
            e = s + L; sc = (c[e] - c[s]) / L                    # mean score inside
            if mode == "contrast":                              # minus mean of the flanks
                fl = max(2, L // 2); a0, b1 = max(0, s - fl), min(T, e + fl)
                n = (s - a0) + (b1 - e)
                sc -= ((c[s] - c[a0]) + (c[b1] - c[e])) / n if n else 0.0
            out.append((round(sc, 9), s, e))       # round: stable ties
    return sorted(out, key=lambda p: (-p[0], p[1], p[2]))     # deterministic tie-break

def nms(props, thr=0.5, k=10):                 # greedy temporal non-maximum suppression
    keep = []
    for p in props:
        if all(tiou(p[1:], q[1:]) < thr for q in keep): keep.append(p)
        if len(keep) == k: break
    return keep

def average_precision(preds, gts, th):          # interpolated AP for one query
    used = [False] * len(gts); tp = []
    for p in preds:
        ious = [-1 if used[j] else tiou(p[1:], g) for j, g in enumerate(gts)]
        j = max(range(len(gts)), key=lambda j: ious[j])
        hit = ious[j] >= th; tp.append(hit)
        if hit: used[j] = True
    ap, best, hits = 0.0, 0.0, 0
    prec = [sum(tp[:i + 1]) / (i + 1) for i in range(len(tp))]
    for i in range(len(tp)):                      # area under the monotone PR curve
        if tp[i]: ap += max(prec[i:]) / len(gts)
    return ap

TH = [round(0.5 + 0.05 * i, 2) for i in range(10)]
def evaluate(w=0.5, mode="contrast", thr=0.5, k=10, enc="siglip"):
    n = len(B["videos"]); r5 = r7 = miou = 0.0; ap = {t: 0.0 for t in TH}
    for v in B["videos"]:
        a, b = curves(v, enc); f = w * z(a) + (1 - w) * z(b)
        P = nms(windows(f, mode), thr, k) if thr < 1 else windows(f, mode)[:k]
        g = v["moments"]; best = max(tiou(P[0][1:], gg) for gg in g)
        r5 += best >= 0.5; r7 += best >= 0.7; miou += best
        for t in TH: ap[t] += average_precision(P, g, t)
    m = {t: x / n for t, x in ap.items()}
    return {"R1@0.5": r5 / n, "R1@0.7": r7 / n, "mIoU": miou / n,
            "mAP@0.5": m[0.5], "mAP@0.75": m[0.75], "mAP": sum(m.values()) / len(TH)}

if __name__ == "__main__":
    torch.manual_seed(0)
    for name, w in [("similarity only", 1.0), ("VLM only", 0.0), ("fused 50/50", 0.5)]:
        r = evaluate(w); print(f"{name:16s}", " ".join(f"{k} {x:.3f}" for k, x in r.items()))

Our results on the 30 videos (\(w = 0.5\), contrast score, \(\tau = 0.5\), \(K = 10\), unless stated):

DetectorR1@0.5R1@0.7mIoUmAP@0.5mAP@0.75mAP
CLIP B/32 only0.6000.5000.5520.6570.5030.499
SigLIP only (\(w = 1\))0.7000.6330.6510.7730.6330.613
VLM only (\(w = 0\))0.7670.7330.7340.8380.7640.715
CLIP + VLM0.7670.7670.7280.8360.6970.706
SigLIP + VLM0.8000.7670.7460.8670.7460.738
SigLIP + VLM, plain mean score0.4000.0670.3870.5620.2320.255
SigLIP + VLM, no NMS0.8000.7670.7460.7870.7320.697

Four lessons. (1) Fusion helps: SigLIP + VLM beats each scorer alone on R1 and on mAP, and CLIP + VLM beats CLIP alone by a lot. (2) The window score matters most: with the plain mean, R1@0.7 falls to 0.067, because the top window is a short piece of the moment. (3) NMS does not change R1 at all, since the top-1 candidate is always kept, but it raises mAP@0.5 from 0.787 to 0.867, because the second true moment can now enter the top 10. (4) A good per-photo AUC (0.966 for SigLIP) does not mean a good detector (R1@0.5 = 0.700): one high-scoring wrong shot is enough to take the top rank.

How sure are these differences? With 30 videos, one video is 3.3 points of R1. We resampled the 30 videos with replacement 10,000 times (a bootstrap). The 95% interval of the fused mAP is [0.632, 0.834]. The paired gain of fusion over SigLIP alone is +0.125, interval [0.046, 0.211]: fusion is better on 13 videos, worse on 2, equal on 15. The gain over the VLM alone is +0.023, interval [−0.048, 0.094]: better on 8 videos, worse on 6. So “fusion beats SigLIP” is solid here; “fusion beats the VLM” is not proven by 30 videos.

mAP (0.5:0.95)mAP@0.5R1@0.5
Figure 4. The full detector and evaluator of detect.py, rerun live in your browser on the measured scores of all 30 videos. The curves show three metrics for every \(w\) from 0 to 1 in steps of 0.05; the vertical line is the current \(w\). Change the window score or the NMS threshold to rerun everything.
How to read Figure 4

At the default (\(w = 0.5\), contrast, \(\tau = 0.5\)) the readout gives R1@0.5 = 0.800, R1@0.7 = 0.767, mAP@0.5 = 0.867 and mAP = 0.738, the bold row of the table. The left end of the curves is the VLM alone (mAP 0.715), the right end SigLIP alone (0.613). The best mAP is 0.750 at \(w = 0.4\), but the curve is bumpy: on 30 videos, picking the best \(w\) on the same videos you report is overfitting. Click “no NMS”: R1 does not move, mAP@0.5 drops to 0.787. Click “plain mean”: every curve collapses.

A note on highlight detection, the second task of QVHighlights: there, the output is a score per clip, and HIT@1 asks whether the single highest-scoring clip is a true highlight. The per-second curves of this lab give that for free. The top second falls inside a true moment in 25 of 30 videos (0.833) for SigLIP alone and in 26 of 30 (0.867) for the VLM alone and for the fused score.

Step 7: study the failures

Averages hide the reasons. Six videos have a fused top-1 below tIoU 0.5. Looking at them one by one is the most useful hour of this lab:

Video, queryTop-1 momentWhat is on screen (first caption)Why
q5v4 and q5v2, “people playing soccer”[84, 88] and [60, 64]A bunch of men play rugby on a muddy fieldNear-miss scene: SigLIP cosine 0.106, VLM log-odds 7.13 (P(Yes) = 0.999). Both models see “team sport on grass”.
q1v2, “a person riding a bike”[109, 113]A man racing an orange motorcycleNear-miss object; the VLM says Yes with log-odds 7.39. Is a motorcycle a “bike”? English says “sometimes”.
q2v4, “people on a beach by the ocean”[94, 106]A man and a woman at the waterfront; a child on a wooden dockNear-miss scene: water but no beach; the VLM says Yes to both (\(P = 0.91\) and 0.90).
q2v0, “people on a beach by the ocean”[22, 26]Two men sitting on the bank of a lake with an ice chestSame family: SigLIP likes the lake (cosine 0.065) while the VLM leans No (log-odds −1.77). The VLM alone gets this video right; here fusion hurts.
q0v3, “a dog playing in the snow”[82, 86], tIoU 0.29A dog sled team seen from behind (labeled positive, at 79–82)Label versus question: the captions say dog and snow, but a sled team is not “playing”. The VLM says No (log-odds −5.48), so the detector cuts the moment in half.

There are three kinds of error here, and each has a different fix.

  • Near-miss scenes and objects (rugby, motorcycles, lakes). Both scorers make the same mistake, so fusion cannot fix it. Fixes: a more specific query (“a person pedaling a bicycle”), a negative prompt (subtract the similarity to “a photo of rugby”), or a stronger VLM as the verifier for the top candidates only.
  • Labels that disagree with the question. Our labels come from words in captions. “A boy dressed in soccer attire … getting out of a car” is labeled as soccer (the VLM says No, log-odds −3.70, and it is right). Dirt bikes are labeled as bikes. A real benchmark has the same problem in another form: the annotator’s idea of the event is not the user’s. Read your false positives and false negatives before you trust a 3-point change.
  • Boundaries. When one shot inside a moment scores low (the dog sled), the best-scoring window covers only part of the moment. Fixes: smooth the curve before windowing, add a gap-tolerant grouping step, or train a small boundary regressor as in TALL or UniVTG (session 207).

Step 8: make it cheaper with a cascade

The VLM cost us 27 minutes for 793 photo–query pairs; SigLIP cost seconds. In production, the VLM is the bill. A cascade runs the expensive model only where the cheap model says “maybe”. We ask the VLM only about shots whose SigLIP z-score is at least a threshold \(\tau_c\). The other shots get the lowest VLM score of the video, as if the VLM had said a clear No.

Cascade threshold \(\tau_c\) on z(a)Shots sent to the VLMR1@0.5mAP
none (all shots)100%0.8000.738
0.044.8%0.7330.711
0.526.3%0.7670.739
1.015.6%0.6670.624
1.59.9%0.5330.505

At \(\tau_c = 0.5\) the VLM sees about a quarter of the shots and the mAP is unchanged (0.739 against 0.738). Going further hurts quickly, because event shots that SigLIP scores low never reach the VLM (at \(\tau_c = 1.0\), the dirt-bike shot of q1v4 with z(a) = 0.53 is never asked). The numbers are not monotone in \(\tau_c\) (0.0 is worse than 0.5): filling unasked shots with the minimum changes the z-scores of the VLM curve, and 30 videos are few. Treat the shape, not the exact values, as the result. The same idea runs at scale in session 211 pipelines: cheap embeddings for everything, expensive models only on candidates.

Where this is used

ApplicationWhat from this lab carries over
Sports highlights (goals, dunks, crashes)Per-segment scores against event prompts, a VLM verifier for top candidates, temporal NMS so one goal gives one clip; near-miss scenes (rugby for soccer) are the main error
Video chapters and “key moments” in search resultsWindows with a contrast score (215.4) give clean boundaries; mAP over tIoU thresholds measures them
Content moderation of long videosThe VLM yes/no log-odds (215.2) as a policy check per segment, with a cascade (Step 8) to control cost; absolute thresholds instead of per-video z-scores, since most videos contain nothing
Security and body-camera reviewText queries for rare events over hours of footage; frame embeddings computed once and reused for every new query
Video-RAGThe same scores retrieve the moments a VLM should watch before it answers a question (session 210)
Text-to-video retrievalAveraging segment embeddings over a moment gives clip vectors for a search index (session 208)
Training data for trained detectorsTraining-free detections as pseudo-labels, the idea behind the large-scale pseudo supervision of UniVTG [Likely]

Common mistakes

  • Adding raw scores from different models. A cosine (span about 0.26) plus log-odds (span about 20) is just the log-odds. Normalize each score first, per video or with statistics from a validation set.
  • Ranking windows by their mean score. Short windows inside a moment always win. In our run this alone took R1@0.7 from 0.767 to 0.067. Use a contrast score, a length prior, or a boundary model.
  • Forgetting NMS, or believing NMS removes all duplicates. Without NMS mAP@0.5 fell from 0.867 to 0.787. With NMS, a short window inside a long one still survives, because its tIoU with the long one is small.
  • Reading the wrong token for “Yes”. After “Assistant:” the answer token has a leading space (“ Yes”). Check the top-1 next tokens on a few examples before you trust \(P(\text{Yes})\).
  • Using \(P(\text{Yes})\) instead of log-odds. Probabilities saturate near 0 and 1, which flattens exactly the differences between strong candidates.
  • Tuning the fusion weight on the test videos. The best \(w\) on our 30 videos (0.4) is partly noise. Choose \(w\) on separate validation videos and report the test set once.
  • Reporting R1 or mAP without the protocol. State the tIoU thresholds, \(\ge\) or \(\gt\), how many predictions per query enter mAP (10 here), and how a top-1 is matched when there are several true moments.
  • Trusting labels you made from text. Caption words label a dog sled as “a dog playing” and a dirt bike as a “bike”. Look at errors before you decide that a model is wrong.

Interview questions

Design a key-moment detector for a new event type with no labeled training data.core
Sample frames (1 to 2 per second) and group them into segments. Cheap scorer: embed each segment once with a dual encoder (SigLIP or CLIP) and compute the cosine with one or more event prompts. Expensive scorer: ask a VLM a fixed yes/no question for each promising segment and use the log-odds of “Yes”. Normalize both scores and fuse them with a weight chosen on a few labeled validation videos. Turn the curve into moments with multi-scale windows and a contrast score, then temporal NMS, and return the top \(K\). Evaluate with R1@0.5 / R1@0.7 and mAP over tIoU 0.5:0.95 on a small labeled set, and read the errors. In our lab this gave R1@0.5 = 0.800 and mAP = 0.738 on 30 videos without any training.
Why normalize the two scores before fusing them, and what is the risk of a per-video z-score?core
The scores live on different scales: SigLIP cosines span about 0.26, VLM log-odds about 20 in our data. Without normalization the sum is just the VLM, and the weight \(w\) means nothing. A per-video z-score puts both curves in “standard deviations above this video’s average”, which also removes video-wide offsets. The risk: it assumes every video contains something. In a video with no event, the best background second still gets a high z-score, so the detector always finds a “moment”. For open-world use, also keep an absolute threshold on the raw scores, or normalize with statistics from a validation set instead of the video itself.
Why does ranking windows by their mean score give poor tIoU, and how do you fix it?core
The mean over a window is at most the mean over its best sub-window, so a short window over the two strongest seconds of a moment beats the window that covers the whole moment, whose weaker shots pull the mean down. The top-1 is then a short piece of the moment with low tIoU. In our run R1@0.7 was 0.067 with the plain mean. Fixes: subtract the mean of the neighbouring flanks (our contrast score, which raised R1@0.7 to 0.767), add a length prior, group consecutive seconds above a threshold, or train a boundary regressor (TALL, UniVTG).
What does temporal NMS do, and why did it change mAP but not R1?basic
Sliding windows produce many overlapping versions of each good moment. Greedy NMS walks down the score-sorted list and keeps a candidate only if its tIoU with every kept moment is below a threshold \(\tau\). The top-1 candidate is always kept, so R1@\(m\) cannot change. mAP uses the top 10: without NMS they are near-copies of one moment, so a second true moment never appears and recall stays at 1/2. In our run NMS raised mAP@0.5 from 0.787 to 0.867 and left R1@0.5 at 0.800.
How do you turn a generative VLM into a per-frame scorer, and what can go wrong?core
Ask a fixed yes/no question with the chat template, run one forward pass, and read the logits of the “Yes” and “No” tokens at the first answer position; use \(b = \ell(\text{Yes}) - \ell(\text{No})\), the log-odds. No generation is needed. Pitfalls: the wrong token id (“Yes” versus “ Yes” with a space), a template that does not end where the answer starts, probabilities that saturate (use log-odds), a model that is biased toward “Yes” for near-miss scenes (rugby for soccer at log-odds 7.13 in our run), and a cost of one forward pass per frame and query.
The VLM is far more expensive per frame than the embedding scorer. How do you cut its cost?advanced
Use a cascade: the cheap scorer runs everywhere, and the VLM runs only on segments whose cheap score is promising, or only on the top-\(K\) proposals after NMS. In our lab, asking the VLM only about shots with SigLIP z-score at least 0.5 sent 26.3% of the shots to it and kept mAP at 0.739 (0.738 with all shots); stricter thresholds lost recall, because event shots that SigLIP scores low never reach the VLM. Other levers: lower frame rate for the VLM, fewer image tiles, batching, caching per (frame, question), and a smaller verifier distilled from the large one.
Your detector reports R1@0.5 = 0.80 on 30 videos. A change brings it to 0.83. Ship it?core
Not on that evidence. One video is 3.3 points, so the change is one video. Compare paired per-video results (how many videos got better, how many worse) and use a bootstrap over videos. In our lab, fusion versus SigLIP alone had a paired mAP gain of +0.125 with a 95% interval [0.046, 0.211] (13 better, 2 worse), a real effect; fusion versus the VLM alone had +0.023 with interval [−0.048, 0.094], not proven. Also make sure the change was chosen on validation videos, not on these 30.
Moment-DETR needs no NMS. Why does your pipeline need it?core
Moment-DETR predicts a fixed set of moments with learned queries and is trained with Hungarian matching: each true moment is matched to one prediction, and the others are pushed toward “no moment”. Duplicates are punished by the training loss, so the model learns not to produce them. Our sliding windows are generated by a fixed rule that knows nothing about the other windows, so near-copies of every good window appear and must be removed afterwards. Many trained detectors with dense per-clip outputs (UniVTG-style heads) also use NMS at inference [Likely].
What is Soft-NMS, and when would it help here?advanced
Greedy NMS deletes every candidate that overlaps a kept one by at least \(\tau\). Soft-NMS instead lowers the overlapping candidate’s score, for example by a factor that falls with its IoU, and keeps it in the list. It helps when true moments are close together or overlap, where hard NMS would delete the second true moment. It also smooths the trade-off we saw in Figure 4, where \(\tau = 0.3\) favored mAP@0.5 and \(\tau = 0.7\) favored mAP@0.75. In our benchmark moments are at least 30 s apart, so we did not need it.
Your training-free detector works on the slideshow benchmark. What changes on real video?advanced
Boundaries are no longer hard cuts, so scores ramp up and down and the contrast score must find soft edges; smoothing and boundary regression matter more. Many events need motion (“starts running”, “scores a goal”), which a single frame cannot show: use clip-level video encoders or a video-LLM with several frames per segment (session 205). Long videos (hours) make the VLM cascade essential. Many videos contain no event, so per-video z-scores must be replaced by calibrated absolute thresholds. Labels become subjective, so measure human agreement first to know the ceiling (session 213).

Exercises

  1. A prediction [63, 75] is compared with the truth [63, 76], and a prediction [72, 76] with the same truth. Compute both tIoUs. Which R1 thresholds (0.5, 0.7) does each pass?
  2. Window A has inside mean 1.5313 and flank mean −0.3263; window B has inside mean 1.9213 and flank mean 0.3321. Which wins with the plain mean, and which with the contrast score (215.4)?
  3. A video has 2 true moments. The ranked predictions are TP, FP, FP, TP at some threshold \(\theta\). Compute the precisions and the AP of formula (215.7).
  4. The VLM gives log-odds 7.39 for one shot and −7.44 for another. What is \(P(\text{Yes})\) for each? Why is 0.9994 against 0.9999 a bad scale to compare two strong candidates?
  5. Our VLM run took 27 minutes for 793 pairs. Estimate the time for 1 hour of video at 1 frame per second and 5 queries, without and with the cascade at \(\tau_c = 0.5\) (26.3% of shots).
  6. Our window lengths are all even, so the true moment [63, 76] (13 s) cannot be matched exactly. Name two ways to get better boundaries without training a model.
Answerscheck
1. [63, 75]: overlap 12, union 13, tIoU \(12/13 = 0.923\), passes 0.5 and 0.7. [72, 76]: overlap 4, union 13, tIoU \(4/13 = 0.308\), passes neither. 2. Plain mean: B wins (1.9213 against 1.5313). Contrast: A has \(1.5313 + 0.3263 = 1.8577\), B has \(1.9213 - 0.3321 = 1.5892\), so A wins. These are windows [63, 75) and [72, 76) of video q1v4. 3. Precisions after each prediction: \(1/1 = 1\), \(1/2 = 0.5\), \(1/3 = 0.333\), \(2/4 = 0.5\). The non-increasing version at the two true positives is 1 (first) and 0.5 (fourth). AP \(= (1 + 0.5)/2 = 0.75\). 4. \(\sigma(7.39) = 0.99938\) and \(\sigma(-7.44) = 0.00059\). Near 1, large differences in log-odds become differences in the fourth decimal of the probability; when you average or z-score, such differences vanish, while log-odds keep them at their real size. 5. Our rate was 27 min / 793 pairs ≈ 2.05 s per pair. One hour at 1 fps is 3,600 frames, times 5 queries = 18,000 pairs, about \(18{,}000 \times 2.05\) s ≈ 10.2 hours. With the cascade, about 26.3% of that, ≈ 2.7 hours (on a busy laptop GPU; a server GPU with batching is much faster). 6. Any two of: add odd lengths or a finer grid of lengths; refine the edges of the top windows by moving each edge ±1 to 3 s and keeping the best contrast score; group consecutive seconds whose fused score is above a threshold and use the group’s edges; snap edges to shot cuts found by a shot-boundary detector.

Key takeaways

  • A training-free key-moment detector has five parts: a cheap per-segment scorer (215.1), an expensive verifier (215.2), a fusion rule (215.3), windows with a boundary-aware score plus temporal NMS (215.4), and tIoU-based evaluation (215.5 to 215.7).
  • On our 30 videos, SigLIP + SmolVLM-500M reached R1@0.5 = 0.800 and mAP = 0.738, against 0.700 / 0.613 for SigLIP alone and 0.767 / 0.715 for the VLM alone. The gain over SigLIP is solid (bootstrap interval [0.046, 0.211]); the gain over the VLM is not proven by 30 videos.
  • The window score mattered more than either model: the plain mean gave mAP 0.255 and R1@0.7 0.067, because short windows win.
  • NMS never changes R1 but raised mAP@0.5 from 0.787 to 0.867; a short window inside a long one survives NMS because their tIoU is small.
  • Read the VLM as log-odds of the “ Yes” token, with the leading space. Normalize scores before you add them.
  • Most remaining errors are near-miss scenes (rugby for soccer, motorcycles for bikes, lakes for beaches) and labels made from words. A cascade sent only 26.3% of shots to the VLM at the same mAP.

Key formulas

\[ a_t = \mathbf{u}_t^\top \mathbf{w}_q, \qquad b_t = \ell_t(\text{Yes}) - \ell_t(\text{No}), \qquad f_t = w\, z(a)_t + (1-w)\, z(b)_t \]
\[ S(s,e) = \operatorname{mean}_{[s,e)} f \;-\; \operatorname{mean}_{N(s,e)} f \]
\[ \operatorname{tIoU} = \frac{\max(0, \min(\hat e, e) - \max(\hat s, s))}{\max(\hat e, e) - \min(\hat s, s)}, \qquad \text{AP} = \frac{1}{G}\sum_{i:\,\text{TP}_i} \max_{j \ge i} \text{prec}_j \]

References for this session

  1. Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
  2. Lin et al. (2023). UniVTG: Towards Unified Video-Language Temporal Grounding
  3. Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
  4. Lin et al. (2023). UniVTG: Towards Unified Video-Language Temporal Grounding
  5. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  6. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
  7. Hodosh, Young & Hockenmaier (2013). Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (Journal of Artificial Intelligence Research)
  8. Lei et al. (code). moment_detr repository: standalone_eval/eval.py and utils.py (QVHighlights evaluation)
  9. Bodla et al. (2017). Soft-NMS: Improving Object Detection With One Line of Code
  10. Gao et al. (2017). TALL: Temporal Activity Localization via Language Query
  11. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  12. Hugging Face (model card). HuggingFaceTB/SmolVLM-500M-Instruct
  13. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models