LLM & VLM courseBlog
207Track 8: Video and multimodal systems · session 7 of 21

Temporal grounding and key-moment detection

Temporal grounding finds when something happens in a video: given a sentence, return its start and end time. Key-moment (highlight) detection scores every clip by how important it is. This session covers the metrics (R1@IoU, mAP, HIT@1), a training-free baseline that thresholds a similarity curve and cleans it with NMS, three model families (sliding windows in TALL, per-clip heads in UniVTG, DETR-style set prediction in Moment-DETR), video LLMs that write timestamps (TimeChat), and why noisy human labels put a ceiling on every score.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions on action recognition (203) and long video (206); CLIP and Q-Former basics; basic PyTorch

What you will learn

  • The two tasks: moment retrieval (text in, start and end time out) and highlight detection (a score for every clip), and the main datasets, Charades-STA and QVHighlights
  • How to compute temporal IoU, R1@0.5 / R1@0.7, mIoU, mAP over IoU thresholds, and HIT@1 averaged over several annotators
  • How a zero-shot baseline works step by step: cosine similarity per clip, smoothing, thresholding into segments, temporal IoU with the truth, and 1-D NMS or Soft-NMS to remove duplicates
  • How TALL scores sliding windows and regresses boundary offsets, and how UniVTG lets every clip predict foreground, offsets and saliency
  • How Moment-DETR predicts a set of moments with learned queries, Hungarian matching and a margin ranking loss for saliency, and why it needs no NMS
  • How TimeChat binds timestamps to frame tokens and writes times as text, and why frame spacing L/T limits its precision
  • Why moment labels are noisy and biased, how much humans disagree, and how to measure the ceiling before trusting small gains

Why this matters

People rarely want a whole video. They want the part where the dog catches the frisbee, the moment the price is announced, or the best five seconds for a preview. Temporal grounding answers “where in time?” for a text query. Key-moment detection answers “which parts matter most?”. Together they turn a long video into something you can search and skim.

This session covers two tasks that are usually trained together:

  1. Moment retrieval (also called temporal sentence grounding or video grounding): given a video and a sentence, return the start and end time of the matching part, for example “12.4 s to 20.4 s”.
  2. Highlight detection: give every short clip of the video a score for how important (or how well it matches the query) it is, and return the top clips.

We start with a training-free baseline that you can watch step by step: a similarity curve over time, a threshold that cuts it into segments, and NMS that removes duplicates. We then follow three generations of models: sliding windows with boundary regression (TALL), DETR-style set prediction (Moment-DETR) and one-head-per-clip models (UniVTG), and finally video LLMs that simply write the timestamps as text (TimeChat). At the end we look hard at the labels themselves: two careful people often do not agree on where a moment starts, and that sets a ceiling for every metric.

This session uses the 1-D detection ideas (temporal IoU, mAP, NMS) from session 203 and the long-video token budget from session 206. The next session, session 208, asks the step before this one: which video contains the moment.

The two tasks and their data

All modern methods start the same way. A frozen encoder cuts the video into short clips of fixed length \(l\) (often 1 or 2 seconds) and turns each clip into a feature vector. The query sentence becomes a sequence of token features. Everything after that works on these two sequences.

\[ V = (v_1, \dots, v_{L_v}), \quad t_i = \big(i - \tfrac12\big)\, l, \qquad \text{MR: } \{(\hat s_k, \hat e_k, c_k)\}_{k}, \qquad \text{HD: } \hat{\mathbf{S}} \in \R^{L_v} \tag{207.1} \]
Read the symbols
\(v_i\)
the feature vector of clip \(i\) (for example CLIP and SlowFast features of 2 seconds of video)
\(L_v\)
the number of clips; a 150-s video with 2-s clips has \(L_v = 75\)
\(t_i\)
the center time of clip \(i\), in seconds
\(\hat s_k, \hat e_k, c_k\)
start, end and confidence of predicted moment \(k\) (moment retrieval returns a ranked list)
\(\hat{\mathbf{S}}\)
one saliency score per clip (highlight detection)

Moment retrieval is like temporal action localization, but the “class” is a free sentence that changes for every query, so the model must compare video and text. Highlight detection is a ranking problem over clips. It comes in two forms: query-based (“highlights about the beach”) and generic (“the best parts of this video”, as used for thumbnails and previews).

DatasetSize and contentNotes
Charades-STA (2017)About 16k query–moment pairs on 6.7k indoor videos; videos about 30 s, moments about 8 s [Certain]Built semi-automatically from Charades activity labels and descriptions (keyword matching, then a check by two authors) [Certain]
QVHighlights (2021)10,310 queries, 18,367 moments, 10,148 YouTube videos of 150 s (vlogs and news) [Certain]Both tasks: 1.8 moments per query on average, and a 5-point saliency score for every relevant 2-s clip from 3 annotators [Certain]
ActivityNet Captions, DiDeMo, TACoSLonger YouTube videos with dense captions; Flickr videos in 5-s units; cooking videos [Likely]Classic moment retrieval benchmarks
Ego4D NLQEgocentric (head-camera) video with natural-language questions [Likely]Long videos, very short answers: hard
TVSum, YouTube HighlightsSmall sets with per-segment importance scores [Likely]Generic highlight detection and summarization

In QVHighlights, annotators first wrote a query and clicked on all relevant 2-s clips in a grid; then other workers rated each selected clip from “Very Bad” to “Very Good”. [Certain] So one query can have several separate moments, and every clip inside a moment has a quality score. This makes it possible to train and test both tasks on the same data.

Metrics: R1@IoU, mAP and HIT@1

Temporal IoU

A predicted moment \(\hat{M} = [\hat s, \hat e]\) is compared with the true moment \(M = [s, e]\) by temporal intersection over union, exactly as in action localization:

\[ \operatorname{IoU}(\hat M, M) = \frac{\max\!\big(0,\ \min(\hat e, e) - \max(\hat s, s)\big)}{\max(\hat e, e) - \min(\hat s, s)} \tag{207.2} \]

For overlapping intervals the bottom is the length of the union. IoU is 1 for a perfect match and 0 when the intervals do not touch.

Recall at the top results

The classic single-moment metric is R@n, IoU=m: the fraction of queries for which at least one of the top \(n\) predictions has IoU of at least \(m\) with the truth. Most papers report \(n = 1\), written R1@m, at \(m \in \{0.3, 0.5, 0.7\}\), plus the mean IoU of the top-1 prediction:

\[ \text{R}n@m = \frac{1}{N_q}\sum_{q=1}^{N_q} \mathbb{1}\Big[\max_{k \le n} \operatorname{IoU}\big(\hat M_{q,k}, M_q\big) \ge m\Big], \qquad \text{mIoU} = \frac{1}{N_q}\sum_{q=1}^{N_q} \operatorname{IoU}\big(\hat M_{q,1}, M_q\big) \tag{207.3} \]
Read the symbols
\(N_q\)
the number of test queries
\(\hat M_{q,k}\)
the \(k\)-th ranked prediction for query \(q\)
\(m\)
the IoU threshold; a larger \(m\) demands tighter boundaries

Papers differ on “\(\ge m\)” versus “\(\gt m\)” (TALL writes “larger than”, QVHighlights writes “\(\ge\)”). [Certain] It rarely matters, but it does when predictions are snapped to a grid and ties are common. Figure 1 lets you move a prediction and see the thresholds pass and fail.

true momentpredictionintersection
Figure 1. One query on a 30-s Charades-style video. The true moment is [8.0, 14.5] s (6.5 s long, close to the Charades-STA average). Move the predicted start and end; the readout gives the IoU of formula (207.2) and whether this prediction would count as a hit for R1@0.3, R1@0.5 and R1@0.7.
How to read Figure 1

The default prediction [9.0, 16.0] s looks good to the eye: it starts 1 s late and ends 1.5 s late. The overlap is 5.5 s and the union 8 s, so IoU = 0.6875. It passes 0.3 and 0.5 but fails 0.7. Now move only the end back to 14.5: IoU becomes 5.5/6.5 = 0.846 and all three pass. Then try a very long prediction, [0, 30]: it covers the whole moment, but IoU is only 6.5/30 = 0.217. IoU punishes both missing parts and extra parts.

Worked example: R1@m and mIoU on five queries

Five queries, one top-1 prediction each:

  • truth [8.0, 14.5], prediction [9.0, 16.0]: IoU 0.688
  • truth [2.1, 9.8], prediction [2.0, 10.0]: IoU \(7.7/8.0 = 0.963\)
  • truth [15.0, 21.3], prediction [11.0, 18.0]: overlap 3, union 10.3, IoU 0.291
  • truth [0.0, 6.4], prediction [0.5, 5.5]: IoU \(5/6.4 = 0.781\)
  • truth [20.2, 27.0], prediction [3.0, 9.0]: IoU 0

R1@0.3 = 3/5 = 0.6 (the third query just misses with 0.291), R1@0.5 = 0.6, R1@0.7 = 2/5 = 0.4, mIoU = 0.5445. One bad query (the last) lowers mIoU a lot, but it costs R1 only one hit: the two metrics answer different questions.

mAP when a query has several moments

In QVHighlights a query can match several separate moments, so R1 is not enough. The benchmark also uses mean average precision, as in detection: rank the predictions, mark each one as a true positive if it has IoU \(\ge \theta\) with a still-unmatched true moment, compute the area under the precision–recall curve, and average:

\[ \text{mAP}@\theta = \frac{1}{N_q}\sum_{q=1}^{N_q} \text{AP}_q(\theta), \qquad \text{avg mAP} = \frac{1}{10}\sum_{\theta \in \{0.5, 0.55, \dots, 0.95\}} \text{mAP}@\theta \tag{207.4} \]

QVHighlights reports mAP@0.5, mAP@0.75 and the average over the ten thresholds 0.5:0.05:0.95. [Certain] Example: true moments [20, 34] and [60, 70] s; ranked predictions [21, 35] (IoU 0.867), [40, 48] (IoU 0) and [58, 69.5] (IoU 0.792). For \(\theta \le 0.75\) the hits are TP, FP, TP and AP \(= \tfrac12 \cdot 1 + \tfrac12 \cdot \tfrac23 = 0.833\). At 0.8 and 0.85 only the first hits (AP 0.5); at 0.9 and 0.95 nothing hits. The average over the ten thresholds is \((6 \cdot 0.833 + 2 \cdot 0.5)/10 = 0.6\).

Highlight detection: mAP and HIT@1 against each annotator

For highlight detection the truth is a score per clip. QVHighlights counts a clip as positive if its score is “Very Good”, ranks clips by the predicted saliency, and reports mAP and HIT@1: is the single top-ranked clip a positive? Because the saliency scores come from 3 people, the metric is computed against each annotator separately and then averaged: [Certain]

\[ \text{HIT@1} = \frac{1}{N_q}\sum_{q=1}^{N_q} \frac{1}{A}\sum_{a=1}^{A} \mathbb{1}\Big[\, y^{(a)}_{q,\,i^*_q} = \text{Very Good} \Big], \qquad i^*_q = \argmax_i \hat S_{q,i} \tag{207.5} \]
Read the symbols
\(A\)
the number of annotators (3 in QVHighlights)
\(y^{(a)}_{q,i}\)
annotator \(a\)’s rating of clip \(i\) for query \(q\)
\(i^*_q\)
the clip the model ranks first

Averaging over annotators is an honest way to say “the truth is not one number”. If the three people disagree about the best clip, even a perfect model cannot reach HIT@1 = 1. We return to this in the section on label noise.

A training-free baseline: similarity curve, threshold, NMS

Before any grounding model, there is a simple method that needs no training at all. Take an image–text model such as CLIP (see session 166). Encode every clip and the query sentence into the same vector space. The cosine similarity of each clip with the query gives one number per clip: a similarity curve over time. Where the curve is high, the video probably shows the query. Three steps turn the curve into moments:

  1. Smooth the curve, so that one bad clip (the dog hidden behind a tree for 2 seconds) does not cut a moment in two.
  2. Threshold it: every maximal run of clips whose smoothed score is at least \(\tau\) becomes one predicted moment.
  3. Remove duplicates with 1-D non-maximum suppression (NMS) when several thresholds or several methods propose overlapping moments.

This baseline matters for three reasons. It is how many real “search inside a video” features start, because clip embeddings are computed once and every new query costs only one dot product per clip. It is the zero-shot reference that trained models must beat. And its last two steps (threshold, then NMS) are exactly the post-processing that per-clip models such as UniVTG use on their learned scores (next section). In the Moment-DETR paper, plain CLIP scoring of single frames even had the best HIT@1 on QVHighlights (details below). [Certain]

\[ \begin{aligned} s_i &= \frac{\mathbf{v}_i^\top \mathbf{q}}{\lVert \mathbf{v}_i \rVert\, \lVert \mathbf{q} \rVert}, \\ \tilde s_i &= \frac{1}{|N_w(i)|} \sum_{j \in N_w(i)} s_j, \quad N_w(i) = \{\max(1, i-w), \dots, \min(L_v, i+w)\} \end{aligned} \tag{207.6} \]
Read the symbols
\(\mathbf{v}_i, \mathbf{q}\)
the embedding of clip \(i\) and of the query sentence, from the same image–text model
\(s_i\)
the raw similarity of clip \(i\) with the query (cosine, between −1 and 1)
\(w\)
the smoothing half-width in clips; \(w = 0\) means no smoothing, \(w = 1\) averages each clip with its two neighbors
\(N_w(i)\)
the window of clips around \(i\), cut at the two ends of the video
\(\tilde s_i\)
the smoothed similarity (a moving average)
\[ \begin{aligned} \hat M &= [\,(a-1)\,l,\ b\,l\,] \ \text{ for each maximal run } a..b \text{ with } \tilde s_i \ge \tau, \\ c &= \frac{1}{b-a+1} \sum_{i=a}^{b} s_i, \qquad c^{\text{oic}} = c - \bar s_{\text{out}} \end{aligned} \tag{207.7} \]
Read the symbols
\(a, b\)
the first and last clip of a run; “maximal” means the clips just before \(a\) and just after \(b\) are below \(\tau\) (or outside the video)
\(l\)
the clip length (2 s here), so clip \(i\) covers \([(i-1)\,l,\ i\,l]\)
\(\tau\)
the threshold on the smoothed curve
\(c\)
the confidence of the moment: the mean raw similarity inside it
\(\bar s_{\text{out}}\)
the mean raw similarity of the \(m = \lceil (b-a+1)/4 \rceil\) clips just before and just after the moment; \(c^{\text{oic}}\) (outer–inner contrast) is high only when the inside is clearly better than its surroundings

Why two scores? The plain mean \(c\) prefers short moments: a 2-s run on the very peak always has a higher mean than the full moment around it. The contrast \(c^{\text{oic}}\) also asks that the boundaries sit where the curve drops. This idea comes from weakly supervised action localization (the outer–inner contrast of AutoLoc). [Likely] Figure 2 runs the full pipeline on a toy video; the numbers are real cosines of small 4-D vectors that you can read in the live panel.

dogfrisbeepersongrassraw \(s_i\)smoothed \(\tilde s_i\)threshold \(\tau\)predicted momenttrue moment
  1. for \(i = 1..L_v\): \(s_i \leftarrow \cos(\mathbf{v}_i, \mathbf{q})\) // (207.6)
  2. for \(i = 1..L_v\): \(\tilde s_i \leftarrow\) mean of \(s\) over \(N_w(i)\)
  3. if \(\tilde s_i \ge \tau\) and no run is open: open a run at \(a = i\)
  4. if \(\tilde s_i \lt \tau\) and a run is open: close it, \(b = i - 1\)
  5. if a run is still open at the end: close it, \(b = L_v\)
  6. for each run: \(\hat M \leftarrow [(a-1)l, bl]\), \(c \leftarrow\) mean \(s\) // (207.7); IoU with the truth // (207.2)
  7. return the moments sorted by \(c\)
Figure 2. Zero-shot grounding of “a dog catches a frisbee” on a 60-s park video cut into 30 clips of \(l = 2\) s. Each clip is a 4-D vector (dog, frisbee, person, grass), drawn as four small bars; the query is \(\mathbf{q} = (0.7, 0.6, 0.1, 0.3)\). The annotator marked the true moment as [21, 35] s, which does not fall on the 2-s clip grid. Other scenes: a person throws a frisbee alone (4–10 s), the dog hides behind a tree (26–28 s), a dog runs without a frisbee (42–48 s). The vectors are made up; every cosine, average, run and IoU is computed live.
How to read Figure 2

Press Play (or Step) and follow the highlighted line; the panel next to it shows the formula with the real numbers. Line 1 fills the gray dots, one cosine per clip. Clip 14 (26–28 s, the dog behind the tree) has \(\mathbf{v} = (0.12, 0.45, 0.62, 0.58)\): dot product 0.590, norms 0.968 and 0.975, so \(s_{14} = 0.625\), far below its neighbors (0.984 and 0.989). Lines 2–5 sweep the smoothed curve against \(\tau\). With the defaults (\(\tau = 0.7\), \(w = 1\)) the average at clip 14 is \((0.984 + 0.625 + 0.989)/3 = 0.866\), so the run stays open and one moment [20, 34] s comes out, with \(c = 0.896\) and IoU \(13/15 = 0.867\) with the truth: a hit at R1@0.7. The dog without a frisbee leaves a second, weaker moment [44, 46] s (\(c = 0.797\), IoU 0). Now set \(w = 0\): the dip cuts the moment in two, [20, 26] s (\(c = 0.936\), IoU 0.333) and [28, 36] s (IoU 0.467), and the top-1 now misses even R1@0.5. Set \(w = 1\) and \(\tau = 0.6\): one long moment [18, 36] s, IoU 0.778. At \(\tau = 0.9\) only two 2-s peaks survive (IoU 0.143 each).

Removing duplicates: 1-D NMS and Soft-NMS

One threshold rarely fits every video and every query. A common fix is to run the thresholding at several values of \(\tau\) and pool all the moments; this is the idea of temporal actionness grouping in SSN. [Likely] The pool now has many nested, overlapping copies of the same event. Non-maximum suppression (NMS) cleans it: sort by score, keep the best moment, delete every moment that overlaps it too much, and repeat with the best of the rest. Soft-NMS (Bodla et al., 2017) does not delete; it lowers the score of an overlapping moment, more for a larger overlap. [Certain]

\[ \text{hard: drop } \hat M_k \ \text{ if } \ \max_{j \in \mathcal{K}} \operatorname{IoU}(\hat M_k, \hat M_j) \gt \theta, \qquad \text{soft: } c_k \leftarrow c_k\, e^{-\operatorname{IoU}(\hat M_k,\, \hat M_m)^2 / \sigma} \tag{207.8} \]
Read the symbols
\(\mathcal{K}\)
the moments kept so far, each with a higher score than \(\hat M_k\)
\(\theta\)
the NMS IoU threshold; UniVTG uses 0.7 for moment retrieval [Certain]
\(\hat M_m\)
the moment just picked (the current best) in Soft-NMS
\(\sigma\)
the width of the Gaussian penalty; 0.5 is the usual value in the Soft-NMS paper [Likely]

NMS is greedy: only kept moments can suppress others, and the order is the score order. So it keeps the moment with the higher score, not the one closer to the truth (it cannot know the truth). This is the main reason Moment-DETR was built to avoid it. Figure 3 runs both versions on the pooled moments of Figure 2.

true momentkeptcurrent best \(m\)being compared \(b\)intersectionsuppressedwaiting
    Figure 3. NMS (207.8) on 11 candidate moments, A to K. They come from the curve of Figure 2 (\(w = 1\)) thresholded at \(\tau = 0.5, 0.6, 0.7, 0.8\) and 0.9, and are scored with the contrast \(c^{\text{oic}}\) of (207.7); the number on the left of each row is its current score. The truth is [21, 35] s. Every IoU and every Soft-NMS decay is computed live.
    How to read Figure 3

    Hard NMS at \(\theta = 0.5\): A = [18, 36] s has the best contrast (0.444) and is kept. Its IoU with C = [20, 34] is \(14/18 = 0.778 \gt 0.5\), so C is suppressed, although C fits the truth better (IoU 0.867 against 0.778 for A). D (an exact copy of C from \(\tau = 0.8\)) and E = [16, 38] (IoU \(18/22 = 0.818\)) go too. The short peaks H and I sit inside A but have IoU only 0.111 with it, so they survive: NMS does not remove moments that are contained in a kept one if they are short. B, the dog without a frisbee, removes F (IoU \(6/10 = 0.6\)). Result: 7 of 11 kept, top-1 A, a hit at R1@0.7. At \(\theta = 0.3\) J and K also go (IoU 0.333 each): 5 kept. At \(\theta = 0.9\) C and E survive: 10 kept. Switch to Soft-NMS: nothing is deleted. C is multiplied by \(e^{-0.778^2/0.5} = 0.298\) when A is picked and ends fifth with score 0.098; its exact copy D is hit again by C itself (IoU 1, factor \(e^{-2} = 0.135\)) and drops to 0.005.

    Algorithm 1: zero-shot moment retrieval from a similarity curve
    1. input: clip embeddings \(\mathbf{v}_{1..L_v}\), query embedding \(\mathbf{q}\), thresholds \(\{\tau\}\), smoothing \(w\), NMS threshold \(\theta\)
    2. \(s_i \leftarrow \cos(\mathbf{v}_i, \mathbf{q})\); \(\tilde s \leftarrow\) moving average of \(s\) // (207.6)
    3. for each \(\tau\): every maximal run of \(\tilde s_i \ge \tau\) becomes a moment with score \(c^{\text{oic}}\) // (207.7)
    4. sort all moments by score; greedy NMS with threshold \(\theta\) // (207.8)
    5. return the top \(n\) moments (for R@n); for highlights, return the clips with the highest \(s_i\)

    The weak points are clear from the figures. The curve knows only what a single clip looks like, so it cannot ground order or change (“after the dog drops the frisbee”). Scores of different queries are not on the same scale, so one \(\tau\) does not fit all queries. And a visually similar scene (a dog without a frisbee) becomes a confident false positive. The models in the next sections learn the boundaries and the scores instead of fixing them by hand.

    Generation 1: sliding windows and boundary regression (TALL)

    The first deep approach treated grounding like early object detection. TALL (Gao et al., 2017) introduced the task “temporal activity localization via language” and the Charades-STA dataset. [Certain] Its model, CTRL (Cross-modal Temporal Regression Localizer), works in three steps:

    1. Cut the video into many candidate windows with a multi-scale sliding window (in training, 64, 128, 256 and 512 frames with 80% overlap). [Certain]
    2. For each (window, sentence) pair, fuse the window feature (plus some context before and after) with the sentence feature and predict an alignment score.
    3. Also predict two offsets that move the window’s start and end toward the true boundaries.
    \[ t^s = s - s_c, \quad t^e = e - e_c, \qquad \mathcal{L} = \mathcal{L}_{\text{aln}} + \alpha\, \mathcal{L}_{\text{reg}} \tag{207.9} \]
    Read the symbols
    \(s_c, e_c\)
    start and end of the candidate window
    \(s, e\)
    start and end of the true moment
    \(t^s, t^e\)
    the regression targets (offsets in time, not scaled by the window length)
    \(\alpha\)
    a weight between the two losses

    The alignment loss is a logistic loss over a batch of \(N\) aligned pairs: the matched pair \((c_i, s_i)\) should score high and all mismatched pairs \((c_i, s_j)\) low:

    \[ \mathcal{L}_{\text{aln}} = \frac{1}{N}\sum_{i=1}^{N}\Big[\alpha_c \log\big(1 + e^{-cs_{i,i}}\big) + \sum_{j \ne i} \alpha_w \log\big(1 + e^{cs_{i,j}}\big)\Big] \tag{207.10} \]

    Here \(cs_{i,j}\) is the alignment score of window \(i\) with sentence \(j\), and \(\alpha_c, \alpha_w\) weight positives and negatives. The paper found that these plain (“non-parameterized”) offsets worked better than the object-detection style offsets scaled by window length: for actions, “time itself is the standard scale”. [Certain] On Charades-STA, CTRL reached R1@0.5 = 23.63 and R1@0.7 = 8.89. [Certain]

    Algorithm 2: sliding-window grounding (CTRL style) at test time
    1. input: video, sentence \(q\), window lengths \(\{w\}\), overlap, trained scorer and regressor
    2. for each window length \(w\) and each start position do
    3.   \(c \leftarrow\) features of the window, plus pre- and post-context windows
    4.   score \(\leftarrow\) alignment\((c, q)\); \((t^s, t^e) \leftarrow\) regressor\((c, q)\)
    5.   refined moment \(\leftarrow [s_c + t^s,\ e_c + t^e]\) // formula (207.9)
    6. sort all refined moments by score; optionally remove duplicates with 1-D NMS // (207.8), Figure 3
    7. return the top \(n\) moments (for R@n)

    The weak point is cost and coverage: a short moment between two windows is found only if the regressor can move far enough, and the number of windows grows with video length times the number of scales.

    Generation 2: one prediction per clip (the UniVTG view)

    A simpler idea, borrowed from anchor-free detectors such as ActionFormer: let every clip make a prediction. UniVTG (Lin et al., 2023) writes all grounding labels in one format. For every clip \(v_i\) with center \(t_i\) it defines three targets: [Certain]

    \[ v_i \mapsto (f_i,\ \mathbf{d}_i,\ s_i), \quad f_i \in \{0, 1\}, \quad \mathbf{d}_i = (d^s_i, d^e_i), \quad b_i = [\,t_i - d^s_i,\ t_i + d^e_i\,], \quad s_i \in [0, 1] \tag{207.11} \]
    Read the symbols
    \(f_i\)
    foreground indicator: 1 if the clip is inside a moment that matches the query
    \(d^s_i, d^e_i\)
    distance from the clip center back to the moment start and forward to its end (only used when \(f_i = 1\))
    \(b_i\)
    the moment this clip “votes for”
    \(s_i\)
    saliency: how relevant the clip is to the query (0 = not at all, 1 = highly)

    The point of this format is that every kind of label can be converted into it. An interval label (moment retrieval) gives \(f\) and \(\mathbf{d}\) directly. A curve label (a score per clip, highlight detection) gives \(s\), and thresholding it gives intervals. A point label (one timestamp, as in Ego4D narrations) can be grown into a pseudo-interval. [Certain] This let UniVTG pretrain on 4.2M temporal labels: 1.8M point labels from Ego4D, 0.9M interval labels from VideoCC and 1.5M pseudo curve labels made by a CLIP teacher. [Certain]

    The model has a light multimodal transformer and three small heads (foreground with binary cross-entropy, boundary with smooth-L1 plus gIoU loss, and saliency as the cosine similarity between clip and sentence features trained with a contrastive loss). At test time: [Certain]

    • Moment retrieval: rank the clip votes \(b_i\) by \(\tilde f_i\), then 1-D NMS with threshold 0.7 (as in Figure 3).
    • Highlight detection: rank clips by \(\tilde f_i + \tilde s_i\) and return the top ones.

    Step through this on a toy example in Figure 4.

    true momentforeground \(\tilde f_i\)saliency \(\tilde s_i\)
    Figure 4. UniVTG-style decoding for the query “a dog catches a frisbee” on a 60-s video cut into 30 clips of 2 s. The true moment is [20, 34] s. A second scene at 42–48 s shows a dog without a frisbee, so the model gives it a medium score. The scores and offsets are made-up numbers; the steps (threshold, vote, NMS, rank by \(f + s\)) are the real procedure.
    How to read Figure 4

    Step 2 shows the foreground head: high near 27 s, a smaller bump near 45 s. Step 3 keeps 6 clips with \(\tilde f \ge 0.4\); each votes for an interval with its two offsets. Step 4 runs NMS at IoU 0.7: the five votes around the true moment collapse into one, [21.1, 34.6] s with IoU 0.884, and the dog-only scene stays as a second, lower-ranked moment (it would be a false positive for mAP but does not hurt R1). Step 5 ranks clips by \(f + s\) for highlights: the top clip is 28–30 s (score 1.682), inside the moment.

    On QVHighlights test, UniVTG without pretraining reached R1@0.5 = 58.86, avg mAP 35.47 and HIT@1 60.96; with its large-scale pretraining, 65.43, 43.63 and 66.28. [Certain] Its zero-shot model (no QVHighlights training at all) reached 25.16 R1@0.5, below the supervised models but above some older supervised baselines. [Certain]

    Generation 3: DETR-style set prediction (Moment-DETR)

    Per-clip voting gives many near-duplicate moments, so it needs NMS. DETR (Carion et al., 2020) showed for object detection that a transformer decoder with a fixed set of learned queries can output a set of boxes directly, trained with a one-to-one matching so that duplicates are punished. Moment-DETR (Lei, Berg and Bansal, 2021), proposed together with QVHighlights, brings this to time. [Certain]

    1. Inputs: video features every 2 s (SlowFast and CLIP ViT-B/32, concatenated to 2816 numbers) and CLIP text token features (512). Both are projected to \(d = 256\) and concatenated into one sequence. [Certain]
    2. Encoder: 2 transformer layers over the joint video+text sequence. A linear head on the video positions gives the saliency score of each clip. [Certain]
    3. Decoder: 2 layers with \(N = 10\) learned moment queries (trainable embeddings, not the text query). Each query outputs a normalized (center, width) and a foreground/background probability. [Certain]

    The model returns 10 moments for every video. Training must decide which of the 10 predictions is responsible for which true moment. That is a bipartite matching (an assignment problem), solved with the Hungarian algorithm:

    \[ \hat\sigma = \argmin_{\sigma} \sum_{i} \Big[ -\hat p_{\sigma(i)}(\text{fg}) + \lambda_{L1}\, \big\lVert m_i - \hat m_{\sigma(i)} \big\rVert_1 + \lambda_{\text{iou}}\, \big(1 - \operatorname{gIoU}(m_i, \hat m_{\sigma(i)})\big) \Big] \tag{207.12} \]
    Read the symbols
    \(m_i\)
    true moment \(i\), as (center, width) divided by the video length
    \(\sigma\)
    an assignment: true moment \(i\) is given to prediction \(\sigma(i)\); each prediction gets at most one
    \(\hat p(\text{fg})\)
    the predicted probability that the moment query is foreground
    \(\operatorname{gIoU}\)
    generalized IoU: IoU minus the fraction of the smallest enclosing interval that is not covered; it is negative for far-apart intervals, so it still gives a useful signal when there is no overlap

    After matching, the loss pulls each matched prediction toward its moment, makes it say “foreground”, and makes every unmatched prediction say “background” (with its log-probability down-weighted by 10 to balance the many background slots): [Certain]

    \[ \mathcal{L} = \lambda_{s}\,\mathcal{L}_{\text{sal}} + \sum_{j=1}^{N} \Big[ -\lambda_{\text{cls}}\, w_j \log \hat p_j(c_j) + \mathbb{1}[c_j = \text{fg}]\, \big(\lambda_{L1} \lVert m - \hat m_j \rVert_1 + \lambda_{\text{iou}} \mathcal{L}_{\text{gIoU}}\big) \Big] \tag{207.13} \]

    with \(w_j = 1\) for foreground and \(0.1\) for background, and \(\lambda_{L1} = 10\), \(\lambda_{\text{iou}} = 1\), \(\lambda_{\text{cls}} = 4\), \(\lambda_s = 1\) in the paper. [Certain] The saliency loss is a hinge (margin) ranking loss on two pairs of clips: a high-rated and a low-rated clip inside the moment, and a clip inside and a clip outside:

    \[ \mathcal{L}_{\text{sal}} = \max\!\big(0,\ \Delta + S(t_{\text{low}}) - S(t_{\text{high}})\big) + \max\!\big(0,\ \Delta + S(t_{\text{out}}) - S(t_{\text{in}})\big), \quad \Delta = 0.2 \tag{207.14} \]

    The loss is zero once the better clip beats the worse one by at least the margin \(\Delta\). It only needs an order between clips, not exact scores, which suits noisy human ratings.

    true momentpredicted momentoptimal matchyour choice
    Figure 5. Bipartite matching for 5 moment queries (Moment-DETR uses 10) and 2 true moments on a normalized timeline. Top: the moments; the number after each prediction is its foreground probability. Bottom: the cost matrix of formula (207.12) with \(\lambda_{\text{iou}} = 1\). Green cells are the optimal (Hungarian) assignment. Click a cell to make your own assignment for that true moment and compare the total cost.
    How to read Figure 5

    At the default \(\lambda_{L1} = 10\) (the paper’s value) the optimum gives true moment 1 to query 1 and true moment 2 to query 4, total cost −0.079. Query 4 fits the second moment best (L1 distance 0.035) even though it is not confident (0.30). Lower \(\lambda_{L1}\) to 6 or less (the switch point is about 6.49): the matching switches to query 2, which is more confident (0.55) but fits worse (L1 0.065). Query 3, a wide box that covers both moments, is never chosen: it is confident but fits neither. Click query 3 in column 2 to see how much worse that would be.

    Worked example: the matched loss in Figure 5

    With the optimal matching (query 1 → moment 1, query 4 → moment 2): L1 distances 0.03 and 0.035 (sum 0.065); gIoU 0.857 and 0.722, so the gIoU loss is \(0.143 + 0.278 = 0.421\). Cross-entropy: foreground \(-\ln 0.85 - \ln 0.30 = 1.366\); background for queries 2, 3, 5: \(-(\ln 0.45 + \ln 0.30 + \ln 0.85) = 2.165\), times 0.1 = 0.216. Total: \(10 \cdot 0.065 + 1 \cdot 0.421 + 4 \cdot (1.366 + 0.216) = 7.403\). Most of the loss is the classifier: query 4 must learn to be more confident, and query 3 must learn to say “background”.

    Algorithm 3: one Moment-DETR training step
    1. input: clip features \(E_v\), text token features \(E_q\), true moments \(\{m_i\}\), clip ratings
    2. \(X \leftarrow [\text{proj}(E_v);\ \text{proj}(E_q)]\); \(H \leftarrow \text{Encoder}(X)\) // 2 layers
    3. saliency \(S \leftarrow\) Linear\((H_{\text{video positions}})\)
    4. \(Z \leftarrow \text{Decoder}(\text{moment queries}, H)\); for each of \(N\) queries: \((\hat c, \hat w) \leftarrow\) FFN, \(\hat p \leftarrow \softmax(\text{Linear})\)
    5. build the \(N \times |\{m_i\}|\) cost matrix; \(\hat\sigma \leftarrow\) Hungarian // formula (207.12)
    6. loss \(\leftarrow\) formula (207.13) with \(\mathcal{L}_{\text{sal}}\) from (207.14); apply it at every decoder layer (auxiliary losses)
    7. AdamW step // lr 1e-4, weight decay 1e-4 in the paper
    8. at test time: return the \(N\) moments sorted by \(\hat p(\text{fg})\); no NMS

    Results on QVHighlights test: Moment-DETR reached R1@0.5 = 52.89, R1@0.7 = 33.02, avg mAP 30.73, highlight mAP 35.69 and HIT@1 55.60. With weakly supervised pretraining on 236K ASR (automatic speech recognition) caption–timestamp pairs from the training videos, it rose to 59.78, 40.33, 36.14, 37.43 and 60.17. [Certain] One surprise in the same table: plain CLIP, scoring each clip by the similarity of its center frame to the query, had the best HIT@1 (61.04), though its highlight mAP (31.30) was lower. [Certain] Picking the single best frame is mostly an image–text matching problem.

    Generation 4: LLMs that write timestamps (TimeChat)

    Video LLMs (see session 205) answer questions in text. Early ones could describe a video but not say when things happen: in TimeChat’s zero-shot tests on Charades-STA, Video-LLaMA (7B) scored R1@0.5 = 2.7 and VideoChat-Embed (7B) 3.2. [Certain] Purely random uniform intervals score 10.77 on the same benchmark. [Certain] The models had seen frames, but nothing told them which frame came at which second.

    TimeChat (Ren et al., CVPR 2024) adds time to the input and trains the model to output time as plain text: [Certain]

    1. Timestamp-aware frame encoder. Each frame goes through an EVA-CLIP ViT-G/14 and an image Q-Former (from InstructBLIP). The Q-Former also reads a sentence such as “This frame is sampled at 2s.”, so the visual tokens of the frame carry its time.
    2. Sliding video Q-Former. A second Q-Former looks at a window of \(L_W\) frames and outputs \(N_V\) tokens, then slides with stride \(S\). Longer videos get more tokens instead of a fixed 32.
    3. LLM. LLaMA-2 7B, frozen, with LoRA (rank 32). The image and video Q-Formers and the projection are trained.
    4. Data. TimeIT, 125K instruction examples from 6 time-related tasks (dense captioning, grounding, step localization, summarization, highlight detection, transcribed speech), with answers in fixed formats such as “The given query happens in 0.0 - 6.9 seconds.”
    \[ N_{\text{tok}} = \frac{T}{S}\, N_V, \qquad R' = \frac{T\, N_P}{(T/S)\, N_V} = \frac{S\, N_P}{N_V} \tag{207.15} \]
    Read the symbols
    \(T\)
    number of sampled frames (96 in TimeChat)
    \(S, N_V\)
    window stride and tokens per window (both 32; the window length \(L_W\) is also 32)
    \(N_P\)
    patches per frame; \(R'\) is the compression rate (patches in, tokens out), which stays constant as \(T\) grows

    With TimeChat’s settings, \(96/32 \cdot 32 = 96\) video tokens go to the LLM. Training is the usual next-token loss on the answer, timestamps included:

    \[ \mathcal{L} = -\sum_{i=1}^{L_T} \log P_\theta\big(x_i \mid X_v,\ X_q,\ x_{\lt i}\big) \tag{207.16} \]

    Here \(X_v\) are the video tokens, \(X_q\) the instruction and query, and \(x_i\) the answer tokens. There is no IoU loss: the digits “2”, “5”, “0”, “.”, “0” of “250.0” are ordinary tokens. A prediction 1 s off and one 100 s off can have similar loss if they differ in one digit. This is the main difference from Moment-DETR, whose loss knows that time is a continuous quantity.

    Zero-shot results (no training on the test datasets): on Charades-STA, TimeChat reached R1@0.5 = 32.2 and R1@0.7 = 13.4; on QVHighlights highlight detection, mAP 14.5 and HIT@1 23.9. [Certain] That is far better than earlier video LLMs, but still below supervised specialists such as Moment-DETR (53.63 R1@0.5 on Charades-STA). The answers must also be parsed from free text; the authors wrote “a considerable number of heuristic rules” for this. [Certain]

    What limits the precision of a written timestamp

    A model can only point at moments it has seen. If \(T\) frames are spread evenly over a video of length \(L\), each frame stands for a slot of

    \[ \Delta = \frac{L}{T}, \qquad \operatorname{IoU}\Big([s - \delta,\ e + \delta],\ [s, e]\Big) = \frac{d}{d + 2\delta}, \quad d = e - s \tag{207.17} \]

    The second formula is the IoU when both boundaries are \(\delta\) seconds too wide. It passes 0.7 only if \(\delta \le 3d/14\): for an 8-s moment, at most 1.7 s on each side. With 96 frames on a 10-minute video, \(\Delta = 6.25\) s; an error of half a slot at each end is already 3.1 s. Figure 6 shows this ceiling.

    true momentbest frame-aligned answerframe slot
    Figure 6. A toy ceiling for timestamp precision. \(T\) frames are sampled evenly; frame \(k\) stands for the slot \([k\Delta, (k+1)\Delta]\). The true moment starts at 41.3% of the video. The “best answer” is the union of the slots whose frame lies inside the moment (or the one slot at its middle if none does): the most a perfect model could say from these frames. The view is zoomed to the area around the moment.
    How to read Figure 6

    The default is TimeChat’s 96 frames on a 600-s video with an 8-s moment: \(\Delta = 6.25\) s, the best answer is [250.00, 256.25] s for a true [247.80, 255.80], IoU 0.686, so even a perfect model fails R1@0.7. On a 30-s Charades-length video the same 96 frames give \(\Delta = 0.31\) s and IoU 0.977. At 30 minutes \(\Delta = 18.75\) s and the ceiling drops to 0.328. With 8 frames, a 10-minute video has 75-s slots and the IoU falls to 0.107. Make the moment longer and the ceiling rises: what matters is \(d / \Delta\).

    This is why long-video grounding needs more frames near the answer: either more tokens, or a two-pass approach (coarse search over the whole video, then dense frames around the candidate), as discussed in session 206. Later time-aware video LLMs (for example VTimeLLM, LITA, TimeSuite) explore other ways to represent time, such as special time tokens or relative positions. [Likely]

    Label noise: when is a “key moment”?

    Every metric above compares a prediction with one human answer. But moments have soft edges. When does “the person opens the door” start: when the hand touches the handle, or when the door moves? Is the third goal replay a “highlight”? Different careful people give different answers, and that is label noise that no model can remove.

    Otani et al. (2020) measured it on Charades-STA and ActivityNet Captions. They collected 5 new annotations for each of 1,000 Charades-STA samples and 1,288 ActivityNet Captions samples, then scored the humans with the same R1@0.5 metric. [Certain]

    R1@0.5 against the original labelCharades-STAActivityNet Captions
    Random human annotator42.835.4
    Most consistent annotator of five52.144.4
    2D-TAN (model)39.744.8
    SCDM (model)51.735.5

    Source: Otani et al., Table 1. [Certain] A typical human scores only about 43% at IoU 0.5 against the official label on Charades-STA, and models are close to or above that. The same paper found strong location biases: many moments start near the beginning of the video, and some verbs are tied to positions (“leave” near the end). Blind baselines that never look at the video beat several published models on Charades-STA. [Certain] QVHighlights was designed with this in mind: moments are spread almost evenly over the video, and on a set of 600 queries annotated by 3 workers, about 90% of queries had an average pairwise IoU above 0.9. [Certain] Its clip-based annotation (2-s cells, click-and-drag) leaves less room for disagreement than free start and end times.

    Figure 7 simulates the effect. Each annotator marks the true moment with a random error on each boundary (Gaussian, standard deviation \(\sigma\)). We compare two annotators with each other, and a perfect model that predicts the exact true moment with one random annotator.

    true momentannotatorsIoU histogram
    Figure 7. Simulated annotation noise. Top: five annotators mark one moment; each boundary is shifted by a Gaussian error with standard deviation \(\sigma\). Bottom: the histogram of IoU over 2,000 pairs of annotators (seeded random numbers, so the result is the same on every load). The readout gives the fraction of pairs with IoU \(\ge 0.5\) and \(\ge 0.7\), and the same for a perfect model scored against one annotator: the best R1 any model can expect.
    How to read Figure 7

    With \(\sigma = 2\) s on an 8-s moment, only 27.7% of annotator pairs agree at IoU 0.7, and a perfect model reaches IoU 0.7 against a random annotator only 49.5% of the time. At \(\sigma = 1\) s the perfect model’s ceiling is 91.3%. Now keep \(\sigma = 2\) and make the moment 30 s long: the ceiling is 99.8%. The same boundary error hurts short moments much more, which matches formula (207.17). Set \(\sigma = 0\) and everything is 1.

    Three practical lessons follow:

    • Measure the ceiling. Before you chase R1@0.7, have two people label a few hundred of your queries and compute their agreement with the same metric.
    • Prefer metrics that use several annotators, as QVHighlights does for highlights (207.5), or soft labels such as averaged ratings.
    • Choose losses that tolerate noise. The ranking loss (207.14) only needs an order; IoU-based losses do not care about the exact position of a single boundary as much as a per-frame classifier does.

    Code: metrics, matching, saliency loss, decoding and parsing

    The code below runs on CPU in well under a second and prints the numbers used in this page.

    import itertools, math, re, torch
    torch.manual_seed(0)
    
    # 1) temporal IoU (207.2) and R1@IoU, mIoU (207.3) for single-moment queries
    def tiou(a, b):
        inter = max(0.0, min(a[1], b[1]) - max(a[0], b[0]))
        union = max(a[1], b[1]) - min(a[0], b[0])
        return inter / union if union > 0 else 0.0
    
    gts   = [(8.0, 14.5), (2.1, 9.8), (15.0, 21.3), (0.0, 6.4), (20.2, 27.0)]
    top1  = [(9.0, 16.0), (2.0, 10.0), (11.0, 18.0), (0.5, 5.5), (3.0, 9.0)]
    ious = [tiou(p, g) for p, g in zip(top1, gts)]
    r1 = {m: sum(v >= m for v in ious) / len(ious) for m in (0.3, 0.5, 0.7)}
    print([round(v, 3) for v in ious], r1, "mIoU", round(sum(ious) / len(ious), 4))
    
    # 2) highlight detection: HIT@1 against each of 3 annotators, then averaged (207.5)
    pred = torch.tensor([0.1, 0.7, 0.9, 0.4, 0.2])            # predicted saliency, 5 clips
    ann = torch.tensor([[1, 4, 5, 3, 1],                     # 1 = Very Bad ... 5 = Very Good
                        [1, 5, 4, 3, 2],
                        [2, 3, 3, 5, 1]])
    top = pred.argmax()
    print("HIT@1 per annotator", (ann[:, top] == 5).int().tolist(), "mean", round((ann[:, top] == 5).float().mean().item(), 3))
    
    # 3) Moment-DETR style bipartite matching (207.12): (center, width) spans, normalized time
    def to_se(cw): return (cw[0] - cw[1] / 2, cw[0] + cw[1] / 2)
    def giou(a, b):
        inter = max(0.0, min(a[1], b[1]) - max(a[0], b[0]))
        union = (a[1] - a[0]) + (b[1] - b[0]) - inter
        hull = max(a[1], b[1]) - min(a[0], b[0])
        return inter / union - (hull - union) / hull
    
    gt = torch.tensor([[0.26, 0.12], [0.675, 0.15]])
    spans = torch.tensor([[0.27, 0.14], [0.66, 0.10], [0.50, 0.40], [0.70, 0.16], [0.10, 0.08]])
    p_fg = torch.tensor([0.85, 0.55, 0.70, 0.30, 0.15])
    lam_l1, lam_iou, lam_cls = 10.0, 1.0, 4.0
    cost = torch.zeros(5, 2)
    for j in range(5):
        for g in range(2):
            l1 = (spans[j] - gt[g]).abs().sum().item()
            cost[j, g] = -p_fg[j] + lam_l1 * l1 + lam_iou * (1 - giou(to_se(spans[j].tolist()), to_se(gt[g].tolist())))
    best = min(itertools.permutations(range(5), 2), key=lambda q: sum(cost[q[g], g] for g in range(2)))
    print("matched queries", [b + 1 for b in best], "cost", round(sum(cost[best[g], g].item() for g in range(2)), 3))
    
    # loss (207.13): L1 + gIoU on matched pairs, foreground/background cross-entropy (background weight 0.1)
    fg = torch.zeros(5); fg[list(best)] = 1
    w = torch.where(fg == 1, 1.0, 0.1)
    ce = -(w * (fg * p_fg.log() + (1 - fg) * (1 - p_fg).log())).sum()
    l1 = sum((spans[best[g]] - gt[g]).abs().sum() for g in range(2))
    gl = sum(1 - giou(to_se(spans[best[g]].tolist()), to_se(gt[g].tolist())) for g in range(2))
    print("L1", round(l1.item(), 3), "gIoU loss", round(gl, 3), "CE", round(ce.item(), 3),
          "total", round((lam_l1 * l1 + lam_iou * gl + lam_cls * ce).item(), 3))
    
    # 4) saliency hinge loss (207.14), margin 0.2
    S = torch.tensor([0.30, 0.55, 0.62, 0.20], requires_grad=True)   # clip scores
    t_high, t_low, t_in, t_out = 2, 1, 1, 3
    loss_sal = torch.relu(0.2 + S[t_low] - S[t_high]) + torch.relu(0.2 + S[t_out] - S[t_in])
    loss_sal.backward()
    print("saliency loss", round(loss_sal.item(), 3), "grad", [round(g, 2) for g in S.grad.tolist()])
    
    # 5) UniVTG-style decoding (207.11): per-clip foreground f, offsets (ds, de); rank by f, 1-D NMS at 0.7
    t = torch.arange(30) * 2.0 + 1.0                                   # clip centers of 2-s clips
    f = torch.maximum(0.93 * torch.exp(-((t - 27) / 6) ** 2), 0.62 * torch.exp(-((t - 45) / 2.6) ** 2)).clamp(min=0.03)
    near = (t - 45).abs() < (t - 27).abs()
    a = torch.where(near, 42.0, 20.0); b = torch.where(near, 48.0, 34.0)
    i = torch.arange(30.0)
    ds = (t - a + 1.2 * torch.sin(1.3 * i)).clamp(min=0.5); de = (b - t + 1.0 * torch.cos(0.9 * i)).clamp(min=0.5)
    cand = [(round(f[k].item(), 3), round((t[k] - ds[k]).item(), 1), round((t[k] + de[k]).item(), 1))
            for k in torch.nonzero(f >= 0.4).flatten().tolist()]
    kept = []
    for c in sorted(cand, key=lambda c: -c[0]):
        if all(tiou(c[1:], k[1:]) < 0.7 for k in kept):
            kept.append(c)
    print("moments", kept, "IoU of top-1", round(tiou(kept[0][1:], (20, 34)), 3))
    
    # 6) an LLM writes timestamps as text: parse them, then score (TimeChat output format)
    answer = "The given query happens in 250.0 - 256.3 seconds."
    s, e = map(float, re.findall(r"(\d+(?:\.\d+)?)\s*-\s*(\d+(?:\.\d+)?)\s*seconds", answer)[0])
    print("parsed", (s, e), "IoU", round(tiou((s, e), (247.8, 255.8)), 3))
    
    # 7) zero-shot grounding from a similarity curve (207.6-207.8): 30 clips of 2 s, 4-D (dog, frisbee, person, grass)
    V = ([(0.07, 0.06, 0.58, 0.51), (0.12, 0.01, 0.54, 0.54), (0.05, 0.59, 0.81, 0.45), (0.12, 0.62, 0.77, 0.41), (0.1, 0.66, 0.75, 0.48)]
         + [(0.12, 0.01, 0.63, 0.51), (0.08, 0.01, 0.64, 0.5), (0.13, 0.1, 0.63, 0.55), (0.09, 0.09, 0.59, 0.55), (0.35, 0.05, 0.46, 0.47)]
         + [(0.56, 0.29, 0.42, 0.48), (0.7, 0.54, 0.28, 0.51), (0.86, 0.75, 0.32, 0.5), (0.12, 0.45, 0.62, 0.58), (0.89, 0.86, 0.3, 0.51)]
         + [(0.83, 0.62, 0.34, 0.51), (0.57, 0.4, 0.39, 0.61), (0.4, 0.29, 0.44, 0.46), (0.08, 0.08, 0.64, 0.45), (0.11, 0.01, 0.63, 0.48)]
         + [(0.15, 0.11, 0.6, 0.56), (0.78, 0.01, 0.21, 0.49), (0.76, 0.04, 0.21, 0.51), (0.75, 0.09, 0.18, 0.61), (0.15, 0.04, 0.6, 0.5)]
         + [(0.12, 0.06, 0.61, 0.51), (0.15, 0.05, 0.59, 0.53), (0.07, 0.03, 0.66, 0.5), (0.11, 0.01, 0.59, 0.51), (0.04, 0.06, 0.62, 0.45)])
    qv = (0.7, 0.6, 0.1, 0.3)
    s = [sum(x * y for x, y in zip(v, qv)) / math.hypot(*v) / math.hypot(*qv) for v in V]   # cosine per clip
    def smooth(s, w): return [sum(s[max(0, i - w):i + w + 1]) / len(s[max(0, i - w):i + w + 1]) for i in range(len(s))]
    def runs(x, tau):
        out, a = [], None
        for i, v in enumerate(x + [-1.0]):                     # sentinel closes a run at the end
            if v >= tau and a is None: a = i
            if v < tau and a is not None: out.append((a, i - 1)); a = None
        return out
    mean = lambda xs: sum(xs) / len(xs)
    ss = smooth(s, 1)
    print("moments", [((2 * a, 2 * b + 2), round(mean(s[a:b + 1]), 3), round(tiou((2 * a, 2 * b + 2), (21, 35)), 3)) for a, b in runs(ss, 0.7)])
    def oic(a, b):
        m = math.ceil((b - a + 1) / 4)
        return mean(s[a:b + 1]) - mean(s[max(0, a - m):a] + s[b + 1:b + 1 + m])
    cand = sorted([(oic(a, b), (2 * a, 2 * b + 2)) for tau in (0.5, 0.6, 0.7, 0.8, 0.9) for a, b in runs(ss, tau)], key=lambda c: -c[0])
    kept = []
    for c in cand:                                             # greedy hard NMS, theta = 0.5
        if all(tiou(c[1], k[1]) <= 0.5 for k in kept): kept.append(c)
    print(len(cand), "candidates, NMS@0.5 keeps", len(kept), "top-1", kept[0][1], "IoU", round(tiou(kept[0][1], (21, 35)), 3))

    Output (PyTorch 2.8, CPU):

    [0.688, 0.963, 0.291, 0.781, 0.0] {0.3: 0.6, 0.5: 0.6, 0.7: 0.4} mIoU 0.5445
    HIT@1 per annotator [1, 0, 0] mean 0.333
    matched queries [1, 4] cost -0.079
    L1 0.065 gIoU loss 0.421 CE 1.583 total 7.403
    saliency loss 0.13 grad [0.0, 1.0, -1.0, 0.0]
    moments [(0.93, 21.1, 34.6), (0.62, 42.4, 48.6)] IoU of top-1 0.884
    parsed (250.0, 256.3) IoU 0.682
    moments [((20, 34), 0.896, 0.867), ((44, 46), 0.797, 0.0)]
    11 candidates, NMS@0.5 keeps 7 top-1 (18, 36) IoU 0.778

    Read it line by line: the five-query metrics match the worked example; the top clip is “Very Good” for only one of three annotators, so HIT@1 is 0.333 for a clip that two people rated 4 and 5; the matching and loss match Figure 5; only the first hinge term is active (0.2 + 0.55 − 0.62 = 0.13), so the gradient pushes the low clip down and the high clip up; the decoding matches Figure 4; and a written answer 0.2 s too wide at the start and 0.5 s at the end still scores 0.682, below 0.7. The last two lines are Figures 2 and 3: the curve gives [20, 34] s with IoU 0.867, and NMS over the 11 multi-threshold moments keeps 7 with A = [18, 36] s on top.

    Where this is used

    ApplicationTask formTypical approach
    “Jump to the moment” search inside a videoMoment retrieval with a user queryClip features indexed once; a light grounding head (Moment-DETR / UniVTG style) per query
    Chapters and key moments for long videosSegmentation + short titles with timestampsShot and topic boundaries, ASR transcript, a video LLM that writes “start - end, title” lines
    Thumbnails, previews and trailersGeneric highlight detectionPer-clip saliency (often CLIP-based), aesthetics and face/motion cues; pick top clips with diversity
    Sports key moments (goals, dunks)Action spotting with tight toleranceDense per-frame features and a temporal model (see session 203); text queries for rare events
    Video-RAG and question answeringGround the question, then read only that partRetrieve the video (session 208), ground the moment, pass dense frames to a VLM (session 210)
    Surveillance and body-camera review“Find when the red car enters”Query-based grounding over hours; coarse-to-fine search to keep cost low
    Egocentric assistants (smart glasses)“Where did I put my keys?” (Ego4D NLQ)Very short answer windows in very long video; memory banks and streaming models (session 206)

    Common mistakes

    • Reporting only R1@0.5. It hides loose boundaries. Report R1@0.7 and mIoU as well, and avg mAP when queries have several moments.
    • Comparing numbers across different features. Grounding results depend strongly on the frozen encoder (SlowFast, CLIP, I3D, InternVideo). Compare models only with the same features, or say clearly that you did not.
    • Ignoring dataset priors. On Charades-STA, blind baselines that never look at the video beat several published models. Always run a “video shuffled” or “text only” baseline.
    • Dropping unparsable LLM answers. Removing the failures from the denominator inflates R1. Count them as misses and report the parse rate.
    • Sampling too few frames for long videos. With \(T\) frames, a perfect model cannot be more precise than \(L/T\) (207.17). Check \(d / \Delta\) for your typical moment before blaming the model.
    • Treating one annotator as the truth for highlights. Ratings differ between people. Evaluate against each annotator (207.5) or against averaged ratings, and measure human agreement.
    • Forgetting to clip and sort predicted boundaries. Regression heads and LLMs can produce \(\hat e \lt \hat s\) or times past the video end.
    • Using NMS with DETR-style models, or forgetting it with per-clip models. Moment-DETR is trained to avoid duplicates; per-clip voting (UniVTG) needs NMS.

    Interview questions

    What is the difference between moment retrieval and highlight detection?basic
    Moment retrieval takes a video and a sentence and returns one or more intervals \([s, e]\) that match the sentence, ranked by confidence; it is measured with R1@IoU and mAP over IoU thresholds. Highlight detection gives every clip a score (saliency) and returns the top clips; it is measured with mAP over clips and HIT@1 against “Very Good” clips. Highlights can be query-based or generic. QVHighlights labels both on the same videos, so one model (Moment-DETR, UniVTG) can learn both.
    What does “R1@0.7 = 40” mean?basic
    For 40% of the test queries, the single top-ranked predicted moment has temporal IoU of at least 0.7 with the true moment. For an 8-s moment, IoU 0.7 allows only about 1.7 s of extra length on each side if both ends are too wide (\(d/(d+2\delta) \ge 0.7 \Rightarrow \delta \le 3d/14\)). R1@0.5 is the looser, more common number; R@5 counts a hit if any of the top 5 predictions matches.
    How does Moment-DETR work, and why does it not need NMS?core
    Clip features (SlowFast + CLIP every 2 s) and CLIP text token features are projected and concatenated into one sequence, encoded by a 2-layer transformer. A linear head on the video positions gives clip saliency. A 2-layer decoder with 10 learned moment queries outputs (center, width) and a foreground probability for each query. In training, the Hungarian algorithm assigns each true moment to exactly one query using a cost of −p(fg) + L1 + gIoU loss; only matched queries are trained to be foreground and to fit their moment, all others are trained to be background. Because two queries cannot both be rewarded for the same moment, the model learns not to produce duplicates, so NMS is not needed.
    Why use a hinge (margin) loss for saliency instead of regressing the ratings?core
    Highlight ratings are subjective and their scale differs between people, but their order within a video is more stable. The hinge loss \(\max(0, \Delta + S(t_{\text{low}}) - S(t_{\text{high}}))\) only asks that a better clip beat a worse one by a margin (0.2 in Moment-DETR); it is zero once that holds. Ranking is also exactly what the metric (mAP, HIT@1) measures. UniVTG uses a contrastive version: the positive clip should beat less-relevant clips in the same video (intra-video) and the same clip should match its own query better than other queries in the batch (inter-video).
    What is UniVTG’s unified formulation and why is it useful?core
    Every clip gets three targets: a foreground flag \(f_i\), offsets \((d^s_i, d^e_i)\) to the boundaries of its moment, and a saliency \(s_i \in [0,1]\). Interval labels, curve labels (per-clip scores) and point labels (single timestamps) can all be converted into this format, sometimes with pseudo-labels. So one model and one loss can train on moment retrieval, highlight and summarization data together, which allowed pretraining on 4.2M labels (Ego4D points, VideoCC intervals, CLIP-teacher curves) and even zero-shot grounding. Inference: rank clip votes by \(f\) and apply NMS for moments; rank clips by \(f + s\) for highlights.
    How does a video LLM like TimeChat output time, and what are the weaknesses?core
    TimeChat binds time to the visual tokens (the image Q-Former reads “This frame is sampled at 2s.” for each frame), compresses frames with a sliding video Q-Former (96 frames → 96 tokens with window, stride and tokens per window all 32), and is instruction-tuned on TimeIT (125K examples, 6 tasks) to write answers like “The given query happens in 12.0 - 18.5 seconds.” Weaknesses: the next-token loss does not know that 250 and 251 are close; precision is limited by frame spacing \(L/T\); answers must be parsed from free text; and zero-shot accuracy (32.2 R1@0.5 on Charades-STA) is still below supervised specialists. Strengths: one model for many tasks, free-form queries and explanations.
    Why does a perfect model not reach R1@0.7 = 100 on Charades-STA?advanced
    Because the labels are noisy. Moment boundaries are ambiguous and different annotators mark them differently. Otani et al. re-annotated 1,000 Charades-STA samples five times: a random human scored only 42.8 R1@0.5 against the official label (52.1 for the most consistent annotator). Charades-STA was also built semi-automatically from activity labels and descriptions. With boundary noise of a couple of seconds on an 8-s moment, the IoU between two annotators is often below 0.7. So above a point, higher numbers may reflect fitting the annotation style or dataset priors, not better understanding.
    What dataset biases affect moment retrieval, and how do you test for them?advanced
    Moment positions are not uniform: on Charades-STA many moments start near the beginning, and some verbs go with positions (“leave” near the end). A model can learn these priors from the text alone. Otani et al. showed that blind baselines (prior-only, one verb, or a text-only network) beat several published models on Charades-STA. Tests: run the model with shuffled or replaced video features, run text-only and prior-only baselines, and evaluate on splits where the moment distribution differs from training. QVHighlights spreads moments evenly to reduce this.
    Design a system that adds chapter markers and “key moments” to long uploaded videos.advanced
    Decode at 1–2 fps and compute clip embeddings once (a CLIP-style encoder) plus an ASR transcript with timestamps. Propose chapter boundaries from shot changes, topic changes in the transcript (embedding similarity drops) and silence; merge to a sensible minimum length. For each chapter, a video LLM or text LLM writes a short title from the transcript and a few frames. For key moments, score clips with a highlight model (saliency from query-free features, motion, audio energy) and pick the top clips with a diversity constraint. Snap all times to shot boundaries so they look clean. Evaluate with human raters on a sample, measure inter-rater agreement first, and track click-through on the markers in production.
    Your grounding model works on 30-s clips but fails on 1-hour videos. What do you check?advanced
    (1) Frame spacing: with a fixed frame or token budget, \(\Delta = L/T\) grows from under a second to tens of seconds, which caps IoU for short moments (207.17). (2) Normalized coordinates: a regression head that outputs (center, width) in [0, 1] has to be 120× more precise in relative terms. (3) Distractors: an hour has many similar scenes, so the top-1 can be a look-alike; use R@5 and a reranker. (4) Training distribution: moments in training may cover a large fraction of the video. Fixes: coarse-to-fine search (retrieve candidate windows with embeddings, then ground densely inside them), more frames around candidates, and training on long videos with short moments.
    Explain gIoU and why it is used for moment regression.core
    Generalized IoU is \(\text{IoU} - |C \setminus (A \cup B)| / |C|\), where \(C\) is the smallest interval that contains both \(A\) and \(B\). For overlapping intervals in 1-D, \(C\) equals the union, so gIoU equals IoU; for disjoint intervals the extra term makes gIoU negative and larger in magnitude the farther apart they are. Plain IoU is 0 for any non-overlapping pair and gives no gradient; \(1 - \text{gIoU}\) still tells the model which way to move. It is paired with an L1 loss on (center, width) because IoU alone ignores scale.
    How would you do zero-shot moment retrieval with a CLIP-style model, and where does it fail?core
    Embed every clip (for example its center frame) and the query, and compute the cosine per clip: a similarity curve (207.6). Smooth it with a short moving average so that one occluded or blurry clip does not split a moment. Threshold it: every maximal run of clips above \(\tau\) becomes a moment \([(a-1)l, bl]\), scored by its mean similarity or, better, by the outer–inner contrast (207.7), because the plain mean prefers tiny moments on the peak. Since no single \(\tau\) fits every query, run several thresholds, pool the moments and remove duplicates with 1-D NMS or Soft-NMS (207.8). Failure modes: (1) it scores single clips, so it cannot ground order or change (“after he sits down”); (2) cosine values are not calibrated across queries, so a fixed \(\tau\) gives no answer for some queries and huge moments for others; (3) look-alike scenes (a dog without a frisbee) become confident false positives; (4) boundaries snap to the clip grid, which caps IoU for short moments; (5) NMS keeps the higher-scored moment, not the better-fitting one, and it does not remove short moments contained in a kept one. It is still a strong cheap baseline and a good HIT@1 baseline, and its post-processing is the same as in per-clip models like UniVTG.

    Exercises

    1. The true moment is [12, 20] s and the prediction is [14, 22] s. Compute the IoU. Is it a hit at 0.5 and at 0.7?
    2. A 4-s moment is predicted with both boundaries \(\delta\) seconds too wide. What is the largest \(\delta\) for IoU \(\ge 0.5\)? For IoU \(\ge 0.7\)?
    3. TimeChat samples 96 frames. What is the frame spacing on a 20-minute video? Can a 10-s moment be located reliably at IoU 0.7? Check with Figure 6 (\(L = 1200\) s).
    4. In Figure 5, set \(\lambda_{L1} = 0\). Which query is matched to true moment 2, and why? Compute its cost.
    5. Saliency scores: \(S(t_{\text{high}}) = 0.6\), \(S(t_{\text{low}}) = 0.5\), \(S(t_{\text{in}}) = 0.5\), \(S(t_{\text{out}}) = 0.45\), margin 0.2. Compute the hinge loss (207.14).
    6. A model with fixed 32 video tokens reads 96 frames of 256 patches each. What is its compression rate? What is TimeChat’s rate \(R'\) with \(S = 32\), \(N_V = 32\)?
    7. In Figure 7 with \(\sigma = 2\) s, find the moment length at which a perfect model’s chance of IoU \(\ge 0.7\) first passes 90%.
    8. In Figure 3 (hard NMS, \(\theta = 0.5\)), compute the IoU of E = [16, 38] s with A = [18, 36] s. Is E suppressed? What is the smallest \(\theta\) on the slider at which E survives?
    Answerscheck
    1. Overlap [14, 20] = 6 s, union [12, 22] = 10 s, IoU 0.6: a hit at 0.5, a miss at 0.7. 2. \(4/(4 + 2\delta) \ge 0.5 \Rightarrow \delta \le 2\) s; \(\ge 0.7 \Rightarrow \delta \le 3 \cdot 4/14 = 0.857\) s. 3. \(\Delta = 1200/96 = 12.5\) s, longer than the moment itself; the best frame-aligned answer is one or two 12.5-s slots, so IoU 0.7 is out of reach; Figure 6 gives a ceiling of 0.331 for this case. 4. Query 2. Without the L1 term the cost is \(-\hat p + 1 - \text{gIoU}\): query 2 gives \(-0.55 + 1 - 0.667 = -0.217\), query 4 gives \(-0.30 + 1 - 0.722 = -0.022\); confidence wins. 5. \(\max(0, 0.2 + 0.5 - 0.6) + \max(0, 0.2 + 0.45 - 0.5) = 0.1 + 0.15 = 0.25\). 6. \(96 \cdot 256 / 32 = 768\) patches per token; TimeChat: \(R' = 32 \cdot 256 / 32 = 256\), and it stays 256 for any \(T\) (here \(N_P = (224/14)^2 = 256\) from TimeChat’s 224-px frames and 14-px patches). 7. The ceiling is 0.495 at 8 s, 0.643 at 10 s, 0.844 at 14 s, 0.887 at 15 s and first passes 90% at 16 s (0.913). For IoU 0.7 to survive 2-s boundary noise, the moment must be about twice as long as Charades-STA’s average. 8. Intersection [18, 36] = 18 s, union [16, 38] = 22 s, IoU \(18/22 = 0.818 \gt 0.5\): suppressed. It survives once \(\theta \ge 0.818\), so at 0.85 on the slider (its IoU with C = [20, 34], 0.636, is then also below \(\theta\)).

    Key takeaways

    • Moment retrieval returns intervals for a sentence; highlight detection scores every clip. Both work on a sequence of clip features (207.1).
    • R1@m counts top-1 predictions with IoU \(\ge m\) (207.2, 207.3); multi-moment queries use mAP averaged over IoU 0.5–0.95 (207.4); highlights use mAP and HIT@1 averaged over annotators (207.5).
    • A training-free baseline: cosine similarity per clip, a moving average, a threshold that cuts runs into moments, and 1-D NMS or Soft-NMS to remove duplicates (207.6–207.8). NMS keeps the higher score, not the better fit.
    • TALL/CTRL scored sliding windows and regressed boundary offsets (207.9, 207.10). UniVTG lets every clip predict foreground, offsets and saliency (207.11), with NMS.
    • Moment-DETR predicts a set of moments with learned queries and Hungarian matching (207.12, 207.13), plus a margin ranking loss for saliency (207.14); it needs no NMS.
    • TimeChat binds timestamps to frame tokens and writes times as text (207.15, 207.16). Its precision is capped by frame spacing \(L/T\) (207.17), and answers must be parsed.
    • Labels are noisy: random human annotators score about 43 R1@0.5 on Charades-STA against the official label, and dataset priors let blind models score well. Measure agreement and run blind baselines before trusting small gains.

    Key formulas

    \[ s_i = \cos(\mathbf{v}_i, \mathbf{q}), \quad \hat M = [(a-1)l,\ bl] \text{ for runs } \tilde s_i \ge \tau, \quad c_k \leftarrow c_k\, e^{-\operatorname{IoU}^2/\sigma} \]
    \[ \operatorname{IoU} = \frac{|\hat M \cap M|}{|\hat M \cup M|}, \quad \text{R}1@m = \frac{1}{N_q}\sum_q \mathbb{1}[\operatorname{IoU}_q \ge m], \quad b_i = [t_i - d^s_i,\ t_i + d^e_i] \]
    \[ \hat\sigma = \argmin_\sigma \sum_i \mathcal{C}_{\text{match}}(m_i, \hat m_{\sigma(i)}), \quad \mathcal{L}_{\text{sal}} = \max(0, \Delta + S_{\text{low}} - S_{\text{high}}) + \max(0, \Delta + S_{\text{out}} - S_{\text{in}}) \]
    \[ N_{\text{tok}} = \frac{T}{S} N_V, \quad \Delta = \frac{L}{T}, \quad \operatorname{IoU}_{\pm\delta} = \frac{d}{d + 2\delta} \]

    References for this session

    1. Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
    2. Lin et al. (2023). UniVTG: Towards Unified Video-Language Temporal Grounding
    3. Ren et al. (2023). TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
    4. Gao et al. (2017). TALL: Temporal Activity Localization via Language Query
    5. Gao et al. (2017). TALL: Temporal Activity Localization via Language Query
    6. Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
    7. Lin et al. (2023). UniVTG: Towards Unified Video-Language Temporal Grounding
    8. Ren et al. (2023). TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
    9. Carion, Massa, Synnaeve, Usunier, Kirillov, Zagoruyko (2020). End-to-End Object Detection with Transformers
    10. Rezatofighi et al. (2019). Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression
    11. Otani, Nakashima, Rahtu & Heikkilä (2020). Uncovering Hidden Challenges in Query-Based Video Moment Retrieval
    12. Sigurdsson, Varol, Wang, Farhadi, Laptev & Gupta (2016). Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
    13. Hendricks, Wang, Shechtman, Sivic, Darrell & Russell (2017). Localizing Moments in Video with Natural Language
    14. Krishna, Hata, Ren, Fei-Fei & Niebles (2017). Dense-Captioning Events in Videos
    15. Zhang et al. (2022). ActionFormer: Localizing Moments of Actions with Transformers
    16. Feichtenhofer et al. (2018). SlowFast Networks for Video Recognition
    17. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
    18. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
    19. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
    20. Bodla et al. (2017). Soft-NMS: Improving Object Detection With One Line of Code
    21. Zhao, Xiong, Wang, Wu, Tang & Lin (2017). Temporal Action Detection with Structured Segment Networks
    22. Shou, Gao, Zhang, Miyazawa & Chang (2018). AutoLoc: Weakly-supervised Temporal Action Localization