LLM & VLM courseBlog
208Track 8: Video and multimodal systems · session 8 of 21

Text-to-video retrieval

Text-to-video retrieval ranks a whole video library by how well each video, or each moment, matches a sentence. This session builds it step by step: CLIP frame embeddings with mean, sequential and attention pooling (CLIP4Clip, X-Pool), video–text encoders trained on video (ViCLIP, InternVideo2), segment-level indexes for long videos, approximate nearest-neighbor search for millions of segments, and re-ranking the top candidates with a VLM.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on CLIP (166) and video encoders (202); basic PyTorch; vector search basics help

What you will learn

  • How a dual encoder scores text against video, and how R@K, median rank and mean rank are computed, and why they depend on gallery size
  • The three CLIP4Clip similarity calculators (mean pooling, sequential transformer, tight cross-encoder), the symmetric contrastive loss, and what the MSR-VTT results say about each
  • Text-conditioned attention pooling (X-Pool): what it gains, and why its query-dependent video vector cannot go into a standard index
  • What ViCLIP and InternVideo2 change by training the encoder on video–text data, and the false-negative problem of many clips per video
  • Clip-level versus segment-level indexes: the dilution formula, how to choose segment length, overlap and shot boundaries, and merging neighboring hits
  • Scaling to tens of millions of segments with IVF, PQ and HNSW, measuring ANN recall, and adding ASR and OCR with reciprocal rank fusion
  • Two-stage retrieval: re-ranking the top candidates with a VLM yes/no score and fusing it with the first-stage score

Why this matters

A user types “a goalkeeper saves a penalty in the rain” and expects the right ten seconds out of a library of a million hours. That is text-to-video retrieval: given a sentence, rank all videos (or all moments of all videos) by how well they match it. The reverse direction, given a video find the best caption, is video-to-text retrieval.

The task looks like image search with more frames. It is harder for three reasons:

  1. A video says much more than a sentence. A 30-second clip may show a kitchen, a person talking, a dog walking in, and a pan on fire. A query usually names one of these things. The model must decide which part of the video the query is about.
  2. Videos are long. A benchmark clip is 10–30 seconds; a real video is minutes or hours. If you squeeze an hour into one vector, a 4-second event disappears.
  3. The library is huge. Every video must be encoded once, offline, into vectors that a query can be compared with in milliseconds. That rules out models that need to see the query and the video together, unless they run only on a short list.

This session builds the standard answer step by step: frame embeddings from CLIP plus pooling (CLIP4Clip), segment-level indexes for long videos, video–text encoders trained on video (ViCLIP, InternVideo2), approximate nearest-neighbor search for millions of segments, and re-ranking the top candidates with a VLM.

We use CLIP from session 166 and the video encoders of session 202. The previous session, Session 207, found the moment that matches a sentence inside one video. Here we search across a whole library. The next session, Session 209, searches for the same content (copies), not the same meaning.

The task, the data and the metrics

Almost every system uses a dual encoder (also called a two-tower or bi-encoder model): a text encoder \(g\) turns the query into a vector \(\mathbf{t}\), a video encoder \(f\) turns each video into a vector \(\mathbf{v}\), both of length \(d\) and normalized to length 1. The score is the cosine similarity:

\[ s(\mathbf{t}, \mathbf{v}) = \mathbf{t}^\top \mathbf{v} = \cos\angle(\mathbf{t}, \mathbf{v}), \qquad \lVert \mathbf{t} \rVert = \lVert \mathbf{v} \rVert = 1 \tag{208.1} \]
Read the symbols
\(\mathbf{t} = g(\text{query})\)
the text embedding of the query
\(\mathbf{v} = f(\text{video})\)
the video embedding, computed once and stored in an index
\(d\)
the embedding size (512 for CLIP ViT-B/32 [Certain])

Because \(\mathbf{v}\) does not depend on the query, all video vectors can be computed offline. At query time we run the text encoder once and compute dot products. This is what makes search over millions of items possible.

Benchmarks

The most used benchmark is MSR-VTT (Microsoft Research Video to Text): 10,000 web video clips, 10 to 32 seconds long, about 41 hours in total, from 20 categories, with 20 human captions per clip (200,000 captions). [Certain] Results are usually reported on the 1k-A test split: 1,000 clip–caption pairs. Models train on 9,000 clips (“Training-9K”) or 7,000 (“Training-7K”). [Certain] Other common sets are MSVD (1,970 clips), LSMDC (movie clips), DiDeMo and ActivityNet Captions; on the last two, all captions of a video are joined into a paragraph, so they test video–paragraph retrieval. [Certain]

Recall at K and median rank

For each test query \(i\) we sort all \(N\) videos by score and find the position (the rank) \(r_i\) of the correct video; rank 1 is the top. Then:

\[ \text{R@}K = \frac{100}{Q} \sum_{i=1}^{Q} \mathbb{1}\big[r_i \le K\big], \qquad \text{MdR} = \operatorname{median}_i\, r_i, \qquad \text{MnR} = \frac{1}{Q} \sum_{i=1}^{Q} r_i \tag{208.2} \]
Read the symbols
\(Q\)
the number of test queries (1,000 on MSR-VTT 1k-A)
\(r_i\)
the rank of the correct video for query \(i\)
R@K
recall at K: the percentage of queries whose correct video is in the top \(K\) (K = 1, 5, 10)
MdR, MnR
median and mean rank; lower is better. MnR is sensitive to a few very bad queries

One thing is easy to forget: R@K depends on the size of the gallery. With 100 videos there are 99 wrong answers that can beat the right one; with 1,000 there are 999. The same model gets a lower R@1 on a bigger gallery. A number from MSR-VTT 1k-A tells you how a model ranks 1,000 clips, not a million. Figure 1 shows this with a toy model.

cosine similarity (darker = higher)correct videotop-1 is wrong
Figure 1. A toy retrieval benchmark. Each video is a random unit vector in 32 dimensions (seeded). The caption of video \(i\) is \(\operatorname{normalize}(a\,\mathbf{v}_i + (1-a)\,\mathbf{n}_i)\), with \(\mathbf{n}_i\) a random unit vector: \(a\) controls how well text and video agree. Left: the top-left 24×24 corner of the similarity matrix (rows = queries, columns = videos). Right: R@1, R@5, R@10 over all queries, formula (208.2).
How to read Figure 1

Each row is one query. The correct video sits on the diagonal (orange frame). If another cell in the row is higher, the row gets a red mark: its R@1 is a miss. At the default (\(a = 0.35\), 1,000 videos) R@1 is 33.1%, R@10 72.0% and the median rank is 3. Now click “100 videos” without moving the slider: R@1 jumps to 59.0%. Nothing about the model changed; only the number of distractors did. Move \(a\) to 0.45 and both galleries become easy (R@1 83.3% on 1,000 videos, 92.0% on 100). This is why you must compare models on the same test split and size.

Worked example: the metrics by hand

Ten queries give the ranks 1, 3, 1, 12, 2, 1, 6, 1, 1, 25. Five ranks equal 1, so \(\text{R@}1 = 50\%\). Seven are \(\le 5\) (the five 1s, the 3 and the 2), so \(\text{R@}5 = 70\%\). Eight are \(\le 10\) (add the 6): \(\text{R@}10 = 80\%\). Sorted: 1, 1, 1, 1, 1, 2, 3, 6, 12, 25; the median of an even count is the mean of the two middle values, \(\text{MdR} = (1 + 2)/2 = 1.5\). The mean rank is \(\text{MnR} = 53/10 = 5.3\), pulled up by the two bad queries.

From frames to a video vector: CLIP4Clip

CLIP (session 166) already maps images and text into one space. The simplest video encoder is therefore: sample frames, embed each with CLIP’s image encoder, and combine the frame vectors. CLIP4Clip (Luo et al., 2021) studied exactly this. [Certain] It starts from CLIP ViT-B/32, samples frames uniformly at 1 frame per second, uses 12 frames, and fine-tunes on each benchmark. [Certain] For video \(i\) the image encoder gives frame embeddings \(\mathbf{z}_1, \dots, \mathbf{z}_T\) (the [CLS] output of each frame). CLIP4Clip compares three ways to get a similarity from them, which it calls similarity calculators.

1. Parameter-free: mean pooling

\[ \bar{\mathbf{z}} = \frac{1}{T} \sum_{j=1}^{T} \mathbf{z}_j, \qquad s(\mathbf{t}, V) = \frac{\mathbf{t}^\top \bar{\mathbf{z}}}{\lVert \mathbf{t} \rVert\, \lVert \bar{\mathbf{z}} \rVert} \tag{208.3} \]

No new weights: the “average frame” lives in CLIP’s space from the first step. It ignores order, and every frame gets the same weight.

2. Sequential: a temporal model before pooling

\[ \tilde{Z} = \operatorname{TransformerEnc}\big(Z + P\big), \qquad s(\mathbf{t}, V) = \cos\!\big(\mathbf{t},\ \operatorname{mean}_j \tilde{\mathbf{z}}_j\big) \tag{208.4} \]
Read the symbols
\(Z = [\mathbf{z}_1, \dots, \mathbf{z}_T]\)
the frame embeddings, a \(T \times d\) matrix
\(P\)
learned position embeddings, one per frame position, so the model knows the order
\(\tilde{Z}\)
frame embeddings after they have exchanged information with each other

CLIP4Clip tries an LSTM (“seqLSTM”) and a small transformer encoder (“seqTransf”) here. The video vector still does not depend on the query, so it can still be indexed.

3. Tight: a joint transformer over text and frames

The “tightTransf” type concatenates the text vector and the frame vectors, \([\mathbf{w}, \mathbf{z}_1, \dots, \mathbf{z}_T]\), adds position and type embeddings, runs a transformer encoder over all of them, and maps the first output token through FC–ReLU–FC to one score. [Certain] This is a cross-encoder: text and video meet inside the network. It is the most expressive and the most expensive, because the network must run once for every (query, video) pair. Nothing can be precomputed per video except the frame embeddings.

Training: the symmetric contrastive loss

Training uses the same loss as CLIP. A batch has \(B\) (video, caption) pairs; we compute all \(B \times B\) similarities and ask each caption to pick its video and each video to pick its caption:

\[ \mathcal{L} = -\frac{1}{B} \sum_{i=1}^{B} \left[ \log \frac{e^{s_{ii}/\tau}}{\sum_{j} e^{s_{ij}/\tau}} + \log \frac{e^{s_{ii}/\tau}}{\sum_{j} e^{s_{ji}/\tau}} \right], \qquad s_{ij} = s(\mathbf{t}_i, V_j) \tag{208.5} \]
Read the symbols
\(s_{ij}\)
similarity of caption \(i\) and video \(j\); the true pairs are on the diagonal
\(\tau\)
the temperature; CLIP learns the logit scale \(1/\tau\) and caps it at 100
first term
text-to-video: caption \(i\) must pick video \(i\) out of the batch
second term
video-to-text: video \(i\) must pick caption \(i\)

CLIP4Clip sums the two directions. [Certain] Every other video in the batch is a negative. Larger batches give more negatives; CLIP4Clip found 128 and 256 about equal and used 128. [Certain]

Algorithm 1: one CLIP4Clip training step (mean pooling)
  1. input: a batch of \(B\) (video, caption) pairs; CLIP image encoder \(\phi\), text encoder \(g\); logit scale \(1/\tau\)
  2. for each video do decode at 1 fps and keep \(T = 12\) uniformly spaced frames, resize to 224×224
  3. \(Z \leftarrow \phi(\text{frames})\) // (B, T, d): every frame goes through the image ViT
  4. \(\mathbf{v}_j \leftarrow \operatorname{normalize}(\operatorname{mean}_t Z_{j,t})\) // formula (208.3)
  5. \(\mathbf{t}_i \leftarrow \operatorname{normalize}(g(\text{caption}_i))\) // the [EOS] token output
  6. \(S \leftarrow [\mathbf{t}_i^\top \mathbf{v}_j]_{B \times B}\); loss from formula (208.5)
  7. backpropagate into both encoders with a very small learning rate (the paper’s best was \(10^{-7}\)) // do not destroy CLIP

What the experiments showed

Text-to-video results on MSR-VTT 1k-A after training on Training-9K (all numbers from the CLIP4Clip paper, ViT-B/32, 12 frames): [Certain]

MethodNew weightsR@1R@5R@10MdR
CLIP zero-shot, mean of frames (“CLIP-straight”)none, no training31.253.764.24
CLIP4Clip meanPnone43.170.480.82
CLIP4Clip seqLSTM1-layer LSTM42.570.880.72
CLIP4Clip seqTransf4-layer transformer [Certain]44.571.481.62
CLIP4Clip tightTransfcross transformer + FC40.271.580.52

Three lessons. First, fine-tuning CLIP on video captions is worth about 12 points of R@1 over zero-shot CLIP, with no new weights at all. Second, the sequential type helps a little on the 9K split, but on the smaller Training-7K split mean pooling was best (42.1 vs 42.0 for seqTransf). [Certain] New weights need data. Third, the tight cross-encoder was the worst on almost every dataset, even though it is the most expressive; the authors explain this by too little data to learn cross-modal interaction from scratch. [Certain] They also tried a 3D patch embedding (patches across time) and got lower results than CLIP’s 2D one (41.6 vs 43.1 R@1 on MSR-VTT), because the 3D weights could not start from CLIP. [Certain] Going from 1 to 6 frames gave a large gain; they chose 12. [Certain]

Attention pooling: which frames does the query talk about?

Mean pooling gives every frame the same weight. But a caption often describes only some frames. There are two ways to learn weights:

  • Text-agnostic attention: a learned query vector (or self-attention) scores the frames. Weights depend only on the video, so the video vector can still be indexed.
  • Text-conditioned attention: the query text decides the weights. X-Pool (Gorti et al., 2022) does this with scaled dot-product attention where the text is the query and the frames are the keys and values: [Certain]
\[ \alpha = \softmax\!\left( \frac{(\mathbf{t} W_Q)(Z W_K)^\top}{\sqrt{D_p}} \right), \qquad \mathbf{v}_{|t} = \operatorname{LN}\!\big( \alpha\, (Z W_V)\, W_O \big) \tag{208.6} \]
Read the symbols
\(W_Q, W_K, W_V\)
projection matrices for the text (query) and the frames (keys and values); X-Pool applies LayerNorm before them and uses \(D_p = 512\) [Certain]
\(\alpha\)
one weight per frame, summing to 1: how relevant each frame is to this text
\(\mathbf{v}_{|t}\)
a video vector conditioned on the text; X-Pool adds a small FC block with a residual on top

X-Pool, with the same CLIP ViT-B/32 and 12 frames, reported 46.9 R@1 on MSR-VTT 9K, against 44.5 for CLIP4Clip seqTransf. [Certain] The price is in the index: \(\mathbf{v}_{|t}\) is different for every query, so you cannot store one vector per video. You store all \(T\) frame vectors and run the attention for every (query, video) pair, or you use X-Pool only to re-rank a short list. A cheap middle ground is to store frame vectors and score a video by its best frame, \(\max_j \mathbf{t}^\top \mathbf{z}_j\): this is text-conditioned, needs no learned weights, and works with a standard vector index (search frames, then group hits by video).

frame matches the query (darker = higher cosine)frame weight used for poolingcorrect video
Figure 2. Pooling eight frame embeddings into one score, in a toy 6-dimensional space whose axes mean “kitchen”, “food”, “dog”, “ball”, “beach” and “person talking”. Mean pooling is formula (208.3); “best frame” is \(\max_j \mathbf{t}^\top\mathbf{z}_j\); attention uses weights \(\softmax_j(\mathbf{t}^\top \mathbf{z}_j / \tau_a)\), a parameter-free version of formula (208.6).
How to read Figure 2

Start with the dog query and mean pooling. The correct video (A, a cooking vlog) shows the dog in only one of eight frames, so its average is mostly “person talking in a kitchen”: score 0.580. The kitchen tour (E) scores 0.693 and wins. Switch to “best frame” (0.988) or attention (0.986 at \(\tau_a = 0.1\)): video A now wins. Now pick the chef query. Mean pooling ranks the cooking show (C) first, 0.962 against 0.952. With best frame, the vlog A wins (0.970) because one of its frames is a perfect match. So text-conditioned pooling helps when the query names a short part of the video and can hurt when the query describes the whole video. Raise \(\tau_a\) toward 1: the attention weights flatten and the scores move back toward mean pooling.

Video–text encoders: ViCLIP and InternVideo2

CLIP4Clip and X-Pool start from an image model and add time at the end. The next step is to train the encoder itself on video–text pairs, so that motion and events are inside the embedding. Two widely used examples come from the InternVideo line of work.

InternVid and ViCLIP

InternVid (Wang et al., 2023) is a video–text dataset of over 7 million videos, nearly 760,000 hours, cut into 234 million clips with generated captions totaling 4.1 billion words. [Certain] ViCLIP is the model trained on it: a ViT-L video encoder and a text encoder, both initialized from CLIP, with the image attention changed to spatiotemporal attention (tokens from all frames attend to each other). [Certain] To save compute it randomly masks video patches during pretraining, as in MAE, and trains briefly without masking at the end so that training matches inference. [Certain] The loss is the contrastive loss (208.5) between video and text embeddings.

In zero-shot retrieval (no training on the benchmark, 8 frames), the paper reports on MSR-VTT text-to-video R@1: [Certain]

ModelPretraining dataMSR-VTT T2V R@1, zero-shot
CLIP ViT-L, mean of 8 framesOpenAI 400M image–text pairs29.0
CLIP ViT-L, mean of 8 framesDataComp-1B30.4
ViCLIPWebVid-10M35.6
ViCLIPInternVid-10M-FLT (a filtered 10M subset)42.4
ViCLIPInternVid-200M39.3

Two things stand out. Training on video–text data adds about 12 points over mean-pooled CLIP of the same size. And the filtered 10M subset beats the full 200M set on zero-shot retrieval: data quality beat data size here. [Certain] The authors suspect one cause: clips cut from the same video look alike and have similar captions, but the contrastive loss treats them as negatives. These false negatives hurt more when many clips come from the same video and when the batch is large. [Certain] Remember this when you build your own training set from long videos.

InternVideo2

InternVideo2 (Wang et al., 2024) scales the video encoder to 6 billion parameters and trains it in three stages: [Certain]

  1. Unmasked token reconstruction. Mask 80% of the video tokens and train the encoder to match, on the visible tokens, the features of two teachers: InternVL-6B (an image–text model, for meaning) and VideoMAEv2-g (for motion).
  2. Cross-modal contrastive learning. Align video with text, audio and speech, using contrastive, matching and masked-language losses. The text encoder is the first 19 layers of BERT-Large.
  3. Next-token prediction. Connect the encoder to an LLM and train on instruction data, for chat and question answering.

The stage-2 model is the one used for retrieval. Its data has 402M entries: 2M videos, 50M video–text pairs, 50M video–audio–speech–text pairs and 300M image–text pairs. [Certain] Zero-shot MSR-VTT text-to-video R@1 is 51.9 for the 1B model and 55.9 for the 6B model; after fine-tuning, the 6B model reports 62.8. [Certain] The paper also found that cutting training videos at learned shot boundaries (with the AutoShot model) instead of FFmpeg’s simple scene filter clearly improved zero-shot retrieval: how you cut a video into clips matters. [Certain] We come back to that in the next section.

For a product, the choice is a trade-off between quality and cost. A CLIP-style encoder with mean pooling is cheap and very strong for “what is in the video” queries. A video–text encoder such as ViCLIP or InternVideo2 is better at actions and events, but each segment costs more to encode, and a 6B encoder is a real bill when you index a million hours. [Likely] The architectures behind these models are covered in session 202 and session 212.

Clip-level and segment-level indexes

MSR-VTT clips are short and each caption describes the whole clip. Real libraries are different: a news broadcast is 30 minutes, a football match 2 hours, a lecture 1 hour. What should one vector in the index stand for?

  • Clip-level (video-level) index: one vector per video, for example the mean of frame embeddings over the whole video. Small and simple. Good for short videos (social clips, ads, stock footage) and for queries about the whole video (“a cooking tutorial for pasta”).
  • Segment-level index: cut each video into segments, embed each segment, and store one vector per segment with its video ID and start and end times. A query returns moments, not just videos. This is what long-video search needs.

With fixed windows of length \(L\) seconds and stride \(S\) seconds (\(S \lt L\) means overlap), a video of \(D\) seconds gives

\[ n_{\text{seg}} = \left\lfloor \frac{D - L}{S} \right\rfloor + 1, \qquad s(\mathbf{t}, V) = \max_{k} \ \mathbf{t}^\top \mathbf{v}_k, \qquad \mathbf{v}_k = \operatorname{normalize}\!\Big( \frac{1}{L} \sum_{j \in \text{seg}_k} \mathbf{z}_j \Big) \tag{208.7} \]
Read the symbols
\(D, L, S\)
video length, segment length and stride, in seconds (one frame embedding per second here)
\(\mathbf{v}_k\)
the embedding of segment \(k\): here, mean-pooled frames; with a video encoder, the encoder’s output for that segment
\(\max_k\)
a video is as relevant as its best segment; the best segment’s times are returned too

For a 1-hour video: 10-second segments without overlap give 360 vectors; with 50% overlap (\(S = 5\)) 719; 2-second segments give 1,800.

Why short events vanish in long segments

Here is the core problem in one formula. Take the simplest case: the event frames have embedding \(\mathbf{e}\) (exactly the query, \(\mathbf{t} = \mathbf{e}\)), the other frames show a background \(\mathbf{b}\) orthogonal to it, and the event fills a fraction \(f\) of the segment. The mean-pooled segment vector is \(f\mathbf{e} + (1 - f)\mathbf{b}\), so

\[ \cos(\mathbf{t}, \mathbf{v}) = \frac{f}{\sqrt{f^2 + (1 - f)^2}} \tag{208.8} \]

The similarity falls fast as the event gets relatively shorter.

Worked example: dilution

A 4-second goal inside a 4-second segment: \(f = 1\), cosine 1. Inside an 8-second segment: \(f = 0.5\), cosine \(0.5/\sqrt{0.5} = 0.707\). Inside a 16-second segment: \(f = 0.25\), cosine \(0.25/\sqrt{0.0625 + 0.5625} = 0.316\). Inside a 40-second segment: \(f = 0.1\), cosine \(0.1/\sqrt{0.01 + 0.81} = 0.110\). A different video that is only “somewhat related” everywhere (cosine 0.5 in all its segments) beats the true match once \(f\) drops below about 0.37 (solve \(f/\sqrt{f^2+(1-f)^2} = 0.5\)).

So why not make segments tiny? Because short segments have their own costs:

  • More vectors: the index grows as \(1/S\) (formula 208.7). Overlap doubles it again.
  • Less context: “a man opens the door, then the dog runs out” spans several seconds. A 1-second segment sees only part of it, and video–text encoders are trained on clips of a few seconds or more.
  • Fragmented results: a 40-second interview becomes 20 near-identical hits that fill the result page. You must merge neighbors (see Algorithm 2).
  • Boundary effects: without overlap, an event can be cut in half by the grid. Overlap of 25–50% is a common fix.

Common choices in practice are segments of a few seconds to about 30 seconds, matched to the encoder’s training clip length and to how long the searched events are. [Likely] A good alternative to fixed windows is to cut at shot boundaries (camera cuts) so segments do not mix two scenes, and to split long shots further; InternVideo2’s result on AutoShot boundaries points the same way. Many systems also keep two levels: a coarse video-level vector to find candidate videos, and segment vectors to localize inside them.

event named by the querysegment similaritybest segmentbest score of another video (0.50)
Figure 3. A 120-second sports clip with one embedding per second: a studio scene (0–60 s) with a 4-second goal replay at 30–34 s, and a street scene (60–120 s) with a 40-second interview at 60–100 s. Each horizontal line is one segment, drawn at the height of its mean-pooled cosine with the query, formula (208.7). The dashed red line is the best score of a distractor video.
How to read Figure 3

With the goal query and \(L = 10\) s (no overlap), the grid happens to put a segment at 30–40 s: cosine 0.672, above the distractor. Set \(L = 8\): the grid cuts the goal in half (32–40 s and 24–32 s each hold 2 seconds of it) and the best cosine drops to 0.504, barely above 0.50. Turn on 50% overlap: a segment at 28–36 s now holds the whole goal and the cosine is 0.774. At \(L = 30\) s the goal is diluted to 0.376 and the video is lost; at \(L = 120\) (one vector for the whole video) it is 0.232. Now switch to the interview query: it survives up to \(L = 60\) (0.903), but with \(L = 2\) the best hit is a 2-second piece of a 40-second answer.

Algorithm 2: build and query a segment-level index
  1. input: videos; encoder \(f\); segment length \(L\), stride \(S\); text encoder \(g\); top-\(K\)
  2. offline, for each video do
  3.   detect shot boundaries; cut into segments of length \(L\) with stride \(S\) inside shots // formula (208.7)
  4.   \(\mathbf{v}_k \leftarrow \operatorname{normalize}(f(\text{segment}_k))\); store \((\mathbf{v}_k, \text{video id}, t_{\text{start}}, t_{\text{end}})\)
  5. build an ANN index over all \(\mathbf{v}_k\) (IVF-PQ or HNSW, next section)
  6. online, for a query do
  7.   \(\mathbf{t} \leftarrow \operatorname{normalize}(g(\text{query}))\); fetch the top \(K' \gt K\) segments from the index
  8.   merge hits from the same video whose times overlap or touch; keep the max score // avoid 20 hits for one event
  9.   optionally re-rank the top candidates with a cross-encoder or VLM // formula (208.12)
  10.   return the top \(K\) (video id, start, end, score)

Scaling to millions of segments

Now count. A library of 100,000 hours, cut into 10-second segments without overlap, gives \(100{,}000 \times 3600 / 10 = 36\) million vectors. The memory of the raw index is

\[ \text{bytes} = n \cdot d \cdot b \tag{208.9} \]

where \(n\) is the number of vectors, \(d\) the dimension and \(b\) the bytes per number. With \(d = 768\) (an assumed size for this example):

Worked example: memory and compute for 36M segments

float32: \(36 \times 10^6 \cdot 768 \cdot 4 = 110.6\) GB. float16: 55.3 GB. int8: 27.6 GB. With product quantization (PQ) to 64 bytes per vector: 2.3 GB, small enough for one machine’s RAM. A brute-force search compares the query with every vector: \(2 \cdot 36 \times 10^6 \cdot 768 = 55.3\) GFLOP per query. That is fine for a few queries on a GPU, but not for thousands of queries per second on CPU. Adding 50% overlap doubles everything (72M vectors).

The standard tools are the same as for text search (see session 151):

  • IVF (inverted file): run k-means on the vectors to get \(n_{\text{list}}\) centroids (cells). Each vector is stored in the list of its nearest centroid. At query time, compare the query with all centroids, and scan only the \(n_{\text{probe}}\) closest lists.
  • PQ (product quantization): split each vector into \(m\) sub-vectors and replace each by the index of its nearest code word (1 byte for 256 code words). A vector becomes \(m\) bytes; distances are computed from small lookup tables. [Certain]
  • HNSW: a graph where each vector links to near neighbors on several levels; search walks the graph greedily. Very fast and accurate, but uses more memory than IVF-PQ. [Likely]

Libraries such as FAISS implement all of these on CPU and GPU. [Certain] The cost of one IVF query is roughly

\[ \text{cost} \approx n_{\text{list}}\, d \ +\ \frac{n_{\text{probe}}}{n_{\text{list}}}\, n\, d \tag{208.10} \]
Read the symbols
\(n_{\text{list}}\, d\)
comparing the query with every centroid
\(\frac{n_{\text{probe}}}{n_{\text{list}}}\, n\)
the number of vectors in the scanned lists, if lists have equal size

With \(n = 36\)M, \(n_{\text{list}} = 65{,}536\) and \(n_{\text{probe}} = 32\), a query compares with 65,536 centroids and about 17,600 vectors: about 433 times fewer than brute force. The price is recall: a true neighbor that sits in a list we did not probe is lost. Figure 4 shows this trade-off.

points in probed cellsquerytrue 10 nearest (filled = found by IVF, hollow = missed)
Figure 4. IVF search on 4,000 seeded points in 2D (real embeddings have hundreds of dimensions, but the logic is the same). k-means with 10 iterations makes \(n_{\text{list}}\) cells. Left: one query, its true 10 nearest neighbors, and those found by scanning only the probed cells. Right: recall@10 averaged over 100 queries (solid) and the fraction of the index scanned (dashed), for every \(n_{\text{probe}}\) from 1 to 16.
How to read Figure 4

With 16 cells and one probe, IVF scans 7.7% of the points and finds 90.8% of the true neighbors on average. Two probes: 99.1% recall for 14.1% of the work. With 64 cells and one probe the scan drops to 1.9% but recall falls to 83.5%; four probes bring it to 100% for 7.2%. Queries near a cell border lose neighbors that sit just across the border; more probes fix that. In production you tune \(n_{\text{probe}}\) on a held-out query set until recall@K of the ANN search (against brute force) is, say, 95%.

Hybrid search: not everything is in the pixels

Many queries are best answered by other signals: what people say (speech transcribed by ASR), what is written on screen (OCR: scoreboards, slides, signs), and metadata (title, date, channel). A strong system runs a text search (BM25 or a text embedding, see session 150) over transcripts and OCR next to the visual search, and fuses the ranked lists. A simple, robust fusion is reciprocal rank fusion (RRF): [Certain]

\[ \text{RRF}(x) = \sum_{m \in \text{lists}} \frac{1}{k + \operatorname{rank}_m(x)}, \qquad k = 60 \tag{208.11} \]

It uses only ranks, so the scores of different systems do not need to be on the same scale. \(k = 60\) is the value from the original paper. [Certain]

Re-ranking with a VLM

The dual encoder compresses a video into one vector before it sees the query. That is why it is fast, and also why it makes errors that a model reading both together would not make: negation (“a man without a hat”), counting (“three dogs”), order (“opens the door, then sits down”) and fine details (“in the rain”). The fix is the same as in text search: a two-stage pipeline.

  1. Retrieve: the dual encoder and the ANN index return the top \(K\) candidates (say 50–200) out of millions, in milliseconds.
  2. Re-rank: a stronger model scores each (query, candidate) pair. This can be a cross-encoder (like CLIP4Clip’s tight type or a matching head), or a VLM (session 205) that sees the frames of the candidate and the query.

A simple VLM re-ranker asks a yes/no question and reads the probabilities of the two answer tokens:

\[ p_{\text{yes}} = \frac{e^{\ell_{\text{yes}}}}{e^{\ell_{\text{yes}}} + e^{\ell_{\text{no}}}}, \qquad s_{\text{final}} = \alpha\, \hat{s}_{\text{dual}} + (1 - \alpha)\, p_{\text{yes}} \tag{208.12} \]
Read the symbols
\(\ell_{\text{yes}}, \ell_{\text{no}}\)
the VLM’s logits for the tokens “Yes” and “No” after the prompt “Does this clip show: query? Answer yes or no.”
\(\hat{s}_{\text{dual}}\)
the dual-encoder score, min–max normalized over the \(K\) candidates so it lies in [0, 1]
\(\alpha\)
the weight of the first stage; tune it on labeled queries

Keeping some of the first-stage score (\(\alpha \gt 0\)) protects against VLM mistakes, and RRF (208.11) is an alternative when the two scores are hard to compare. Cost: \(K\) VLM calls per query. If one batch of 25 candidates (8 frames each) takes 120 ms on your GPU, re-ranking 50 candidates adds 240 ms; that is a latency budget you must plan, and the frames of candidates should be cached as pre-computed visual tokens if possible. Use the re-ranker where the first stage is weak (long, detailed queries) and skip it for short keyword queries.

dual-encoder scoreVLM \(p_{\text{yes}}\)fused scorethe segment the user wanted
Figure 5. A two-stage search for “a goalkeeper saves a penalty in the rain”. The first stage returns 8 segments with close cosine scores. A VLM gives each a probability of “yes”. The fused score is formula (208.12). The scores are invented for the illustration; the procedure is the real one.
How to read Figure 5

Step through. In step 2 the wanted segment (penalty save in heavy rain) is only 4th: the dual encoder sees “penalty”, “goalkeeper” and “rain” in many candidates and cannot tell which has all three. In step 3 the VLM gives it 0.93 while the others are at most 0.55. In step 4, with \(\alpha = 0.5\), it moves to rank 1. Move \(\alpha\): at 1.0 you get back the first-stage order; at 0.7 it is 2nd; from 0.6 down it is 1st. At \(\alpha = 0\) you trust only the VLM.

Code: pooling, metrics, segments, IVF and fusion

The code runs on CPU in under a second. It uses synthetic “frozen CLIP” features so that the result is exact and reproducible: each video has 8 frames showing one scene, except one frame that shows the event named in its caption.

import math, torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)

# 1) three ways to turn frame embeddings Z (B, T, d) into one video vector
class MeanPool(nn.Module):                      # parameter-free, formula (208.3)
    def forward(self, Z, t=None):
        return F.normalize(Z.mean(1), dim=-1)

class SeqTransf(nn.Module):                     # sequential type, formula (208.4)
    def __init__(self, d, T, layers=2):
        super().__init__()
        self.pos = nn.Parameter(torch.zeros(1, T, d))
        layer = nn.TransformerEncoderLayer(d, nhead=4, dim_feedforward=2 * d, batch_first=True)
        self.enc = nn.TransformerEncoder(layer, layers)
    def forward(self, Z, t=None):
        return F.normalize((Z + self.enc(Z + self.pos)).mean(1), dim=-1)   # residual keeps CLIP's space

class TextCondPool(nn.Module):                  # X-Pool-style attention, formula (208.5)
    def __init__(self, d):
        super().__init__()
        self.ln = nn.LayerNorm(d)
        self.q, self.k, self.v = (nn.Linear(d, d, bias=False) for _ in range(3))
        for m in (self.q, self.k, self.v): nn.init.eye_(m.weight)          # start as identity
    def forward(self, Z, t):                    # Z: (B, T, d), t: (Q, d) -> (Q, B, d)
        Zn, tn = self.ln(Z), self.ln(t)         # LayerNorm: vectors of length ~ sqrt(d)
        w = torch.einsum("qd,btd->qbt", self.q(tn), self.k(Zn)) / math.sqrt(Z.shape[-1])
        return F.normalize(torch.einsum("qbt,btd->qbd", w.softmax(-1), self.v(Zn)), dim=-1)

# 2) symmetric contrastive loss over a batch, formula (208.6)
def clip_loss(S, logit_scale=100.0):         # S: (B, B) cosines, positives on the diagonal
    S = logit_scale * S
    y = torch.arange(len(S))
    return 0.5 * (F.cross_entropy(S, y) + F.cross_entropy(S.T, y))

# 3) retrieval metrics, formula (208.2): rank of the true video for each text
def recall_at_k(S, ks=(1, 5, 10)):
    rank = 1 + (S > S.diag()[:, None]).sum(1)
    out = {f"R@{k}": round(100 * (rank <= k).float().mean().item(), 1) for k in ks}
    out["MdR"] = rank.float().median().item()
    return out

# synthetic "frozen CLIP" features: each video has 8 frames; its caption names ONE short event
B, T, d = 256, 8, 64
scene = F.normalize(torch.randn(B, 1, d), dim=-1).expand(B, T, d)          # what most frames show
event = F.normalize(torch.randn(B, d), dim=-1)                              # the captioned event
pos = torch.randint(0, T, (B,))                                             # where the event happens
Z = scene.clone(); Z[torch.arange(B), pos] = event
Z = F.normalize(Z + 0.3 * torch.randn(B, T, d) / math.sqrt(d), dim=-1)
text = F.normalize(event + 0.3 * torch.randn(B, d) / math.sqrt(d), dim=-1)

S_mean = text @ MeanPool()(Z).T
print("mean pooling     ", recall_at_k(S_mean))
S_max = torch.einsum("qd,btd->qbt", text, Z).max(-1).values                # best single frame
print("max over frames  ", recall_at_k(S_max))
xp = TextCondPool(d)
with torch.no_grad():
    Vq = xp(Z, text)                                                        # one video vector per (text, video)
    S_x = torch.einsum("qd,qbd->qb", text, Vq)
print("text-cond. attn  ", recall_at_k(S_x), tuple(Vq.shape))
seq = SeqTransf(d, T).eval()                                               # untrained: shapes only
print("seqTransf output", tuple(seq(Z).shape), "new params", sum(p.numel() for p in seq.parameters()))
print("loss: mean", round(clip_loss(S_mean).item(), 3), " text-cond.", round(clip_loss(S_x).item(), 3),
      " uniform guess log B =", round(math.log(B), 3))

# 4) a segment index for long videos, formula (208.7): windows of L s, stride S, score = max
def segment(frame_emb, L, S):                   # frame_emb: (D, d), one row per second
    D = frame_emb.shape[0]
    starts = list(range(0, D - L + 1, S))
    if starts[-1] + L < D: starts.append(D - L) # cover the tail
    segs = torch.stack([F.normalize(frame_emb[s:s + L].mean(0), dim=0) for s in starts])
    return segs, torch.tensor(starts)

vid = torch.zeros(120, 4); vid[:60, 0] = 1; vid[60:, 1] = 1                 # two background scenes
vid[30:34] = torch.tensor([0.3, 0, 1, 0]); vid[60:100] = torch.tensor([0, 0.3, 0, 1])
vid = F.normalize(vid, dim=-1)
q_goal = F.normalize(torch.tensor([0.25, 0, 1, 0]), dim=0)
for L, S in [(4, 2), (10, 10), (30, 30), (120, 120)]:
    segs, starts = segment(vid, L, S)
    sims = segs @ q_goal
    i = sims.argmax().item()
    print(f"L={L:3d} S={S:3d}: {len(starts):3d} vectors, best [{starts[i].item()}, {starts[i].item() + L}] s, cos {sims[i].item():.3f}")

# 5) IVF: cluster the index, search only the nprobe closest cells, formula (208.10)
def kmeans(X, k, iters=10):
    C = X[torch.linspace(0, len(X) - 1, k).long()].clone()
    for _ in range(iters):
        a = torch.cdist(X, C).argmin(1)
        for j in range(k):
            if (a == j).any(): C[j] = X[a == j].mean(0)
    return C, torch.cdist(X, C).argmin(1)

N = 20000
X = F.normalize(torch.randn(16, 32)[torch.randint(0, 16, (N,))] + 0.5 * torch.randn(N, 32), dim=-1)
Qv = F.normalize(X[:200] + 0.1 * torch.randn(200, 32), dim=-1)
C, assign = kmeans(X, 64)
exact = (Qv @ X.T).topk(10).indices
for nprobe in (1, 4, 16):
    cells = (Qv @ C.T).topk(nprobe).indices
    hits, scanned = 0, 0
    for qi in range(len(Qv)):
        cand = torch.nonzero(torch.isin(assign, cells[qi])).squeeze(1)
        scanned += len(cand)
        top = cand[(Qv[qi] @ X[cand].T).topk(10).indices]
        hits += len(set(top.tolist()) & set(exact[qi].tolist()))
    print(f"nprobe={nprobe:2d}: recall@10 {hits / (10 * len(Qv)):.3f}, scanned {scanned / len(Qv) / N:.3f} of the index")

# 6) re-ranking: fuse the dual-encoder score with a VLM's p(yes), formula (208.11)
dual = torch.tensor([.412, .405, .398, .391, .380, .372, .361, .355])
p_yes = torch.tensor([.18, .35, .41, .93, .02, .55, .03, .47])
for alpha in (1.0, 0.5):
    fused = alpha * (dual - dual.min()) / (dual.max() - dual.min()) + (1 - alpha) * p_yes
    print(f"alpha={alpha}: order {(fused.argsort(descending=True, stable=True) + 1).tolist()}")

Output (PyTorch 2.8, CPU):

mean pooling      {'R@1': 3.9, 'R@5': 17.2, 'R@10': 26.2, 'MdR': 36.0}
max over frames   {'R@1': 100.0, 'R@5': 100.0, 'R@10': 100.0, 'MdR': 1.0}
text-cond. attn   {'R@1': 100.0, 'R@5': 100.0, 'R@10': 100.0, 'MdR': 1.0} (256, 256, 64)
seqTransf output (256, 64) new params 67456
loss: mean 21.856  text-cond. 0.0  uniform guess log B = 5.545
L=  4 S=  2:  59 vectors, best [30, 34] s, cos 0.999
L= 10 S= 10:  12 vectors, best [30, 40] s, cos 0.672
L= 30 S= 30:   4 vectors, best [30, 60] s, cos 0.376
L=120 S=120:   1 vectors, best [0, 120] s, cos 0.232
nprobe= 1: recall@10 0.559, scanned 0.023 of the index
nprobe= 4: recall@10 0.954, scanned 0.067 of the index
nprobe=16: recall@10 1.000, scanned 0.253 of the index
alpha=1.0: order [1, 2, 3, 4, 5, 6, 7, 8]
alpha=0.5: order [4, 2, 1, 3, 6, 8, 5, 7]

Read it line by line. Mean pooling fails on this data (R@1 3.9%) because each caption names one frame out of eight: exactly the dilution of formula (208.8). The best frame and text-conditioned attention find every video, but note the shape (256, 256, 64): one video vector per query, which is why it cannot be indexed. The contrastive loss of the mean-pooled scores (21.9) is far above \(\log B = 5.5\), the loss of a uniform guess, because the model is confidently wrong. The segment part reproduces Figure 3 (0.999 at 4 s, 0.672 at 10 s, 0.232 for one vector per video). The IVF part shows recall rising with \(n_{\text{probe}}\) while the scanned fraction stays small. The last line is Figure 5 at \(\alpha = 0.5\).

Where this is used

ApplicationIndex unitTypical design
Stock footage and social clip searchOne vector per short clipCLIP or video–text encoder, mean pooling, ANN index, metadata filters
Broadcast and sports archives (“all goals by player X in the rain”)Segments of a few seconds, plus video-level vectorsShot-based segments, hybrid search with ASR and on-screen OCR, VLM re-ranking; links to key-moment detection in session 207
Video-RAG for question answeringSegments with timestampsRetrieve top segments for the question, feed their frames and transcripts to a VLM (session 210)
Content moderation and policy searchSegmentsText queries describing a policy (“a weapon pointed at a person”), high recall, human review
Dataset curation for trainingClipsFind clips that match a concept to build or balance training sets; remove near-duplicates (session 209)
Personal video and surveillance searchSegments, often per tracked objectOn-device or private index; time and camera filters; privacy rules
Zero-shot classification and taggingClipsRetrieval with class names as queries, as in session 203

Common mistakes

  • Comparing R@1 across different gallery sizes or splits. MSR-VTT 1k-A, the full test set and your 10M-segment index are different problems (Figure 1). Report the gallery size.
  • One vector for a long video. Mean pooling over an hour dilutes every short event (formula 208.8). Use segments, or at least a two-level index.
  • Segments without overlap and without merging. Events get cut by the grid, and long events flood the results with neighbors. Use 25–50% overlap or shot boundaries, and merge adjacent hits.
  • Indexing a text-conditioned model. X-Pool-style pooling depends on the query, so it cannot produce one stored vector per video. Use it to re-rank a short list, or store frame vectors and score by the best frame.
  • Mismatched frame sampling. Encoding the index at 1 frame per 4 seconds when the model was trained with 12 frames at 1 fps changes the embeddings. Use the same sampling, resolution and number of frames at indexing time as in training.
  • Tuning ANN parameters without measuring recall. Always measure the ANN top-K against brute force on held-out queries; a fast index with 70% recall silently drops results.
  • Ignoring speech and text on screen. Many queries are about what is said or written. Visual embeddings alone miss them; add ASR and OCR search and fuse.
  • Training on many clips of the same video in one batch. They act as false negatives in the contrastive loss, as ViCLIP’s authors suspected. Limit clips per video per batch.

Interview questions

What do R@1, R@10 and median rank measure, and why can the same model get a very different R@1 on two test sets?basic
For each query, rank all gallery videos and find the position of the correct one. R@K is the percentage of queries with that position ≤ K; MdR is the median position (lower is better). R@K depends on the number of distractors: with 1,000 videos there are 999 chances for a wrong video to score higher, with 100 only 99. In the toy of Figure 1, the same embeddings give 33.1% R@1 on 1,000 videos and 59.0% on 100. So compare only on the same split and gallery size, and expect lower numbers on a production index of millions.
How do you get a single video embedding from an image–text model like CLIP?basic
Sample \(T\) frames uniformly (CLIP4Clip: 1 fps, 12 frames), encode each with the image encoder, average the normalized frame embeddings, and normalize again (208.3). This already works zero-shot (31.2 R@1 on MSR-VTT for CLIP ViT-B/32) and, after fine-tuning both encoders with the symmetric contrastive loss, reached 43.1. Options beyond the mean: a temporal transformer before pooling (seqTransf), learned attention pooling, or a video–text encoder trained on video.
CLIP4Clip compared parameter-free, sequential and tight similarity calculators. What did it find, and why?core
On MSR-VTT 9K: meanP 43.1, seqLSTM 42.5, seqTransf 44.5, tightTransf 40.2 R@1. The sequential transformer helps slightly when there is enough data; on the 7K split mean pooling was best. The tight type, a cross-encoder over text and frames, was worst on almost all datasets. The reason given: new, randomly initialized weights need lots of data, and the tight type throws away CLIP’s ready-made alignment by learning a new interaction and a new scoring head. It is also expensive: it must run for every query–video pair, so it can only be a re-ranker.
Explain X-Pool. What does it gain and what does it cost in a real search system?core
X-Pool pools frames with cross-attention: the text embedding is the query, the frame embeddings are keys and values (208.6), so frames that match the caption get high weights. It improved MSR-VTT 9K R@1 from 44.5 (CLIP4Clip seqTransf) to 46.9 with the same backbone. The cost: the video vector depends on the query, so you cannot precompute one vector per video and use an ANN index. You must store all frame vectors and run attention for every candidate, which is fine for re-ranking the top few hundred and impossible for millions. The cheap approximation is best-frame scoring over a frame-level index.
How do you choose the segment length for a segment-level index?core
Balance four things. Dilution: an event filling a fraction \(f\) of a segment loses similarity roughly as \(f/\sqrt{f^2 + (1-f)^2}\) (208.8), so segments should not be much longer than the events users search for. Context: segments should be long enough for the action and match the encoder’s training clip length. Index size: vectors scale as \(D/S\). Result quality: long events split into many hits must be merged. In practice: a few seconds to about 30 seconds, 25–50% overlap or shot-based cuts, merging of neighboring hits, and often a second, video-level vector. Tune \(L\) on real queries with labeled moments.
Dual encoder or cross-encoder for text-to-video retrieval?core
Both, in two stages. The dual encoder encodes videos offline and compares vectors with one dot product, so it scales to millions of segments with ANN search, but it compresses each video before seeing the query and is weak on negation, counting, order and small details. A cross-encoder or VLM reads the query and the video together and is much more accurate on those, but costs one forward pass per pair. So: dual encoder for the top 50–200, cross-encoder or VLM to re-rank them, fused with the first-stage score (208.12).
How much memory does an index of 36 million 768-dimensional segment vectors need, and how do you make it smaller and faster?core
\(n d b\) (208.9): 110.6 GB in float32, 55.3 GB in float16, 27.6 GB in int8, 2.3 GB with 64-byte PQ codes. Brute-force search costs about 55 GFLOP per query. Use IVF to scan only \(n_{\text{probe}}/n_{\text{list}}\) of the vectors (with 65,536 lists and 32 probes, about 17,600 vectors plus 65,536 centroids per query), PQ to shrink each vector, or HNSW when memory allows. Re-score the final top candidates with full-precision vectors, and measure ANN recall against brute force.
Why might more contrastive training data from long videos make a video–text model worse?advanced
Clips cut from the same video look alike and get similar captions. In a contrastive batch, they are treated as negatives of each other even though they are nearly positives: false negatives. The loss then pushes apart things that should be close, and the effect grows with batch size and with the number of clips per video. ViCLIP’s results fit this: a filtered 10M subset of InternVid gave better zero-shot MSR-VTT retrieval (42.4) than the 200M set (39.3), and the authors suspect false negatives as one reason. Fixes: limit clips per video per batch, deduplicate near-identical clips and captions, use soft targets or mask same-video pairs in the loss.
Design text search over 100,000 hours of sports broadcasts.advanced
Ingest: decode at 1–2 fps, detect shots, cut into ~5–10-second segments with overlap inside shots (≈36–72M vectors). Encode each segment with a video–text encoder (or CLIP frames + mean pooling as a cheap baseline); also run ASR on commentary and OCR on the scoreboard, and store match metadata (teams, date, minute). Index: IVF-PQ or HNSW for visual vectors, BM25 for transcripts and OCR, filters for metadata. Query: encode the text, get the top few hundred segments from each source, fuse with RRF (208.11), merge neighboring hits, re-rank the top 50 with a VLM yes/no score (208.12), and return clips with timestamps. Evaluate with a labeled set of real queries (R@K, nDCG), ANN recall against brute force, and latency at p95; monitor queries with no clicks.
How would you add a VLM re-ranker, and how do you keep it from making things worse?advanced
Take the top \(K\) candidates, give the VLM a fixed number of frames per candidate plus a prompt like “Does this clip show: {query}? Answer yes or no.”, and read \(p_{\text{yes}}\) from the yes/no logits (208.12). Fuse with the normalized first-stage score using \(\alpha\) tuned on labeled queries, or with RRF. Risks: the VLM is biased toward “yes”, sensitive to prompt wording, and limited by the frames it is shown (it cannot confirm an event between sampled frames). Mitigate with calibration on labeled pairs, comparing logits rather than generated text, enough frames around the hit, and an A/B test. Watch latency: \(K\) calls per query; batch them, cache visual tokens, and only re-rank queries where the first stage is unsure.
What do ViCLIP and InternVideo2 add over CLIP with mean pooling?core
They train the video encoder on video–text pairs so time is modeled inside the encoder. ViCLIP turns CLIP ViT-L’s attention into spatiotemporal attention, trains contrastively on InternVid with random patch masking, and raised zero-shot MSR-VTT T2V R@1 from about 29–30 (mean-pooled CLIP ViT-L) to 42.4. InternVideo2 scales to a 6B encoder trained in three stages (distillation from InternVL and VideoMAEv2 on masked video, contrastive alignment with text, audio and speech, then next-token prediction with an LLM) and reports 55.9 zero-shot R@1. The price is encoding cost per segment, which matters when indexing millions of hours.

Exercises

  1. A 1-hour video is cut into segments of \(L = 8\) s with stride \(S = 4\) s. How many vectors? How many for 10,000 such videos?
  2. Using formula (208.8), what is the cosine when a 3-second event sits in a 12-second segment? Would it beat a distractor at 0.5?
  3. An index has 5 million segments with \(d = 512\). How much memory in float16? With 32-byte PQ codes?
  4. With \(n = 36\)M vectors, \(n_{\text{list}} = 16{,}384\) and \(n_{\text{probe}} = 16\), how many vectors does one IVF query scan if lists are equal in size?
  5. In Figure 3 (no overlap), the goal query gives 0.504 with \(L = 8\) s but 0.672 with \(L = 10\) s. Explain why the longer segment does better here.
  6. In Figure 1, find the smallest \(a\) at which R@1 on 1,000 videos is above 50%. Why is the answer different for 100 videos?
  7. Your search must return in 300 ms. The first stage takes 40 ms; a VLM batch of 25 candidates takes 120 ms. What is the largest \(K\) you can re-rank (in whole batches, run one after another)?
Answerscheck
1. \(\lfloor (3600 - 8)/4 \rfloor + 1 = 898 + 1 = 899\) vectors; 8.99 million for 10,000 videos. 2. \(f = 0.25\): \(0.25/\sqrt{0.0625 + 0.5625} = 0.316\); no, it loses to 0.5. 3. \(5 \times 10^6 \cdot 512 \cdot 2 = 5.12\) GB; with PQ \(5 \times 10^6 \cdot 32 = 0.16\) GB. 4. \(36 \times 10^6 \cdot 16 / 16{,}384 \approx 35{,}156\) vectors (plus 16,384 centroid comparisons). 5. Alignment with the grid: with \(L = 8\) the segments are 24–32 and 32–40, each holding only 2 of the 4 goal seconds (\(f = 0.25\)); with \(L = 10\) one segment is 30–40 and holds all 4 seconds (\(f = 0.4\)). Overlap removes this luck. 6. Move the slider: R@1 is 45.9% at \(a = 0.38\) and 51.2% at 0.39, so the answer is 0.39; on 100 videos it is already 59.0% at 0.35, because there are fewer distractors. 7. \(300 - 40 = 260\) ms leaves two batches (240 ms), so \(K = 50\).

Key takeaways

  • Text-to-video retrieval is a dual-encoder problem: video vectors are computed offline, a query is one text encoding and many dot products (208.1). Measure with R@K and median rank (208.2), always with the gallery size.
  • CLIP frame embeddings + mean pooling (208.3), fine-tuned with the symmetric contrastive loss (208.5), is a strong baseline (CLIP4Clip, 43.1 R@1 on MSR-VTT). A temporal transformer helps a little with enough data; a tight cross-encoder needs much more data and cannot be indexed.
  • Text-conditioned pooling (X-Pool, 208.6) finds the frames the query is about, but makes the video vector query-dependent: use it to re-rank, or score by best frame.
  • Video–text encoders trained on video (ViCLIP, InternVideo2) beat mean-pooled CLIP by large margins in zero-shot retrieval; data quality and how videos are cut into clips matter as much as size.
  • Long videos need a segment-level index (208.7). Short events are diluted in long segments (208.8); short segments cost vectors and context. Use overlap or shot boundaries and merge neighboring hits.
  • Scale with ANN search (IVF, PQ, HNSW; 208.9, 208.10) and measure recall against brute force. Add ASR and OCR search with rank fusion (208.11), and re-rank the top candidates with a VLM (208.12).

Key formulas

\[ s = \mathbf{t}^\top \mathbf{v}, \quad \text{R@}K = \tfrac{100}{Q}\textstyle\sum_i \mathbb{1}[r_i \le K], \quad \bar{\mathbf{z}} = \tfrac1T \textstyle\sum_j \mathbf{z}_j, \quad \alpha = \softmax\!\big(\tfrac{\mathbf{q}K^\top}{\sqrt{D_p}}\big) \]
\[ n_{\text{seg}} = \left\lfloor \tfrac{D - L}{S} \right\rfloor + 1, \quad \cos = \tfrac{f}{\sqrt{f^2 + (1-f)^2}}, \quad \text{bytes} = n d b, \quad \text{RRF} = \textstyle\sum_m \tfrac{1}{60 + \operatorname{rank}_m} \]

References for this session

  1. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  2. Wang et al. (2023). InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
  3. Wang et al. (2024). InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
  4. Xu et al. (2016). MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
  5. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  6. Gorti, Vouitsis, Ma, Golestan, Volkovs, Garg & Yu (2022). X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval
  7. Wang et al. (2023). InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
  8. Wang et al. (2024). InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
  9. Xu et al. (2016). MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
  10. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  11. Johnson et al. (2017). Billion-scale similarity search with GPUs
  12. Jégou et al. (2011). Product Quantization for Nearest Neighbor Search
  13. Malkov & Yashunin (2016). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
  14. Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
  15. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models