Text-to-video retrieval
Text-to-video retrieval ranks a whole video library by how well each video, or each moment, matches a sentence. This session builds it step by step: CLIP frame embeddings with mean, sequential and attention pooling (CLIP4Clip, X-Pool), video–text encoders trained on video (ViCLIP, InternVideo2), segment-level indexes for long videos, approximate nearest-neighbor search for millions of segments, and re-ranking the top candidates with a VLM.
What you will learn
- How a dual encoder scores text against video, and how R@K, median rank and mean rank are computed, and why they depend on gallery size
- The three CLIP4Clip similarity calculators (mean pooling, sequential transformer, tight cross-encoder), the symmetric contrastive loss, and what the MSR-VTT results say about each
- Text-conditioned attention pooling (X-Pool): what it gains, and why its query-dependent video vector cannot go into a standard index
- What ViCLIP and InternVideo2 change by training the encoder on video–text data, and the false-negative problem of many clips per video
- Clip-level versus segment-level indexes: the dilution formula, how to choose segment length, overlap and shot boundaries, and merging neighboring hits
- Scaling to tens of millions of segments with IVF, PQ and HNSW, measuring ANN recall, and adding ASR and OCR with reciprocal rank fusion
- Two-stage retrieval: re-ranking the top candidates with a VLM yes/no score and fusing it with the first-stage score
Why this matters
A user types “a goalkeeper saves a penalty in the rain” and expects the right ten seconds out of a library of a million hours. That is text-to-video retrieval: given a sentence, rank all videos (or all moments of all videos) by how well they match it. The reverse direction, given a video find the best caption, is video-to-text retrieval.
The task looks like image search with more frames. It is harder for three reasons:
- A video says much more than a sentence. A 30-second clip may show a kitchen, a person talking, a dog walking in, and a pan on fire. A query usually names one of these things. The model must decide which part of the video the query is about.
- Videos are long. A benchmark clip is 10–30 seconds; a real video is minutes or hours. If you squeeze an hour into one vector, a 4-second event disappears.
- The library is huge. Every video must be encoded once, offline, into vectors that a query can be compared with in milliseconds. That rules out models that need to see the query and the video together, unless they run only on a short list.
This session builds the standard answer step by step: frame embeddings from CLIP plus pooling (CLIP4Clip), segment-level indexes for long videos, video–text encoders trained on video (ViCLIP, InternVideo2), approximate nearest-neighbor search for millions of segments, and re-ranking the top candidates with a VLM.
We use CLIP from session 166 and the video encoders of session 202. The previous session, Session 207, found the moment that matches a sentence inside one video. Here we search across a whole library. The next session, Session 209, searches for the same content (copies), not the same meaning.
The task, the data and the metrics
Almost every system uses a dual encoder (also called a two-tower or bi-encoder model): a text encoder \(g\) turns the query into a vector \(\mathbf{t}\), a video encoder \(f\) turns each video into a vector \(\mathbf{v}\), both of length \(d\) and normalized to length 1. The score is the cosine similarity:
- Read the symbols
- \(\mathbf{t} = g(\text{query})\)
- the text embedding of the query
- \(\mathbf{v} = f(\text{video})\)
- the video embedding, computed once and stored in an index
- \(d\)
- the embedding size (512 for CLIP ViT-B/32 [Certain])
Because \(\mathbf{v}\) does not depend on the query, all video vectors can be computed offline. At query time we run the text encoder once and compute dot products. This is what makes search over millions of items possible.
Benchmarks
The most used benchmark is MSR-VTT (Microsoft Research Video to Text): 10,000 web video clips, 10 to 32 seconds long, about 41 hours in total, from 20 categories, with 20 human captions per clip (200,000 captions). [Certain] Results are usually reported on the 1k-A test split: 1,000 clip–caption pairs. Models train on 9,000 clips (“Training-9K”) or 7,000 (“Training-7K”). [Certain] Other common sets are MSVD (1,970 clips), LSMDC (movie clips), DiDeMo and ActivityNet Captions; on the last two, all captions of a video are joined into a paragraph, so they test video–paragraph retrieval. [Certain]
Recall at K and median rank
For each test query \(i\) we sort all \(N\) videos by score and find the position (the rank) \(r_i\) of the correct video; rank 1 is the top. Then:
- Read the symbols
- \(Q\)
- the number of test queries (1,000 on MSR-VTT 1k-A)
- \(r_i\)
- the rank of the correct video for query \(i\)
- R@K
- recall at K: the percentage of queries whose correct video is in the top \(K\) (K = 1, 5, 10)
- MdR, MnR
- median and mean rank; lower is better. MnR is sensitive to a few very bad queries
One thing is easy to forget: R@K depends on the size of the gallery. With 100 videos there are 99 wrong answers that can beat the right one; with 1,000 there are 999. The same model gets a lower R@1 on a bigger gallery. A number from MSR-VTT 1k-A tells you how a model ranks 1,000 clips, not a million. Figure 1 shows this with a toy model.
How to read Figure 1
Each row is one query. The correct video sits on the diagonal (orange frame). If another cell in the row is higher, the row gets a red mark: its R@1 is a miss. At the default (\(a = 0.35\), 1,000 videos) R@1 is 33.1%, R@10 72.0% and the median rank is 3. Now click “100 videos” without moving the slider: R@1 jumps to 59.0%. Nothing about the model changed; only the number of distractors did. Move \(a\) to 0.45 and both galleries become easy (R@1 83.3% on 1,000 videos, 92.0% on 100). This is why you must compare models on the same test split and size.
Ten queries give the ranks 1, 3, 1, 12, 2, 1, 6, 1, 1, 25. Five ranks equal 1, so \(\text{R@}1 = 50\%\). Seven are \(\le 5\) (the five 1s, the 3 and the 2), so \(\text{R@}5 = 70\%\). Eight are \(\le 10\) (add the 6): \(\text{R@}10 = 80\%\). Sorted: 1, 1, 1, 1, 1, 2, 3, 6, 12, 25; the median of an even count is the mean of the two middle values, \(\text{MdR} = (1 + 2)/2 = 1.5\). The mean rank is \(\text{MnR} = 53/10 = 5.3\), pulled up by the two bad queries.
From frames to a video vector: CLIP4Clip
CLIP (session 166) already maps images and text into one space. The simplest video encoder is therefore: sample frames, embed each with CLIP’s image encoder, and combine the frame vectors. CLIP4Clip (Luo et al., 2021) studied exactly this. [Certain] It starts from CLIP ViT-B/32, samples frames uniformly at 1 frame per second, uses 12 frames, and fine-tunes on each benchmark. [Certain] For video \(i\) the image encoder gives frame embeddings \(\mathbf{z}_1, \dots, \mathbf{z}_T\) (the [CLS] output of each frame). CLIP4Clip compares three ways to get a similarity from them, which it calls similarity calculators.
1. Parameter-free: mean pooling
No new weights: the “average frame” lives in CLIP’s space from the first step. It ignores order, and every frame gets the same weight.
2. Sequential: a temporal model before pooling
- Read the symbols
- \(Z = [\mathbf{z}_1, \dots, \mathbf{z}_T]\)
- the frame embeddings, a \(T \times d\) matrix
- \(P\)
- learned position embeddings, one per frame position, so the model knows the order
- \(\tilde{Z}\)
- frame embeddings after they have exchanged information with each other
CLIP4Clip tries an LSTM (“seqLSTM”) and a small transformer encoder (“seqTransf”) here. The video vector still does not depend on the query, so it can still be indexed.
3. Tight: a joint transformer over text and frames
The “tightTransf” type concatenates the text vector and the frame vectors, \([\mathbf{w}, \mathbf{z}_1, \dots, \mathbf{z}_T]\), adds position and type embeddings, runs a transformer encoder over all of them, and maps the first output token through FC–ReLU–FC to one score. [Certain] This is a cross-encoder: text and video meet inside the network. It is the most expressive and the most expensive, because the network must run once for every (query, video) pair. Nothing can be precomputed per video except the frame embeddings.
Training: the symmetric contrastive loss
Training uses the same loss as CLIP. A batch has \(B\) (video, caption) pairs; we compute all \(B \times B\) similarities and ask each caption to pick its video and each video to pick its caption:
- Read the symbols
- \(s_{ij}\)
- similarity of caption \(i\) and video \(j\); the true pairs are on the diagonal
- \(\tau\)
- the temperature; CLIP learns the logit scale \(1/\tau\) and caps it at 100
- first term
- text-to-video: caption \(i\) must pick video \(i\) out of the batch
- second term
- video-to-text: video \(i\) must pick caption \(i\)
CLIP4Clip sums the two directions. [Certain] Every other video in the batch is a negative. Larger batches give more negatives; CLIP4Clip found 128 and 256 about equal and used 128. [Certain]
- input: a batch of \(B\) (video, caption) pairs; CLIP image encoder \(\phi\), text encoder \(g\); logit scale \(1/\tau\)
- for each video do decode at 1 fps and keep \(T = 12\) uniformly spaced frames, resize to 224×224
- \(Z \leftarrow \phi(\text{frames})\) // (B, T, d): every frame goes through the image ViT
- \(\mathbf{v}_j \leftarrow \operatorname{normalize}(\operatorname{mean}_t Z_{j,t})\) // formula (208.3)
- \(\mathbf{t}_i \leftarrow \operatorname{normalize}(g(\text{caption}_i))\) // the [EOS] token output
- \(S \leftarrow [\mathbf{t}_i^\top \mathbf{v}_j]_{B \times B}\); loss from formula (208.5)
- backpropagate into both encoders with a very small learning rate (the paper’s best was \(10^{-7}\)) // do not destroy CLIP
What the experiments showed
Text-to-video results on MSR-VTT 1k-A after training on Training-9K (all numbers from the CLIP4Clip paper, ViT-B/32, 12 frames): [Certain]
| Method | New weights | R@1 | R@5 | R@10 | MdR |
|---|---|---|---|---|---|
| CLIP zero-shot, mean of frames (“CLIP-straight”) | none, no training | 31.2 | 53.7 | 64.2 | 4 |
| CLIP4Clip meanP | none | 43.1 | 70.4 | 80.8 | 2 |
| CLIP4Clip seqLSTM | 1-layer LSTM | 42.5 | 70.8 | 80.7 | 2 |
| CLIP4Clip seqTransf | 4-layer transformer [Certain] | 44.5 | 71.4 | 81.6 | 2 |
| CLIP4Clip tightTransf | cross transformer + FC | 40.2 | 71.5 | 80.5 | 2 |
Three lessons. First, fine-tuning CLIP on video captions is worth about 12 points of R@1 over zero-shot CLIP, with no new weights at all. Second, the sequential type helps a little on the 9K split, but on the smaller Training-7K split mean pooling was best (42.1 vs 42.0 for seqTransf). [Certain] New weights need data. Third, the tight cross-encoder was the worst on almost every dataset, even though it is the most expressive; the authors explain this by too little data to learn cross-modal interaction from scratch. [Certain] They also tried a 3D patch embedding (patches across time) and got lower results than CLIP’s 2D one (41.6 vs 43.1 R@1 on MSR-VTT), because the 3D weights could not start from CLIP. [Certain] Going from 1 to 6 frames gave a large gain; they chose 12. [Certain]
Attention pooling: which frames does the query talk about?
Mean pooling gives every frame the same weight. But a caption often describes only some frames. There are two ways to learn weights:
- Text-agnostic attention: a learned query vector (or self-attention) scores the frames. Weights depend only on the video, so the video vector can still be indexed.
- Text-conditioned attention: the query text decides the weights. X-Pool (Gorti et al., 2022) does this with scaled dot-product attention where the text is the query and the frames are the keys and values: [Certain]
- Read the symbols
- \(W_Q, W_K, W_V\)
- projection matrices for the text (query) and the frames (keys and values); X-Pool applies LayerNorm before them and uses \(D_p = 512\) [Certain]
- \(\alpha\)
- one weight per frame, summing to 1: how relevant each frame is to this text
- \(\mathbf{v}_{|t}\)
- a video vector conditioned on the text; X-Pool adds a small FC block with a residual on top
X-Pool, with the same CLIP ViT-B/32 and 12 frames, reported 46.9 R@1 on MSR-VTT 9K, against 44.5 for CLIP4Clip seqTransf. [Certain] The price is in the index: \(\mathbf{v}_{|t}\) is different for every query, so you cannot store one vector per video. You store all \(T\) frame vectors and run the attention for every (query, video) pair, or you use X-Pool only to re-rank a short list. A cheap middle ground is to store frame vectors and score a video by its best frame, \(\max_j \mathbf{t}^\top \mathbf{z}_j\): this is text-conditioned, needs no learned weights, and works with a standard vector index (search frames, then group hits by video).
How to read Figure 2
Start with the dog query and mean pooling. The correct video (A, a cooking vlog) shows the dog in only one of eight frames, so its average is mostly “person talking in a kitchen”: score 0.580. The kitchen tour (E) scores 0.693 and wins. Switch to “best frame” (0.988) or attention (0.986 at \(\tau_a = 0.1\)): video A now wins. Now pick the chef query. Mean pooling ranks the cooking show (C) first, 0.962 against 0.952. With best frame, the vlog A wins (0.970) because one of its frames is a perfect match. So text-conditioned pooling helps when the query names a short part of the video and can hurt when the query describes the whole video. Raise \(\tau_a\) toward 1: the attention weights flatten and the scores move back toward mean pooling.
Video–text encoders: ViCLIP and InternVideo2
CLIP4Clip and X-Pool start from an image model and add time at the end. The next step is to train the encoder itself on video–text pairs, so that motion and events are inside the embedding. Two widely used examples come from the InternVideo line of work.
InternVid and ViCLIP
InternVid (Wang et al., 2023) is a video–text dataset of over 7 million videos, nearly 760,000 hours, cut into 234 million clips with generated captions totaling 4.1 billion words. [Certain] ViCLIP is the model trained on it: a ViT-L video encoder and a text encoder, both initialized from CLIP, with the image attention changed to spatiotemporal attention (tokens from all frames attend to each other). [Certain] To save compute it randomly masks video patches during pretraining, as in MAE, and trains briefly without masking at the end so that training matches inference. [Certain] The loss is the contrastive loss (208.5) between video and text embeddings.
In zero-shot retrieval (no training on the benchmark, 8 frames), the paper reports on MSR-VTT text-to-video R@1: [Certain]
| Model | Pretraining data | MSR-VTT T2V R@1, zero-shot |
|---|---|---|
| CLIP ViT-L, mean of 8 frames | OpenAI 400M image–text pairs | 29.0 |
| CLIP ViT-L, mean of 8 frames | DataComp-1B | 30.4 |
| ViCLIP | WebVid-10M | 35.6 |
| ViCLIP | InternVid-10M-FLT (a filtered 10M subset) | 42.4 |
| ViCLIP | InternVid-200M | 39.3 |
Two things stand out. Training on video–text data adds about 12 points over mean-pooled CLIP of the same size. And the filtered 10M subset beats the full 200M set on zero-shot retrieval: data quality beat data size here. [Certain] The authors suspect one cause: clips cut from the same video look alike and have similar captions, but the contrastive loss treats them as negatives. These false negatives hurt more when many clips come from the same video and when the batch is large. [Certain] Remember this when you build your own training set from long videos.
InternVideo2
InternVideo2 (Wang et al., 2024) scales the video encoder to 6 billion parameters and trains it in three stages: [Certain]
- Unmasked token reconstruction. Mask 80% of the video tokens and train the encoder to match, on the visible tokens, the features of two teachers: InternVL-6B (an image–text model, for meaning) and VideoMAEv2-g (for motion).
- Cross-modal contrastive learning. Align video with text, audio and speech, using contrastive, matching and masked-language losses. The text encoder is the first 19 layers of BERT-Large.
- Next-token prediction. Connect the encoder to an LLM and train on instruction data, for chat and question answering.
The stage-2 model is the one used for retrieval. Its data has 402M entries: 2M videos, 50M video–text pairs, 50M video–audio–speech–text pairs and 300M image–text pairs. [Certain] Zero-shot MSR-VTT text-to-video R@1 is 51.9 for the 1B model and 55.9 for the 6B model; after fine-tuning, the 6B model reports 62.8. [Certain] The paper also found that cutting training videos at learned shot boundaries (with the AutoShot model) instead of FFmpeg’s simple scene filter clearly improved zero-shot retrieval: how you cut a video into clips matters. [Certain] We come back to that in the next section.
For a product, the choice is a trade-off between quality and cost. A CLIP-style encoder with mean pooling is cheap and very strong for “what is in the video” queries. A video–text encoder such as ViCLIP or InternVideo2 is better at actions and events, but each segment costs more to encode, and a 6B encoder is a real bill when you index a million hours. [Likely] The architectures behind these models are covered in session 202 and session 212.
Clip-level and segment-level indexes
MSR-VTT clips are short and each caption describes the whole clip. Real libraries are different: a news broadcast is 30 minutes, a football match 2 hours, a lecture 1 hour. What should one vector in the index stand for?
- Clip-level (video-level) index: one vector per video, for example the mean of frame embeddings over the whole video. Small and simple. Good for short videos (social clips, ads, stock footage) and for queries about the whole video (“a cooking tutorial for pasta”).
- Segment-level index: cut each video into segments, embed each segment, and store one vector per segment with its video ID and start and end times. A query returns moments, not just videos. This is what long-video search needs.
With fixed windows of length \(L\) seconds and stride \(S\) seconds (\(S \lt L\) means overlap), a video of \(D\) seconds gives
- Read the symbols
- \(D, L, S\)
- video length, segment length and stride, in seconds (one frame embedding per second here)
- \(\mathbf{v}_k\)
- the embedding of segment \(k\): here, mean-pooled frames; with a video encoder, the encoder’s output for that segment
- \(\max_k\)
- a video is as relevant as its best segment; the best segment’s times are returned too
For a 1-hour video: 10-second segments without overlap give 360 vectors; with 50% overlap (\(S = 5\)) 719; 2-second segments give 1,800.
Why short events vanish in long segments
Here is the core problem in one formula. Take the simplest case: the event frames have embedding \(\mathbf{e}\) (exactly the query, \(\mathbf{t} = \mathbf{e}\)), the other frames show a background \(\mathbf{b}\) orthogonal to it, and the event fills a fraction \(f\) of the segment. The mean-pooled segment vector is \(f\mathbf{e} + (1 - f)\mathbf{b}\), so
The similarity falls fast as the event gets relatively shorter.
A 4-second goal inside a 4-second segment: \(f = 1\), cosine 1. Inside an 8-second segment: \(f = 0.5\), cosine \(0.5/\sqrt{0.5} = 0.707\). Inside a 16-second segment: \(f = 0.25\), cosine \(0.25/\sqrt{0.0625 + 0.5625} = 0.316\). Inside a 40-second segment: \(f = 0.1\), cosine \(0.1/\sqrt{0.01 + 0.81} = 0.110\). A different video that is only “somewhat related” everywhere (cosine 0.5 in all its segments) beats the true match once \(f\) drops below about 0.37 (solve \(f/\sqrt{f^2+(1-f)^2} = 0.5\)).
So why not make segments tiny? Because short segments have their own costs:
- More vectors: the index grows as \(1/S\) (formula 208.7). Overlap doubles it again.
- Less context: “a man opens the door, then the dog runs out” spans several seconds. A 1-second segment sees only part of it, and video–text encoders are trained on clips of a few seconds or more.
- Fragmented results: a 40-second interview becomes 20 near-identical hits that fill the result page. You must merge neighbors (see Algorithm 2).
- Boundary effects: without overlap, an event can be cut in half by the grid. Overlap of 25–50% is a common fix.
Common choices in practice are segments of a few seconds to about 30 seconds, matched to the encoder’s training clip length and to how long the searched events are. [Likely] A good alternative to fixed windows is to cut at shot boundaries (camera cuts) so segments do not mix two scenes, and to split long shots further; InternVideo2’s result on AutoShot boundaries points the same way. Many systems also keep two levels: a coarse video-level vector to find candidate videos, and segment vectors to localize inside them.
How to read Figure 3
With the goal query and \(L = 10\) s (no overlap), the grid happens to put a segment at 30–40 s: cosine 0.672, above the distractor. Set \(L = 8\): the grid cuts the goal in half (32–40 s and 24–32 s each hold 2 seconds of it) and the best cosine drops to 0.504, barely above 0.50. Turn on 50% overlap: a segment at 28–36 s now holds the whole goal and the cosine is 0.774. At \(L = 30\) s the goal is diluted to 0.376 and the video is lost; at \(L = 120\) (one vector for the whole video) it is 0.232. Now switch to the interview query: it survives up to \(L = 60\) (0.903), but with \(L = 2\) the best hit is a 2-second piece of a 40-second answer.
- input: videos; encoder \(f\); segment length \(L\), stride \(S\); text encoder \(g\); top-\(K\)
- offline, for each video do
- detect shot boundaries; cut into segments of length \(L\) with stride \(S\) inside shots // formula (208.7)
- \(\mathbf{v}_k \leftarrow \operatorname{normalize}(f(\text{segment}_k))\); store \((\mathbf{v}_k, \text{video id}, t_{\text{start}}, t_{\text{end}})\)
- build an ANN index over all \(\mathbf{v}_k\) (IVF-PQ or HNSW, next section)
- online, for a query do
- \(\mathbf{t} \leftarrow \operatorname{normalize}(g(\text{query}))\); fetch the top \(K' \gt K\) segments from the index
- merge hits from the same video whose times overlap or touch; keep the max score // avoid 20 hits for one event
- optionally re-rank the top candidates with a cross-encoder or VLM // formula (208.12)
- return the top \(K\) (video id, start, end, score)
Scaling to millions of segments
Now count. A library of 100,000 hours, cut into 10-second segments without overlap, gives \(100{,}000 \times 3600 / 10 = 36\) million vectors. The memory of the raw index is
where \(n\) is the number of vectors, \(d\) the dimension and \(b\) the bytes per number. With \(d = 768\) (an assumed size for this example):
float32: \(36 \times 10^6 \cdot 768 \cdot 4 = 110.6\) GB. float16: 55.3 GB. int8: 27.6 GB. With product quantization (PQ) to 64 bytes per vector: 2.3 GB, small enough for one machine’s RAM. A brute-force search compares the query with every vector: \(2 \cdot 36 \times 10^6 \cdot 768 = 55.3\) GFLOP per query. That is fine for a few queries on a GPU, but not for thousands of queries per second on CPU. Adding 50% overlap doubles everything (72M vectors).
The standard tools are the same as for text search (see session 151):
- IVF (inverted file): run k-means on the vectors to get \(n_{\text{list}}\) centroids (cells). Each vector is stored in the list of its nearest centroid. At query time, compare the query with all centroids, and scan only the \(n_{\text{probe}}\) closest lists.
- PQ (product quantization): split each vector into \(m\) sub-vectors and replace each by the index of its nearest code word (1 byte for 256 code words). A vector becomes \(m\) bytes; distances are computed from small lookup tables. [Certain]
- HNSW: a graph where each vector links to near neighbors on several levels; search walks the graph greedily. Very fast and accurate, but uses more memory than IVF-PQ. [Likely]
Libraries such as FAISS implement all of these on CPU and GPU. [Certain] The cost of one IVF query is roughly
- Read the symbols
- \(n_{\text{list}}\, d\)
- comparing the query with every centroid
- \(\frac{n_{\text{probe}}}{n_{\text{list}}}\, n\)
- the number of vectors in the scanned lists, if lists have equal size
With \(n = 36\)M, \(n_{\text{list}} = 65{,}536\) and \(n_{\text{probe}} = 32\), a query compares with 65,536 centroids and about 17,600 vectors: about 433 times fewer than brute force. The price is recall: a true neighbor that sits in a list we did not probe is lost. Figure 4 shows this trade-off.
How to read Figure 4
With 16 cells and one probe, IVF scans 7.7% of the points and finds 90.8% of the true neighbors on average. Two probes: 99.1% recall for 14.1% of the work. With 64 cells and one probe the scan drops to 1.9% but recall falls to 83.5%; four probes bring it to 100% for 7.2%. Queries near a cell border lose neighbors that sit just across the border; more probes fix that. In production you tune \(n_{\text{probe}}\) on a held-out query set until recall@K of the ANN search (against brute force) is, say, 95%.
Hybrid search: not everything is in the pixels
Many queries are best answered by other signals: what people say (speech transcribed by ASR), what is written on screen (OCR: scoreboards, slides, signs), and metadata (title, date, channel). A strong system runs a text search (BM25 or a text embedding, see session 150) over transcripts and OCR next to the visual search, and fuses the ranked lists. A simple, robust fusion is reciprocal rank fusion (RRF): [Certain]
It uses only ranks, so the scores of different systems do not need to be on the same scale. \(k = 60\) is the value from the original paper. [Certain]
Re-ranking with a VLM
The dual encoder compresses a video into one vector before it sees the query. That is why it is fast, and also why it makes errors that a model reading both together would not make: negation (“a man without a hat”), counting (“three dogs”), order (“opens the door, then sits down”) and fine details (“in the rain”). The fix is the same as in text search: a two-stage pipeline.
- Retrieve: the dual encoder and the ANN index return the top \(K\) candidates (say 50–200) out of millions, in milliseconds.
- Re-rank: a stronger model scores each (query, candidate) pair. This can be a cross-encoder (like CLIP4Clip’s tight type or a matching head), or a VLM (session 205) that sees the frames of the candidate and the query.
A simple VLM re-ranker asks a yes/no question and reads the probabilities of the two answer tokens:
- Read the symbols
- \(\ell_{\text{yes}}, \ell_{\text{no}}\)
- the VLM’s logits for the tokens “Yes” and “No” after the prompt “Does this clip show: query? Answer yes or no.”
- \(\hat{s}_{\text{dual}}\)
- the dual-encoder score, min–max normalized over the \(K\) candidates so it lies in [0, 1]
- \(\alpha\)
- the weight of the first stage; tune it on labeled queries
Keeping some of the first-stage score (\(\alpha \gt 0\)) protects against VLM mistakes, and RRF (208.11) is an alternative when the two scores are hard to compare. Cost: \(K\) VLM calls per query. If one batch of 25 candidates (8 frames each) takes 120 ms on your GPU, re-ranking 50 candidates adds 240 ms; that is a latency budget you must plan, and the frames of candidates should be cached as pre-computed visual tokens if possible. Use the re-ranker where the first stage is weak (long, detailed queries) and skip it for short keyword queries.
How to read Figure 5
Step through. In step 2 the wanted segment (penalty save in heavy rain) is only 4th: the dual encoder sees “penalty”, “goalkeeper” and “rain” in many candidates and cannot tell which has all three. In step 3 the VLM gives it 0.93 while the others are at most 0.55. In step 4, with \(\alpha = 0.5\), it moves to rank 1. Move \(\alpha\): at 1.0 you get back the first-stage order; at 0.7 it is 2nd; from 0.6 down it is 1st. At \(\alpha = 0\) you trust only the VLM.
Code: pooling, metrics, segments, IVF and fusion
The code runs on CPU in under a second. It uses synthetic “frozen CLIP” features so that the result is exact and reproducible: each video has 8 frames showing one scene, except one frame that shows the event named in its caption.
import math, torch import torch.nn as nn import torch.nn.functional as F torch.manual_seed(0) # 1) three ways to turn frame embeddings Z (B, T, d) into one video vector class MeanPool(nn.Module): # parameter-free, formula (208.3) def forward(self, Z, t=None): return F.normalize(Z.mean(1), dim=-1) class SeqTransf(nn.Module): # sequential type, formula (208.4) def __init__(self, d, T, layers=2): super().__init__() self.pos = nn.Parameter(torch.zeros(1, T, d)) layer = nn.TransformerEncoderLayer(d, nhead=4, dim_feedforward=2 * d, batch_first=True) self.enc = nn.TransformerEncoder(layer, layers) def forward(self, Z, t=None): return F.normalize((Z + self.enc(Z + self.pos)).mean(1), dim=-1) # residual keeps CLIP's space class TextCondPool(nn.Module): # X-Pool-style attention, formula (208.5) def __init__(self, d): super().__init__() self.ln = nn.LayerNorm(d) self.q, self.k, self.v = (nn.Linear(d, d, bias=False) for _ in range(3)) for m in (self.q, self.k, self.v): nn.init.eye_(m.weight) # start as identity def forward(self, Z, t): # Z: (B, T, d), t: (Q, d) -> (Q, B, d) Zn, tn = self.ln(Z), self.ln(t) # LayerNorm: vectors of length ~ sqrt(d) w = torch.einsum("qd,btd->qbt", self.q(tn), self.k(Zn)) / math.sqrt(Z.shape[-1]) return F.normalize(torch.einsum("qbt,btd->qbd", w.softmax(-1), self.v(Zn)), dim=-1) # 2) symmetric contrastive loss over a batch, formula (208.6) def clip_loss(S, logit_scale=100.0): # S: (B, B) cosines, positives on the diagonal S = logit_scale * S y = torch.arange(len(S)) return 0.5 * (F.cross_entropy(S, y) + F.cross_entropy(S.T, y)) # 3) retrieval metrics, formula (208.2): rank of the true video for each text def recall_at_k(S, ks=(1, 5, 10)): rank = 1 + (S > S.diag()[:, None]).sum(1) out = {f"R@{k}": round(100 * (rank <= k).float().mean().item(), 1) for k in ks} out["MdR"] = rank.float().median().item() return out # synthetic "frozen CLIP" features: each video has 8 frames; its caption names ONE short event B, T, d = 256, 8, 64 scene = F.normalize(torch.randn(B, 1, d), dim=-1).expand(B, T, d) # what most frames show event = F.normalize(torch.randn(B, d), dim=-1) # the captioned event pos = torch.randint(0, T, (B,)) # where the event happens Z = scene.clone(); Z[torch.arange(B), pos] = event Z = F.normalize(Z + 0.3 * torch.randn(B, T, d) / math.sqrt(d), dim=-1) text = F.normalize(event + 0.3 * torch.randn(B, d) / math.sqrt(d), dim=-1) S_mean = text @ MeanPool()(Z).T print("mean pooling ", recall_at_k(S_mean)) S_max = torch.einsum("qd,btd->qbt", text, Z).max(-1).values # best single frame print("max over frames ", recall_at_k(S_max)) xp = TextCondPool(d) with torch.no_grad(): Vq = xp(Z, text) # one video vector per (text, video) S_x = torch.einsum("qd,qbd->qb", text, Vq) print("text-cond. attn ", recall_at_k(S_x), tuple(Vq.shape)) seq = SeqTransf(d, T).eval() # untrained: shapes only print("seqTransf output", tuple(seq(Z).shape), "new params", sum(p.numel() for p in seq.parameters())) print("loss: mean", round(clip_loss(S_mean).item(), 3), " text-cond.", round(clip_loss(S_x).item(), 3), " uniform guess log B =", round(math.log(B), 3)) # 4) a segment index for long videos, formula (208.7): windows of L s, stride S, score = max def segment(frame_emb, L, S): # frame_emb: (D, d), one row per second D = frame_emb.shape[0] starts = list(range(0, D - L + 1, S)) if starts[-1] + L < D: starts.append(D - L) # cover the tail segs = torch.stack([F.normalize(frame_emb[s:s + L].mean(0), dim=0) for s in starts]) return segs, torch.tensor(starts) vid = torch.zeros(120, 4); vid[:60, 0] = 1; vid[60:, 1] = 1 # two background scenes vid[30:34] = torch.tensor([0.3, 0, 1, 0]); vid[60:100] = torch.tensor([0, 0.3, 0, 1]) vid = F.normalize(vid, dim=-1) q_goal = F.normalize(torch.tensor([0.25, 0, 1, 0]), dim=0) for L, S in [(4, 2), (10, 10), (30, 30), (120, 120)]: segs, starts = segment(vid, L, S) sims = segs @ q_goal i = sims.argmax().item() print(f"L={L:3d} S={S:3d}: {len(starts):3d} vectors, best [{starts[i].item()}, {starts[i].item() + L}] s, cos {sims[i].item():.3f}") # 5) IVF: cluster the index, search only the nprobe closest cells, formula (208.10) def kmeans(X, k, iters=10): C = X[torch.linspace(0, len(X) - 1, k).long()].clone() for _ in range(iters): a = torch.cdist(X, C).argmin(1) for j in range(k): if (a == j).any(): C[j] = X[a == j].mean(0) return C, torch.cdist(X, C).argmin(1) N = 20000 X = F.normalize(torch.randn(16, 32)[torch.randint(0, 16, (N,))] + 0.5 * torch.randn(N, 32), dim=-1) Qv = F.normalize(X[:200] + 0.1 * torch.randn(200, 32), dim=-1) C, assign = kmeans(X, 64) exact = (Qv @ X.T).topk(10).indices for nprobe in (1, 4, 16): cells = (Qv @ C.T).topk(nprobe).indices hits, scanned = 0, 0 for qi in range(len(Qv)): cand = torch.nonzero(torch.isin(assign, cells[qi])).squeeze(1) scanned += len(cand) top = cand[(Qv[qi] @ X[cand].T).topk(10).indices] hits += len(set(top.tolist()) & set(exact[qi].tolist())) print(f"nprobe={nprobe:2d}: recall@10 {hits / (10 * len(Qv)):.3f}, scanned {scanned / len(Qv) / N:.3f} of the index") # 6) re-ranking: fuse the dual-encoder score with a VLM's p(yes), formula (208.11) dual = torch.tensor([.412, .405, .398, .391, .380, .372, .361, .355]) p_yes = torch.tensor([.18, .35, .41, .93, .02, .55, .03, .47]) for alpha in (1.0, 0.5): fused = alpha * (dual - dual.min()) / (dual.max() - dual.min()) + (1 - alpha) * p_yes print(f"alpha={alpha}: order {(fused.argsort(descending=True, stable=True) + 1).tolist()}")
Output (PyTorch 2.8, CPU):
mean pooling {'R@1': 3.9, 'R@5': 17.2, 'R@10': 26.2, 'MdR': 36.0}
max over frames {'R@1': 100.0, 'R@5': 100.0, 'R@10': 100.0, 'MdR': 1.0}
text-cond. attn {'R@1': 100.0, 'R@5': 100.0, 'R@10': 100.0, 'MdR': 1.0} (256, 256, 64)
seqTransf output (256, 64) new params 67456
loss: mean 21.856 text-cond. 0.0 uniform guess log B = 5.545
L= 4 S= 2: 59 vectors, best [30, 34] s, cos 0.999
L= 10 S= 10: 12 vectors, best [30, 40] s, cos 0.672
L= 30 S= 30: 4 vectors, best [30, 60] s, cos 0.376
L=120 S=120: 1 vectors, best [0, 120] s, cos 0.232
nprobe= 1: recall@10 0.559, scanned 0.023 of the index
nprobe= 4: recall@10 0.954, scanned 0.067 of the index
nprobe=16: recall@10 1.000, scanned 0.253 of the index
alpha=1.0: order [1, 2, 3, 4, 5, 6, 7, 8]
alpha=0.5: order [4, 2, 1, 3, 6, 8, 5, 7]
Read it line by line. Mean pooling fails on this data (R@1 3.9%) because each caption names one frame out of eight: exactly the dilution of formula (208.8). The best frame and text-conditioned attention find every video, but note the shape (256, 256, 64): one video vector per query, which is why it cannot be indexed. The contrastive loss of the mean-pooled scores (21.9) is far above \(\log B = 5.5\), the loss of a uniform guess, because the model is confidently wrong. The segment part reproduces Figure 3 (0.999 at 4 s, 0.672 at 10 s, 0.232 for one vector per video). The IVF part shows recall rising with \(n_{\text{probe}}\) while the scanned fraction stays small. The last line is Figure 5 at \(\alpha = 0.5\).
Where this is used
| Application | Index unit | Typical design |
|---|---|---|
| Stock footage and social clip search | One vector per short clip | CLIP or video–text encoder, mean pooling, ANN index, metadata filters |
| Broadcast and sports archives (“all goals by player X in the rain”) | Segments of a few seconds, plus video-level vectors | Shot-based segments, hybrid search with ASR and on-screen OCR, VLM re-ranking; links to key-moment detection in session 207 |
| Video-RAG for question answering | Segments with timestamps | Retrieve top segments for the question, feed their frames and transcripts to a VLM (session 210) |
| Content moderation and policy search | Segments | Text queries describing a policy (“a weapon pointed at a person”), high recall, human review |
| Dataset curation for training | Clips | Find clips that match a concept to build or balance training sets; remove near-duplicates (session 209) |
| Personal video and surveillance search | Segments, often per tracked object | On-device or private index; time and camera filters; privacy rules |
| Zero-shot classification and tagging | Clips | Retrieval with class names as queries, as in session 203 |
Common mistakes
- Comparing R@1 across different gallery sizes or splits. MSR-VTT 1k-A, the full test set and your 10M-segment index are different problems (Figure 1). Report the gallery size.
- One vector for a long video. Mean pooling over an hour dilutes every short event (formula 208.8). Use segments, or at least a two-level index.
- Segments without overlap and without merging. Events get cut by the grid, and long events flood the results with neighbors. Use 25–50% overlap or shot boundaries, and merge adjacent hits.
- Indexing a text-conditioned model. X-Pool-style pooling depends on the query, so it cannot produce one stored vector per video. Use it to re-rank a short list, or store frame vectors and score by the best frame.
- Mismatched frame sampling. Encoding the index at 1 frame per 4 seconds when the model was trained with 12 frames at 1 fps changes the embeddings. Use the same sampling, resolution and number of frames at indexing time as in training.
- Tuning ANN parameters without measuring recall. Always measure the ANN top-K against brute force on held-out queries; a fast index with 70% recall silently drops results.
- Ignoring speech and text on screen. Many queries are about what is said or written. Visual embeddings alone miss them; add ASR and OCR search and fuse.
- Training on many clips of the same video in one batch. They act as false negatives in the contrastive loss, as ViCLIP’s authors suspected. Limit clips per video per batch.
Interview questions
What do R@1, R@10 and median rank measure, and why can the same model get a very different R@1 on two test sets?basic
How do you get a single video embedding from an image–text model like CLIP?basic
CLIP4Clip compared parameter-free, sequential and tight similarity calculators. What did it find, and why?core
Explain X-Pool. What does it gain and what does it cost in a real search system?core
How do you choose the segment length for a segment-level index?core
Dual encoder or cross-encoder for text-to-video retrieval?core
How much memory does an index of 36 million 768-dimensional segment vectors need, and how do you make it smaller and faster?core
Why might more contrastive training data from long videos make a video–text model worse?advanced
Design text search over 100,000 hours of sports broadcasts.advanced
How would you add a VLM re-ranker, and how do you keep it from making things worse?advanced
What do ViCLIP and InternVideo2 add over CLIP with mean pooling?core
Exercises
- A 1-hour video is cut into segments of \(L = 8\) s with stride \(S = 4\) s. How many vectors? How many for 10,000 such videos?
- Using formula (208.8), what is the cosine when a 3-second event sits in a 12-second segment? Would it beat a distractor at 0.5?
- An index has 5 million segments with \(d = 512\). How much memory in float16? With 32-byte PQ codes?
- With \(n = 36\)M vectors, \(n_{\text{list}} = 16{,}384\) and \(n_{\text{probe}} = 16\), how many vectors does one IVF query scan if lists are equal in size?
- In Figure 3 (no overlap), the goal query gives 0.504 with \(L = 8\) s but 0.672 with \(L = 10\) s. Explain why the longer segment does better here.
- In Figure 1, find the smallest \(a\) at which R@1 on 1,000 videos is above 50%. Why is the answer different for 100 videos?
- Your search must return in 300 ms. The first stage takes 40 ms; a VLM batch of 25 candidates takes 120 ms. What is the largest \(K\) you can re-rank (in whole batches, run one after another)?
Answerscheck
Key takeaways
- Text-to-video retrieval is a dual-encoder problem: video vectors are computed offline, a query is one text encoding and many dot products (208.1). Measure with R@K and median rank (208.2), always with the gallery size.
- CLIP frame embeddings + mean pooling (208.3), fine-tuned with the symmetric contrastive loss (208.5), is a strong baseline (CLIP4Clip, 43.1 R@1 on MSR-VTT). A temporal transformer helps a little with enough data; a tight cross-encoder needs much more data and cannot be indexed.
- Text-conditioned pooling (X-Pool, 208.6) finds the frames the query is about, but makes the video vector query-dependent: use it to re-rank, or score by best frame.
- Video–text encoders trained on video (ViCLIP, InternVideo2) beat mean-pooled CLIP by large margins in zero-shot retrieval; data quality and how videos are cut into clips matter as much as size.
- Long videos need a segment-level index (208.7). Short events are diluted in long segments (208.8); short segments cost vectors and context. Use overlap or shot boundaries and merge neighboring hits.
- Scale with ANN search (IVF, PQ, HNSW; 208.9, 208.10) and measure recall against brute force. Add ASR and OCR search with rank fusion (208.11), and re-rank the top candidates with a VLM (208.12).
Key formulas
References for this session
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Wang et al. (2023). InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
- Wang et al. (2024). InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
- Xu et al. (2016). MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Gorti, Vouitsis, Ma, Golestan, Volkovs, Garg & Yu (2022). X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval
- Wang et al. (2023). InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
- Wang et al. (2024). InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
- Xu et al. (2016). MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Johnson et al. (2017). Billion-scale similarity search with GPUs
- Jégou et al. (2011). Product Quantization for Nearest Neighbor Search
- Malkov & Yashunin (2016). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
- Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models