LLM & VLM courseBlog
214Track 8: Video and multimodal systems · session 14 of 21

text-to-video retrieval

In this lab you build a small text-to-video search system and measure it. You cut videos into segments, embed every frame with SigLIP, pool each segment into one vector, index the segments with FAISS, search with text queries, and compute recall@k. Then you train a one-layer temporal model on top of the frozen frames and see why mean pooling cannot tell which way something moves.

Time: 3 to 4 hours (about 25 minutes of it is CPU time)Level: Intermediate · hands-on labBefore this: Sessions 167 (SigLIP), 208 (text-to-video retrieval) and 213 (evaluating video understanding); Python and basic PyTorch

What you will learn

  • How to turn a video library into a segment index: frame sampling, segment length and stride, pooling, normalization and a metadata table
  • How to embed frames and queries with SigLIP correctly (padding to 64 tokens, unit vectors) and how to read its small cosine values
  • How to index and search with FAISS (IndexFlatIP, IndexIDMap) and how IVF trades recall for speed through nlist and nprobe
  • How to compute recall@k with an explicit hit rule for overlapping segments, and how noisy 40 queries are
  • Why mean and best-frame pooling are blind to the order of frames, measured: R@1 near 50% when both directions of a motion exist
  • How a one-layer temporal transformer with position embeddings, trained with a symmetric contrastive loss, raises R@1, and how much seeds move the result

Why this matters

In session 208 you learned the ideas: embed frames, pool them, put the vectors in an index, and rank by cosine similarity. In this lab you build the whole system and measure it. You cut videos into segments, embed every frame with SigLIP, pool the frames of each segment into one vector, index the segments with FAISS, search with text queries, and compute recall@k. Then you train a small temporal model and check whether it beats mean pooling.

Every number on this page comes from running the code below on a CPU laptop. Some results will surprise you:

  • Mean pooling cannot tell direction. On our library, mean pooling of SigLIP frames gets R@1 = 45.0% with 8-second segments. Most misses are not random: 17 of the 22 misses return the right object moving the wrong way.
  • A one-layer transformer over the frames fixes most of it. With the same frozen SigLIP features, the temporal model reaches R@1 = 95.0% (one seed), 85.8% on average over three seeds.
  • The loss matters as much as the model. Training the same model with only the video-to-text half of the contrastive loss gave 65.0% on average.

The previous session, Session 213, explained how to score video systems, including R@k. Here you apply it to your own system. The next lab, Session 215, reuses the same segment scores to find moments inside one video.

Setup

You need Python 3.9 or newer and about 1 GB of disk for the model. No GPU is needed.

pip install torch transformers faiss-cpu sentencepiece pillow numpy protobuf

We use google/siglip-base-patch16-224: a ViT-B/16 image encoder at \(224 \times 224\) pixels and a text encoder, both giving 768-dimensional embeddings. [Certain: checked with transformers 4.57] SigLIP (Zhai et al., 2023) is a CLIP-style dual encoder trained with a sigmoid loss instead of a softmax loss, so each image–text pair is scored on its own, as a yes/no question (session 167).

A toy library with known answers

To measure recall you need queries whose correct answer you know exactly. Real benchmarks such as MSR-VTT give that, but they need large downloads. So we generate a small library: a shape of one color moves across a light background for 4 to 10 seconds, and the rest of the video is empty background. The vocabulary is 5 colors × 4 shapes × 2 directions = 40 events. Each event has one caption, for example “a green circle moving from left to right”.

  • Test library: 10 videos of 64 seconds at 1 frame per second (640 frames). Each video holds 4 events; every one of the 40 events appears exactly once. The events cover 247 of the 640 seconds; they last 6.2 seconds on average.
  • Queries: the 40 captions. Each has exactly one correct moment in the whole library.
  • Training clips (only for the temporal model in Step 6): 600 short clips of 4, 8 or 16 frames, rendered with a different random seed. They never appear in the test library.

The hard part is built in. For every object, the library holds both directions, for example a green circle moving left to right and a green circle moving right to left. A system that ignores time can find the object but must guess the direction.

Everything is deterministic: rendering uses numpy.random.default_rng(214) (test) and default_rng(7) (training); SigLIP runs in eval mode on CPU; training uses torch.manual_seed. Running the scripts twice gave bit-identical embeddings.

Step 1: the pipeline

Figure 1 shows the whole system. The top row runs offline, once per video. The bottom row runs online, once per query. They meet in the index.

offline, once per video videodecode 1 fps SigLIP image768-d per frame segments L, Spool: mean / model FAISS index + metadataid → (video, start, end) online, once per query text query“a green circle…” SigLIP textpad to 64 tokens search top-kinner product ranked momentsvideo 8, 52–60 s, score 0.148 evaluation: compare the ranked moments with the labeled events → R@1, R@5, R@10
Figure 1. The lab system. Top row (key color): the index vectors, computed offline. Middle row (query color): the query vector, computed per search. The metadata returned for each hit (value color) turns an index id into a video and a time span.

Three design choices decide almost everything: the encoder (here SigLIP, frozen), the unit of retrieval (here segments of \(L\) seconds with stride \(S\)), and the pooling that turns the frames of a segment into one vector (mean, best frame, or a learned temporal model). Steps 2 to 6 build and measure each one.

Step 2: embed frames and queries with SigLIP

Every frame goes through the image encoder; every caption through the text encoder. We normalize both to length 1, so the dot product is the cosine similarity:

\[ c_j = \mathbf{t}^\top \mathbf{z}_j, \qquad \lVert \mathbf{t} \rVert = \lVert \mathbf{z}_j \rVert = 1, \qquad p_j = \sigma\big(e^{t'} c_j + b\big) \tag{214.1} \]
Read the symbols
\(\mathbf{z}_j\)
the SigLIP embedding of frame \(j\) (768 numbers, length 1)
\(\mathbf{t}\)
the SigLIP embedding of the query text
\(c_j\)
cosine similarity of frame \(j\) and the query
\(e^{t'}, b\)
SigLIP’s learned scale and bias; the paper initializes \(t' = \log 10\) and \(b = -10\) [Certain]. In this checkpoint \(e^{t'} = 117.33\) and \(b = -12.93\) [Certain: read from the model]
\(p_j\)
the model’s probability that frame and text match; \(\sigma\) is the logistic sigmoid

Two practical details. First, the SigLIP text encoder was trained with every caption padded to 64 tokens, and the Hugging Face documentation says to pass padding="max_length" for that reason. [Certain] With other padding you get different text vectors. Second, SigLIP cosines are small. In our library the best frame–caption cosines are about 0.15, and the whole range is about −0.07 to 0.19. That is normal: with scale 117.33 and bias −12.93, a cosine of 0.110 already gives \(p = 0.5\). Do not compare raw cosine thresholds across models.

Worked example: what a cosine of 0.148 means

In video 8, the segment from 52 to 60 s (the green circle moving left to right) has cosine 0.1484 with the query “a green circle moving from left to right”. Then \(117.33 \times 0.1484 - 12.93 = 4.48\) and \(\sigma(4.48) = 0.989\). The segment from 8 to 16 s shows the green circle moving the other way; its cosine is 0.1388, which gives \(p = 0.974\). Both are “yes” with high confidence. The model sees the green circle, but direction barely moves the score.

One frame per second is enough here, because each event lasts at least 4 seconds. For real video, 1 fps is a common starting point for retrieval (CLIP4Clip also samples 1 frame per second [Certain]); fast actions need more.

Step 3: cut segments and pool

A segment is a window of \(L\) seconds that starts every \(S\) seconds (the stride). If \(S \lt L\), neighboring segments overlap. A video of \(D\) seconds gives

\[ n_{\text{seg}} = \left\lfloor \frac{D - L}{S} \right\rfloor + 1 \tag{214.2} \]

For our 64-second videos with \(L = 8, S = 4\): \(\lfloor 56/4 \rfloor + 1 = 15\) segments per video, 150 in the library. With \(L = 4, S = 2\) it is 31 per video (310 in total); with \(L = 16, S = 8\) it is 7 (70 in total).

Mean pooling

The simplest segment vector is the normalized mean of its frame vectors, as in CLIP4Clip’s “meanP” (session 208). Its score with a query can be written with only frame cosines and frame–frame similarities:

\[ s_k = \mathbf{t}^\top \frac{\bar{\mathbf{z}}_k}{\lVert \bar{\mathbf{z}}_k \rVert} = \frac{\frac{1}{L}\sum_{j \in k} c_j}{\sqrt{\frac{1}{L^2} \sum_{i \in k} \sum_{j \in k} \mathbf{z}_i^\top \mathbf{z}_j}} \tag{214.3} \]
Read the symbols
\(k\)
one segment; \(j \in k\) are its \(L\) frames
\(\bar{\mathbf{z}}_k\)
the mean of the frame vectors of segment \(k\)
numerator
the average frame cosine with the query
denominator
the length of the mean vector: near 1 if the frames look alike, smaller if they differ (then normalization lifts the score)

Formula (214.3) is how Figure 2 computes scores in your browser: it stores the \(64 \times 64\) frame similarity matrix of one real video and the frame cosines of six queries, and it gives exactly the same numbers as the Python code.

Worked example: one segment score

Segment [52, 60) of video 8 and the query “a green circle moving from left to right”. The eight frame cosines are 0.1482, 0.1486, 0.1479, 0.1479, 0.1479, 0.1456, 0.1477, 0.1495; their mean is 0.14789. The frames look almost the same to SigLIP, so the mean vector has length 0.99679. The segment score is \(0.14789 / 0.99679 = 0.1484\).

Best-frame scoring

A text-conditioned alternative with no training is to score a segment by its best frame, \(\max_{j \in k} c_j\). You index frames, not segments, and map each frame hit to the segments that contain it. It helps when the query names a short part of a long segment. It has the same blind spot as the mean: a single frame shows no motion.

Query
\(L\)
Figure 2. Real SigLIP scores on video 8 of the test library. Top: its four events (→ = left to right, ← = right to left); the event the query asks for is dark, its opposite-direction twin has a red outline. Middle: the cosine of each frame with the query. Bottom: the segments for the chosen \(L\) and \(S\), shaded by their score from formula (214.3) or by their best frame.
How to read Figure 2

Start with “green ● →”, \(L = 8\), \(S = L/2\): 15 segments, and the top-1 is [52, 60) with 0.1484, a hit. Now click “green ● ←”. The correct event is now the one at 9–19 s, but the top-1 is still [52, 60) (0.1505), a miss; the best segment on the correct event scores only 0.1412. Both queries prefer the same segment: the scores follow the object, not the direction. Try “red ▲ →”: its correct event is in another video, and the best segment here is its twin. Set \(L = 32\): scores drop to about 0.09 or less, because empty frames dilute the mean. Set \(L = 2\): the curve is followed closely, and the green circle queries hit or miss by about 0.001. With “best frame”, two overlapping segments that share the best frame get the same score; the tie goes to the earlier one, which can be a miss (try “purple ● →”).

Algorithm 1: build the segment index
  1. input: videos, segment length \(L\), stride \(S\), image encoder \(\phi\)
  2. vectors \(\leftarrow\) [ ];  meta \(\leftarrow\) [ ]
  3. for each video \(v\) do
  4. frames \(\leftarrow\) decode \(v\) at 1 fps;  \(Z \leftarrow\) normalize(\(\phi\)(frames)) // \((D, 768)\), batch the frames
  5. for \(a = 0, S, 2S, \dots\) while \(a + L \le D\) do // formula (214.2)
  6. \(\mathbf{x} \leftarrow\) normalize(pool(\(Z[a : a+L]\))) // mean, or the temporal model
  7. vectors.append(\(\mathbf{x}\));  meta.append((\(v\), \(a\), \(a + L\)))
  8. index \(\leftarrow\) IndexIDMap(IndexFlatIP(768));  index.add_with_ids(vectors, \(0, \dots, n-1\))
  9. return index, meta // save both; the index alone does not know times

Step 4: index with FAISS and search

FAISS (Facebook AI Similarity Search; Johnson et al., 2017) is a library for nearest-neighbor search over dense vectors. [Certain] For a library of our size we use the exact index IndexFlatIP: it stores all vectors and computes the inner product of the query with every one of them. For unit vectors, the inner product is the cosine, so the ranking is exactly formula (214.3). Wrapping it in IndexIDMap lets us attach our own 64-bit ids to the vectors; a separate table maps each id to (video, start, end).

Algorithm 2: answer one query
  1. input: query text, index, meta, \(k\)
  2. \(\mathbf{t} \leftarrow\) normalize(SigLIP_text(query, padding = max_length))
  3. scores, ids \(\leftarrow\) index.search(\(\mathbf{t}\), \(k'\)) // \(k' \gt k\): ask for extra, overlaps will be merged
  4. hits \(\leftarrow\) [ (meta[id], score) for id, score ]
  5. for each hit in score order: drop it if it overlaps a kept hit of the same video by more than half // temporal non-maximum suppression
  6. return the first \(k\) kept hits as (video, start, end, score)

Our evaluation in Step 5 does not merge hits, so that the numbers match the plain index. In a product you should merge: with 50% overlap, one event usually appears in two neighboring segments, and they fill two of the top 5 places.

How big does the index get?

\[ \text{bytes} = n_{\text{seg}} \cdot d \cdot b \tag{214.4} \]
Read the symbols
\(n_{\text{seg}}\)
the number of segments in the whole library
\(d\)
the embedding size, 768 for SigLIP base
\(b\)
bytes per number: 4 for float32, 2 for float16; product quantization (PQ) stores a code of a few bytes per vector instead

Our library: \(150 \times 768 \times 4 = 460{,}800\) bytes (segments × dimensions × bytes per float). Tiny. Now take 1,000 hours of video with \(L = 8, S = 4\): about \(3{,}600{,}000 / 4 = 900{,}000\) segments, which is 2.76 GB in float32, 1.38 GB in float16, or 57.6 MB with 64-byte PQ codes. A frame-level index at 1 fps would hold 3.6 million vectors, 11.06 GB in float32. These are simple products, so check them for your own numbers.

Approximate search: IVF

An exact index compares the query with every vector. That is fine up to a few million vectors on a CPU [Likely], but it grows linearly. An inverted file index (IVF) first clusters the vectors with k-means into nlist lists. At query time it compares the query with the nlist centroids, opens only the nprobe closest lists, and searches inside them. If the lists have similar sizes, it scans about

\[ \text{scanned} \approx \frac{\text{nprobe}}{\text{nlist}} \cdot N \ +\ \text{nlist} \tag{214.5} \]

vectors instead of \(N\). The price is recall of the index: a true neighbor that sits in a list you did not open is lost. In FAISS, IndexIVFFlat has nprobe = 1 by default [Certain: checked with faiss 1.13], which is often too low. Figure 4 lets you see the trade-off.

4
Figure 4. An IVF index built in your browser: 4,000 unit vectors in 16 dimensions drawn around 40 cluster centers (seeded), spherical k-means with 10 iterations, and 200 queries. Recall@10 is the share of the exact top-10 that the IVF search also returns. Each point is one value of nprobe (1, 2, 4, …).
How to read Figure 4

With nlist = 64 and nprobe = 1, the search scans only 1.85% of the vectors but finds 71.15% of the true top-10. Doubling nprobe to 2 gives 89.45%; at nprobe = 8 it is 98.80% for 12.73% of the work. With nlist = 16, every list is bigger, so the same recall costs more scanning: nprobe = 2 gives 89.40% but scans 12.88%. Both curves reach 100% when every list is opened, which is just an exact search. On real embeddings, measure this curve on your own data: the shape depends on how clustered the vectors are.

For millions of segments you would use IVF with PQ codes (IndexIVFPQ) or a graph index such as HNSW (IndexHNSWFlat; its default efSearch is 16 in faiss 1.13 [Certain: checked]). Session session 208 explains both. The rule does not change: measure index recall against an exact search on a sample of queries before you trust the fast index.

Step 5: measure recall@k

For each query we sort all segments by score and find the rank \(r_i\) of the first correct segment. With overlapping segments of different lengths, “correct” needs a rule. We use: a segment is a hit for an event if they are in the same video and

\[ \operatorname{hit}(k, e) \iff \lvert k \cap e \rvert \ \ge\ \tfrac{1}{2} \min\big(\lvert k \rvert, \lvert e \rvert\big) \tag{214.6} \]
Read the symbols
\(k, e\)
a segment and the labeled event, as time intervals [start, end)
\(\lvert k \cap e \rvert\)
their overlap in seconds
\(\min(\lvert k \rvert, \lvert e \rvert)\)
the shorter of the two lengths: a long segment must contain at least half of the event; a short segment must lie at least half inside it

Then recall at \(K\) over \(Q\) queries is

\[ \text{R@}K = \frac{100}{Q} \sum_{i=1}^{Q} \mathbb{1}\big[r_i \le K\big] \tag{214.7} \]

With \(Q = 40\), each query is worth 2.5 points. Change the hit rule and the numbers change, so always state it next to your results. Temporal grounding papers use a temporal IoU threshold instead (R1@IoU, session 213).

Algorithm 3: recall@k with a hit rule
  1. input: queries with labeled events \((v_i, e_i)\), ranked id lists, meta
  2. for each query \(i\) do
  3. \(r_i \leftarrow\) 1 + position of the first id whose meta \((v, a, b)\) has \(v = v_i\) and hit\(([a, b), e_i)\) // formula (214.6)
  4. \(\text{R@}K \leftarrow 100 \cdot \operatorname{mean}_i \mathbb{1}[r_i \le K]\) for \(K = 1, 5, 10\);  \(\text{MdR} \leftarrow \operatorname{median}_i(r_i)\) // formula (214.7)

Results for mean pooling and best-frame scoring, with 50% overlap (\(S = L/2\)), on all 40 queries:

Pooling\(L\)segmentsR@1R@5R@10MdR
mean4 s31050.0100.0100.01.5
mean8 s15045.082.597.52
mean16 s7037.565.087.52
best frame4 s31047.582.5100.03
best frame8 s15040.0100.0100.02
best frame16 s7042.5100.0100.02

Read it in two parts. R@1 sits near 50% for both methods at every length: the system finds the object and then guesses the direction. At \(L = 8\), 17 of the 22 top-1 misses of mean pooling are the opposite-direction twin. R@5 falls with longer segments for the mean (100 → 82.5 → 65.0): empty frames dilute the event, as in the dilution formula of session 208. Best frame does not dilute, so its R@5 stays at 100% for 8 and 16 seconds.

Step 6: a small temporal model

Why can mean pooling never learn direction? Because the mean does not depend on the order of the frames. For any reordering \(\pi\) of the frames,

\[ \frac{1}{L} \sum_{j=1}^{L} \mathbf{z}_{\pi(j)} = \frac{1}{L} \sum_{j=1}^{L} \mathbf{z}_j \tag{214.8} \]

A circle moving left to right and the same circle moving right to left are the same set of frames in opposite order, so they get (almost) the same vector. Best-frame scoring is order-blind in the same way. To see motion, the pooling must know where each frame is in time.

We use the CLIP4Clip “seqTransf” idea at a small scale: add a learned position embedding to each frame vector, run one transformer encoder layer over the frames of the segment, add the result back to the frames (a residual connection), and then mean-pool:

\[ H = \operatorname{Enc}\big(Z + P_{1:L}\big), \qquad \mathbf{v} = \operatorname{normalize}\Big( \frac{1}{L} \sum_{j=1}^{L} (\mathbf{z}_j + \mathbf{h}_j) \Big), \qquad \mathbf{t}' = \operatorname{normalize}(\mathbf{t} + W\mathbf{t}) \tag{214.9} \]
Read the symbols
\(Z\)
the \(L \times 768\) frame embeddings of one segment (SigLIP, frozen)
\(P_{1:L}\)
learned position embeddings, one per position, starting at zero
\(\operatorname{Enc}\)
one pre-norm transformer encoder layer, 8 heads, feed-forward size 1,024
\(\mathbf{h}_j\)
the output for frame \(j\): what it learned from the other frames, including their order
\(W\)
a \(768 \times 768\) linear layer on the text side, initialized to zero, so \(\mathbf{t}' = \mathbf{t}\) at the start

Why also adapt the text? The SigLIP text vectors of “… moving from left to right” and “… moving from right to left” have cosine 0.993 for the red circle. If the text side cannot separate them, a perfect video side does not help. The small residual \(W\) lets training stretch that difference.

The loss

Each training batch has \(B = 32\) clips of 32 different events with their captions. Choosing different events matters: two clips of the same event in one batch would count as negatives for each other (a false negative). We use the symmetric contrastive loss with temperature \(\tau = 0.05\):

\[ \mathcal{L} = \frac{1}{2B} \sum_{i=1}^{B} \left[ -\log \frac{e^{\mathbf{v}_i^\top \mathbf{t}'_i/\tau}}{\sum_{j} e^{\mathbf{v}_i^\top \mathbf{t}'_j/\tau}} \ -\ \log \frac{e^{\mathbf{v}_i^\top \mathbf{t}'_i/\tau}}{\sum_{j} e^{\mathbf{v}_j^\top \mathbf{t}'_i/\tau}} \right] \tag{214.10} \]
Read the symbols
first term
video → text: each clip must pick its own caption among the 32 captions
second term
text → video: each caption must pick its own clip among the 32 clips. This is exactly the retrieval task
\(\tau\)
the temperature: smaller values make the softmax sharper

Training runs 1,500 steps of AdamW (learning rate \(3 \cdot 10^{-4}\), weight decay 0.01), with the segment length drawn from {4, 8, 16} each step. It takes about 1.5 minutes on a laptop CPU, because SigLIP is frozen and its features are computed once.

\(L\)
Figure 3. Measured results of the lab code on the 40 test queries (segments with \(S = L/2\)). Bars: R@1, R@5 and R@10 for mean pooling, best frame, and the temporal model of formula (214.9) trained with only the video→text term of (214.10) or with both terms (seed 0). Strip: the rank of every query for the selected method; hover a cell to see the query.
How to read Figure 3

At \(L = 8\), mean pooling has 18 green cells out of 40 (R@1 45.0%), and 15 of its 22 misses are amber (rank 2 to 5): usually the twin came first. The symmetric temporal model turns 38 of 40 green (95.0%). The model trained with only the video→text term reaches 65.0%. At \(L = 4\) and \(L = 16\) the gap shrinks (77.5% and 67.5% for symmetric), and at \(L = 16\) the v→t model is even ahead (72.5%). These are single runs on 40 queries; read the next table before you draw conclusions.

How stable is this? Three seeds

We trained each variant with seeds 0, 1 and 2 (only the model initialization, dropout and batch order change). R@1 in percent:

Model\(L\) = 4 s\(L\) = 8 s\(L\) = 16 s
mean pooling (no training)50.045.037.5
temporal, v→t loss only: seeds 0, 1, 262.5, 62.5, 80.0 (mean 68.3)65.0, 50.0, 80.0 (mean 65.0)72.5, 57.5, 80.0 (mean 70.0)
temporal, symmetric loss: seeds 0, 1, 277.5, 87.5, 80.0 (mean 81.7)95.0, 80.0, 82.5 (mean 85.8)67.5, 80.0, 80.0 (mean 75.8)

Three conclusions are safe. (1) The temporal model beats mean pooling in all 18 runs, by 5 to 50 points of R@1. (2) On average the symmetric loss is better than the video→text loss alone at every length, but single runs overlap: at \(L = 8\) the best v→t run (80.0) equals the worst symmetric run (80.0). (3) The spread between seeds is up to 15 points. With 40 queries, the 95% confidence interval of a recall near 85% is about ±11 points (\(1.96\sqrt{0.85 \cdot 0.15 / 40} = 0.111\)). A real comparison needs more queries and several seeds.

Why should the text→video term help? Retrieval compares one caption against many segments. The video→text term only teaches each clip to prefer its caption over other captions; the scores of different clips for the same caption are never compared, so they need not be calibrated against each other. The text→video term trains exactly that comparison. [Likely: this is the usual explanation; our three seeds support it on average but do not prove it.]

The full program

Three files. build_library.py renders the videos and runs SigLIP (about 20 minutes on a laptop CPU, mostly the 5,628 training frames). search.py builds FAISS indexes and computes recall; it imports only numpy and faiss. temporal.py trains the temporal model and writes its segment and query vectors. Keep FAISS and PyTorch in separate processes on macOS: with the pip wheels we used (torch 2.8, faiss-cpu 1.13), importing both in one process aborted with “OMP: Error #15” (two copies of the OpenMP runtime). [Certain: seen in this lab]

# build_library.py: render a toy video library, embed every frame and query with SigLIP
import itertools, json, numpy as np, torch
from PIL import Image, ImageDraw
from transformers import AutoModel, AutoProcessor
torch.manual_seed(0)
COLORS = {"red": (220, 30, 30), "green": (30, 160, 40), "blue": (30, 60, 220),
          "yellow": (240, 210, 20), "purple": (140, 40, 180)}
SHAPES = ["circle", "square", "triangle", "star"]
EVENTS = [(c, s, d) for c in COLORS for s in SHAPES for d in ("lr", "rl")]      # 40 events
BGS = [(235, 235, 235), (225, 235, 245), (245, 235, 220), (230, 245, 230), (240, 228, 240)]
def caption(c, s, d):
    return f"a {c} {s} moving from " + ("left to right" if d == "lr" else "right to left")

def frame(rng, bg, shape=None, color=None, x=0.5, size=0.18):     # one 224x224 RGB frame
    a = np.clip(np.array(bg)[None, None] + rng.normal(0, 6, (224, 224, 3)), 0, 255)
    im = Image.fromarray(a.astype(np.uint8)); dr = ImageDraw.Draw(im)
    cx, cy, r, col = 224 * x, 112, 224 * size, COLORS.get(color)
    if shape == "circle":   dr.ellipse([cx - r, cy - r, cx + r, cy + r], fill=col)
    if shape == "square":   dr.rectangle([cx - r, cy - r, cx + r, cy + r], fill=col)
    if shape == "triangle": dr.polygon([(cx, cy - r), (cx - r, cy + r), (cx + r, cy + r)], fill=col)
    if shape == "star":
        pts = [(cx + (r if k % 2 == 0 else 0.45 * r) * np.cos(np.pi / 2 + k * np.pi / 5),
                cy - (r if k % 2 == 0 else 0.45 * r) * np.sin(np.pi / 2 + k * np.pi / 5)) for k in range(10)]
        dr.polygon(pts, fill=col)
    return im

def event_frames(rng, c, s, d, dur, bg):          # 1 frame per second, the shape crosses the screen
    size = rng.uniform(0.12, 0.2)
    xs = np.linspace(0.2, 0.8, dur)
    return [frame(rng, bg, s, c, x, size) for x in (xs if d == "lr" else xs[::-1])]

def make_video(rng, events, D=64):                # D seconds = D frames at 1 fps
    bg = BGS[rng.integers(len(BGS))]
    durs = [int(rng.integers(4, 11)) for _ in events]                      # 4 to 10 s per event
    free = D - sum(durs)
    gaps = np.diff(np.concatenate([[0], np.sort(rng.integers(0, free + 1, len(events))), [free]]))
    frames, spans = [], []
    for k, e in enumerate(events):
        frames += [frame(rng, bg) for _ in range(gaps[k])]                 # empty background
        st = len(frames); frames += event_frames(rng, *e, durs[k], bg)
        spans.append((st, len(frames)))                                     # [start, end) in seconds
    frames += [frame(rng, bg) for _ in range(gaps[-1])]
    return frames, spans

model = AutoModel.from_pretrained("google/siglip-base-patch16-224").eval()
proc = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")
@torch.no_grad()
def embed_images(ims, bs=64):
    z = torch.cat([model.get_image_features(**proc(images=ims[i:i + bs], return_tensors="pt"))
                   for i in range(0, len(ims), bs)])
    return torch.nn.functional.normalize(z, dim=-1).numpy()                # (n, 768), unit length
@torch.no_grad()
def embed_texts(texts):                           # SigLIP was trained with padding="max_length" (64 tokens)
    t = model.get_text_features(**proc(text=texts, padding="max_length", return_tensors="pt"))
    return torch.nn.functional.normalize(t, dim=-1).numpy()

if __name__ == "__main__":
    rng = np.random.default_rng(214)              # test library: 10 videos, 4 events each, all 40 events once
    perm = rng.permutation(len(EVENTS)); Z, meta = [], []
    for v in range(10):
        evs = [EVENTS[i] for i in perm[4 * v:4 * v + 4]]
        frames, spans = make_video(rng, evs)
        Z.append(embed_images(frames)); meta.append({"events": evs, "spans": spans})
    T = embed_texts([caption(*e) for e in EVENTS])
    np.savez("library.npz", Z=np.stack(Z), T=T)                            # Z: (10, 64, 768), T: (40, 768)
    json.dump({"meta": meta, "events": EVENTS}, open("library.json", "w"), default=int)
    rng = np.random.default_rng(7)                # training clips: other renders, never seen at test time
    clips, labels = [], []
    for n in range(600):
        L = int(rng.choice([4, 8, 16])); ei = int(rng.integers(len(EVENTS)))
        dur = int(rng.integers(max(3, L // 2), min(L, 10) + 1)); bg = BGS[rng.integers(len(BGS))]
        off = int(rng.integers(0, L - dur + 1))
        fr = ([frame(rng, bg) for _ in range(off)] + event_frames(rng, *EVENTS[ei], dur, bg)
              + [frame(rng, bg) for _ in range(L - dur - off)])
        clips.append(embed_images(fr)); labels.append(ei)
    np.savez("train.npz", **{f"c{n}": c for n, c in enumerate(clips)}, y=np.array(labels))
# search.py: cut segments, pool, index with FAISS, search, measure recall@k (numpy + faiss only)
import json, sys, numpy as np, faiss
lib = np.load("library.npz"); J = json.load(open("library.json"))
Z, T = lib["Z"].astype("float32"), lib["T"].astype("float32")          # (10, 64, 768), (40, 768)
EVENTS = [tuple(e) for e in J["events"]]
gt = {tuple(e): (v, tuple(sp)) for v, m in enumerate(J["meta"]) for e, sp in zip(m["events"], m["spans"])}

def segments(D, L, S):                            # windows [a, a+L) with stride S
    return [(a, a + L) for a in range(0, D - L + 1, S)]

def is_hit(seg, ev):                              # overlap >= half of the shorter interval
    inter = max(0, min(seg[1], ev[1]) - max(seg[0], ev[0]))
    return inter >= 0.5 * min(seg[1] - seg[0], ev[1] - ev[0])

def build_index(vecs, ids):
    index = faiss.IndexIDMap(faiss.IndexFlatIP(vecs.shape[1]))         # exact inner product
    index.add_with_ids(vecs, np.asarray(ids, dtype="int64"))
    return index

def recall(rank_lists, segs_of_id, ks=(1, 5, 10)):
    ranks = []
    for e, ids in zip(EVENTS, rank_lists):
        v, ev = gt[e]
        ranks.append(next(r + 1 for r, i in enumerate(ids) if segs_of_id[i][0] == v and is_hit(segs_of_id[i][1:], ev)))
    ranks = np.array(ranks)
    return {f"R@{k}": 100 * float(np.mean(ranks <= k)) for k in ks} | {"MdR": float(np.median(ranks))}, ranks

def run(L, S, method, seg_vecs=None, queries=None):
    segs_of_id = [(v, a, b) for v in range(Z.shape[0]) for a, b in segments(Z.shape[1], L, S)]
    if method == "mean":                          # one vector per segment: the normalized mean frame
        X = np.stack([Z[v, a:b].mean(0) for v, a, b in segs_of_id]); faiss.normalize_L2(X)
        index = build_index(X, range(len(segs_of_id)))
        _, I = index.search(T, len(segs_of_id))
    elif method == "maxframe":                    # index frames, a segment scores its best frame
        index = build_index(Z.reshape(-1, Z.shape[2]), range(Z.shape[0] * Z.shape[1]))
        Sf, If = index.search(T, Z.shape[0] * Z.shape[1])
        I = []
        for q in range(len(T)):
            best = {}
            for s, f in zip(Sf[q], If[q]):        # frames come best first
                v, t = divmod(int(f), Z.shape[1])
                for j, (vv, a, b) in enumerate(segs_of_id):
                    if vv == v and a <= t < b and j not in best: best[j] = s
            I.append(sorted(best, key=lambda j: -best[j]))
    else:                                         # vectors written by temporal.py
        index = build_index(seg_vecs, range(len(segs_of_id)))
        _, I = index.search(queries, len(segs_of_id))
    return recall(I, segs_of_id)

if __name__ == "__main__":
    for L, S in [(4, 2), (8, 4), (16, 8)]:
        for m in ["mean", "maxframe"]:
            print(L, S, m, run(L, S, m)[0])
# temporal.py: a small transformer over the frames of a segment (CLIP4Clip "seqTransf" style)
import json, sys, numpy as np, torch, torch.nn as nn, torch.nn.functional as F
SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
SYMMETRIC = (sys.argv[2] != "v2t") if len(sys.argv) > 2 else True
torch.manual_seed(SEED); np.random.seed(SEED)
lib = np.load("library.npz"); tr = np.load("train.npz")
Z, T = torch.tensor(lib["Z"]), torch.tensor(lib["T"])                    # frozen SigLIP features
y = tr["y"]; clips = [tr[f"c{n}"] for n in range(len(y))]

class Temporal(nn.Module):                        # video side: frames -> one segment vector
    def __init__(self, d=768, max_len=32):
        super().__init__()
        self.pos = nn.Parameter(torch.zeros(max_len, d))                   # learned position embeddings
        layer = nn.TransformerEncoderLayer(d, 8, 1024, dropout=0.1, batch_first=True, norm_first=True)
        self.enc = nn.TransformerEncoder(layer, num_layers=1)
    def forward(self, z):                         # z: (B, L, d)
        h = self.enc(z + self.pos[: z.shape[1]])
        return F.normalize((z + h).mean(1), dim=-1)                        # residual, then mean pool

class TextAdapter(nn.Module):                     # text side: t + W t, starts as the identity
    def __init__(self, d=768):
        super().__init__(); self.lin = nn.Linear(d, d)
        nn.init.zeros_(self.lin.weight); nn.init.zeros_(self.lin.bias)
    def forward(self, t): return F.normalize(t + self.lin(t), dim=-1)

vm, tm = Temporal(), TextAdapter()
opt = torch.optim.AdamW([*vm.parameters(), *tm.parameters()], lr=3e-4, weight_decay=0.01)
by = {}                                           # (event, length) -> clip indices
for i, c in enumerate(clips): by.setdefault((int(y[i]), len(c)), []).append(i)
tau = 0.05
for step in range(1500):
    L = int(np.random.choice([4, 8, 16]))
    evs = [e for e in range(len(T)) if (e, L) in by]
    evs = list(np.random.choice(evs, min(32, len(evs)), replace=False))   # 32 DIFFERENT events: no false negatives
    z = torch.tensor(np.stack([clips[int(np.random.choice(by[(e, L)]))] for e in evs]))
    logits = vm(z) @ tm(T[evs]).T / tau           # (32 clips, 32 captions)
    target = torch.arange(len(evs))
    loss = F.cross_entropy(logits, target)        # video -> text
    if SYMMETRIC:
        loss = (loss + F.cross_entropy(logits.T, target)) / 2              # + text -> video
    opt.zero_grad(); loss.backward(); opt.step()

vm.eval(); tm.eval(); out = {}
with torch.no_grad():
    out["T"] = tm(T).numpy()
    for L, S in [(4, 2), (8, 4), (16, 8)]:
        segs = torch.stack([Z[v, a:a + L] for v in range(Z.shape[0]) for a in range(0, Z.shape[1] - L + 1, S)])
        out[f"X{L}_{S}"] = vm(segs).numpy()
np.savez(f"temporal_s{SEED}_{'sym' if SYMMETRIC else 'v2t'}.npz", **out)
# eval_temporal.py: python eval_temporal.py temporal_s0_sym.npz
import sys, numpy as np; from search import run
f = np.load(sys.argv[1])
for L, S in [(4, 2), (8, 4), (16, 8)]:
    print(L, S, run(L, S, "temporal", f[f"X{L}_{S}"].astype("float32"), f["T"].astype("float32"))[0])
# python temporal.py 0 sym  && python eval_temporal.py temporal_s0_sym.npz
# 4 2 {'R@1': 77.5, 'R@5': 100.0, 'R@10': 100.0, 'MdR': 1.0}
# 8 4 {'R@1': 95.0, 'R@5': 100.0, 'R@10': 100.0, 'MdR': 1.0}
# 16 8 {'R@1': 67.5, 'R@5': 92.5, 'R@10': 97.5, 'MdR': 1.0}
# python temporal.py 0 v2t  && python eval_temporal.py temporal_s0_v2t.npz
# 4 2 {'R@1': 62.5, ...}   8 4 {'R@1': 65.0, ...}   16 8 {'R@1': 72.5, ...}

Running python search.py prints the mean and best-frame rows of the table in Step 5. To use real videos, replace make_video with a decoder (for example PyAV or ffmpeg -vf fps=1) and replace the generated captions and spans with your labeled moments; the rest does not change.

Where this is used

ApplicationWhat this lab’s pieces do there
Search in a video archive (news, sports, stock footage)Segment index plus metadata returns moments, not whole files; merge overlapping hits before showing results
Video-RAGThe same index retrieves segments whose frames (and transcript) are then given to a VLM to answer a question (session 210)
Key-moment detectionSegment scores against event prompts, computed exactly as in Figure 2, become a score curve over time; the next lab adds non-maximum suppression and a VLM check
Action recognition and direction-sensitive events“Opening” vs “closing”, “entering” vs “leaving” need order: mean pooling fails as in Step 5; a temporal model or a video encoder is needed
Near-duplicate detectionThe same FAISS machinery with video–video similarity instead of text–video (session 209)
Data curation for trainingText queries over a large unlabeled pool find candidate clips to label or to filter

Common mistakes

  • Forgetting to normalize. IndexFlatIP ranks by inner product. Without unit vectors, long vectors win, and a pooled mean is shorter than its frames. Normalize after pooling, and normalize queries.
  • Default text padding with SigLIP. The model was trained with captions padded to 64 tokens; use padding="max_length".
  • Using a fixed cosine threshold from another model. SigLIP cosines of about 0.15 are strong matches here; CLIP cosines live in a different range. Calibrate thresholds per model, or use ranks.
  • Losing the metadata. A FAISS id means nothing without the table that maps it to (video, start, end). Save both together, and rebuild both together.
  • No hit rule, or an unreported one. With overlapping segments, “correct” must be defined (formula 214.6). Different rules give different R@k.
  • Trusting one run on a small test set. 40 queries give ±11 points of noise at 85%; seeds moved our R@1 by up to 15 points.
  • Only one half of the contrastive loss, or repeated events in a batch. Train text→video as well, and keep false negatives out of the batch.
  • Measuring an approximate index without an exact baseline. IVF with the default nprobe = 1 lost about 29% of the true top-10 in Figure 4.

Interview questions

Why can mean pooling of frame embeddings not tell “left to right” from “right to left”?basic
The mean is invariant to the order of the frames (formula 214.8). The two motions show the same set of frames in opposite order, so they give almost the same vector. Any order-free pooling (mean, max, best frame) has this blind spot. In our lab, 17 of the 22 top-1 misses of mean pooling at 8-second segments were the opposite-direction twin. You need position information: a temporal model over frames, or a video encoder with temporal attention.
You use FAISS IndexFlatIP. What must be true about the vectors for this to be cosine search?basic
Both the stored vectors and the query must have length 1. Then the inner product equals the cosine. If you mean-pool frames, the mean is shorter than 1 (less so when frames look alike), so normalize after pooling, not only before. faiss.normalize_L2 does it in place on a float32 array.
How do you choose the segment length \(L\) and the stride \(S\)?core
Match \(L\) to the length of the events people search for and to the clip length the encoder was trained on. Too long dilutes short events: in our lab, mean-pooling R@5 dropped from 100% at 4 s to 65% at 16 s. Too short multiplies the index (formula 214.2), loses context for actions, and floods the results with near-duplicates. Overlap (\(S = L/2\) is common) avoids cutting events at a boundary but doubles the index. Cutting at shot boundaries is a good alternative. Measure R@k on a labeled set for two or three choices.
What exactly does R@5 = 82.5% mean in this lab?core
For 33 of the 40 queries, at least one of the top 5 segments is a hit, where hit means same video and overlap of at least half of the shorter interval (formula 214.6). It depends on the gallery (150 segments here), on the hit rule, and on overlap: with \(S = L/2\) one event often has two hit segments, while its twin also has two, so they compete for the top places. Always report gallery size and hit rule with R@k.
SigLIP gives a cosine of 0.15 for a perfect match. Is something wrong?core
No. SigLIP is trained with a sigmoid loss and a learned scale and bias. In this checkpoint the scale is 117.33 and the bias −12.93, so \(p = \sigma(117.33 \cdot 0.15 - 12.93) \approx 0.99\), and a cosine of 0.110 already means \(p = 0.5\). Ranking only needs the order of the scores. If you need a yes/no threshold, apply the scale and bias, and calibrate on labeled data.
Your temporal model gets 95% R@1 with one seed. Is it better than mean pooling? Is the symmetric loss better than video→text only?advanced
Better than mean pooling: yes, in all 18 runs, by 5 to 50 points. Symmetric vs v→t: probably, because it wins on average at every length (85.8 vs 65.0 at 8 s), but single runs overlap and seeds move R@1 by up to 15 points. With 40 queries the 95% interval near 85% is about ±11 points. To be sure: more queries, several seeds, a paired test on the same queries, and report mean and spread.
Why should the text→video term of the contrastive loss matter for retrieval?advanced
Retrieval ranks many segments for one caption. The video→text term only compares captions for a fixed clip, so it never forces the scores of different clips for the same caption to be comparable. The text→video term trains exactly that comparison: for caption i, clip i must beat the other clips. CLIP and CLIP4Clip use both terms. [Likely] Also keep repeated events out of a batch, or their clips become false negatives.
Design the index for 1,000 hours of video. What sizes and choices?advanced
With 8-second segments and 4-second stride, about 900,000 segments. With 768-d float32 that is 2.76 GB, 1.38 GB in float16: an exact flat index still fits in RAM on one machine. For 100× more video, use IVF-PQ (for example 64-byte codes: 57.6 MB per 900,000 segments) or HNSW, tune nprobe or efSearch against an exact baseline on sample queries, and re-rank the top candidates with exact vectors or a VLM. Store (video, start, end) per id, merge overlapping hits, and update the index incrementally as new videos arrive.
Best-frame scoring or mean pooling?core
Best frame (index frames, score a segment or video by its top frame) is text-conditioned and does not dilute short events: in our lab its R@5 stayed at 100% for 8- and 16-second segments while the mean fell to 82.5% and 65%. But it indexes every frame (more memory), it is noisy on single bad frames, it ties overlapping segments that share the best frame, and it is as order-blind as the mean. Mean pooling is better when the query describes the whole segment.
Your offline R@k looks great, but users complain. What do you check?core
That the test set looks like real queries (length, vocabulary, language) and real videos (length, domain); that test captions and videos were not in training; that the index in production is the same one you evaluated (same encoder version, same normalization, same nprobe); that overlapping hits are merged so the result page is not five copies of one event; and whether users need moments that a fixed segment grid cuts in half. Then add a labeled set from real query logs.

Exercises

  1. How many segments does one 64-second video give with \(L = 16, S = 4\)?
  2. Estimate the float16 index size for 500 hours of video with \(L = 10, S = 5\) and 768-d vectors.
  3. Implement the merge step of Algorithm 2 and count how many of the top 5 results of mean pooling at \(L = 8\) are duplicates of the same event before and after merging.
  4. Run search.py with best-frame scoring at \(L = 8\). Explain why R@5 reaches 100% while R@1 stays at 40%.
  5. Add queries without a direction, such as “a green circle”. What should count as correct, and how must the evaluation change?
  6. Train the symmetric temporal model with seeds 3 and 4. Report the mean and the range of R@1 at \(L = 8\) over five seeds.
Answerscheck
1. \(\lfloor (64 - 16)/4 \rfloor + 1 = 13\). 2. \(500 \times 3600 / 5 = 360{,}000\) segments; \(360{,}000 \times 768 \times 2 = 552{,}960{,}000\) bytes, about 553 MB. 3. With 50% overlap one event usually covers two neighboring segments with almost equal scores, so duplicates are common; after merging, the top 5 hold more distinct events (count it: the answer depends on your merge threshold). 4. One frame always shows the right object, so a hit always appears within the top 5 (the worst rank is 5), which gives R@5 = 100%; but a single frame carries no direction, so the order between event and twin is a coin flip: R@1 = 40.0%. 5. Both directions of the green circle are correct; the ground truth becomes a set of events, and a query is a hit at rank \(r\) if the segment at \(r\) hits any of them. R@1 should rise close to 100% for order-free pooling. 6. Depends on your runs; seeds 0 to 2 gave 95.0, 80.0 and 82.5. Expect a spread of 10 to 15 points.

Key takeaways

  • A text-to-video system is: frames → SigLIP vectors → segments (214.2) → pooled, normalized vectors → FAISS index with a metadata table → text query → ranked moments.
  • For unit vectors, IndexFlatIP is exact cosine search; mean-pooled segment scores follow formula (214.3). Index memory is \(n_{\text{seg}} \cdot d \cdot b\) (214.4); IVF scans about \(\text{nprobe}/\text{nlist}\) of it (214.5) at a measurable loss of recall.
  • R@\(k\) needs a stated hit rule (214.6, 214.7). With 40 queries, each query is 2.5 points and noise is about ±11 points.
  • Mean and best-frame pooling are order-blind (214.8): on our library they find the object but guess the direction (R@1 45.0% and 40.0% at 8 s).
  • A one-layer temporal transformer with position embeddings over frozen SigLIP frames (214.9), trained with the symmetric loss (214.10), raised R@1 to 85.8% on average over three seeds.

Key formulas

\[ n_{\text{seg}} = \left\lfloor \tfrac{D - L}{S} \right\rfloor + 1, \quad s_k = \mathbf{t}^\top \tfrac{\bar{\mathbf{z}}_k}{\lVert \bar{\mathbf{z}}_k \rVert}, \quad \text{R@}K = \tfrac{100}{Q} \textstyle\sum_i \mathbb{1}[r_i \le K], \quad \mathbf{v} = \operatorname{normalize}\big(\operatorname{mean}_j(\mathbf{z}_j + \mathbf{h}_j)\big) \]

References for this session

  1. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  2. Meta (code). FAISS library
  3. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  4. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  5. Johnson et al. (2017). Billion-scale similarity search with GPUs
  6. Meta (code). FAISS library
  7. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  8. Jégou et al. (2011). Product Quantization for Nearest Neighbor Search
  9. Malkov & Yashunin (2016). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
  10. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models