text-to-video retrieval
In this lab you build a small text-to-video search system and measure it. You cut videos into segments, embed every frame with SigLIP, pool each segment into one vector, index the segments with FAISS, search with text queries, and compute recall@k. Then you train a one-layer temporal model on top of the frozen frames and see why mean pooling cannot tell which way something moves.
What you will learn
- How to turn a video library into a segment index: frame sampling, segment length and stride, pooling, normalization and a metadata table
- How to embed frames and queries with SigLIP correctly (padding to 64 tokens, unit vectors) and how to read its small cosine values
- How to index and search with FAISS (IndexFlatIP, IndexIDMap) and how IVF trades recall for speed through nlist and nprobe
- How to compute recall@k with an explicit hit rule for overlapping segments, and how noisy 40 queries are
- Why mean and best-frame pooling are blind to the order of frames, measured: R@1 near 50% when both directions of a motion exist
- How a one-layer temporal transformer with position embeddings, trained with a symmetric contrastive loss, raises R@1, and how much seeds move the result
Why this matters
In session 208 you learned the ideas: embed frames, pool them, put the vectors in an index, and rank by cosine similarity. In this lab you build the whole system and measure it. You cut videos into segments, embed every frame with SigLIP, pool the frames of each segment into one vector, index the segments with FAISS, search with text queries, and compute recall@k. Then you train a small temporal model and check whether it beats mean pooling.
Every number on this page comes from running the code below on a CPU laptop. Some results will surprise you:
- Mean pooling cannot tell direction. On our library, mean pooling of SigLIP frames gets R@1 = 45.0% with 8-second segments. Most misses are not random: 17 of the 22 misses return the right object moving the wrong way.
- A one-layer transformer over the frames fixes most of it. With the same frozen SigLIP features, the temporal model reaches R@1 = 95.0% (one seed), 85.8% on average over three seeds.
- The loss matters as much as the model. Training the same model with only the video-to-text half of the contrastive loss gave 65.0% on average.
The previous session, Session 213, explained how to score video systems, including R@k. Here you apply it to your own system. The next lab, Session 215, reuses the same segment scores to find moments inside one video.
Setup
You need Python 3.9 or newer and about 1 GB of disk for the model. No GPU is needed.
pip install torch transformers faiss-cpu sentencepiece pillow numpy protobuf
We use google/siglip-base-patch16-224: a ViT-B/16 image encoder at \(224 \times 224\) pixels and a text encoder, both giving 768-dimensional embeddings. [Certain: checked with transformers 4.57] SigLIP (Zhai et al., 2023) is a CLIP-style dual encoder trained with a sigmoid loss instead of a softmax loss, so each image–text pair is scored on its own, as a yes/no question (session 167).
A toy library with known answers
To measure recall you need queries whose correct answer you know exactly. Real benchmarks such as MSR-VTT give that, but they need large downloads. So we generate a small library: a shape of one color moves across a light background for 4 to 10 seconds, and the rest of the video is empty background. The vocabulary is 5 colors × 4 shapes × 2 directions = 40 events. Each event has one caption, for example “a green circle moving from left to right”.
- Test library: 10 videos of 64 seconds at 1 frame per second (640 frames). Each video holds 4 events; every one of the 40 events appears exactly once. The events cover 247 of the 640 seconds; they last 6.2 seconds on average.
- Queries: the 40 captions. Each has exactly one correct moment in the whole library.
- Training clips (only for the temporal model in Step 6): 600 short clips of 4, 8 or 16 frames, rendered with a different random seed. They never appear in the test library.
The hard part is built in. For every object, the library holds both directions, for example a green circle moving left to right and a green circle moving right to left. A system that ignores time can find the object but must guess the direction.
Everything is deterministic: rendering uses numpy.random.default_rng(214) (test) and default_rng(7) (training); SigLIP runs in eval mode on CPU; training uses torch.manual_seed. Running the scripts twice gave bit-identical embeddings.
Step 1: the pipeline
Figure 1 shows the whole system. The top row runs offline, once per video. The bottom row runs online, once per query. They meet in the index.
Three design choices decide almost everything: the encoder (here SigLIP, frozen), the unit of retrieval (here segments of \(L\) seconds with stride \(S\)), and the pooling that turns the frames of a segment into one vector (mean, best frame, or a learned temporal model). Steps 2 to 6 build and measure each one.
Step 2: embed frames and queries with SigLIP
Every frame goes through the image encoder; every caption through the text encoder. We normalize both to length 1, so the dot product is the cosine similarity:
- Read the symbols
- \(\mathbf{z}_j\)
- the SigLIP embedding of frame \(j\) (768 numbers, length 1)
- \(\mathbf{t}\)
- the SigLIP embedding of the query text
- \(c_j\)
- cosine similarity of frame \(j\) and the query
- \(e^{t'}, b\)
- SigLIP’s learned scale and bias; the paper initializes \(t' = \log 10\) and \(b = -10\) [Certain]. In this checkpoint \(e^{t'} = 117.33\) and \(b = -12.93\) [Certain: read from the model]
- \(p_j\)
- the model’s probability that frame and text match; \(\sigma\) is the logistic sigmoid
Two practical details. First, the SigLIP text encoder was trained with every caption padded to 64 tokens, and the Hugging Face documentation says to pass padding="max_length" for that reason. [Certain] With other padding you get different text vectors. Second, SigLIP cosines are small. In our library the best frame–caption cosines are about 0.15, and the whole range is about −0.07 to 0.19. That is normal: with scale 117.33 and bias −12.93, a cosine of 0.110 already gives \(p = 0.5\). Do not compare raw cosine thresholds across models.
In video 8, the segment from 52 to 60 s (the green circle moving left to right) has cosine 0.1484 with the query “a green circle moving from left to right”. Then \(117.33 \times 0.1484 - 12.93 = 4.48\) and \(\sigma(4.48) = 0.989\). The segment from 8 to 16 s shows the green circle moving the other way; its cosine is 0.1388, which gives \(p = 0.974\). Both are “yes” with high confidence. The model sees the green circle, but direction barely moves the score.
One frame per second is enough here, because each event lasts at least 4 seconds. For real video, 1 fps is a common starting point for retrieval (CLIP4Clip also samples 1 frame per second [Certain]); fast actions need more.
Step 3: cut segments and pool
A segment is a window of \(L\) seconds that starts every \(S\) seconds (the stride). If \(S \lt L\), neighboring segments overlap. A video of \(D\) seconds gives
For our 64-second videos with \(L = 8, S = 4\): \(\lfloor 56/4 \rfloor + 1 = 15\) segments per video, 150 in the library. With \(L = 4, S = 2\) it is 31 per video (310 in total); with \(L = 16, S = 8\) it is 7 (70 in total).
Mean pooling
The simplest segment vector is the normalized mean of its frame vectors, as in CLIP4Clip’s “meanP” (session 208). Its score with a query can be written with only frame cosines and frame–frame similarities:
- Read the symbols
- \(k\)
- one segment; \(j \in k\) are its \(L\) frames
- \(\bar{\mathbf{z}}_k\)
- the mean of the frame vectors of segment \(k\)
- numerator
- the average frame cosine with the query
- denominator
- the length of the mean vector: near 1 if the frames look alike, smaller if they differ (then normalization lifts the score)
Formula (214.3) is how Figure 2 computes scores in your browser: it stores the \(64 \times 64\) frame similarity matrix of one real video and the frame cosines of six queries, and it gives exactly the same numbers as the Python code.
Segment [52, 60) of video 8 and the query “a green circle moving from left to right”. The eight frame cosines are 0.1482, 0.1486, 0.1479, 0.1479, 0.1479, 0.1456, 0.1477, 0.1495; their mean is 0.14789. The frames look almost the same to SigLIP, so the mean vector has length 0.99679. The segment score is \(0.14789 / 0.99679 = 0.1484\).
Best-frame scoring
A text-conditioned alternative with no training is to score a segment by its best frame, \(\max_{j \in k} c_j\). You index frames, not segments, and map each frame hit to the segments that contain it. It helps when the query names a short part of a long segment. It has the same blind spot as the mean: a single frame shows no motion.
How to read Figure 2
Start with “green ● →”, \(L = 8\), \(S = L/2\): 15 segments, and the top-1 is [52, 60) with 0.1484, a hit. Now click “green ● ←”. The correct event is now the one at 9–19 s, but the top-1 is still [52, 60) (0.1505), a miss; the best segment on the correct event scores only 0.1412. Both queries prefer the same segment: the scores follow the object, not the direction. Try “red ▲ →”: its correct event is in another video, and the best segment here is its twin. Set \(L = 32\): scores drop to about 0.09 or less, because empty frames dilute the mean. Set \(L = 2\): the curve is followed closely, and the green circle queries hit or miss by about 0.001. With “best frame”, two overlapping segments that share the best frame get the same score; the tie goes to the earlier one, which can be a miss (try “purple ● →”).
- input: videos, segment length \(L\), stride \(S\), image encoder \(\phi\)
- vectors \(\leftarrow\) [ ]; meta \(\leftarrow\) [ ]
- for each video \(v\) do
- frames \(\leftarrow\) decode \(v\) at 1 fps; \(Z \leftarrow\) normalize(\(\phi\)(frames)) // \((D, 768)\), batch the frames
- for \(a = 0, S, 2S, \dots\) while \(a + L \le D\) do // formula (214.2)
- \(\mathbf{x} \leftarrow\) normalize(pool(\(Z[a : a+L]\))) // mean, or the temporal model
- vectors.append(\(\mathbf{x}\)); meta.append((\(v\), \(a\), \(a + L\)))
- index \(\leftarrow\) IndexIDMap(IndexFlatIP(768)); index.add_with_ids(vectors, \(0, \dots, n-1\))
- return index, meta // save both; the index alone does not know times
Step 4: index with FAISS and search
FAISS (Facebook AI Similarity Search; Johnson et al., 2017) is a library for nearest-neighbor search over dense vectors. [Certain] For a library of our size we use the exact index IndexFlatIP: it stores all vectors and computes the inner product of the query with every one of them. For unit vectors, the inner product is the cosine, so the ranking is exactly formula (214.3). Wrapping it in IndexIDMap lets us attach our own 64-bit ids to the vectors; a separate table maps each id to (video, start, end).
- input: query text, index, meta, \(k\)
- \(\mathbf{t} \leftarrow\) normalize(SigLIP_text(query, padding = max_length))
- scores, ids \(\leftarrow\) index.search(\(\mathbf{t}\), \(k'\)) // \(k' \gt k\): ask for extra, overlaps will be merged
- hits \(\leftarrow\) [ (meta[id], score) for id, score ]
- for each hit in score order: drop it if it overlaps a kept hit of the same video by more than half // temporal non-maximum suppression
- return the first \(k\) kept hits as (video, start, end, score)
Our evaluation in Step 5 does not merge hits, so that the numbers match the plain index. In a product you should merge: with 50% overlap, one event usually appears in two neighboring segments, and they fill two of the top 5 places.
How big does the index get?
- Read the symbols
- \(n_{\text{seg}}\)
- the number of segments in the whole library
- \(d\)
- the embedding size, 768 for SigLIP base
- \(b\)
- bytes per number: 4 for float32, 2 for float16; product quantization (PQ) stores a code of a few bytes per vector instead
Our library: \(150 \times 768 \times 4 = 460{,}800\) bytes (segments × dimensions × bytes per float). Tiny. Now take 1,000 hours of video with \(L = 8, S = 4\): about \(3{,}600{,}000 / 4 = 900{,}000\) segments, which is 2.76 GB in float32, 1.38 GB in float16, or 57.6 MB with 64-byte PQ codes. A frame-level index at 1 fps would hold 3.6 million vectors, 11.06 GB in float32. These are simple products, so check them for your own numbers.
Approximate search: IVF
An exact index compares the query with every vector. That is fine up to a few million vectors on a CPU [Likely], but it grows linearly. An inverted file index (IVF) first clusters the vectors with k-means into nlist lists. At query time it compares the query with the nlist centroids, opens only the nprobe closest lists, and searches inside them. If the lists have similar sizes, it scans about
vectors instead of \(N\). The price is recall of the index: a true neighbor that sits in a list you did not open is lost. In FAISS, IndexIVFFlat has nprobe = 1 by default [Certain: checked with faiss 1.13], which is often too low. Figure 4 lets you see the trade-off.
How to read Figure 4
With nlist = 64 and nprobe = 1, the search scans only 1.85% of the vectors but finds 71.15% of the true top-10. Doubling nprobe to 2 gives 89.45%; at nprobe = 8 it is 98.80% for 12.73% of the work. With nlist = 16, every list is bigger, so the same recall costs more scanning: nprobe = 2 gives 89.40% but scans 12.88%. Both curves reach 100% when every list is opened, which is just an exact search. On real embeddings, measure this curve on your own data: the shape depends on how clustered the vectors are.
For millions of segments you would use IVF with PQ codes (IndexIVFPQ) or a graph index such as HNSW (IndexHNSWFlat; its default efSearch is 16 in faiss 1.13 [Certain: checked]). Session session 208 explains both. The rule does not change: measure index recall against an exact search on a sample of queries before you trust the fast index.
Step 5: measure recall@k
For each query we sort all segments by score and find the rank \(r_i\) of the first correct segment. With overlapping segments of different lengths, “correct” needs a rule. We use: a segment is a hit for an event if they are in the same video and
- Read the symbols
- \(k, e\)
- a segment and the labeled event, as time intervals [start, end)
- \(\lvert k \cap e \rvert\)
- their overlap in seconds
- \(\min(\lvert k \rvert, \lvert e \rvert)\)
- the shorter of the two lengths: a long segment must contain at least half of the event; a short segment must lie at least half inside it
Then recall at \(K\) over \(Q\) queries is
With \(Q = 40\), each query is worth 2.5 points. Change the hit rule and the numbers change, so always state it next to your results. Temporal grounding papers use a temporal IoU threshold instead (R1@IoU, session 213).
- input: queries with labeled events \((v_i, e_i)\), ranked id lists, meta
- for each query \(i\) do
- \(r_i \leftarrow\) 1 + position of the first id whose meta \((v, a, b)\) has \(v = v_i\) and hit\(([a, b), e_i)\) // formula (214.6)
- \(\text{R@}K \leftarrow 100 \cdot \operatorname{mean}_i \mathbb{1}[r_i \le K]\) for \(K = 1, 5, 10\); \(\text{MdR} \leftarrow \operatorname{median}_i(r_i)\) // formula (214.7)
Results for mean pooling and best-frame scoring, with 50% overlap (\(S = L/2\)), on all 40 queries:
| Pooling | \(L\) | segments | R@1 | R@5 | R@10 | MdR |
|---|---|---|---|---|---|---|
| mean | 4 s | 310 | 50.0 | 100.0 | 100.0 | 1.5 |
| mean | 8 s | 150 | 45.0 | 82.5 | 97.5 | 2 |
| mean | 16 s | 70 | 37.5 | 65.0 | 87.5 | 2 |
| best frame | 4 s | 310 | 47.5 | 82.5 | 100.0 | 3 |
| best frame | 8 s | 150 | 40.0 | 100.0 | 100.0 | 2 |
| best frame | 16 s | 70 | 42.5 | 100.0 | 100.0 | 2 |
Read it in two parts. R@1 sits near 50% for both methods at every length: the system finds the object and then guesses the direction. At \(L = 8\), 17 of the 22 top-1 misses of mean pooling are the opposite-direction twin. R@5 falls with longer segments for the mean (100 → 82.5 → 65.0): empty frames dilute the event, as in the dilution formula of session 208. Best frame does not dilute, so its R@5 stays at 100% for 8 and 16 seconds.
Step 6: a small temporal model
Why can mean pooling never learn direction? Because the mean does not depend on the order of the frames. For any reordering \(\pi\) of the frames,
A circle moving left to right and the same circle moving right to left are the same set of frames in opposite order, so they get (almost) the same vector. Best-frame scoring is order-blind in the same way. To see motion, the pooling must know where each frame is in time.
We use the CLIP4Clip “seqTransf” idea at a small scale: add a learned position embedding to each frame vector, run one transformer encoder layer over the frames of the segment, add the result back to the frames (a residual connection), and then mean-pool:
- Read the symbols
- \(Z\)
- the \(L \times 768\) frame embeddings of one segment (SigLIP, frozen)
- \(P_{1:L}\)
- learned position embeddings, one per position, starting at zero
- \(\operatorname{Enc}\)
- one pre-norm transformer encoder layer, 8 heads, feed-forward size 1,024
- \(\mathbf{h}_j\)
- the output for frame \(j\): what it learned from the other frames, including their order
- \(W\)
- a \(768 \times 768\) linear layer on the text side, initialized to zero, so \(\mathbf{t}' = \mathbf{t}\) at the start
Why also adapt the text? The SigLIP text vectors of “… moving from left to right” and “… moving from right to left” have cosine 0.993 for the red circle. If the text side cannot separate them, a perfect video side does not help. The small residual \(W\) lets training stretch that difference.
The loss
Each training batch has \(B = 32\) clips of 32 different events with their captions. Choosing different events matters: two clips of the same event in one batch would count as negatives for each other (a false negative). We use the symmetric contrastive loss with temperature \(\tau = 0.05\):
- Read the symbols
- first term
- video → text: each clip must pick its own caption among the 32 captions
- second term
- text → video: each caption must pick its own clip among the 32 clips. This is exactly the retrieval task
- \(\tau\)
- the temperature: smaller values make the softmax sharper
Training runs 1,500 steps of AdamW (learning rate \(3 \cdot 10^{-4}\), weight decay 0.01), with the segment length drawn from {4, 8, 16} each step. It takes about 1.5 minutes on a laptop CPU, because SigLIP is frozen and its features are computed once.
How to read Figure 3
At \(L = 8\), mean pooling has 18 green cells out of 40 (R@1 45.0%), and 15 of its 22 misses are amber (rank 2 to 5): usually the twin came first. The symmetric temporal model turns 38 of 40 green (95.0%). The model trained with only the video→text term reaches 65.0%. At \(L = 4\) and \(L = 16\) the gap shrinks (77.5% and 67.5% for symmetric), and at \(L = 16\) the v→t model is even ahead (72.5%). These are single runs on 40 queries; read the next table before you draw conclusions.
How stable is this? Three seeds
We trained each variant with seeds 0, 1 and 2 (only the model initialization, dropout and batch order change). R@1 in percent:
| Model | \(L\) = 4 s | \(L\) = 8 s | \(L\) = 16 s |
|---|---|---|---|
| mean pooling (no training) | 50.0 | 45.0 | 37.5 |
| temporal, v→t loss only: seeds 0, 1, 2 | 62.5, 62.5, 80.0 (mean 68.3) | 65.0, 50.0, 80.0 (mean 65.0) | 72.5, 57.5, 80.0 (mean 70.0) |
| temporal, symmetric loss: seeds 0, 1, 2 | 77.5, 87.5, 80.0 (mean 81.7) | 95.0, 80.0, 82.5 (mean 85.8) | 67.5, 80.0, 80.0 (mean 75.8) |
Three conclusions are safe. (1) The temporal model beats mean pooling in all 18 runs, by 5 to 50 points of R@1. (2) On average the symmetric loss is better than the video→text loss alone at every length, but single runs overlap: at \(L = 8\) the best v→t run (80.0) equals the worst symmetric run (80.0). (3) The spread between seeds is up to 15 points. With 40 queries, the 95% confidence interval of a recall near 85% is about ±11 points (\(1.96\sqrt{0.85 \cdot 0.15 / 40} = 0.111\)). A real comparison needs more queries and several seeds.
Why should the text→video term help? Retrieval compares one caption against many segments. The video→text term only teaches each clip to prefer its caption over other captions; the scores of different clips for the same caption are never compared, so they need not be calibrated against each other. The text→video term trains exactly that comparison. [Likely: this is the usual explanation; our three seeds support it on average but do not prove it.]
The full program
Three files. build_library.py renders the videos and runs SigLIP (about 20 minutes on a laptop CPU, mostly the 5,628 training frames). search.py builds FAISS indexes and computes recall; it imports only numpy and faiss. temporal.py trains the temporal model and writes its segment and query vectors. Keep FAISS and PyTorch in separate processes on macOS: with the pip wheels we used (torch 2.8, faiss-cpu 1.13), importing both in one process aborted with “OMP: Error #15” (two copies of the OpenMP runtime). [Certain: seen in this lab]
# build_library.py: render a toy video library, embed every frame and query with SigLIP
import itertools, json, numpy as np, torch
from PIL import Image, ImageDraw
from transformers import AutoModel, AutoProcessor
torch.manual_seed(0)
COLORS = {"red": (220, 30, 30), "green": (30, 160, 40), "blue": (30, 60, 220),
"yellow": (240, 210, 20), "purple": (140, 40, 180)}
SHAPES = ["circle", "square", "triangle", "star"]
EVENTS = [(c, s, d) for c in COLORS for s in SHAPES for d in ("lr", "rl")] # 40 events
BGS = [(235, 235, 235), (225, 235, 245), (245, 235, 220), (230, 245, 230), (240, 228, 240)]
def caption(c, s, d):
return f"a {c} {s} moving from " + ("left to right" if d == "lr" else "right to left")
def frame(rng, bg, shape=None, color=None, x=0.5, size=0.18): # one 224x224 RGB frame
a = np.clip(np.array(bg)[None, None] + rng.normal(0, 6, (224, 224, 3)), 0, 255)
im = Image.fromarray(a.astype(np.uint8)); dr = ImageDraw.Draw(im)
cx, cy, r, col = 224 * x, 112, 224 * size, COLORS.get(color)
if shape == "circle": dr.ellipse([cx - r, cy - r, cx + r, cy + r], fill=col)
if shape == "square": dr.rectangle([cx - r, cy - r, cx + r, cy + r], fill=col)
if shape == "triangle": dr.polygon([(cx, cy - r), (cx - r, cy + r), (cx + r, cy + r)], fill=col)
if shape == "star":
pts = [(cx + (r if k % 2 == 0 else 0.45 * r) * np.cos(np.pi / 2 + k * np.pi / 5),
cy - (r if k % 2 == 0 else 0.45 * r) * np.sin(np.pi / 2 + k * np.pi / 5)) for k in range(10)]
dr.polygon(pts, fill=col)
return im
def event_frames(rng, c, s, d, dur, bg): # 1 frame per second, the shape crosses the screen
size = rng.uniform(0.12, 0.2)
xs = np.linspace(0.2, 0.8, dur)
return [frame(rng, bg, s, c, x, size) for x in (xs if d == "lr" else xs[::-1])]
def make_video(rng, events, D=64): # D seconds = D frames at 1 fps
bg = BGS[rng.integers(len(BGS))]
durs = [int(rng.integers(4, 11)) for _ in events] # 4 to 10 s per event
free = D - sum(durs)
gaps = np.diff(np.concatenate([[0], np.sort(rng.integers(0, free + 1, len(events))), [free]]))
frames, spans = [], []
for k, e in enumerate(events):
frames += [frame(rng, bg) for _ in range(gaps[k])] # empty background
st = len(frames); frames += event_frames(rng, *e, durs[k], bg)
spans.append((st, len(frames))) # [start, end) in seconds
frames += [frame(rng, bg) for _ in range(gaps[-1])]
return frames, spans
model = AutoModel.from_pretrained("google/siglip-base-patch16-224").eval()
proc = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")
@torch.no_grad()
def embed_images(ims, bs=64):
z = torch.cat([model.get_image_features(**proc(images=ims[i:i + bs], return_tensors="pt"))
for i in range(0, len(ims), bs)])
return torch.nn.functional.normalize(z, dim=-1).numpy() # (n, 768), unit length
@torch.no_grad()
def embed_texts(texts): # SigLIP was trained with padding="max_length" (64 tokens)
t = model.get_text_features(**proc(text=texts, padding="max_length", return_tensors="pt"))
return torch.nn.functional.normalize(t, dim=-1).numpy()
if __name__ == "__main__":
rng = np.random.default_rng(214) # test library: 10 videos, 4 events each, all 40 events once
perm = rng.permutation(len(EVENTS)); Z, meta = [], []
for v in range(10):
evs = [EVENTS[i] for i in perm[4 * v:4 * v + 4]]
frames, spans = make_video(rng, evs)
Z.append(embed_images(frames)); meta.append({"events": evs, "spans": spans})
T = embed_texts([caption(*e) for e in EVENTS])
np.savez("library.npz", Z=np.stack(Z), T=T) # Z: (10, 64, 768), T: (40, 768)
json.dump({"meta": meta, "events": EVENTS}, open("library.json", "w"), default=int)
rng = np.random.default_rng(7) # training clips: other renders, never seen at test time
clips, labels = [], []
for n in range(600):
L = int(rng.choice([4, 8, 16])); ei = int(rng.integers(len(EVENTS)))
dur = int(rng.integers(max(3, L // 2), min(L, 10) + 1)); bg = BGS[rng.integers(len(BGS))]
off = int(rng.integers(0, L - dur + 1))
fr = ([frame(rng, bg) for _ in range(off)] + event_frames(rng, *EVENTS[ei], dur, bg)
+ [frame(rng, bg) for _ in range(L - dur - off)])
clips.append(embed_images(fr)); labels.append(ei)
np.savez("train.npz", **{f"c{n}": c for n, c in enumerate(clips)}, y=np.array(labels))
# search.py: cut segments, pool, index with FAISS, search, measure recall@k (numpy + faiss only)
import json, sys, numpy as np, faiss
lib = np.load("library.npz"); J = json.load(open("library.json"))
Z, T = lib["Z"].astype("float32"), lib["T"].astype("float32") # (10, 64, 768), (40, 768)
EVENTS = [tuple(e) for e in J["events"]]
gt = {tuple(e): (v, tuple(sp)) for v, m in enumerate(J["meta"]) for e, sp in zip(m["events"], m["spans"])}
def segments(D, L, S): # windows [a, a+L) with stride S
return [(a, a + L) for a in range(0, D - L + 1, S)]
def is_hit(seg, ev): # overlap >= half of the shorter interval
inter = max(0, min(seg[1], ev[1]) - max(seg[0], ev[0]))
return inter >= 0.5 * min(seg[1] - seg[0], ev[1] - ev[0])
def build_index(vecs, ids):
index = faiss.IndexIDMap(faiss.IndexFlatIP(vecs.shape[1])) # exact inner product
index.add_with_ids(vecs, np.asarray(ids, dtype="int64"))
return index
def recall(rank_lists, segs_of_id, ks=(1, 5, 10)):
ranks = []
for e, ids in zip(EVENTS, rank_lists):
v, ev = gt[e]
ranks.append(next(r + 1 for r, i in enumerate(ids) if segs_of_id[i][0] == v and is_hit(segs_of_id[i][1:], ev)))
ranks = np.array(ranks)
return {f"R@{k}": 100 * float(np.mean(ranks <= k)) for k in ks} | {"MdR": float(np.median(ranks))}, ranks
def run(L, S, method, seg_vecs=None, queries=None):
segs_of_id = [(v, a, b) for v in range(Z.shape[0]) for a, b in segments(Z.shape[1], L, S)]
if method == "mean": # one vector per segment: the normalized mean frame
X = np.stack([Z[v, a:b].mean(0) for v, a, b in segs_of_id]); faiss.normalize_L2(X)
index = build_index(X, range(len(segs_of_id)))
_, I = index.search(T, len(segs_of_id))
elif method == "maxframe": # index frames, a segment scores its best frame
index = build_index(Z.reshape(-1, Z.shape[2]), range(Z.shape[0] * Z.shape[1]))
Sf, If = index.search(T, Z.shape[0] * Z.shape[1])
I = []
for q in range(len(T)):
best = {}
for s, f in zip(Sf[q], If[q]): # frames come best first
v, t = divmod(int(f), Z.shape[1])
for j, (vv, a, b) in enumerate(segs_of_id):
if vv == v and a <= t < b and j not in best: best[j] = s
I.append(sorted(best, key=lambda j: -best[j]))
else: # vectors written by temporal.py
index = build_index(seg_vecs, range(len(segs_of_id)))
_, I = index.search(queries, len(segs_of_id))
return recall(I, segs_of_id)
if __name__ == "__main__":
for L, S in [(4, 2), (8, 4), (16, 8)]:
for m in ["mean", "maxframe"]:
print(L, S, m, run(L, S, m)[0])
# temporal.py: a small transformer over the frames of a segment (CLIP4Clip "seqTransf" style)
import json, sys, numpy as np, torch, torch.nn as nn, torch.nn.functional as F
SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
SYMMETRIC = (sys.argv[2] != "v2t") if len(sys.argv) > 2 else True
torch.manual_seed(SEED); np.random.seed(SEED)
lib = np.load("library.npz"); tr = np.load("train.npz")
Z, T = torch.tensor(lib["Z"]), torch.tensor(lib["T"]) # frozen SigLIP features
y = tr["y"]; clips = [tr[f"c{n}"] for n in range(len(y))]
class Temporal(nn.Module): # video side: frames -> one segment vector
def __init__(self, d=768, max_len=32):
super().__init__()
self.pos = nn.Parameter(torch.zeros(max_len, d)) # learned position embeddings
layer = nn.TransformerEncoderLayer(d, 8, 1024, dropout=0.1, batch_first=True, norm_first=True)
self.enc = nn.TransformerEncoder(layer, num_layers=1)
def forward(self, z): # z: (B, L, d)
h = self.enc(z + self.pos[: z.shape[1]])
return F.normalize((z + h).mean(1), dim=-1) # residual, then mean pool
class TextAdapter(nn.Module): # text side: t + W t, starts as the identity
def __init__(self, d=768):
super().__init__(); self.lin = nn.Linear(d, d)
nn.init.zeros_(self.lin.weight); nn.init.zeros_(self.lin.bias)
def forward(self, t): return F.normalize(t + self.lin(t), dim=-1)
vm, tm = Temporal(), TextAdapter()
opt = torch.optim.AdamW([*vm.parameters(), *tm.parameters()], lr=3e-4, weight_decay=0.01)
by = {} # (event, length) -> clip indices
for i, c in enumerate(clips): by.setdefault((int(y[i]), len(c)), []).append(i)
tau = 0.05
for step in range(1500):
L = int(np.random.choice([4, 8, 16]))
evs = [e for e in range(len(T)) if (e, L) in by]
evs = list(np.random.choice(evs, min(32, len(evs)), replace=False)) # 32 DIFFERENT events: no false negatives
z = torch.tensor(np.stack([clips[int(np.random.choice(by[(e, L)]))] for e in evs]))
logits = vm(z) @ tm(T[evs]).T / tau # (32 clips, 32 captions)
target = torch.arange(len(evs))
loss = F.cross_entropy(logits, target) # video -> text
if SYMMETRIC:
loss = (loss + F.cross_entropy(logits.T, target)) / 2 # + text -> video
opt.zero_grad(); loss.backward(); opt.step()
vm.eval(); tm.eval(); out = {}
with torch.no_grad():
out["T"] = tm(T).numpy()
for L, S in [(4, 2), (8, 4), (16, 8)]:
segs = torch.stack([Z[v, a:a + L] for v in range(Z.shape[0]) for a in range(0, Z.shape[1] - L + 1, S)])
out[f"X{L}_{S}"] = vm(segs).numpy()
np.savez(f"temporal_s{SEED}_{'sym' if SYMMETRIC else 'v2t'}.npz", **out)
# eval_temporal.py: python eval_temporal.py temporal_s0_sym.npz import sys, numpy as np; from search import run f = np.load(sys.argv[1]) for L, S in [(4, 2), (8, 4), (16, 8)]: print(L, S, run(L, S, "temporal", f[f"X{L}_{S}"].astype("float32"), f["T"].astype("float32"))[0]) # python temporal.py 0 sym && python eval_temporal.py temporal_s0_sym.npz # 4 2 {'R@1': 77.5, 'R@5': 100.0, 'R@10': 100.0, 'MdR': 1.0} # 8 4 {'R@1': 95.0, 'R@5': 100.0, 'R@10': 100.0, 'MdR': 1.0} # 16 8 {'R@1': 67.5, 'R@5': 92.5, 'R@10': 97.5, 'MdR': 1.0} # python temporal.py 0 v2t && python eval_temporal.py temporal_s0_v2t.npz # 4 2 {'R@1': 62.5, ...} 8 4 {'R@1': 65.0, ...} 16 8 {'R@1': 72.5, ...}
Running python search.py prints the mean and best-frame rows of the table in Step 5. To use real videos, replace make_video with a decoder (for example PyAV or ffmpeg -vf fps=1) and replace the generated captions and spans with your labeled moments; the rest does not change.
Where this is used
| Application | What this lab’s pieces do there |
|---|---|
| Search in a video archive (news, sports, stock footage) | Segment index plus metadata returns moments, not whole files; merge overlapping hits before showing results |
| Video-RAG | The same index retrieves segments whose frames (and transcript) are then given to a VLM to answer a question (session 210) |
| Key-moment detection | Segment scores against event prompts, computed exactly as in Figure 2, become a score curve over time; the next lab adds non-maximum suppression and a VLM check |
| Action recognition and direction-sensitive events | “Opening” vs “closing”, “entering” vs “leaving” need order: mean pooling fails as in Step 5; a temporal model or a video encoder is needed |
| Near-duplicate detection | The same FAISS machinery with video–video similarity instead of text–video (session 209) |
| Data curation for training | Text queries over a large unlabeled pool find candidate clips to label or to filter |
Common mistakes
- Forgetting to normalize.
IndexFlatIPranks by inner product. Without unit vectors, long vectors win, and a pooled mean is shorter than its frames. Normalize after pooling, and normalize queries. - Default text padding with SigLIP. The model was trained with captions padded to 64 tokens; use
padding="max_length". - Using a fixed cosine threshold from another model. SigLIP cosines of about 0.15 are strong matches here; CLIP cosines live in a different range. Calibrate thresholds per model, or use ranks.
- Losing the metadata. A FAISS id means nothing without the table that maps it to (video, start, end). Save both together, and rebuild both together.
- No hit rule, or an unreported one. With overlapping segments, “correct” must be defined (formula 214.6). Different rules give different R@k.
- Trusting one run on a small test set. 40 queries give ±11 points of noise at 85%; seeds moved our R@1 by up to 15 points.
- Only one half of the contrastive loss, or repeated events in a batch. Train text→video as well, and keep false negatives out of the batch.
- Measuring an approximate index without an exact baseline. IVF with the default
nprobe = 1lost about 29% of the true top-10 in Figure 4.
Interview questions
Why can mean pooling of frame embeddings not tell “left to right” from “right to left”?basic
You use FAISS IndexFlatIP. What must be true about the vectors for this to be cosine search?basic
faiss.normalize_L2 does it in place on a float32 array.How do you choose the segment length \(L\) and the stride \(S\)?core
What exactly does R@5 = 82.5% mean in this lab?core
SigLIP gives a cosine of 0.15 for a perfect match. Is something wrong?core
Your temporal model gets 95% R@1 with one seed. Is it better than mean pooling? Is the symmetric loss better than video→text only?advanced
Why should the text→video term of the contrastive loss matter for retrieval?advanced
Design the index for 1,000 hours of video. What sizes and choices?advanced
Best-frame scoring or mean pooling?core
Your offline R@k looks great, but users complain. What do you check?core
Exercises
- How many segments does one 64-second video give with \(L = 16, S = 4\)?
- Estimate the float16 index size for 500 hours of video with \(L = 10, S = 5\) and 768-d vectors.
- Implement the merge step of Algorithm 2 and count how many of the top 5 results of mean pooling at \(L = 8\) are duplicates of the same event before and after merging.
- Run
search.pywith best-frame scoring at \(L = 8\). Explain why R@5 reaches 100% while R@1 stays at 40%. - Add queries without a direction, such as “a green circle”. What should count as correct, and how must the evaluation change?
- Train the symmetric temporal model with seeds 3 and 4. Report the mean and the range of R@1 at \(L = 8\) over five seeds.
Answerscheck
Key takeaways
- A text-to-video system is: frames → SigLIP vectors → segments (214.2) → pooled, normalized vectors → FAISS index with a metadata table → text query → ranked moments.
- For unit vectors,
IndexFlatIPis exact cosine search; mean-pooled segment scores follow formula (214.3). Index memory is \(n_{\text{seg}} \cdot d \cdot b\) (214.4); IVF scans about \(\text{nprobe}/\text{nlist}\) of it (214.5) at a measurable loss of recall. - R@\(k\) needs a stated hit rule (214.6, 214.7). With 40 queries, each query is 2.5 points and noise is about ±11 points.
- Mean and best-frame pooling are order-blind (214.8): on our library they find the object but guess the direction (R@1 45.0% and 40.0% at 8 s).
- A one-layer temporal transformer with position embeddings over frozen SigLIP frames (214.9), trained with the symmetric loss (214.10), raised R@1 to 85.8% on average over three seeds.
Key formulas
References for this session
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Meta (code). FAISS library
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Johnson et al. (2017). Billion-scale similarity search with GPUs
- Meta (code). FAISS library
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Jégou et al. (2011). Product Quantization for Nearest Neighbor Search
- Malkov & Yashunin (2016). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models