LLM & VLM courseBlog
236Track 10: Capstone and interviews · session 3 of 6

capstone build

In this lab you build a small version of the key-moment search system designed in the previous session: an offline indexer, a retrieval service, a VLM re-ranker that reads the probability of “yes”, and a timestamp refiner. You measure every stage on a labeled test archive, profile one query until you find what is slow, fix it step by step (41 times faster on our machine), and see how to serve the re-ranker with vLLM or TensorRT-LLM on an L4.

Time: 5 to 6 hours (mostly hands-on; model downloads about 8 GB)Level: Advanced · hands-on labBefore this: The capstone design (235), text-to-video retrieval (208), temporal grounding (207), serving VLMs (198), runtime profiling (123)

What you will learn

  • How to build a labeled stand-in archive when you cannot get real sports video, and what it can and cannot tell you
  • How to build the offline index: frame embeddings with SigLIP, 2-second segments, one FP16 matrix, and moment grouping at query time
  • How to turn a VLM into a re-ranker: the log-odds of “yes” against “no” from one prefill, z-score fusion with the first-stage score, and how to choose the weight on a dev split
  • How a training-free contrast search moves start and end to the right second, and why it raises mAP but can lower recall at tIoU 0.5
  • How to time each stage correctly on a GPU (synchronize first), read the profile, and remove one bottleneck after another: precision, image tiles, batching, caching, fewer candidates
  • How to serve the re-ranker with vLLM or TensorRT-LLM, and how to project measured numbers to an NVIDIA L4 with the design formulas
  • How to report a small benchmark honestly: dev and test splits, per-query differences and a bootstrap interval

Why this matters

In session 235 you designed, on paper, a system that answers “penalty saved by the keeper” with the exact seconds of video. A design is a set of guesses: that the first stage finds the right moment, that a VLM re-ranker helps, that the timestamps are good, that the latency fits. This lab turns the guesses into measurements. You build a small version of every box in the design, run it on a labeled test archive, and then do the thing that real projects spend most of their time on: find what is slow, fix it, and check that the fix did not break accuracy.

You will build four pieces: an offline indexer (frames to vectors to an index), a retrieval service (query to ranked moments), a VLM re-ranker that reads the probability of “yes”, and a timestamp refiner. Then you profile one query, which takes 70 seconds in the first working version, and bring it to 1.7 seconds on the same laptop, one bottleneck at a time. Finally you put the re-ranker behind vLLM or TensorRT-LLM and project the numbers to the NVIDIA L4 that the design assumed. In the next session, session 237, you take the system apart again and study where and why it fails.

All numbers on this page come from runs of the code shown here. Speeds were measured on one Apple M5 Pro laptop (48 GB, PyTorch MPS backend, FP16), not on an L4. We say clearly where we project to the L4 with the formulas of the design session, and those projections are tagged.

Setup: what you build and what you test it on

Figure 1 shows the whole pipeline running on our test archive for one query. Press Play and follow one query from the frames to the ranked moments. Each step names the code you write for it below.

Query:
shots of the queried eventshots of other eventsbackground shotssegment score and moments
Figure 1. The built pipeline on real data from the test archive: four of the 40 test videos (the ones that hold the best results for the chosen query), each 180 s long. Steps: archive, frame embeddings, 2-s segments in the index, query scores, top-100 hits grouped into moments, VLM re-rank, timestamp refinement. Dashed boxes are the labeled true moments.
How to read Figure 1

Each row is one video; time runs left to right from 0 to 180 s. In the score step, darker blue means a higher cosine between the query and that 2-s segment. Notice that the blue peaks sit on the green shots (the queried event), and that a few background or other-event shots also light up: those are the near misses the re-ranker has to push down. The numbers on the moment boxes are global ranks over all 40 videos. In the last step, compare the solid refined boxes with the dashed true moments.

The test archive

We could not use real sports broadcasts: they are not free to download and nobody has labeled them for our queries. So we built a stand-in with exact labels. Each test “video” is a slideshow: a 180-s sequence of shots, where each shot is one still photo from Flickr8k shown for 2 to 5 seconds. The Flickr8k copy we use (jxie/flickr8k on Hugging Face) has 8,000 photos with 5 human captions each, and many of them show sports. A moment is a run of 2 to 4 shots whose photos all show the same event, for example “a person riding a horse”. A photo counts as showing an event when at least 3 of its 5 captions match a pattern for that event (for example the word “soccer”) and no caption mentions another of our events. Between moments we put background photos, and 40% of those are hard background: photos whose captions use sports words such as rugby, football, volleyball, field or player, without any of our events.

SplitVideosDurationShotsMomentsMoment lengthUsed for
dev201 hour1,04442 (3 per query)5 to 18 schoosing every setting
test402 hours2,08084 (6 per query)4 to 19 s, mean 10.6 sthe numbers we report

There are 14 queries: a soccer game, a basketball game, a baseball game, a person playing tennis, surfing a wave, skiing down a snowy slope, snowboarding, riding a skateboard, riding a bicycle, riding a motorcycle, riding a horse, climbing a rock wall, swimming, and a dog catching a frisbee. Moments cover 12.4% of the test time; the rest is background, as in a real match where key moments are rare.

Software and models

  • Embedding model: SigLIP-B/16 at 224×224 (google/siglip-base-patch16-224), 768-dimensional image and text vectors.
  • Re-rankers: SmolVLM-500M-Instruct (about 0.5 billion parameters) and Qwen2.5-VL-3B-Instruct, the re-ranker of the design session.
  • Libraries: PyTorch, Hugging Face transformers, pandas; vLLM or TensorRT-LLM for serving on an NVIDIA GPU.
  • Seeds: random.Random(0) for the archive and torch.manual_seed(0) everywhere; the model forward passes have no sampling.

Step 1: build the labeled test archive

The archive builder is plain Python. It reads the captions, decides which photos clearly show which event, and lays out videos so that moments are at least 40 s apart. Ground-truth moments are written next to the shots. Nothing here uses a model, so the labels cannot leak the behavior of the system we test.

Algorithm 1: build the slideshow archive (once)
  1. input: photos with 5 captions each; 14 events, each with caption patterns
  2. for each photo and event: votes \(\leftarrow\) number of captions that match all patterns of the event
  3. positives of event \(e\) \(\leftarrow\) photos with votes \(\ge 3\) for \(e\) and 0 for every other event
  4. background \(\leftarrow\) photos with 0 votes for every event; hard background \(\leftarrow\) those with \(\ge 2\) captions that use sports words
  5. for each video: pick the events it contains and start times at least 40 s apart
  6. fill time with shots of 2 to 5 s: inside a moment, 2 to 4 positive photos of its event; elsewhere, background (hard with probability 0.4)
  7. return videos as lists of shots (photo, start, end, event) and the moments (event, start, end)
# bench.py: a small "sports archive" of slideshow videos with exact moment labels
import pandas as pd, re, json, random, glob
from huggingface_hub import snapshot_download
root = snapshot_download("jxie/flickr8k", repo_type="dataset")
fs = sorted(glob.glob(root + "/data/*.parquet"))              # test, train x2, validation: 8,000 photos
d = pd.concat([pd.read_parquet(f, columns=[f"caption_{j}" for j in range(5)]) for f in fs], ignore_index=True)
caps = [[d.iloc[i][f"caption_{j}"].lower() for j in range(5)] for i in range(len(d))]
def votes(p): return [sum(bool(re.search(p, c)) for c in cs) for cs in caps]
EVENTS = {   # query: caption patterns that must ALL match in at least 3 of 5 captions
 "a soccer game": [r"soccer"], "a basketball game": [r"basketball"], "a person surfing a wave": [r"\bsurf"],
 "a person skiing down a snowy slope": [r"\bski(s|er|ers|ing)?\b"], "a person riding a skateboard": [r"skateboard"],
 "a person riding a bicycle": [r"\b(bicycl\w*|cyclist|biker|bmx)\b"],
 "a person climbing a rock wall": [r"\bclimb", r"\b(rock|rocks|wall|cliff|mountain)\b"],
 "a baseball game": [r"baseball"], "a person playing tennis": [r"tennis"], "a person snowboarding": [r"snowboard"],
 "a person riding a motorcycle": [r"\b(motorcycl\w*|motocross|motorbike|dirt ?bike)"],
 "a person swimming": [r"\bswim(s|mer|mers|ming)?\b"], "a dog catching a frisbee": [r"\bdogs?\b", r"frisbee"],
 "a person riding a horse": [r"\bhorses?\b", r"\brid(e|es|er|ers|ing)\b"]}
NEAR = r"\b(rugby|football|volleyball|rodeo|pool|water|dogs?|ball|field|race|racing|jump\w*|snow|wave|ramp|helmet|uniform\w*|player\w*|team)\b"
Q = list(EVENTS); N = len(d)
V = {q: [votes(p) for p in ps] for q, ps in EVENTS.items()}
ev = lambda q, i: min(v[i] for v in V[q])                     # votes for event q: all patterns must match
pos = {q: [i for i in range(N) if ev(q, i) >= 3 and all(ev(o, i) == 0 for o in Q if o != q)] for q in Q}
hv = votes(NEAR)
bg = [i for i in range(N) if max(ev(q, i) for q in Q) == 0]
hard = [i for i in bg if hv[i] >= 2]; easy = [i for i in bg if hv[i] < 2]
rng = random.Random(0)
for L in pos.values(): rng.shuffle(L)
rng.shuffle(hard); rng.shuffle(easy)
def build(prefix, n_vid, per_event, counts, T=180):
    moms = [q for q in Q for _ in range(per_event)]; rng.shuffle(moms)
    cnt = counts[:]; rng.shuffle(cnt); videos = []; k = 0
    for v in range(n_vid):
        evs = moms[k:k + cnt[v]]; k += cnt[v]
        while True:                                            # moment starts at least 40 s apart
            st = sorted(rng.sample(range(5, T - 25), len(evs)))
            if all(b - a >= 40 for a, b in zip(st, st[1:])): break
        shots, moments, t, j = [], [], 0, 0
        while t < T:
            if j < len(evs) and t >= st[j]:                      # a moment: 2 to 4 shots of the event
                s = t
                for _ in range(rng.randint(2, 4)):
                    L = rng.randint(2, 5); shots.append([pos[evs[j]].pop(), t, min(T, t + L), Q.index(evs[j])]); t += L
                moments.append([evs[j], s, min(T, t)]); j += 1; continue
            L = rng.randint(2, 5)                              # background shot, hard with probability 0.4
            shots.append([(hard if rng.random() < 0.4 else easy).pop(), t, min(T, t + L), -1]); t += L
        videos.append({"vid": f"{prefix}{v:02d}", "T": T, "shots": shots, "moments": moments})
    return videos
test = build("t", 40, 6, [0]*4 + [1]*4 + [2]*16 + [3]*16)     # 40 videos, 84 moments
dev = build("d", 20, 3, [0]*2 + [1]*2 + [2]*8 + [3]*8)         # 20 videos, 42 moments
json.dump({"queries": Q, "test": test, "dev": dev}, open("bench.json", "w"))

Run it twice: the file must be byte-identical (we checked with md5). A benchmark that changes between runs cannot tell you whether a change to the system helped.

Step 2: the offline indexer

The indexer does the expensive work once per video, as the design asked. It embeds every frame with the SigLIP image tower, cuts the video into segments, mean-pools the frame vectors of each segment, and writes the result into one matrix. In our archive one second of video is one frame, and each shot shows a single photo, so we embed each of the 3,124 distinct photos once and copy its vector to every second of its shot. With real video you would embed 1 or 2 decoded frames per second instead.

\[ \mathbf{v}_j = \frac{\bar{\mathbf{z}}_j}{\lVert \bar{\mathbf{z}}_j \rVert}, \quad \bar{\mathbf{z}}_j = \frac{1}{|S_j|}\sum_{t \in S_j} \mathbf{z}_t, \qquad s_j(q) = \mathbf{t}_q^\top \mathbf{v}_j \tag{236.1} \]
Read the symbols
\(S_j\)
the seconds inside segment \(j\); our segments are 2 s long with no overlap, so \(|S_j| = 2\)
\(\mathbf{z}_t\)
the unit-length SigLIP image vector of the frame at second \(t\)
\(\mathbf{v}_j\)
the unit-length segment vector, one row of the index
\(\mathbf{t}_q\)
the unit-length SigLIP text vector of the query; \(s_j(q)\) is the cosine between query and segment

This is formula (235.3) of the design with two changes. The design used 6-s windows with a 3-s stride. Here the segments are short (2 s) and do not overlap, because we do two things at query time that the design put elsewhere: we group neighboring hit segments into moments (Step 3) and we refine the boundaries to the second (Step 5). Short segments keep the boundaries sharp; grouping gives long events one result instead of five.

Worked example: how big is the index?

Test archive: \(40 \times 180 / 2 = 3{,}600\) segments (40 videos of 180 s, 2-s segments). In FP16: \(3{,}600 \times 768 \times 2 = 5{,}529{,}600\) bytes, about 5.5 MB. An exact (“flat”) search is one matrix-vector product and fits in any GPU or CPU cache. The design’s 24 million segments would be \(24 \times 10^6 \times 768 \times 2 = 36.9\) GB, which is why it needed product quantization (PQ) and an IVF index (session 151). At lab scale a flat matrix is the right choice: it has no recall loss to debug.

# index.py: offline indexer. SigLIP image vectors per second, 2-s segments, one FP16 matrix
import torch, json, io, glob, pandas as pd, torch.nn.functional as F
from PIL import Image
from transformers import AutoModel, AutoProcessor
torch.manual_seed(0); torch.set_grad_enabled(False)
DEV = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
B = json.load(open("bench.json")); Q = B["queries"]
fs = sorted(glob.glob(snapshot_dir + "/data/*.parquet"))    # snapshot_dir: the Flickr8k download of Step 1
JPG = pd.concat([pd.read_parquet(f, columns=["image"]) for f in fs], ignore_index=True)
img = lambda i: Image.open(io.BytesIO(JPG.iloc[i]["image"]["bytes"])).convert("RGB")
MID = "google/siglip-base-patch16-224"
sig = AutoModel.from_pretrained(MID).eval().to(DEV); sp = AutoProcessor.from_pretrained(MID)
vec = lambda o: o if isinstance(o, torch.Tensor) else o.pooler_output

photos = sorted({s[0] for split in ("test", "dev") for v in B[split] for s in v["shots"]})   # 3,124
U = []
for a in range(0, len(photos), 64):                            # batch 64 frames per call
    x = sp(images=[img(i) for i in photos[a:a + 64]], return_tensors="pt")["pixel_values"].to(DEV)
    U.append(F.normalize(vec(sig.get_image_features(pixel_values=x)).float(), dim=-1).cpu())
U = torch.cat(U); row = {p: n for n, p in enumerate(photos)}

SEG = 2
def build_index(videos):
    frames, segs, meta = {}, [], []
    for v in videos:
        fr = torch.zeros(v["T"], U.shape[1]); ph = [None] * v["T"]   # one frame vector per second
        for p, s, e, _ in v["shots"]:
            fr[s:e] = U[row[p]]
            for t in range(s, e): ph[t] = p
        frames[v["vid"]] = (fr, ph)                            # kept for refinement and keyframes
        for k in range(v["T"] // SEG):
            segs.append(fr[k * SEG:(k + 1) * SEG].mean(0)); meta.append((v["vid"], k * SEG, (k + 1) * SEG))
    X = F.normalize(torch.stack(segs), dim=-1).half()          # formula (236.1): (N, 768) FP16
    return X, meta, frames
X, meta, frames = build_index(B["test"])
print(X.shape, X.numel() * X.element_size())                  # torch.Size([3600, 768]) 5529600

Keep the per-second frame vectors next to the index (the frames dictionary). They cost \(7{,}200 \times 768\) numbers for the test archive, they are never searched, and two later steps need them: keyframe selection for the re-ranker and the timestamp refiner. The design session made the same choice for the 2-fps embeddings on disk.

Step 3: the retrieval service

The online service embeds the query text with the same SigLIP model, scores every segment, and keeps the top 100 segments. A long moment covers several neighboring segments, and each of them is a hit. Showing them as five separate results would be useless, so we group hits: within one video, hits whose segments touch are merged into one moment.

\[ m = [a, b) \text{ = a maximal run of touching hit segments in one video}, \qquad \text{score}(m) = \max_{j \in m} s_j(q) \tag{236.2} \]

We tried the mean and the max of the segment scores, and allowing gaps of 0, 1 or 2 segments, on the dev split; max with no gap was best, so the test run uses it. On the test archive each query gives 31 to 58 moments from its 100 hits.

How we score a ranked list of moments

A predicted moment is correct when its tIoU (temporal intersection over union, formula 235.2) with a true moment of the same query, in the same video, is at least a threshold \(\theta\). Each true moment can be matched once. We report four numbers over the top 10 results, averaged over the 14 queries:

\[ \text{AP}_\theta = \frac{1}{|G|}\sum_{i=1}^{10} \text{tp}_i \cdot \max_{j \ge i} \text{Prec}@j, \qquad \text{mAP} = \frac{1}{10}\sum_{\theta \in \{0.50, 0.55, \dots, 0.95\}} \text{AP}_\theta \tag{236.3} \]
Read the symbols
\(G\)
the true moments of this query (6 in the test archive)
\(\text{tp}_i\)
1 if result \(i\) matches a not-yet-matched true moment at tIoU \(\ge \theta\), else 0
\(\text{Prec}@j\)
precision of the top \(j\) results; the max over \(j \ge i\) is the usual interpolation
mAP
AP averaged over ten tIoU thresholds; it rewards precise boundaries, as in QVHighlights and Moment-DETR

The other two numbers are Hit@1 (the first result is correct at tIoU 0.5) and R@10@\(\theta\) (the share of true moments found in the top 10 at tIoU \(\theta\)).

Algorithm 2: answer one query (online)
  1. input: query text \(q\); index \(X \in \R^{N \times 768}\) (FP16), segment metadata, per-second frame vectors
  2. \(\mathbf{t}_q \leftarrow\) SigLIP text vector of “a photo of \(q\).” // same model version as the index
  3. scores \(\leftarrow X \mathbf{t}_q\); hits \(\leftarrow\) top 100 segments
  4. moments \(\leftarrow\) group touching hits per video; score = max // (236.2)
  5. for the top \(M\) moments: take keyframes at 25%, 50% and 75% of the span; VLM log-odds of “yes” for each // (236.4)
  6. fused score \(\leftarrow w \cdot\) z(first-stage score) + \((1-w) \cdot\) z(mean log-odds); re-sort the top \(M\) // (236.5)
  7. for the top 30: refine \([a, b)\) by the contrast search; drop a result if it overlaps an earlier one at tIoU \(\ge 0.5\) // (236.6)
  8. return the top 10 moments (video, start, end, score) and the time spent in every stage
# search.py: stage 1 (search and grouping) and the metrics of formula (236.3)
def search(q, X, meta, n_hits=100):
    x = sp(text=[f"a photo of {q}."], padding="max_length", max_length=64, return_tensors="pt").to(DEV)
    wq = F.normalize(vec(sig.get_text_features(input_ids=x["input_ids"])).float(), dim=-1)[0]
    top = torch.topk(X.to(DEV).float() @ wq, n_hits)
    byv = {}
    for s, i in zip(top.values.tolist(), top.indices.tolist()):
        vid, a, b = meta[i]; byv.setdefault(vid, []).append((a, b, s))
    res = []
    for vid, hs in byv.items():                               # group touching hits: formula (236.2)
        hs.sort(); cur = [hs[0]]
        for h in hs[1:] + [None]:
            if h is not None and h[0] == cur[-1][1]: cur.append(h); continue
            res.append([max(x[2] for x in cur), vid, cur[0][0], cur[-1][1]])
            if h is not None: cur = [h]
    return sorted(res, key=lambda r: (-r[0], r[1], r[2])), wq.cpu()

def tiou(p, g):
    inter = max(0.0, min(p[1], g[1]) - max(p[0], g[0]))
    return inter / (max(p[1], g[1]) - min(p[0], g[0]))
def ap(P, G, th):                                             # P: [score, vid, s, e]; G: [vid, s, e]
    used, tp = [False] * len(G), []
    for p in P:
        best, bj = -1, -1
        for j, g in enumerate(G):
            if not used[j] and g[0] == p[1] and tiou(p[2:4], g[1:3]) > best: best, bj = tiou(p[2:4], g[1:3]), j
        tp.append(best >= th)
        if best >= th: used[bj] = True
    prec = [sum(tp[:i + 1]) / (i + 1) for i in range(len(tp))]
    return sum(max(prec[i:]) for i in range(len(tp)) if tp[i]) / len(G)
TH = [round(0.5 + 0.05 * i, 2) for i in range(10)]
gts = lambda q: [(v["vid"], m[1], m[2]) for v in B["test"] for m in v["moments"] if m[0] == q]
R, wq = search("a person riding a horse", X, meta)
print(sum(ap(R[:10], gts("a person riding a horse"), th) for th in TH) / 10)

Stage-1 results

SplitHit@1R@10@0.5R@10@0.7mAP@0.5mAP
dev1.0000.9760.9290.9450.771
test1.0000.9050.8450.8590.659

The first stage alone puts a correct moment at rank 1 for all 14 test queries and finds 90.5% of the true moments in its top 10. As we warned, category queries are easy for SigLIP. The weak number is mAP (0.659): the boundaries sit on the 2-s grid, while true moments start and end at any second, so many hits are correct at tIoU 0.5 but not at 0.8 or 0.9. Two later steps attack the two remaining problems: wrong moments in the top 10 (the re-ranker) and imprecise boundaries (the refiner).

Step 4: VLM re-ranking

The re-ranker shows the VLM one keyframe and asks: “Does this image show a person riding a horse? Answer yes or no.” We do not let it write. We run one forward pass over the prompt (the prefill) and read the scores (logits) for the next token. The model may answer “Yes” or “yes”, so we sum over both spellings, and the same for “No” and “no”:

\[ \ell(x, q) = \log \sum_{y \in \mathcal{Y}} p(y \mid x, q) - \log \sum_{n \in \mathcal{N}} p(n \mid x, q) = \operatorname{LSE}(z_{\mathcal{Y}}) - \operatorname{LSE}(z_{\mathcal{N}}) \tag{236.4} \]
Read the symbols
\(x, q\)
a keyframe and the query text, inside the chat prompt
\(\mathcal{Y}, \mathcal{N}\)
the token ids of “Yes”, “yes” and of “No”, “no”
\(z\)
the logits of the next token; LSE is log-sum-exp. The softmax denominator cancels, so we never need the full vocabulary
\(\ell\)
the log-odds of “yes”: 0 means 50/50, +3 means about 20 to 1 for yes

Why the log-odds and not \(p(\text{yes})\)? Both give the same order, but \(p(\text{yes})\) saturates near 0 and 1, while log-odds keep the differences between “sure” and “very sure”, which matters when we add it to another score. A candidate moment gets the mean \(\bar\ell\) over its 3 keyframes (at 25%, 50% and 75% of its span).

Fusing two scores

The first-stage cosine and the VLM log-odds live on different scales (cosines between 0.03 and 0.15 for these candidates, log-odds between about −10 and +8). We convert both to z-scores within the \(M\) candidates of this query and mix them:

\[ f_i = w \cdot \frac{s_i - \mu_s}{\sigma_s} + (1 - w) \cdot \frac{\bar\ell_i - \mu_\ell}{\sigma_\ell}, \qquad i = 1, \dots, M \tag{236.5} \]

With \(w = 1\) we keep the first-stage order; with \(w = 0\) the VLM alone decides. Results below rank \(M\) keep their first-stage order. The weight \(w\) and the number of candidates \(M\) are chosen on dev, then frozen for test.

Re-ranker:
Figure 2. The re-ranker on the real test results. For each query we stored the top 30 first-stage moments with their cosine, the VLM log-odds (mean over 3 keyframes, Qwen2.5-VL-3B with at most 64 visual tokens per image, or SmolVLM-500M with one image tile) and the caption of the middle keyframe. The list shows the top 10 after fusion (236.5); the four boxes are averages over all 14 queries, computed live in your browser with (236.3), next to the first-stage value.
How to read Figure 2

A green bar on the left means the moment matches a true moment at tIoU \(\ge 0.5\); the number on the right is its best tIoU. Start at the default (Qwen, \(M = 30\), \(w = 0.75\)): mean mAP@0.5 is 0.880 against 0.859 for the first stage. Now slide \(w\) to 0: the VLM alone is clearly worse (0.776). Choose “a person swimming”: many candidates show people in or near water (a kid in a swimming pool, a girl wading in a lake), and the VLM and SigLIP disagree about which of them count, so the list changes a lot with \(w\). Switch to SmolVLM: at \(w = 0\) it falls to 0.644, but fused it is almost as good as Qwen.

Re-ranker results on test

All rows use \(M = 30\) candidates and 3 keyframes each, and \(w = 0.75\) (the best value on dev for every re-ranker), except the two rows with \(w = 0\).

ConfigurationHit@1R@10@0.5mAP@0.5mAP
first stage only1.0000.9050.8590.659
SmolVLM-500M alone (\(w = 0\))0.7140.7980.6440.475
Qwen2.5-VL-3B alone (\(w = 0\), \(\le 64\) visual tokens)0.9290.8810.7760.600
fused, SmolVLM-500M (1 tile)0.9290.9170.8740.663
fused, Qwen2.5-VL-3B (native resolution)0.9290.9170.8820.672
fused, Qwen2.5-VL-3B (\(\le 64\) visual tokens)0.9290.9170.8800.676
fused, Qwen \(\le 64\) tokens, \(M = 10\), 1 keyframe1.0000.9050.8680.668

Three things to notice. First, the VLM alone is worse than SigLIP here. Within the 30 candidates of each query, the first-stage cosine separates true moments from the rest with an AUC of 0.971 (AUC: the chance that a random correct candidate scores above a random wrong one), Qwen’s log-odds 0.921 and SmolVLM’s 0.898. A single keyframe judged in isolation is a noisy signal, and the VLM sees one photo while SigLIP has the whole 2-s segment pooled. Second, fusion helps a little. mAP@0.5 goes from 0.859 to 0.880. Third, Hit@1 drops from 1.000 to 0.929: one query lost its correct first result. A re-ranker can hurt as well as help, and you only see it if you measure both.

Worked example: is +0.021 real?

With 14 queries, one query that improves by 0.14 moves the mean by 0.01. So look at queries, not only at the mean. Per query, AP@0.5 with the fused Qwen re-ranker against the first stage: 6 queries better, 2 worse, 6 unchanged; the mean difference is +0.021.

Resample the 14 per-query differences with replacement 10,000 times (seed 0) and take the 2.5% and 97.5% points of the mean: the 95% bootstrap interval is [+0.004, +0.044]. The gain is probably positive but small, and this interval says nothing about other queries or real video. We tuned \(w\) on dev, so the test numbers are not inflated by tuning.

Is the re-ranker worth its cost, then? On this archive, barely. The design session’s case for it was fine-grained queries (“the keeper saves the penalty” against “the penalty is scored”), where a contrastive embedding cannot tell the two apart and a VLM that reads the frames with the question can. [Likely] Our category queries do not test that. This is a real lesson of the lab: measure the re-ranker on the queries you will get, and keep the switch to turn it off.

Can the system say “not found”?

We also asked four queries whose events we did not label: a rugby match, a person playing volleyball, a person playing golf, a person rowing a boat. On the test archive, the Qwen log-odds of their top first-stage result were +3.14, −0.65, +2.38 and +3.21. For the 14 labeled queries, the top result’s log-odds ranged from −1.74 (swimming) to +4.10. So no threshold separates “present” from “absent”. Why? We looked at the photos: the rugby result is a man in a rugby uniform, the golf result is “a girl getting ready to swing the club”, the rowing result is “a man paddling a boat”. They are background photos only by our labels; the VLM was right. Incomplete labels make a system look wrong when it is right. The next session, session 237, builds a failure analysis on exactly this kind of case.

# rerank.py: a VLM verifier that returns the log-odds of "yes" (formula 236.4), and the fusion (236.5)
from transformers import AutoModelForImageTextToText
class Verifier:
    def __init__(s, mid="Qwen/Qwen2.5-VL-3B-Instruct", max_pixels=64 * 28 * 28, batch=30):
        s.proc = AutoProcessor.from_pretrained(mid, max_pixels=max_pixels)   # at most 64 visual tokens
        s.proc.tokenizer.padding_side = "left"                 # position -1 is the answer slot in every row
        s.model = AutoModelForImageTextToText.from_pretrained(mid, dtype=torch.float16).to(DEV).eval()
        tk = s.proc.tokenizer; s.batch = batch
        s.YES = [tk.encode(w, add_special_tokens=False)[0] for w in ("Yes", "yes")]
        s.NO = [tk.encode(w, add_special_tokens=False)[0] for w in ("No", "no")]
    def prompt(s, q):
        m = [{"role": "user", "content": [{"type": "image"},
              {"type": "text", "text": f"Does this image show {q}? Answer yes or no."}]}]
        return s.proc.apply_chat_template(m, add_generation_prompt=True)
    def logodds(s, pics, q):
        out = []
        for a in range(0, len(pics), s.batch):
            chunk = pics[a:a + s.batch]
            x = s.proc(text=[s.prompt(q)] * len(chunk), images=chunk, return_tensors="pt", padding=True).to(DEV)
            x["pixel_values"] = x["pixel_values"].half()
            lg = s.model(**x, logits_to_keep=1).logits[:, -1].float()      # prefill only, last position
            out += (torch.logsumexp(lg[:, s.YES], 1) - torch.logsumexp(lg[:, s.NO], 1)).cpu().tolist()
        return out

def keyframes(vid, s, e):                                     # photos at 25%, 50%, 75% of the span
    ph = frames[vid][1]
    return [ph[t] for t in sorted({min(e - 1, s + int(f * (e - s))) for f in (0.25, 0.5, 0.75)})]
def zs(x):
    x = torch.tensor(x, dtype=torch.float64); return ((x - x.mean()) / (x.std(unbiased=False) + 1e-8)).tolist()
def rerank(R, q, V, M=30, w=0.75):
    top = R[:M]; kfs = [keyframes(*c[1:4]) for c in top]
    lo = V.logodds([img(p) for k in kfs for p in k], q); it = iter(lo)
    vl = [sum(next(it) for _ in k) / len(k) for k in kfs]     # mean log-odds per candidate
    f = [w * a + (1 - w) * b for a, b in zip(zs([c[0] for c in top]), zs(vl))]
    order = sorted(range(len(top)), key=lambda i: (-f[i], i))
    return [[f[i]] + top[i][1:] for i in order] + R[M:]

Step 5: timestamp refinement

A moment from Step 3 starts and ends on the 2-s grid. The refiner moves each boundary by up to \(r\) seconds to the place where the moment looks most different from its surroundings. It uses the stored per-second frame vectors: \(a_t = \mathbf{t}_q^\top \mathbf{z}_t\), z-scored over the whole video to \(\tilde a_t\). For a candidate span \([s, e)\) we compute a contrast: the mean inside minus the mean of two flanks of length \(h\) on each side:

\[ C(s, e) = \frac{1}{e - s}\sum_{t=s}^{e-1} \tilde a_t \;-\; \frac{1}{|L| + |R|}\Big(\sum_{t \in L} \tilde a_t + \sum_{t \in R} \tilde a_t\Big), \quad h = \max\!\Big(2, \Big\lfloor \frac{e - s}{2} \Big\rfloor\Big) \tag{236.6} \]
Read the symbols
\(L, R\)
the flanks: the \(h\) seconds before \(s\) and after \(e\), cut at the video ends
\(\tilde a_t\)
the query score of second \(t\), z-scored over the video so that “high” means high for this video
search
all \(s\) within \(\pm r\) of the old start and \(e\) within \(\pm r\) of the old end, with \(e - s \ge 2\); keep the \((s, e)\) with the largest \(C\)

This is a training-free cousin of the threshold rule (235.10) in the design. A threshold needs a value of \(\alpha\) that works for every query; the contrast only asks for the span that stands out most from its neighbors, which is scale-free. With \(r = 2\) it searches 25 spans or fewer per moment, from a prefix sum, so the cost is negligible. After refinement two results can collapse onto the same moment, so we drop any result that overlaps an earlier one at tIoU \(\ge 0.5\).

Query:
Result:
per-second score \(\tilde a_t\)first-stage spantrue momentrefined spanflanks
Figure 3. The contrast search (236.6) on real test results, computed in your browser from the stored per-second scores. Shots show one photo each, so the curve is flat within a shot and jumps at cuts. The refined span is the one with the largest contrast among all starts and ends within \(\pm r\) seconds of the first-stage span.
How to read Figure 3

Most results move by one second onto the shot cut and reach tIoU 1.0: the 2-s grid was the only error. Now pick “a dog catching a frisbee”, result 3: the true moment is [68, 75), but the neighboring background shots show dogs playing too, so their score is high and the contrast search grows the span; at \(r = 2\) the tIoU falls from 0.875 to 0.583. Raise \(r\): more freedom also means more room to drift. We chose \(r = 2\) on dev.

Refinement results

Test, top 10R@10@0.5R@10@0.7mAP@0.5mAP
first stage0.9050.8450.8590.659
first stage + refine (\(r = 2\))0.8810.8570.8270.749
fused Qwen re-rank0.9170.8570.8800.676
fused Qwen re-rank + refine (\(r = 2\))0.8930.8690.8480.763

Refinement raises mAP from 0.676 to 0.763: boundaries that were one second off become exact, which counts at the strict thresholds 0.8 to 0.95. But it lowers mAP@0.5 from 0.880 to 0.848, because a few moments drift into similar neighbors (the dog case of Figure 3) and fall under tIoU 0.5. On dev the same change was neutral at 0.5 (0.945 to 0.945 for the first stage) and +0.093 in mAP. Which number matters is a product decision: an editor who wants the exact second cares about mAP; a fan who just wants to find the goal cares about recall at 0.5. Report both.

# refine.py: contrast search for start and end (formula 236.6)
def contrast(c, s, e, T):                                     # c: prefix sums of the z-scored scores
    h = max(2, (e - s) // 2); a0, b1 = max(0, s - h), min(T, e + h); n = (s - a0) + (b1 - e)
    inside = (c[e] - c[s]) / (e - s)
    return inside - (((c[s] - c[a0]) + (c[b1] - c[e])) / n if n else 0.0)
def refine_one(vid, s0, e0, wq, r=2):
    a = (frames[vid][0] @ wq).double(); a = (a - a.mean()) / a.std(unbiased=False)
    T = len(a); c = [0.0] + a.cumsum(0).tolist(); best = None
    for s in range(max(0, s0 - r), min(T - 2, s0 + r) + 1):
        for e in range(max(s + 2, e0 - r), min(T, e0 + r) + 1):
            sc = contrast(c, s, e, T)
            if best is None or sc > best[0] + 1e-12: best = (sc, s, e)
    return best[1], best[2]
def refine(R, wq, K=10, r=2):
    out = []
    for c in R:
        s, e = refine_one(c[1], c[2], c[3], wq, r)
        if all(not (o[1] == c[1] and tiou((s, e), o[2:4]) >= 0.5) for o in out): out.append([c[0], c[1], s, e])
        if len(out) == K: break
    return out

Step 6: profile and fix the bottleneck

The pipeline now works. The first version that ran end to end used SmolVLM-500M in FP32 with its default image setting, one keyframe per call, and took 70 seconds per query. The design’s budget was 1.5 s. Before changing anything, measure where the time goes. Wrap each stage in a timer, and know one rule about GPUs: GPU calls return before the GPU finishes. PyTorch queues the work and moves on. If you read the clock without waiting, the GPU time is charged to whichever stage next waits for a result, usually a .cpu() or .tolist() later. So the timer synchronizes before every reading (session 123).

# timer.py: per-stage wall time; T("name") closes the running stage and opens a new one
import time
def sync():
    if DEV == "cuda": torch.cuda.synchronize()
    elif DEV == "mps": torch.mps.synchronize()
class Timer:
    def __init__(s): s.t, s.cur = {}, None
    def __call__(s, name):
        sync(); now = time.perf_counter()                     # wait for the GPU, or its time lands in the wrong stage
        if s.cur: s.t[s.cur] = s.t.get(s.cur, 0) + now - s.last
        s.cur, s.last = name, now
# inside search(): T = Timer(); T("text") ... T("search") ... T("keyframes") ... T("preprocess") ... T("vlm_gpu") ... T("end")
# run 2 warm-up queries first (model load, kernel compilation, caches), then average over the test queries
Algorithm 3: the profiling loop
  1. fix the workload: the same queries, the same candidates, warm-up first
  2. measure every stage with synchronized timers; also record the accuracy on dev
  3. find the largest stage; ask why it is large (too much work? work done badly? waiting?)
  4. change one thing; measure again; check accuracy did not fall more than you accept
  5. repeat until the budget is met; then project to the target hardware and confirm there

The speed-up from one fix is limited by the part it does not touch. If a fraction \(p\) of the time is made \(k\) times faster:

\[ \text{speed-up} = \frac{1}{(1 - p) + p / k} \;\le\; \frac{1}{1 - p} \tag{236.7} \]

This is Amdahl’s law. It explains why every fix below moved the bottleneck to a new place, and why the last fixes were not about the GPU at all.

Re-ranker:Scale:
text + search + groupingkeyframes (JPEG decode)preprocess (CPU)VLM on the GPUfuse + refine
Figure 4. Measured mean time per query, split by stage, for each step of the optimization (Apple M5 Pro, MPS, 30 candidates with 3 keyframes, about 80 VLM images per query, unless the step says otherwise). The slowest settings were averaged over 4 or 6 queries, the others over all 14. Step through to see what each fix changed and where the bottleneck moved.
How to read Figure 4

Each bar is one version of the service; the newest is highlighted. Watch the red part (the VLM on the GPU) shrink, and the orange part (CPU preprocessing: resizing images and building the prompt tensors) become a large share of what is left. The log scale shows the totals of all versions side by side, so you can compare 70 s with 1.7 s. Switch to Qwen: batching barely helps there, but fewer visual tokens do.

The fixes, one at a time

VersionChangeVLM GPU (ms)Total (ms)Speed-upWhy it helped, and what is next
S1SmolVLM FP32, image split into 13 tiles (883 prompt tokens), batch 167,35770,248GPU is 96% of the time
S2FP1621,40524,7512.84×half the bytes, faster matrix units; still GPU-bound
S3one tile (88 prompt tokens)16,47219,4821.27×10× fewer tokens but only 1.27× faster: at batch 1 each call is small and fixed per-call cost dominates [Likely]
S4all keyframes of a query in one batch5,5118,0842.41×fills the GPU; now CPU preprocessing is 2.35 s, 29% of the total
S5vision features computed offline; online runs only the language model1,6091,6804.81×removes the image preprocessing and the vision encoder from the query path
Worked example: why S5 had to leave the GPU alone

In S4 the GPU part is 5,511 of 8,084 ms, so \(p = 0.682\). By (236.7), even an infinitely fast GPU (\(k \to \infty\)) gives at most \(1 / (1 - 0.682) = 3.14\times\). The next fix had to remove the CPU work, not speed up the GPU.

The fix: the images are fixed at ingest time, so their vision features are too. Offline, we ran the vision encoder and the connector once per keyframe and stored the output: \(64 \times 960 \times 2 = 122{,}880\) bytes per keyframe (64 image tokens of 960 numbers, 2 bytes each). For the 2,080 test keyframes this took 216 s once and needs 256 MB. Online, the prompt is tokenized once per query, and the cached features are pasted into the image positions of the input embeddings. The total became 1,680 ms: \(70{,}248 / 1{,}680 = 41.8\times\) faster than S1.

The price: at the design’s scale (24 million segments, one keyframe each) the same cache would be \(24 \times 10^6 \times 122{,}880\) bytes = 2.95 TB, and 6.3 TB for Qwen2.5-VL-3B with its 2,048-wide language model. Possible, but a real storage decision. This is the same trade-off as the offline captions in the design: work moved offline is paid for every segment, used or not.

We did not re-score the whole test set with the cached path. It is the same computation in a different order, so the scores should match the uncached ones up to FP16 rounding; check this on a few hundred pairs before you trust it. [Likely]

The Qwen path

VersionChangeVLM GPU (ms)Total (ms)Speed-upTest mAP@0.5
Q1Qwen2.5-VL-3B FP16, native resolution (prompt 143 to 362 tokens, mean 261), batch 120,42221,2000.882
Q2batch 3018,76319,2261.10×0.882
Q3at most 64 visual tokens per image (prompt 84 to 102 tokens, mean 90.5)9,1909,5692.01×0.880
Q4\(M = 10\) candidates, 1 keyframe each (10 images instead of about 80)2,5042,6753.58×0.868

Batching helped SmolVLM 2.4× but Qwen only 1.1×. With about 260 tokens per prompt and a 3-billion-parameter model, a single Qwen call already gives the GPU enough work: it is compute-bound, so there is little idle capacity for batching to fill. SmolVLM’s one-tile prompt of 88 tokens on a 0.5-billion model is the opposite. [Likely] For Qwen, the levers are the amount of work: fewer visual tokens (2.0× for −0.002 mAP@0.5) and fewer images (3.6× for −0.012). Every row has a cost in the last column; that column is why we kept a dev and a test split.

Step 7: serve the re-ranker with vLLM or TensorRT-LLM

The Hugging Face forward pass is fine for measuring, but not for serving many users. A serving engine adds continuous batching (requests from different users share GPU batches as they arrive), paged KV-cache memory, CUDA graphs and fused kernels (session 198). For our re-ranker every request is a prefill plus exactly one output token, which suits these engines well: no long decodes hold batch slots.

vLLM

vLLM runs Qwen2.5-VL directly from the Hugging Face checkpoint. Its example script for vision-language models says its default settings were tested on a single L4, and for Qwen2.5-VL-3B it uses max_model_len=4096 and max_num_seqs=5 [Certain]. We ask for one token with its log-probabilities and compute (236.4) from them.

# serve_vllm.py: the re-ranker on vLLM (one output token, read log-probabilities)
import math
from vllm import LLM, SamplingParams
llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct", max_model_len=2048, max_num_seqs=32, seed=0,
          limit_mm_per_prompt={"image": 1},
          mm_processor_kwargs={"max_pixels": 64 * 28 * 28})   # the Q3 setting: at most 64 visual tokens
tok = llm.get_tokenizer()
YES = [tok.encode(w, add_special_tokens=False)[0] for w in ("Yes", "yes")]
NO = [tok.encode(w, add_special_tokens=False)[0] for w in ("No", "no")]
P = ("<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n"
     "<|vision_start|><|image_pad|><|vision_end|>Does this image show {q}? Answer yes or no.<|im_end|>\n"
     "<|im_start|>assistant\n")
sp = SamplingParams(max_tokens=1, temperature=0.0, logprobs=20)  # top-20 log-probs of the one new token
def logodds(pics, q):
    outs = llm.generate([{"prompt": P.format(q=q), "multi_modal_data": {"image": im}} for im in pics], sp)
    res = []
    for o in outs:
        lp = o.outputs[0].logprobs[0]                         # dict: token id -> Logprob
        lse = lambda ids: math.log(sum(math.exp(lp[i].logprob) for i in ids if i in lp) or 1e-30)
        res.append(lse(YES) - lse(NO))                         # formula (236.4); a missing token counts as ~0
    return res

Check one thing before you trust it: on 100 (image, query) pairs, the vLLM log-odds must match the Hugging Face ones of Step 4 closely (small FP16 differences are normal). If “No” is outside the top 20 for a confident “yes”, the formula above clips it; that only happens for very large log-odds and does not change the order much. [Likely] vLLM can also take precomputed image embeddings (with enable_mm_embeds=True, and image_grid_thw for Qwen2-VL-style models), which is the serving version of our S5 cache [Certain, vLLM docs]. Keep the question text before the image in the prompt if you want prefix caching to reuse the computation for the shared part across the \(M\) candidates of one query; with the image first, only the system prompt is shared. [Likely]

TensorRT-LLM

The design fixed the hardware: L4 GPUs with TensorRT. TensorRT-LLM’s support matrix lists Qwen2_5_VLForConditionalGeneration with language, image and video inputs, and with chunked prefill, CUDA graphs and KV-cache reuse, on its PyTorch backend [Certain, TensorRT-LLM docs, checked November 2026]. The quickest start is its OpenAI-compatible server, trtllm-serve Qwen/Qwen2.5-VL-3B-Instruct --backend pytorch, and a chat request with max_tokens=1 that asks for the top log-probabilities of the answer token. Check that your version returns them; if not, use its Python LLM API, which follows the same pattern as the vLLM code above. [Likely] On the L4 (an Ada GPU), FP8 weights for the language model are the next lever after the ones in Figure 4; compare the log-odds with the FP16 ones on dev before switching (session 131).

From the laptop to the L4

Absolute times on a laptop say little about an L4. What transfers is the work: tokens and FLOPs. Use the design formulas (235.8) and (235.9) with the measured token counts:

\[ t_{\text{query}} \approx n_{\text{img}} \cdot \frac{2 P_{\text{vit}} \cdot 4 T_v + 2 P_{\text{llm}} (T_v + T_q)}{\eta \, P_{\text{peak}}} \tag{236.8} \]
Read the symbols
\(n_{\text{img}}\)
VLM calls per query: \(M\) candidates \(\times\) keyframes per candidate
\(T_v, T_q\)
visual tokens per image and the other prompt tokens (measured: 64 and about 26.5)
\(P_{\text{vit}}, P_{\text{llm}}\)
0.667 and 2.775 billion weights in the linear layers of Qwen2.5-VL-3B’s vision encoder and language model (from the design session)
\(\eta P_{\text{peak}}\)
effective speed: \(0.5 \times 121\) TFLOP/s dense BF16 on the L4 = 60.5 TFLOP/s
Worked example: the Q4 setting on an L4

One call: \(2 \times 0.667 \times 10^9 \times 256 + 2 \times 2.775 \times 10^9 \times 90.5 = 0.342 + 0.502 = 0.844\) TFLOP. Ten calls: 8.44 TFLOP, so about 140 ms at 60.5 TFLOP/s. GPUs for 5 queries per second at 60% utilization: \(5 \times 0.1395 / 0.6 = 1.16\), so 2 L4s. The design’s default re-ranker (20 candidates, 4 frames at 336) needed 1,154 ms and 10 L4s.

With Q3 (\(n_{\text{img}} = 30 \times 3 = 90\) calls) the same formula gives 1,255 ms and 11 L4s; with native resolution (about 234 visual tokens) and 90 calls, 4,015 ms. These are projections, not measurements. Small batches of short prompts may not reach \(\eta = 0.5\), and the formula ignores attention and the CPU work we found in Step 6. Confirm on an L4 with the timer above. [Likely]

Figure 5. Projection of the measured re-ranker settings to an NVIDIA L4 with (236.8): Qwen2.5-VL-3B in BF16, 26.5 other prompt tokens, 90 ms for the fixed stages of the design, target utilization 0.6. The markers show the design’s limits: 1.5 s per query and 12 GPUs.
How to read Figure 5

The default is our Q4 setting: about 140 ms of re-ranking and 2 L4s at 5 queries per second. Set \(M = 30\) and 3 keyframes (Q3): about 1.26 s and 11 GPUs, near both limits. Then raise \(T_v\) to 236 (about the native resolution of these photos): about 4.0 s of re-ranking, far over the budget. Lower \(\eta\) to 0.2 to see how sensitive the plan is to the efficiency you assume; this is the number you must measure first on a real L4.

The full service

Put the pieces together into one class. This is the program behind every number in Figure 4: change the constructor arguments to get each version.

# service.py: the online search service with a stage timer
class SearchService:
    def __init__(s, split="test", verifier=None, M=30, nkf=3, w=0.75, r=2):
        s.X, s.meta, _ = build_index(B[split]); s.X = s.X.to(DEV)     # (N, 768) FP16 on the GPU
        s.V, s.M, s.nkf, s.w, s.r = verifier, M, nkf, w, r
    def search(s, q, K=10):
        T = Timer(); T("text")
        x = sp(text=[f"a photo of {q}."], padding="max_length", max_length=64, return_tensors="pt").to(DEV)
        wq = F.normalize(vec(sig.get_text_features(input_ids=x["input_ids"])).float(), dim=-1)[0]
        T("search"); top = torch.topk(s.X.float() @ wq, 100)
        vals, idx = top.values.cpu().tolist(), top.indices.cpu().tolist()
        T("group"); R = group(s.meta, vals, idx)              # formula (236.2), as in search.py
        if s.V:
            cand = R[:s.M]; kfs = [keyframes(*c[1:4]) for c in cand]
            if s.nkf == 1: kfs = [[k[len(k) // 2]] for k in kfs]
            T("keyframes"); pics = [img(p) for k in kfs for p in k]          # JPEG decode on the CPU
            lo = s.V.logodds(pics, q, T); it = iter(lo)                       # T("preprocess"), T("vlm_gpu") inside
            T("fuse_refine"); vl = [sum(next(it) for _ in k) / len(k) for k in kfs]
            f = [s.w * a + (1 - s.w) * b for a, b in zip(zs([c[0] for c in cand]), zs(vl))]
            R = [[f[i]] + cand[i][1:] for i in sorted(range(len(cand)), key=lambda i: (-f[i], i))] + R[s.M:]
        T("fuse_refine"); out = refine(R[:3 * K], wq.cpu(), K, s.r)
        T("end"); return out, T.t

S = SearchService("test", Verifier(), M=10, nkf=1)            # the Q4 version
for q in Q[:2]: S.search(q)                                    # warm-up
tot = {}
for q in Q:
    out, t = S.search(q)
    for k, v in t.items(): tot[k] = tot.get(k, 0) + v
print({k: round(1000 * v / len(Q), 1) for k, v in tot.items()})   # ms per stage, mean over the 14 queries

Two details matter. The Verifier’s logodds takes the timer and calls T("preprocess") before the processor and T("vlm_gpu") before the forward pass, so the profile separates CPU preparation from GPU work. And the warm-up queries are not counted: the first call compiles kernels and fills caches, and can be many times slower.

Where this is used

ApplicationWhich part of this lab carries over
Sports archive search for editors (the capstone)all of it; replace photos by decoded frames, add shot detection, scoreboard clock, transcripts and replay merging from the design
Key-moment detection and highlight reelsthe per-second scores and the contrast refiner, with a saliency score instead of a query (session 207)
Text-to-video retrieval in a media libraryindex, grouping and fusion; a VLM verifier for the top results (session 208)
Video-RAG assistantsthe same retrieval feeds moments into a Video-LLM that answers questions (session 210)
Content moderation and brand safety“does this frame show X?” log-odds as a calibrated verifier on retrieved candidates
Any RAG system with a cross-encoder re-rankerretrieve-then-re-rank, z-score fusion, tuning on dev, and the profiling loop of Step 6

Common mistakes

  • Timing GPU code without synchronizing. The time lands in a later stage, and you optimize the wrong thing. Synchronize before each clock reading, and warm up first.
  • Replacing the first-stage score with the VLM score. Here the VLM alone lowered mAP@0.5 from 0.859 to 0.776. Fuse, and tune the weight on dev.
  • Tuning on the test split. \(w\), \(M\), \(r\) and the grouping rule were chosen on dev; otherwise the test numbers measure your tuning, not your system.
  • Reading \(p(\text{yes})\) from one spelling only. Qwen answered “No” 871 times, “Yes” 274 times and “yes” 3 times on our pairs; with native resolution, lower-case answers were much more common (“yes” 152, “no” 219). Sum over the spellings, as in (236.4).
  • Right padding in a batched prefill. With right padding, position −1 is a pad token for shorter prompts. Pad on the left, or index each row at its own last position.
  • Optimizing the GPU when the CPU is the bottleneck. After batching, 29% of the time was CPU preprocessing; Amdahl capped any GPU fix at 3.1×.
  • Reporting one number. A re-ranker can raise mAP@0.5 and lower Hit@1; refinement raised mAP and lowered mAP@0.5. Report several metrics and per-query differences.
  • Trusting “absent” labels. Our “not in the archive” queries were in the archive, just unlabeled. Check the top results by eye before you blame the model.

Interview questions

Describe the system you built and its measured numbers.basic
An offline indexer embeds every second with SigLIP-B/16, mean-pools 2-s segments and stores one FP16 matrix (\(3{,}600 \times 768\), 5.5 MB for 2 hours). The online service embeds the query, takes the top 100 segments, groups touching hits into moments (score = max), re-ranks the top \(M\) with a VLM’s log-odds of “yes” fused with the cosine by z-scores (\(w = 0.75\) from dev), and refines boundaries with a contrast search. On a 40-video, 84-moment test archive: first stage mAP@0.5 0.859, fused 0.880, with refinement mAP 0.763. Profiling took a query from 70 s to 1.7 s on a laptop GPU; the projected L4 re-rank time for the 10-candidate setting is about 140 ms.
Why use the log-odds of “yes” from one forward pass instead of generating an answer?core
It needs only the prefill, no decode loop, so it is cheap and batches well. It gives a continuous score for ranking instead of a binary answer. And it avoids parsing free text. Use log-sum-exp over all spellings (“Yes”, “yes”) on each side; the softmax denominator cancels, so it is just a difference of two log-sum-exps over a few logits. Log-odds do not saturate the way \(p(\text{yes})\) does, which matters when you add it to another score.
Your VLM re-ranker alone is worse than the embedding search. Why might that be, and what do you do?core
On category queries a contrastive model is strong (AUC 0.971 within the candidates here, against 0.921 for the VLM). The VLM sees one keyframe at a time, at reduced resolution, and has its own biases (for example, whether a kid playing in a pool counts as “a person swimming”). Do not replace; fuse with a tuned weight and normalize scales first (z-scores within the candidate list). Then ask whether your real queries are fine-grained enough to need a VLM at all; measure it on those queries, and keep a switch to turn it off.
How do you know a +0.021 mAP gain on 14 queries is real?core
Look at per-query differences: here 6 better, 2 worse, 6 equal. Bootstrap over queries: resample the 14 differences 10,000 times; the 95% interval was [+0.004, +0.044]. Make sure the setting was chosen on a separate dev split. Then say what the test cannot show: other query types, real video, other leagues. A larger, independently labeled test set is the real fix.
How did you find the bottleneck, and what is the classic mistake when timing GPU code?core
A per-stage timer around text encoding, search, grouping, keyframe decode, preprocessing, VLM forward and fusion, averaged over the test queries after warm-up. The classic mistake is not synchronizing: GPU calls are asynchronous, so the GPU time shows up in whatever stage first waits on a result. Call torch.cuda.synchronize() (or the MPS version) before each clock reading, or use CUDA events or a profiler for kernel-level detail.
Batching gave 2.4× for SmolVLM but only 1.1× for Qwen2.5-VL-3B. Explain.advanced
Batching helps when single calls leave the GPU idle: small models and short prompts are dominated by per-call overhead and memory traffic for the weights. SmolVLM with one tile is 88 tokens on 0.5B parameters, so batching 80 images fills the GPU. Qwen at native resolution is about 260 tokens on 3B parameters per call, already enough work to be compute-bound, so batching adds little. The lever for Qwen is less work: cap visual tokens (2.0×) and score fewer images (3.6×).
You cached the vision features offline and got 4.8×. When is that a bad idea?advanced
When storage or freshness costs more than compute. Our cache is 122,880 bytes per keyframe; at 24 million segments that is 2.95 TB for SmolVLM and 6.3 TB for Qwen2.5-VL-3B, paid for every segment, even those no query ever touches. It also ties the cache to one model version and one image resolution: change either and you rebuild everything. It is a good idea when the candidate pool is small or hot (recent games), or when CPU preprocessing is the bottleneck and cannot be moved to the GPU.
Why did timestamp refinement raise mAP but lower mAP@0.5?core
mAP averages over tIoU thresholds from 0.5 to 0.95, so it rewards exact boundaries; the 2-s grid left many boundaries one second off, and the contrast search fixes those (mAP 0.676 to 0.763). But when neighboring shots look like the event (dogs playing next to a frisbee catch), the search can grow the span into them and push tIoU below 0.5 (0.875 to 0.583 in one case), which lowers recall at 0.5. Limit the radius, tune it on dev, and report both metrics.
vLLM or TensorRT-LLM for this re-ranker on L4s?core
Both support Qwen2.5-VL. vLLM is quick to set up from the Hugging Face checkpoint, has continuous batching and prefix caching, and can take precomputed image embeddings. TensorRT-LLM fits the design’s fixed L4 + TensorRT stack, supports Qwen2.5-VL with chunked prefill, CUDA graphs and KV-cache reuse on its PyTorch backend, and gives access to FP8 on Ada GPUs. For a one-token prefill workload, either is fine; choose by the team’s stack, check that log-probabilities are returned, and compare log-odds against the reference implementation before switching.
You measured on a laptop. How do you turn that into an L4 plan?advanced
Transfer the work, not the time: tokens per call (measured: 64 visual and about 26.5 others), calls per query, and parameter counts give FLOPs per query; divide by an assumed effective speed (\(\eta \times\) peak). For 10 calls that is 8.44 TFLOP, about 140 ms at 60.5 TFLOP/s, so 2 L4s at 5 queries per second. Then list what the projection misses: the real \(\eta\) for small batches, attention, CPU preprocessing (which was 29% at one point), and queueing for p95. The first thing to do on an L4 is to measure \(\eta\) with the same timer.
How would the system return “no result” for an event that is not in the archive?advanced
With a calibrated threshold on the verifier score, tuned on dev with real negative queries. Our attempt also showed a trap: the four “absent” queries were in fact present in unlabeled background photos (a rugby uniform, a golf swing, a man paddling a boat), so the VLM was right and the labels were wrong. Before trusting a threshold, audit the top results of negative queries by eye, and label negatives as carefully as positives.

Exercises

  1. Your archive grows to 500 hours. With 2-s segments and FP16 768-d vectors, how big is the flat index? Is a flat search still reasonable?
  2. In version S3, the CPU stages (keyframes and preprocessing) took 2,953 of 19,482 ms. What is the largest speed-up any GPU-only optimization could give?
  3. Re-run Figure 2 with SmolVLM and find the \(w\) where the mean mAP@0.5 is highest. Is it the same \(w\) as for Qwen? Why is it dangerous to pick it on this figure?
  4. Use (236.8): how many L4s at 5 queries per second for \(M = 20\), 2 keyframes, 64 visual tokens, \(\eta = 0.5\), utilization 0.6?
  5. Implement the threshold refinement of (235.10) with \(\alpha = 0.5\) and compare it with the contrast search on dev. Which do you expect to be more robust across queries, and why?
  6. Change the prompt so that the question comes before the image. Measure whether the log-odds change on 100 pairs, and whether vLLM prefix caching lowers the time per query.
Answerscheck
1. \(500 \times 3600 / 2 = 900{,}000\) segments; \(900{,}000 \times 768 \times 2 = 1.38\) GB. A flat search reads 1.38 GB per query: on a GPU with 300 GB/s that is about 5 ms, still fine; on a CPU it is slower, and at tens of millions of segments IVF-PQ becomes necessary. 2. The GPU part is \(p = 16{,}472 / 19{,}482 = 0.846\); the limit is \(1 / (1 - p) = 19{,}482 / 3{,}010 \approx 6.5\times\) (counting all non-GPU stages, 3,010 ms). 3. Use the figure; the important part is the second question: any value picked on the test set is tuned on test, and its score is optimistic. Pick on dev (it was 0.75 for both on dev). 4. \(40 \times 0.844 = 33.8\) TFLOP (40 calls); at 60.5 TFLOP/s, 0.558 s per query; \(5 \times 0.558 / 0.6 = 4.65\), so 5 L4s. 5. The contrast search is scale-free: it compares the span with its own neighbors, so it needs no per-query threshold; a fixed \(\alpha\) depends on how peaked the score curve is, which varies by query. Measure both: expected but not guaranteed. 6. The answer is a measurement; expect small changes in log-odds and a gain from prefix caching only for the shared text part, which is short here.

Key takeaways

  • Build a labeled stand-in when real data is not available, split it into dev and test, and say clearly what it cannot test (here: motion, replays, fine-grained queries).
  • The index is a normalized, mean-pooled FP16 matrix (236.1); group touching hits into moments (236.2); score with tIoU-based AP and mAP (236.3).
  • A VLM re-ranker is one prefill and a log-odds (236.4). Alone it was worse than SigLIP here; fused (236.5) it gave +0.021 mAP@0.5, with a 95% bootstrap interval of [+0.004, +0.044].
  • Contrast refinement (236.6) fixed grid-quantized boundaries (mAP 0.676 to 0.763) but cost some recall at tIoU 0.5.
  • Profile with synchronized timers and fix one bottleneck at a time; Amdahl (236.7) tells you when the GPU is no longer the problem. 70 s became 1.7 s on the same laptop.
  • Serve with vLLM or TensorRT-LLM; project to the L4 through work, not time (236.8), then measure there.

Key formulas

\[ \ell = \operatorname{LSE}(z_{\mathcal{Y}}) - \operatorname{LSE}(z_{\mathcal{N}}), \quad f = w\, z(s) + (1-w)\, z(\bar\ell), \quad \text{speed-up} \le \frac{1}{1-p}, \quad t \approx n_{\text{img}} \frac{2P_{\text{vit}} 4T_v + 2P_{\text{llm}}(T_v + T_q)}{\eta P} \]

References for this session

  1. vLLM project (docs). vLLM documentation
  2. NVIDIA (code). TensorRT-LLM
  3. vLLM project (docs). vLLM documentation
  4. NVIDIA (code). TensorRT-LLM
  5. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  6. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  7. Bai et al. (2025). Qwen2.5-VL Technical Report
  8. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
  9. Hodosh, Young & Hockenmaier (2013). Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (Flickr8k)
  10. Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
  11. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  12. Johnson et al. (2017). Billion-scale similarity search with GPUs
  13. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
  14. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  15. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models