LLM & VLM courseBlog
235Track 10: Capstone and interviews · session 2 of 6

Capstone design: sports key-moment search

A sports broadcaster wants editors and fans to type “penalty saved by the keeper in the second half” and get the exact seconds of video back. This capstone designs that system on paper before any code: the requirements, the pipeline (segments, embeddings, an index, a VLM re-ranker, timestamps), the latency and cost math on NVIDIA L4 GPUs with TensorRT, and the risks and metrics that decide whether it works.

Time: 3 to 4 hours (reading 2 hours, exercises and your own design 1 to 2 hours)Level: Advanced · system designBefore this: Text-to-video retrieval (208), temporal grounding (207), vector search at scale (151), TensorRT (131), the L4 latency budget (199)

What you will learn

  • How to turn a vague product wish into requirements: query types, latency targets, accuracy metrics, cost limits and fixed hardware
  • A complete design: decode, shot-aware segments, SigLIP embeddings, an IVF-PQ index with metadata filters, transcript fusion, a VLM re-ranker and timestamp refinement
  • How window length and stride decide the best possible tIoU and the size of the index
  • Why metadata filters and the number of re-ranked candidates control recall, and how to reason about it with a simple model
  • Latency and cost math for a VLM re-ranker on an L4: visual tokens, FLOPs, milliseconds per candidate, GPUs and dollars per 1,000 queries
  • The main risks (replays, commentary delay, ambiguous queries, domain shift, model upgrades) and what to measure at each stage

Why this matters

A football broadcaster has 20,000 hours of match video. An editor needs “every penalty saved by a goalkeeper this season” for a show that airs in an hour. A fan types “Salah curling shot into the top corner”. Today, someone scrubs through video by hand, or searches a list of events that a logging team typed during the game. This session designs a system that answers such questions with the exact seconds of video, on a fixed hardware budget: NVIDIA L4 GPUs running TensorRT engines.

It is a capstone because it uses almost everything in Track 8 and much of the inference track: text-to-video retrieval (session 208), temporal grounding (session 207), vector indexes (session 151), compilation with TensorRT (session 131), and the pencil-and-paper latency budget of session 199. In session 234 you took one model apart. Here you put many models together into one service, and you decide on paper whether it can work before anyone writes code. In the next session, session 236, you build a small version of it.

A key moment here means a short span of video (a few seconds to a minute) that matches what the user asked for: a goal, a save, a foul, a celebration, a tactical pattern. The system returns a ranked list of moments, each with a game, a start time and an end time.

Requirements: queries, latency, accuracy, cost, hardware

A design starts from requirements, not from models. Ask questions until every requirement is a number, or a short list. Here is the set we will design for. These are our design inputs for the example, not facts about any real broadcaster.

RequirementOur valueWhy it matters for the design
Archive20,000 hours of football broadcasts (about 10,000 games of about 2 hours each, with pre-game and half-time)sets the index size and the one-time ingest cost
New videoabout 100 hours per day; searchable within 1 hour after the game endssets the daily ingest cost; “1 hour” allows batch ingest, not streaming
Query types(a) event type: “all red cards”; (b) event + entity: “Haaland headers”; (c) free description: “keeper slips and the ball goes in”; (d) filters: team, season, competition, half(a) and (d) can use structured data; (b) and (c) need visual search and a re-ranker
Load5 queries per second (QPS) at peak, 1 QPS on averagesets the number of GPUs for the online path
Latencyfirst results in at most 0.3 s; final re-ranked results with timestamps in at most 1.5 s (95th percentile)limits how much VLM work we can do per query
AccuracyHit@10 at least 0.85 on our own test set of 500 queries; for hits, tIoU at least 0.5 in 85% of casesdefines “works”; we will measure each stage separately
Costat most 12 L4 GPUs for the online path at peakcaps the re-ranker size and the number of candidates
HardwareNVIDIA L4 (fixed by the platform team), models compiled with TensorRT or TensorRT-LLMfixes peak compute, memory and video decoders

Hit@10 means: the right moment is somewhere in the top 10 results. tIoU (temporal intersection over union) measures how well a predicted time span overlaps the true span; we define it in formula (235.2). 95th percentile (p95) latency means that 95% of queries finish faster than this.

The hardware: what an L4 gives us

From NVIDIA’s product page [Certain]: 24 GB of memory, 300 GB/s of memory bandwidth, 72 W, and 242 TFLOP/s BF16 and 485 TFLOP/s FP8 on the tensor cores with 2:4 sparsity; dense numbers are half: 121 TFLOP/s BF16 and 242 TFLOP/s FP8. It also has 4 video decoders (NVDEC), 4 JPEG decoders and 2 video encoders. For video work the decoders matter as much as the tensor cores: frames must be decoded before any model can see them.

As in session 199, we assume real kernels reach \(\eta = 0.5\) of the dense peak, so about 60.5 TFLOP/s in BF16. This is an assumption to replace with a measurement later. [Likely]

TensorRT compiles a model graph into an engine: fused, timed kernels for one GPU model and one TensorRT version. [Certain] For us this means: build the engines on L4s, declare the input shapes you will use (batch sizes, image sizes) as optimization profiles, and rebuild when you upgrade TensorRT or change GPUs. TensorRT-LLM is the matching runtime for the language part of a VLM. Whether a given VLM is supported by TensorRT-LLM must be checked for the exact model and version. [Likely]

The design at a glance

The system has two paths. The offline path (ingest) runs once per game: it decodes the video, cuts it into short segments, computes vectors and text for each segment, and writes them into an index. The online path (query) runs for every user request: it reads the query, searches the index, re-ranks the best candidates with a vision-language model (VLM), and returns moments with precise times. The rule of the whole design: do expensive work offline, once per video; do only cheap or query-specific work online.

offline: once per gameonline: once per query
Figure 1. The two paths of the design. The top row runs at ingest time; the bottom row runs for every query. Step through to see what each box does, which model it uses, and the time or cost we budget for it.
How to read Figure 1

Each step highlights one box and explains its job and its budget. Notice where the GPU time goes: offline, the frame encoder and the decoders; online, almost all of it goes to the VLM re-ranker. That is why most of the latency and cost math later in this page is about the re-ranker. The arrow from the index down to the online row is the only link between the two paths: the online path never touches raw video except to fetch a few frames for the re-ranker.

Segmenting: the unit of search

We cannot index a 2-hour video as one vector: the query is about 5 seconds of it. We also should not index single frames: one frame cannot show “a slide tackle”, and frames are 7,200 per hour even at 2 frames per second. The unit of search is a segment: a short window of video, for example 6 seconds. We slide the window along the video with a stride (the step between window starts):

\[ N_{\text{seg}} = \left\lfloor \frac{D - w}{s} \right\rfloor + 1 \;\approx\; \frac{D}{s}, \qquad N_{\text{archive}} \approx \frac{3600\, H}{s} \tag{235.1} \]
Read the symbols
\(D\)
length of one video in seconds
\(w\)
window length in seconds (how much video one vector describes)
\(s\)
stride in seconds; \(s \lt w\) means windows overlap
\(H\)
hours of video in the archive

With \(w = 6\) s and \(s = 3\) s (50% overlap), the 20,000-hour archive gives \(72{,}000{,}000 / 3 = 24.0\) million segments. Halving the stride doubles the index; doubling it halves the index but, as we see next, hurts the timestamps.

How good can a fixed window be?

A moment has a true span \([a^*, b^*]\). A window \([a, b]\) is scored against it with tIoU, the length of the overlap divided by the length of the union:

\[ \text{tIoU}\big([a,b],[a^*,b^*]\big) = \frac{\max\!\big(0,\ \min(b, b^*) - \max(a, a^*)\big)}{\max(b, b^*) - \min(a, a^*)} \tag{235.2} \]

Even a perfect retriever can only return the best window in the grid. Where the true event falls relative to the grid is random, so the best tIoU is a random number. Figure 2 computes its average and its worst case. Two facts come out. First, with a 6-s event, a 6-s window and a 3-s stride, the worst case is 0.60 and the average is 0.79: every event can be hit at tIoU 0.5. With a 6-s stride (no overlap), the worst case drops to 0.33 and only 67% of positions reach 0.5. Second, windows much shorter than the event can never reach 0.5 alone (a 2-s window on a 6-s event gives at most 0.33), so short windows need a later step that merges neighbors. That step is the timestamp refinement below.

true eventbest windowother windows
Figure 2. Window length and stride against the best possible tIoU. Top: the window grid and one event; the best window is highlighted. Bottom: the best tIoU for every possible event start within one stride (the pattern repeats every stride), with its average. The readout also gives the index size for the 20,000-hour archive.
How to read Figure 2

Move “event start” to see the best and worst positions. Then raise the stride to the window length: the dip in the bottom curve gets deep, because some events fall between two windows. Lower the stride: the curve flattens, but the number of segments, and so the index memory, grows as \(1/s\). Set the event to 15 s and the window to 6 s: no single window can reach tIoU 0.5, so long events need merged windows or a refinement step.

Two practical rules improve on a plain grid. Snap to shots: broadcast video is cut into camera shots, and a replay or a close-up starts at a cut. A shot-boundary detector such as TransNet V2 (session 211) lets us end windows at cuts, so a window does not mix the wide game camera with a crowd shot. Keep finer data on the side: we store the per-frame embeddings at 2 frames per second (not in the index, just on disk), so the timestamp step can work at 0.5-s resolution without a bigger index.

Embedding and indexing

Each segment needs a vector that a text query can be compared with. We use a contrastive image–text model such as SigLIP: it maps an image and a sentence into the same space, so that matching pairs have a high cosine similarity (session 208). We embed frames, then pool them into one segment vector. CLIP4Clip showed that plain mean pooling of frame features is a strong, parameter-free baseline for video–text retrieval. [Likely]

\[ \mathbf{v}_j = \frac{\bar{\mathbf{z}}_j}{\lVert \bar{\mathbf{z}}_j \rVert}, \quad \bar{\mathbf{z}}_j = \frac{1}{|S_j|} \sum_{f \in S_j} \mathbf{z}_f, \qquad \text{score}(q, j) = \mathbf{t}_q^\top \mathbf{v}_j \tag{235.3} \]
Read the symbols
\(S_j\)
the sampled frames inside segment \(j\) (12 frames for 6 s at 2 frames per second)
\(\mathbf{z}_f\)
the image embedding of frame \(f\)
\(\mathbf{v}_j\)
the unit-length segment vector stored in the index
\(\mathbf{t}_q\)
the unit-length text embedding of the query; the dot product of two unit vectors is their cosine

Text from the broadcast

Sports video comes with strong text signals. The commentary says “GOAL!” and the player’s name; a speech recognizer such as Whisper turns it into timed text. The scoreboard shows the game clock and the score; reading it (OCR) gives a game time for every second of video, and a score change is a near-certain goal signal. Official event feeds (from the league or a data provider) list goals, cards and substitutions with game times. We index the transcript with a keyword index (BM25) next to the vector index and merge the two ranked lists with reciprocal rank fusion (RRF), which needs only ranks, not comparable scores:

\[ \text{RRF}(j) = \sum_{m \in \text{lists}} \frac{1}{k + \operatorname{rank}_m(j)}, \qquad k = 60 \tag{235.4} \]

A segment missing from a list adds nothing for that list. \(k = 60\) is the value from the original RRF paper. [Certain]

Index size

Memory for the vectors is simple arithmetic:

\[ \text{Mem} = N \cdot (b_{\text{code}} + b_{\text{id}}), \qquad b_{\text{code}} = d \cdot \text{bytes per number (flat)}, \;\; \text{or } m \text{ bytes (PQ)} \tag{235.5} \]
Worked example: can the index live on one L4?

\(N = 24.0\) million segments, \(d = 768\) (SigLIP base).

FP32 flat: \(24 \times 10^6 \times 768 \times 4 = 73.7\) GB. FP16 flat: 36.9 GB. INT8: 18.4 GB. None of these fits next to a VLM on a 24 GB L4, and FP16 does not fit at all.

IVF-PQ with \(m = 96\) bytes per vector plus an 8-byte id: \(24 \times 10^6 \times 104 = 2.50\) GB. This fits easily in CPU memory or on a GPU. Product quantization (PQ) stores each vector as 96 one-byte codes, and IVF searches only the few clusters near the query (session 151). We keep the FP16 vectors on disk and re-score the top few hundred PQ results exactly.

Two more index features matter as much as speed. Metadata filters: most queries name a team, a player, a season or a competition. Each segment carries the game id, teams, date, half and game clock, and the search runs only over the matching games. Versioning: every vector belongs to one embedding model version. A new model means re-embedding the archive into a new index, not mixing two models in one index.

Ingest cost

The offline cost per hour of video is the encoder FLOPs for all sampled frames divided by the effective GPU speed:

\[ t_{\text{ingest}} \text{ per video-hour} = \frac{3600 \cdot f_s \cdot C_{\text{frame}}}{\eta\, P} \tag{235.6} \]
Worked example: embedding the archive with SigLIP-B/16 at 224×224

The vision tower has 12 layers of width 768 and MLP width 3,072: about 85 million weights in its linear layers. A 224×224 frame gives \(14 \times 14 = 196\) patches. FLOPs per frame \(\approx 2 \times 85 \times 10^6 \times 196\) plus attention \(\approx 34.7\) GFLOP.

At \(f_s = 2\) frames per second: \(7{,}200 \times 34.7 \text{ GFLOP} = 250\) TFLOP per video-hour. At 60.5 TFLOP/s, that is about 4.1 s of one L4 per video-hour. The whole archive: \(20{,}000 \times 4.1\) s \(\approx\) 23 GPU-hours. The daily 100 hours: about 7 minutes.

The catch is decoding. A 25 frames-per-second broadcast has 90,000 frames per hour, and a decoder must decode nearly all of them to output 2 per second (only keyframes can be decoded alone). That is 12.5 times more frames decoded than embedded. On an L4 the 4 hardware decoders, not the tensor cores, often set the ingest speed; measure it with your codec and resolution before planning. [Likely]

Retrieve, then re-rank

A single embedding comparison is fast but shallow. It cannot reliably tell “the keeper saves the penalty” from “the penalty is scored”: both have a keeper, a ball and a penalty spot. A VLM that looks at the frames and the query together can tell, but it costs tens of milliseconds per candidate, so it cannot look at 24 million segments. The standard answer is a cascade: a cheap first stage returns the top \(K\) candidates, an expensive second stage re-orders only those \(K\). The second stage can never find a moment the first stage missed, so recall@K of the first stage is the ceiling of the whole system.

A simple model of first-stage recall

To reason about \(K\), use a toy model. The true segment gets a score \(x \sim \mathcal{N}(d, 1)\); each of \(M\) other segments in the search space gets an independent score from \(\mathcal{N}(0, 1)\). The number of other segments that beat the true one is then about Poisson with mean \(M\, Q(x)\), where \(Q\) is the tail of the standard normal. The true segment is in the top \(K\) when fewer than \(K\) others beat it:

\[ \text{R@}K(d, M) = \E_{x \sim \mathcal{N}(d,1)} \Big[ \Pr\big( \text{Poisson}(M\, Q(x)) \lt K \big) \Big], \qquad Q(x) = \Pr\big(\mathcal{N}(0,1) \gt x\big) \tag{235.7} \]
Read the symbols
\(d\)
separation: how many noise units the right segment scores above a typical wrong one. A better embedding model or a better query gives a larger \(d\)
\(M\)
size of the search space after filters
\(K\)
number of candidates passed to the re-ranker

This is a toy: real wrong segments are not independent (replays and similar plays score high together), so real recall is lower. But it shows the two levers clearly. With \(d = 5\): searching the whole archive (\(M = 24\) million) gives R@20 = 0.58. Filtering to one team’s season (38 games, \(M = 91{,}200\)) gives 0.93. Filtering to one game (\(M = 2{,}400\)) gives 0.995. Metadata filters are the cheapest accuracy you can buy. Raising \(K\) also helps, but each extra candidate costs re-ranker time.

Search space:
selected search spaceother search spacesyour \(K\)
Figure 3. First-stage recall@K from the toy model (235.7), against \(K\) on a log axis. A league season is 380 games and a team season 38 games, at about 2,400 segments per game (2 hours, 3-s stride). The readout also gives the VLM re-rank time for \(K\) candidates with the default re-ranker of Figure 5 (57.7 ms each on one L4).
How to read Figure 3

The curves rise with \(K\) and flatten: going from 10 to 20 candidates helps much more than going from 100 to 200. Switch the search space: the curve for one game is near 1 already at \(K = 10\), while the whole archive needs a much better model (larger \(d\)) to get there. Compare the cost line in the readout: re-ranking 200 candidates would take more than 11 s. In practice you choose the smallest \(K\) whose recall is close to the flat part.

The VLM re-ranker

The re-ranker gets, for each candidate, a few frames and a prompt such as: “Does this clip show: penalty saved by the goalkeeper? Answer yes or no.” We do not let it write an answer. We read the probability of the token “yes” after the prompt, and sort candidates by it. That needs only the prefill (the forward pass over the prompt), not a long decode, so it is compute-bound and easy to batch. (For prefill and decode, see session 196.)

Take Qwen2.5-VL-3B as the example re-ranker. From its config.json [Certain]: patch size 14, a 2×2 spatial merge, a temporal patch size of 2 (two frames become one group), a vision encoder of 32 layers of width 1,280, and a language model of 36 layers of width 2,048. Counting the weights: about 0.667 billion in the vision encoder and merger, and 2.775 billion in the language model’s linear layers. For \(F\) frames at \(R \times R\) pixels:

\[ T_v = \frac{F}{2} \left( \frac{R}{28} \right)^{2}, \qquad n_{\text{patch}} = 4\, T_v \tag{235.8} \]
\[ C_{\text{cand}} \approx 2 P_{\text{vit}}\, n_{\text{patch}} + 2 P_{\text{llm}} \,(T_v + T_q), \qquad t_{\text{cand}} = \frac{C_{\text{cand}}}{\eta\, P_{\text{peak}}} \tag{235.9} \]
Read the symbols
\(T_v\)
visual tokens the language model reads; each token covers 2 frames × 28 × 28 pixels
\(n_{\text{patch}}\)
14×14-pixel patches (two frames deep) that the vision encoder processes
\(P_{\text{vit}}, P_{\text{llm}}\)
weights in the linear layers of the vision encoder and the language model
\(T_q\)
text tokens of the prompt and query (we use 64)
\(C_{\text{cand}}\)
FLOPs to score one candidate; “2 × weights × tokens” is the usual forward cost, ignoring attention (small here, as the vision encoder mostly uses windowed attention) [Likely]
Worked example: the cost of one candidate

8 frames at 448×448: \(T_v = 4 \times 16^2 = 1{,}024\) tokens, \(n_{\text{patch}} = 4{,}096\). Encoder \(2 \times 0.667 \times 10^9 \times 4{,}096 = 5.46\) TFLOP; language model \(2 \times 2.775 \times 10^9 \times 1{,}088 = 6.05\) TFLOP. Total 11.5 TFLOP, so 190 ms at 60.5 TFLOP/s. Twenty candidates: 3.8 s. Too slow for a 1.5 s target.

4 frames at 336×336: \(T_v = 2 \times 12^2 = 288\), \(n_{\text{patch}} = 1{,}152\). Encoder 1.54 TFLOP, language model 1.96 TFLOP, total 3.49 TFLOP: 57.7 ms per candidate, 1.15 s for 20. The frame budget is the first knob, exactly as for images.

Timestamps: from a segment to a moment

The re-ranker tells us which segments match. The user wants a start and an end. We refine around the best segment using the per-second scores we stored at ingest (the cosine between the query and each second’s frames, formula 235.3 without pooling). Take a window of about ±15 s around the segment, find the peak second \(p\), and grow a region left and right while the score stays above a fraction \(\alpha\) of the peak:

\[ [\hat a, \hat b] = \text{the longest run of seconds } t \ni p \text{ with } s(t) \ge \alpha\, s(p), \qquad \text{clip} = [\hat a - \Delta_{\text{pre}},\ \hat b + \Delta_{\text{post}}] \tag{235.10} \]

The detected moment \([\hat a, \hat b]\) is what we evaluate with tIoU. The clip we play adds a little context: editors want the build-up before a goal, so we add a pre-roll \(\Delta_{\text{pre}}\) and a short post-roll. If you have labeled data, a trained boundary model does better than a threshold: a per-clip saliency and boundary head (the UniVTG view) or a DETR-style moment decoder as in Moment-DETR, trained on QVHighlights-style data (session 207). The threshold rule is the training-free baseline you start with. It is also what the lab session 215 built.

true event (20–28 s)detected momentclip to play (with pre-roll, 2 s post-roll)
Figure 4. Timestamp refinement on a synthetic per-second score curve (a fixed formula, not real data). The true goal is at 20–28 s. The second bump near 44 s is a replay of the same goal from another camera.
How to read Figure 4

A low threshold grows the region into the noise; a high one keeps only the core of the event. Around \(\alpha = 0.35\)–0.5 the detected moment is [20, 27] s, tIoU 0.875. At \(\alpha = 0.8\) it shrinks to [22, 26] s and tIoU falls to 0.5. The replay bump is a separate region: the system must recognize it as the same event (same game clock on the scoreboard, a replay graphic, or a shot-type classifier) and either merge it or show it as “replay”, not as a second goal.

Latency and cost math before building

Now add up the online path. Everything except the re-ranker is small, so we give each part a fixed budget (a target we will hold the implementation to, not a measurement):

StageBudgetNotes
Parse the query, extract filters (team and player dictionary, rules)5 msa small LLM here would add 100+ ms; use it only when the rules fail
Text embedding (SigLIP text tower, TensorRT FP16)5 msabout 11 GFLOP for 64 tokens: tiny
IVF-PQ search with filters, exact re-score of top results20 msCPU or GPU; filtered searches are small
Transcript search and RRF fusion10 msruns in parallel with the vector search in practice
Fetch candidate frames (stored as JPEG at ingest)40 msnever decode the source video online
Timestamp refinement and replay merge10 msuses stored per-second scores
Fixed part90 ms
VLM re-rank of \(N\) candidates\(N \cdot t_{\text{cand}}\)formula (235.9); the dominant term
\[ T_{\text{query}} \approx T_{\text{fixed}} + N \cdot t_{\text{cand}} \tag{235.11} \]

The number of GPUs follows from the work per query. Each query keeps one GPU busy for \(N \cdot t_{\text{cand}}\) seconds. We do not run GPUs at 100%: queues grow without bound near full load and the 95th percentile explodes, so we plan for a target utilization \(u\) (we use 0.6):

\[ G = \left\lceil \frac{\lambda \cdot N \cdot t_{\text{cand}}}{u} \right\rceil, \qquad \text{cost per 1,000 queries} = \frac{1000 \cdot N \cdot t_{\text{cand}}}{u \cdot 3600} \cdot \text{price per GPU-hour} \tag{235.12} \]
Read the symbols
\(\lambda\)
peak queries per second
\(N\)
candidates re-ranked per query (the \(K\) of Figure 3)
\(u\)
planned average utilization of each GPU at peak
\(G\)
number of L4s for the re-ranker
Worked example: the default plan

Qwen2.5-VL-3B in BF16, 4 frames at 336×336, \(N = 20\): \(t_{\text{cand}} = 57.7\) ms, re-rank 1,154 ms, total \(90 + 1{,}154 = 1{,}244\) ms. Under the 1.5 s target, with about 250 ms left for queueing and variation.

GPUs at 5 QPS: \(5 \times 1.154 / 0.6 = 9.6\), so 10 L4s, within the cap of 12. At an example price of $0.80 per GPU-hour (prices vary by cloud and contract; use yours), one query costs \(1.154 / 0.6 = 1.92\) GPU-seconds, so $0.43 per 1,000 queries. Keeping 10 GPUs up all month costs \(10 \times 720 \times 0.80 = \$5{,}760\).

Weights: about 3.75 billion parameters in BF16 is 7.5 GB, well within 24 GB, so one L4 holds one re-ranker with room for large batches.

Re-ranker:Language model in:
fixed stages (90 ms)VLM vision encoderVLM language model (prefill)
Figure 5. A latency and cost calculator for the online path on L4s (formulas 235.8–235.12), with \(\eta = 0.5\) of the dense peak (121 TFLOP/s BF16, 242 TFLOP/s FP8), 64 prompt tokens, the vision encoder always in BF16, and target utilization 0.6. Estimates of the mean, not measurements; the 95th percentile must be measured.
How to read Figure 5

Start from the default (1,244 ms, 10 GPUs). Switch to 8 frames at 448: the bar passes 3.8 s and the GPU count passes 30. Switch to the 7B model: about 2.1 s, over budget. FP8 for the language model cuts its part in half (on the L4, an Ada GPU, FP8 tensor cores exist), giving 831 ms at the default settings. Lowering \(N\) to 10 halves both latency and cost, but read Figure 3 first: it also lowers the recall ceiling. Raise the load: latency per query does not change, but the GPU count grows linearly.

Moving work offline

The numbers suggest another option: run the VLM offline on every segment once, write a caption (“Penalty kick; goalkeeper in yellow dives left and saves”), and search captions as text. The prefill alone for 24 million segments at 3.49 TFLOP each is about 385 GPU-hours at 60.5 TFLOP/s, plus the caption decoding. That is a one-time cost a little smaller than two days of the 10 online re-rankers (480 GPU-hours). Captions make many queries cheap, but they only contain what the caption model chose to say; a query about a detail it skipped (“the ball hits both posts”) still needs the online, query-specific re-ranker. A good design uses both: captions and transcripts as extra first-stage lists in RRF, and the online re-ranker on a small \(N\).

The algorithms and the code

Algorithm 1: ingest one game (offline)
  1. input: video file, game metadata (teams, date, competition), event feed if available
  2. decode with the hardware decoder; keep 2 frames per second; store one JPEG per second for the re-ranker
  3. detect shot boundaries; read the scoreboard clock every second (OCR); flag replay shots
  4. embed every kept frame with the image tower (TensorRT engine, FP16, batch 64) // formula (235.6) for the cost
  5. cut segments of \(w\) seconds with stride \(s\), snapped to shot boundaries; mean-pool and normalize // (235.1), (235.3)
  6. transcribe the commentary; align transcript and event feed to video time with the clock map
  7. write segment vectors (PQ codes) with metadata to the index version \(v\); write transcripts to the keyword index; store the 2-fps frame embeddings on disk for the per-second scores
  8. run validation: number of segments, clock coverage, a few known events must be found
Algorithm 2: answer one query (online)
  1. input: query text, optional filters; index version \(v\)
  2. extract filters from the text (teams, players, season, half) with a dictionary; merge with explicit filters
  3. \(\mathbf{t}_q \leftarrow\) text embedding of the query (same model version \(v\) as the index)
  4. list A \(\leftarrow\) IVF-PQ top 200 within the filters, re-scored with exact FP16 vectors
  5. list B \(\leftarrow\) keyword search over transcripts and event feed within the filters
  6. candidates \(\leftarrow\) top \(N\) by RRF over lists A and B // (235.4)
  7. fetch 4 stored frames per candidate; VLM scores \(p(\text{yes})\) for all \(N\) in one batch // (235.8), (235.9)
  8. for the top results: refine \([\hat a, \hat b]\) from per-second scores; merge replays by game clock // (235.10)
  9. return moments (game, start, end, clip with pre-roll, score), and log every stage’s time and output

The PyTorch program below runs the whole pipeline on a fake half of football: synthetic frame embeddings with three events and a replay, segments and mean pooling, search, RRF fusion with a fake transcript, threshold refinement, tIoU, and the re-rank budget. It is deterministic; the printed values are shown in the comments.

import math, torch
import torch.nn.functional as F
torch.manual_seed(0)
D, FPS, DUR = 64, 2, 1800                 # embedding size, sampled frames per second, a 30-minute half

# 1) a fake broadcast: frame embeddings = camera noise + event "concepts" during the events
concepts = {e: F.normalize(torch.randn(D), dim=0) for e in ["goal", "corner", "yellow card"]}
truth = {"goal": (600, 608), "corner": (300, 306), "yellow card": (1210, 1215)}
frames = 0.35 * torch.randn(DUR * FPS, D) + F.normalize(torch.randn(D), dim=0)    # shared "pitch" look
for e, (s, t) in truth.items():
    frames[s * FPS:t * FPS] += 1.2 * concepts[e]
frames[700 * FPS:706 * FPS] += 0.8 * concepts["goal"]        # a replay of the goal, weaker and shorter
frames = F.normalize(frames, dim=-1)

# 2) segments: window w, stride s, mean pooling then L2-normalize (formulas 235.1 and 235.3)
w, s = 6, 3
starts = torch.arange(0, DUR - w + 1, s)
seg = torch.stack([frames[a * FPS:(a + w) * FPS].mean(0) for a in starts.tolist()])
seg = F.normalize(seg, dim=-1)
print("segments:", len(starts))                               # 599 = (1800 - 6) // 3 + 1

# 3) query: a "text embedding" near the concept (a real system uses the SigLIP text tower)
q = F.normalize(concepts["goal"] + 0.6 * torch.randn(D) / math.sqrt(D), dim=0)
sim = seg @ q                                                  # exact search; IVF-PQ at archive scale
top = sim.topk(5).indices
print("top segment starts:", starts[top].tolist())            # [600, 603, 702, 699, 597]: 702 and 699 are the replay

# 4) fuse with a transcript list by reciprocal rank fusion (formula 235.4)
asr_hits = [598, 601, 604, 700]                               # seconds where commentary says "goal"
asr_rank = {}
for r, t in enumerate(asr_hits):
    j = int(torch.argmin((starts - t).abs()))
    asr_rank.setdefault(j, r + 1)
vis_rank = {int(j): r + 1 for r, j in enumerate(sim.argsort(descending=True)[:50])}
rrf = {j: sum(1 / (60 + rk[j]) for rk in (vis_rank, asr_rank) if j in rk)
       for j in set(vis_rank) | set(asr_rank)}
fused = sorted(rrf, key=rrf.get, reverse=True)[:5]
print("fused starts:", [int(starts[j]) for j in fused])     # [600, 603, 597, 699, 702]

# 5) timestamps: per-second scores around the best segment, threshold at alpha * peak (formula 235.10)
per_sec = (frames @ q).view(DUR, FPS).mean(1)
c = int(starts[fused[0]]); lo, hi = max(0, c - 15), min(DUR, c + 21)
a = per_sec[lo:hi]; p = int(a.argmax()); tau = 0.5 * float(a[p])
L = p
while L > 0 and a[L - 1] >= tau: L -= 1
R = p
while R < len(a) - 1 and a[R + 1] >= tau: R += 1
st, en = lo + L, lo + R + 1
def tiou(a, b):                                               # formula 235.2
    inter = max(0, min(a[1], b[1]) - max(a[0], b[0]))
    return inter / (max(a[1], b[1]) - min(a[0], b[0]))
print("moment:", (st, en), "tIoU:", round(tiou((st, en), truth["goal"]), 3))   # (600, 608) tIoU: 1.0

# 6) the re-rank budget (formulas 235.8, 235.9, 235.12) for a 3B VLM on one L4
def rerank_ms(frames=4, res=336, n=20, p_vit=0.667e9, p_llm=2.775e9, text=64, eta=0.5, peak=121e12):
    tv = frames / 2 * (res / 28) ** 2                           # visual tokens after 2x2 merge, 2-frame groups
    flops = 2 * p_vit * 4 * tv + 2 * p_llm * (tv + text)
    return n * flops / (eta * peak) * 1e3
ms = rerank_ms()
print("re-rank ms:", round(ms), " GPUs for 5 QPS at 60% busy:", math.ceil(5 * ms / 1e3 / 0.6))   # 1154, 10

Notice what the toy already shows: the replay (starting near 700 s) is in the top 5 twice. Without a replay merge, the user would see the same goal three times.

Risks, failure modes, and what to measure

A design review spends as long on what can go wrong as on the happy path. The table lists the main risks for this system, how each one shows up, and what reduces it.

RiskHow it shows upMitigation
Replays and highlightsthe same goal appears 2–4 times in the top 10replay detection (shot type, replay graphics), merge by game clock; SoccerNet-v2 even has a replay grounding task [Certain]
Celebrations and reactionsclose-ups of players celebrating rank above the event itselfre-ranker prompt asks for the action, not the reaction; timestamp step anchors on the action
Commentary delaytranscript-based moments start 5–20 s lateuse transcripts for candidates only; place the start from frames
Ambiguous queries“best goal”, “crazy moment”: no visual meaningmap to popularity or editor tags; ask a follow-up; do not let the VLM guess
Fine-grained identity“Haaland header” returns any headerplayer names come from the event feed, jersey numbers or the transcript, not from the frame embedding
Domain shiftworks on the main league, fails on women’s league, youth games, another sport, night games, low-resolution feedsslice the test set by competition, camera and resolution; report each slice
Model upgradenew text tower with old segment vectors gives nonsenseversioned indexes; re-embed, compare on the test set, then switch
Latency tailsp95 far above the mean when the re-ranker batch is large or GPUs are busyplan utilization at 0.6; cap \(N\); a timeout that returns first-stage results
VLM failure modesalways says “yes”, prefers the first candidate in a batch, or is fooled by on-screen textcalibrate \(p(\text{yes})\) on labeled pairs; score candidates independently; check with blind (text-only) baselines
Rights and privacyvideo from some competitions may not be stored or shownrights metadata as an index filter, enforced on the server

Measure each stage, not only the end

The end-to-end hit rate is roughly a product of stage success rates. For a query to succeed, the right moment must be in the first-stage list, the re-ranker must move it into the top 10, and the timestamp must be good:

\[ P(\text{success}) \approx \text{R@}K_{\text{stage 1}} \times P(\text{top 10} \mid \text{in candidates}) \times P(\text{tIoU} \ge 0.5 \mid \text{found}) \tag{235.13} \]

With example values 0.93, 0.90 and 0.85, the product is 0.71. Each factor looks fine; the system does not. And when the end number is bad, only per-stage numbers tell you which stage to fix. So we log and measure:

  • Offline quality: first-stage recall@\(K\) (the ceiling); re-ranker nDCG@10 and Hit@10 on the candidates; timestamp R1@IoU=0.5 and the mean start error in seconds; duplicate rate in the top 10. All on a fixed test set of real queries with labeled spans, split by query type, competition and camera (session 213).
  • Online behavior: click-through and “clip used in an edit” rates; reformulated queries (the user did not find it); zero-result queries.
  • System: p50, p95 and p99 latency per stage; GPU utilization; queue length; cost per 1,000 queries; ingest lag after the final whistle; decoder throughput.

Where this is used

ApplicationWhat changes in the design
Broadcaster archive search for editorsthis page; filters and accuracy matter more than cost per query
Fan-facing highlight search in a sports appmuch higher QPS: cache popular queries, lower \(N\), more offline captions
Automatic highlight reelsno query: score every segment for importance (saliency), then pick and order the best moments (session 207)
Coaching and scouting toolstactical queries (“high press after a goal kick”) need tracking data and player positions, not only frames (session 204)
Content moderation and rights checksnear-duplicate search instead of text search (session 209)
Video-RAG assistants (“what happened before the red card?”)the same retrieval feeds moments into a Video-LLM that writes an answer (session 210)

Common mistakes

  • Choosing the model before the requirements. A 7B re-ranker with 8 frames looks great in a notebook and needs 55 L4s at 5 QPS (Figure 5).
  • Running the VLM over the whole archive per query. Expensive work belongs offline, or on a small candidate set.
  • Re-ranking without measuring first-stage recall. If the right moment is not in the top \(K\), no re-ranker can fix it; tune the first stage and the filters first.
  • Stride equal to the window. Events that fall across two windows get tIoU as low as 0.33 for a 6-s event (Figure 2). Overlap by about half.
  • Using commentary time as the moment time. Commentary is late; place the start from frames.
  • Mixing embedding model versions in one index. Vectors from different models are not comparable.
  • Decoding source video online. Store the frames the re-ranker needs at ingest; online, read JPEGs.
  • Planning GPUs at 100% utilization. The mean latency looks fine; the 95th percentile does not.

Interview questions

Walk me through your design for sports key-moment search.basic
Two paths. Offline, per game: decode, sample 2 frames per second, detect shots and read the game clock, embed frames with a SigLIP-style model, pool them into overlapping 6-s segments, transcribe commentary, and write PQ-compressed vectors with metadata (game, teams, half, clock) to a versioned index plus a keyword index. Online, per query: extract filters, embed the text, search the vector and keyword indexes within the filters, fuse with RRF, re-rank the top 20 with a small VLM by \(p(\text{yes})\), refine start and end from per-second scores, merge replays by game clock, and return clips with pre-roll. Then give latency, cost and metrics.
How do you choose the segment window and stride?core
Start from the typical event length. The window should be about that length so one vector describes one event, and the stride about half the window so every event overlaps some window well: with 6-s events, \(w = 6\), \(s = 3\) gives a worst-case best tIoU of 0.60 and an average of 0.79, while \(s = 6\) has a worst case of 0.33. The cost is index size, which grows as \(1/s\): 24 million segments for 20,000 hours at \(s = 3\). Snap windows to shot boundaries, and keep per-frame embeddings on disk for the timestamp step.
Why retrieve-then-re-rank, and how do you choose the number of candidates?core
The embedding search is cheap enough for millions of segments but cannot read the query and the frames together; a VLM can, but costs about 58 ms per candidate on an L4 in our setting. The cascade gets the VLM’s accuracy for the price of a few candidates. The first stage’s recall@K is the ceiling of the whole system, so measure it as a function of \(K\) and pick the smallest \(K\) near the flat part of the curve, then check that \(K \cdot t_{\text{cand}}\) fits the latency budget and the GPU count fits the cost cap.
Estimate the VLM re-rank latency for Qwen2.5-VL-3B on one L4.core
Visual tokens: \(F/2 \cdot (R/28)^2\); with 4 frames at 336, 288 tokens and 1,152 encoder patches. FLOPs: \(2 \times 0.667\text{B} \times 1{,}152 = 1.54\) T for the encoder and \(2 \times 2.775\text{B} \times 352 = 1.96\) T for the language model, 3.49 TFLOP in total. At half of the 121 TFLOP/s dense BF16 peak, 57.7 ms per candidate, so 1.15 s for 20. With 8 frames at 448 it would be 190 ms per candidate and 3.8 s for 20. Only prefill is needed because we read \(p(\text{yes})\) instead of generating text.
The index does not fit on one GPU. What do you do?core
24 million 768-d vectors are 36.9 GB in FP16. Options: product quantization (96 bytes per vector, 2.5 GB with ids) inside an IVF index, with exact re-scoring of the top few hundred from FP16 vectors on disk or in CPU memory; run the ANN search on CPU, since filtered searches are small; shard by season or competition. Also reduce \(N\): metadata filters shrink each search to a few thousand or tens of thousands of segments.
How do you handle replays?core
A goal is typically shown live and then replayed from other cameras, sometimes several times, and every replay matches the query. Detect replays (shot-type classifier, replay transition graphics, a frozen or missing scoreboard clock) at ingest and mark those segments. At query time, merge results that map to the same game-clock time, keep the live one as the main result and attach replays as alternative angles. Measure the duplicate rate in the top 10.
How do you produce precise timestamps?core
Retrieval returns a segment, which is only as precise as the stride. Around it, use stored per-second scores: find the peak, grow a region while the score stays above a fraction of the peak, and report it; then add pre-roll for playback. On the synthetic curve of Figure 4, \(\alpha = 0.5\) gives [20, 27] for a true [20, 28], tIoU 0.875. With labeled data, train a boundary model (saliency plus boundary regression, or a Moment-DETR-style decoder). Use the scoreboard clock to report game time and to align with event feeds.
How do you evaluate the system before launch?core
Build a test set of a few hundred real queries from editors, with labeled spans, independent of the system’s own outputs, sliced by query type, competition, camera and resolution. Measure first-stage recall@K, re-ranker Hit@10 and nDCG@10, timestamp R1@IoU=0.5 and start error, duplicate rate, and per-stage p50/p95 latency under a load test at the peak QPS. Their product approximates end-to-end success (\(0.93 \times 0.90 \times 0.85 = 0.71\) in our example), which shows which stage to improve.
You upgrade the embedding model. What breaks and how do you roll it out?advanced
Old segment vectors and new text vectors live in different spaces, so mixing them gives meaningless scores. Build a new index version by re-embedding the whole archive (about 23 L4-hours of encoder compute for SigLIP-B at 2 frames per second, plus decoding, which is often the real limit), evaluate both versions on the test set and on a shadow share of traffic, then switch atomically. Keep the old index until the new one is proven. The text tower and the index must always carry the same version id.
A product manager now wants live games searchable within one minute. What changes?advanced
Ingest becomes streaming: decode the live feed continuously, embed frames as they arrive (cheap: about 4 s of GPU per video-hour), and append segments to a small in-memory index for live games that is searched together with the main index, then merged into it after the game. Shot detection and segmenting must work on a sliding buffer; transcripts arrive in chunks; replays of a live goal arrive seconds later and must be merged as they come. Offline captioning can run later. Re-ranking is unchanged.
Why TensorRT on the L4, and what are the pitfalls?advanced
TensorRT fuses layers and picks the fastest kernels for the exact GPU, and it supports FP16 and, on an Ada GPU like the L4, FP8; TensorRT-LLM provides batching and the KV cache for the language part. Pitfalls: engines are specific to one GPU model and TensorRT version, so build them on L4s in CI; dynamic shapes need optimization profiles that cover your batch sizes and frame sizes; check that the engine’s outputs match the PyTorch model on a test batch (especially after FP8); and check that your exact VLM version is supported before you commit. [Likely]
Should the VLM caption every segment offline instead of re-ranking online?advanced
It is a trade-off. Captioning 24 million segments is a one-time cost of about 385 L4-hours for prefill at our settings plus decoding, a little less than two days of the online fleet, and it turns many queries into cheap text search. But captions only contain what the caption model wrote; queries about other details still need a query-specific look at the frames. Use captions as an extra list in the first stage, and keep a small online re-ranker for the final order.

Exercises

  1. Your archive is 500 hours. With \(w = 8\) s and \(s = 4\) s, how many segments are there, and how much memory do flat FP16 vectors with \(d = 768\) need?
  2. With formulas (235.8) and (235.9), compute the per-candidate time for Qwen2.5-VL-3B with 8 frames at 448×448 in BF16.
  3. Load grows to 12 QPS. With \(N = 10\), 4 frames at 336, 3B in BF16 and \(u = 0.6\), how many L4s do you need?
  4. First-stage recall@20 is 0.93, the re-ranker puts 90% of found moments in the top 10, and 85% of those have \(\text{tIoU} \ge 0.5\). What is the end-to-end success rate? Which stage would you improve first, and why?
  5. A 6-s event, a 6-s window. What is the worst-case best tIoU with stride 6 s, and with stride 3 s? Check with Figure 2.
  6. How many L4-seconds per day does it take to embed 100 new hours of video with SigLIP-B/16 at 2 frames per second? What part of ingest is more likely to be the bottleneck?
Answerscheck
1. \(500 \times 3600 / 4 = 450{,}000\) segments; \(450{,}000 \times 768 \times 2 = 0.69\) GB. It fits on one GPU; no PQ needed at this size. 2. \(T_v = 4 \times 16^2 = 1{,}024\), \(n_{\text{patch}} = 4{,}096\); FLOPs \(= 5.46 + 6.05 = 11.5\) T; at 60.5 TFLOP/s, 190 ms. 3. \(t_{\text{cand}} = 57.7\) ms, so 0.577 GPU-s per query; \(12 \times 0.577 / 0.6 = 11.5\), so 12 L4s, at the cap. 4. \(0.93 \times 0.90 \times 0.85 = 0.71\). Look at the per-stage numbers on failures; the cheapest gains are usually the first stage (filters, \(K\), transcript fusion) because it caps everything, and the timestamp step, which is cheap to improve with a trained boundary model. 5. Stride 6: \(3/9 = 0.33\) when the event starts in the middle of a window. Stride 3: \(4.5/7.5 = 0.60\). 6. \(100 \times 4.1 \approx 413\) s, about 7 minutes of one L4. Decoding is more likely the bottleneck, since the decoder must process about 90,000 frames per hour of 25-fps video, 12.5 times the frames that are embedded.

Key takeaways

  • Start with numbers: query types, load, latency (1.5 s at p95), accuracy (Hit@10, tIoU), cost (12 L4s) and fixed hardware (L4 + TensorRT).
  • Expensive work offline once per video (decode, embed, transcribe, index); online only cheap search plus a VLM re-ranker on a few candidates.
  • Window and stride set the best possible tIoU and the index size (235.1, 235.2): 6-s windows with a 3-s stride give 24 million segments for 20,000 hours; PQ fits them in 2.5 GB.
  • First-stage recall@K is the ceiling (235.7); metadata filters raise it more cheaply than anything else.
  • The re-ranker dominates latency and cost (235.8–235.12): 57.7 ms per candidate for a 3B VLM with 4 frames at 336, so 1.24 s per query and 10 L4s at 5 QPS.
  • Replays, late commentary, ambiguous queries and model upgrades are the main risks; measure each stage separately because end-to-end success is their product (235.13).

Key formulas

\[ N \approx \frac{3600 H}{s}, \quad T_v = \frac{F}{2}\Big(\frac{R}{28}\Big)^2, \quad t_{\text{cand}} = \frac{2 P_{\text{vit}} \cdot 4T_v + 2 P_{\text{llm}}(T_v + T_q)}{\eta P}, \quad G = \Big\lceil \frac{\lambda N t_{\text{cand}}}{u} \Big\rceil \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  2. Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
  3. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  4. Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
  5. Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
  6. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  7. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  8. Johnson et al. (2017). Billion-scale similarity search with GPUs
  9. Jégou et al. (2011). Product Quantization for Nearest Neighbor Search
  10. Souček & Lokoč (2020). TransNet V2: An effective deep network architecture for fast shot transition detection
  11. Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
  12. Bai et al. (2025). Qwen2.5-VL Technical Report
  13. Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
  14. NVIDIA (2023). NVIDIA L4 Tensor Core GPU (product page and datasheet)
  15. NVIDIA (docs). TensorRT documentation
  16. NVIDIA (code). TensorRT-LLM
  17. Deliège et al. (2021). SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos
  18. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models