Capstone design: sports key-moment search
A sports broadcaster wants editors and fans to type “penalty saved by the keeper in the second half” and get the exact seconds of video back. This capstone designs that system on paper before any code: the requirements, the pipeline (segments, embeddings, an index, a VLM re-ranker, timestamps), the latency and cost math on NVIDIA L4 GPUs with TensorRT, and the risks and metrics that decide whether it works.
What you will learn
- How to turn a vague product wish into requirements: query types, latency targets, accuracy metrics, cost limits and fixed hardware
- A complete design: decode, shot-aware segments, SigLIP embeddings, an IVF-PQ index with metadata filters, transcript fusion, a VLM re-ranker and timestamp refinement
- How window length and stride decide the best possible tIoU and the size of the index
- Why metadata filters and the number of re-ranked candidates control recall, and how to reason about it with a simple model
- Latency and cost math for a VLM re-ranker on an L4: visual tokens, FLOPs, milliseconds per candidate, GPUs and dollars per 1,000 queries
- The main risks (replays, commentary delay, ambiguous queries, domain shift, model upgrades) and what to measure at each stage
Why this matters
A football broadcaster has 20,000 hours of match video. An editor needs “every penalty saved by a goalkeeper this season” for a show that airs in an hour. A fan types “Salah curling shot into the top corner”. Today, someone scrubs through video by hand, or searches a list of events that a logging team typed during the game. This session designs a system that answers such questions with the exact seconds of video, on a fixed hardware budget: NVIDIA L4 GPUs running TensorRT engines.
It is a capstone because it uses almost everything in Track 8 and much of the inference track: text-to-video retrieval (session 208), temporal grounding (session 207), vector indexes (session 151), compilation with TensorRT (session 131), and the pencil-and-paper latency budget of session 199. In session 234 you took one model apart. Here you put many models together into one service, and you decide on paper whether it can work before anyone writes code. In the next session, session 236, you build a small version of it.
A key moment here means a short span of video (a few seconds to a minute) that matches what the user asked for: a goal, a save, a foul, a celebration, a tactical pattern. The system returns a ranked list of moments, each with a game, a start time and an end time.
Requirements: queries, latency, accuracy, cost, hardware
A design starts from requirements, not from models. Ask questions until every requirement is a number, or a short list. Here is the set we will design for. These are our design inputs for the example, not facts about any real broadcaster.
| Requirement | Our value | Why it matters for the design |
|---|---|---|
| Archive | 20,000 hours of football broadcasts (about 10,000 games of about 2 hours each, with pre-game and half-time) | sets the index size and the one-time ingest cost |
| New video | about 100 hours per day; searchable within 1 hour after the game ends | sets the daily ingest cost; “1 hour” allows batch ingest, not streaming |
| Query types | (a) event type: “all red cards”; (b) event + entity: “Haaland headers”; (c) free description: “keeper slips and the ball goes in”; (d) filters: team, season, competition, half | (a) and (d) can use structured data; (b) and (c) need visual search and a re-ranker |
| Load | 5 queries per second (QPS) at peak, 1 QPS on average | sets the number of GPUs for the online path |
| Latency | first results in at most 0.3 s; final re-ranked results with timestamps in at most 1.5 s (95th percentile) | limits how much VLM work we can do per query |
| Accuracy | Hit@10 at least 0.85 on our own test set of 500 queries; for hits, tIoU at least 0.5 in 85% of cases | defines “works”; we will measure each stage separately |
| Cost | at most 12 L4 GPUs for the online path at peak | caps the re-ranker size and the number of candidates |
| Hardware | NVIDIA L4 (fixed by the platform team), models compiled with TensorRT or TensorRT-LLM | fixes peak compute, memory and video decoders |
Hit@10 means: the right moment is somewhere in the top 10 results. tIoU (temporal intersection over union) measures how well a predicted time span overlaps the true span; we define it in formula (235.2). 95th percentile (p95) latency means that 95% of queries finish faster than this.
The hardware: what an L4 gives us
From NVIDIA’s product page [Certain]: 24 GB of memory, 300 GB/s of memory bandwidth, 72 W, and 242 TFLOP/s BF16 and 485 TFLOP/s FP8 on the tensor cores with 2:4 sparsity; dense numbers are half: 121 TFLOP/s BF16 and 242 TFLOP/s FP8. It also has 4 video decoders (NVDEC), 4 JPEG decoders and 2 video encoders. For video work the decoders matter as much as the tensor cores: frames must be decoded before any model can see them.
As in session 199, we assume real kernels reach \(\eta = 0.5\) of the dense peak, so about 60.5 TFLOP/s in BF16. This is an assumption to replace with a measurement later. [Likely]
TensorRT compiles a model graph into an engine: fused, timed kernels for one GPU model and one TensorRT version. [Certain] For us this means: build the engines on L4s, declare the input shapes you will use (batch sizes, image sizes) as optimization profiles, and rebuild when you upgrade TensorRT or change GPUs. TensorRT-LLM is the matching runtime for the language part of a VLM. Whether a given VLM is supported by TensorRT-LLM must be checked for the exact model and version. [Likely]
The design at a glance
The system has two paths. The offline path (ingest) runs once per game: it decodes the video, cuts it into short segments, computes vectors and text for each segment, and writes them into an index. The online path (query) runs for every user request: it reads the query, searches the index, re-ranks the best candidates with a vision-language model (VLM), and returns moments with precise times. The rule of the whole design: do expensive work offline, once per video; do only cheap or query-specific work online.
How to read Figure 1
Each step highlights one box and explains its job and its budget. Notice where the GPU time goes: offline, the frame encoder and the decoders; online, almost all of it goes to the VLM re-ranker. That is why most of the latency and cost math later in this page is about the re-ranker. The arrow from the index down to the online row is the only link between the two paths: the online path never touches raw video except to fetch a few frames for the re-ranker.
Segmenting: the unit of search
We cannot index a 2-hour video as one vector: the query is about 5 seconds of it. We also should not index single frames: one frame cannot show “a slide tackle”, and frames are 7,200 per hour even at 2 frames per second. The unit of search is a segment: a short window of video, for example 6 seconds. We slide the window along the video with a stride (the step between window starts):
- Read the symbols
- \(D\)
- length of one video in seconds
- \(w\)
- window length in seconds (how much video one vector describes)
- \(s\)
- stride in seconds; \(s \lt w\) means windows overlap
- \(H\)
- hours of video in the archive
With \(w = 6\) s and \(s = 3\) s (50% overlap), the 20,000-hour archive gives \(72{,}000{,}000 / 3 = 24.0\) million segments. Halving the stride doubles the index; doubling it halves the index but, as we see next, hurts the timestamps.
How good can a fixed window be?
A moment has a true span \([a^*, b^*]\). A window \([a, b]\) is scored against it with tIoU, the length of the overlap divided by the length of the union:
Even a perfect retriever can only return the best window in the grid. Where the true event falls relative to the grid is random, so the best tIoU is a random number. Figure 2 computes its average and its worst case. Two facts come out. First, with a 6-s event, a 6-s window and a 3-s stride, the worst case is 0.60 and the average is 0.79: every event can be hit at tIoU 0.5. With a 6-s stride (no overlap), the worst case drops to 0.33 and only 67% of positions reach 0.5. Second, windows much shorter than the event can never reach 0.5 alone (a 2-s window on a 6-s event gives at most 0.33), so short windows need a later step that merges neighbors. That step is the timestamp refinement below.
How to read Figure 2
Move “event start” to see the best and worst positions. Then raise the stride to the window length: the dip in the bottom curve gets deep, because some events fall between two windows. Lower the stride: the curve flattens, but the number of segments, and so the index memory, grows as \(1/s\). Set the event to 15 s and the window to 6 s: no single window can reach tIoU 0.5, so long events need merged windows or a refinement step.
Two practical rules improve on a plain grid. Snap to shots: broadcast video is cut into camera shots, and a replay or a close-up starts at a cut. A shot-boundary detector such as TransNet V2 (session 211) lets us end windows at cuts, so a window does not mix the wide game camera with a crowd shot. Keep finer data on the side: we store the per-frame embeddings at 2 frames per second (not in the index, just on disk), so the timestamp step can work at 0.5-s resolution without a bigger index.
Embedding and indexing
Each segment needs a vector that a text query can be compared with. We use a contrastive image–text model such as SigLIP: it maps an image and a sentence into the same space, so that matching pairs have a high cosine similarity (session 208). We embed frames, then pool them into one segment vector. CLIP4Clip showed that plain mean pooling of frame features is a strong, parameter-free baseline for video–text retrieval. [Likely]
- Read the symbols
- \(S_j\)
- the sampled frames inside segment \(j\) (12 frames for 6 s at 2 frames per second)
- \(\mathbf{z}_f\)
- the image embedding of frame \(f\)
- \(\mathbf{v}_j\)
- the unit-length segment vector stored in the index
- \(\mathbf{t}_q\)
- the unit-length text embedding of the query; the dot product of two unit vectors is their cosine
Text from the broadcast
Sports video comes with strong text signals. The commentary says “GOAL!” and the player’s name; a speech recognizer such as Whisper turns it into timed text. The scoreboard shows the game clock and the score; reading it (OCR) gives a game time for every second of video, and a score change is a near-certain goal signal. Official event feeds (from the league or a data provider) list goals, cards and substitutions with game times. We index the transcript with a keyword index (BM25) next to the vector index and merge the two ranked lists with reciprocal rank fusion (RRF), which needs only ranks, not comparable scores:
A segment missing from a list adds nothing for that list. \(k = 60\) is the value from the original RRF paper. [Certain]
Index size
Memory for the vectors is simple arithmetic:
\(N = 24.0\) million segments, \(d = 768\) (SigLIP base).
FP32 flat: \(24 \times 10^6 \times 768 \times 4 = 73.7\) GB. FP16 flat: 36.9 GB. INT8: 18.4 GB. None of these fits next to a VLM on a 24 GB L4, and FP16 does not fit at all.
IVF-PQ with \(m = 96\) bytes per vector plus an 8-byte id: \(24 \times 10^6 \times 104 = 2.50\) GB. This fits easily in CPU memory or on a GPU. Product quantization (PQ) stores each vector as 96 one-byte codes, and IVF searches only the few clusters near the query (session 151). We keep the FP16 vectors on disk and re-score the top few hundred PQ results exactly.
Two more index features matter as much as speed. Metadata filters: most queries name a team, a player, a season or a competition. Each segment carries the game id, teams, date, half and game clock, and the search runs only over the matching games. Versioning: every vector belongs to one embedding model version. A new model means re-embedding the archive into a new index, not mixing two models in one index.
Ingest cost
The offline cost per hour of video is the encoder FLOPs for all sampled frames divided by the effective GPU speed:
The vision tower has 12 layers of width 768 and MLP width 3,072: about 85 million weights in its linear layers. A 224×224 frame gives \(14 \times 14 = 196\) patches. FLOPs per frame \(\approx 2 \times 85 \times 10^6 \times 196\) plus attention \(\approx 34.7\) GFLOP.
At \(f_s = 2\) frames per second: \(7{,}200 \times 34.7 \text{ GFLOP} = 250\) TFLOP per video-hour. At 60.5 TFLOP/s, that is about 4.1 s of one L4 per video-hour. The whole archive: \(20{,}000 \times 4.1\) s \(\approx\) 23 GPU-hours. The daily 100 hours: about 7 minutes.
The catch is decoding. A 25 frames-per-second broadcast has 90,000 frames per hour, and a decoder must decode nearly all of them to output 2 per second (only keyframes can be decoded alone). That is 12.5 times more frames decoded than embedded. On an L4 the 4 hardware decoders, not the tensor cores, often set the ingest speed; measure it with your codec and resolution before planning. [Likely]
Retrieve, then re-rank
A single embedding comparison is fast but shallow. It cannot reliably tell “the keeper saves the penalty” from “the penalty is scored”: both have a keeper, a ball and a penalty spot. A VLM that looks at the frames and the query together can tell, but it costs tens of milliseconds per candidate, so it cannot look at 24 million segments. The standard answer is a cascade: a cheap first stage returns the top \(K\) candidates, an expensive second stage re-orders only those \(K\). The second stage can never find a moment the first stage missed, so recall@K of the first stage is the ceiling of the whole system.
A simple model of first-stage recall
To reason about \(K\), use a toy model. The true segment gets a score \(x \sim \mathcal{N}(d, 1)\); each of \(M\) other segments in the search space gets an independent score from \(\mathcal{N}(0, 1)\). The number of other segments that beat the true one is then about Poisson with mean \(M\, Q(x)\), where \(Q\) is the tail of the standard normal. The true segment is in the top \(K\) when fewer than \(K\) others beat it:
- Read the symbols
- \(d\)
- separation: how many noise units the right segment scores above a typical wrong one. A better embedding model or a better query gives a larger \(d\)
- \(M\)
- size of the search space after filters
- \(K\)
- number of candidates passed to the re-ranker
This is a toy: real wrong segments are not independent (replays and similar plays score high together), so real recall is lower. But it shows the two levers clearly. With \(d = 5\): searching the whole archive (\(M = 24\) million) gives R@20 = 0.58. Filtering to one team’s season (38 games, \(M = 91{,}200\)) gives 0.93. Filtering to one game (\(M = 2{,}400\)) gives 0.995. Metadata filters are the cheapest accuracy you can buy. Raising \(K\) also helps, but each extra candidate costs re-ranker time.
How to read Figure 3
The curves rise with \(K\) and flatten: going from 10 to 20 candidates helps much more than going from 100 to 200. Switch the search space: the curve for one game is near 1 already at \(K = 10\), while the whole archive needs a much better model (larger \(d\)) to get there. Compare the cost line in the readout: re-ranking 200 candidates would take more than 11 s. In practice you choose the smallest \(K\) whose recall is close to the flat part.
The VLM re-ranker
The re-ranker gets, for each candidate, a few frames and a prompt such as: “Does this clip show: penalty saved by the goalkeeper? Answer yes or no.” We do not let it write an answer. We read the probability of the token “yes” after the prompt, and sort candidates by it. That needs only the prefill (the forward pass over the prompt), not a long decode, so it is compute-bound and easy to batch. (For prefill and decode, see session 196.)
Take Qwen2.5-VL-3B as the example re-ranker. From its config.json [Certain]: patch size 14, a 2×2 spatial merge, a temporal patch size of 2 (two frames become one group), a vision encoder of 32 layers of width 1,280, and a language model of 36 layers of width 2,048. Counting the weights: about 0.667 billion in the vision encoder and merger, and 2.775 billion in the language model’s linear layers. For \(F\) frames at \(R \times R\) pixels:
- Read the symbols
- \(T_v\)
- visual tokens the language model reads; each token covers 2 frames × 28 × 28 pixels
- \(n_{\text{patch}}\)
- 14×14-pixel patches (two frames deep) that the vision encoder processes
- \(P_{\text{vit}}, P_{\text{llm}}\)
- weights in the linear layers of the vision encoder and the language model
- \(T_q\)
- text tokens of the prompt and query (we use 64)
- \(C_{\text{cand}}\)
- FLOPs to score one candidate; “2 × weights × tokens” is the usual forward cost, ignoring attention (small here, as the vision encoder mostly uses windowed attention) [Likely]
8 frames at 448×448: \(T_v = 4 \times 16^2 = 1{,}024\) tokens, \(n_{\text{patch}} = 4{,}096\). Encoder \(2 \times 0.667 \times 10^9 \times 4{,}096 = 5.46\) TFLOP; language model \(2 \times 2.775 \times 10^9 \times 1{,}088 = 6.05\) TFLOP. Total 11.5 TFLOP, so 190 ms at 60.5 TFLOP/s. Twenty candidates: 3.8 s. Too slow for a 1.5 s target.
4 frames at 336×336: \(T_v = 2 \times 12^2 = 288\), \(n_{\text{patch}} = 1{,}152\). Encoder 1.54 TFLOP, language model 1.96 TFLOP, total 3.49 TFLOP: 57.7 ms per candidate, 1.15 s for 20. The frame budget is the first knob, exactly as for images.
Timestamps: from a segment to a moment
The re-ranker tells us which segments match. The user wants a start and an end. We refine around the best segment using the per-second scores we stored at ingest (the cosine between the query and each second’s frames, formula 235.3 without pooling). Take a window of about ±15 s around the segment, find the peak second \(p\), and grow a region left and right while the score stays above a fraction \(\alpha\) of the peak:
The detected moment \([\hat a, \hat b]\) is what we evaluate with tIoU. The clip we play adds a little context: editors want the build-up before a goal, so we add a pre-roll \(\Delta_{\text{pre}}\) and a short post-roll. If you have labeled data, a trained boundary model does better than a threshold: a per-clip saliency and boundary head (the UniVTG view) or a DETR-style moment decoder as in Moment-DETR, trained on QVHighlights-style data (session 207). The threshold rule is the training-free baseline you start with. It is also what the lab session 215 built.
How to read Figure 4
A low threshold grows the region into the noise; a high one keeps only the core of the event. Around \(\alpha = 0.35\)–0.5 the detected moment is [20, 27] s, tIoU 0.875. At \(\alpha = 0.8\) it shrinks to [22, 26] s and tIoU falls to 0.5. The replay bump is a separate region: the system must recognize it as the same event (same game clock on the scoreboard, a replay graphic, or a shot-type classifier) and either merge it or show it as “replay”, not as a second goal.
Latency and cost math before building
Now add up the online path. Everything except the re-ranker is small, so we give each part a fixed budget (a target we will hold the implementation to, not a measurement):
| Stage | Budget | Notes |
|---|---|---|
| Parse the query, extract filters (team and player dictionary, rules) | 5 ms | a small LLM here would add 100+ ms; use it only when the rules fail |
| Text embedding (SigLIP text tower, TensorRT FP16) | 5 ms | about 11 GFLOP for 64 tokens: tiny |
| IVF-PQ search with filters, exact re-score of top results | 20 ms | CPU or GPU; filtered searches are small |
| Transcript search and RRF fusion | 10 ms | runs in parallel with the vector search in practice |
| Fetch candidate frames (stored as JPEG at ingest) | 40 ms | never decode the source video online |
| Timestamp refinement and replay merge | 10 ms | uses stored per-second scores |
| Fixed part | 90 ms | |
| VLM re-rank of \(N\) candidates | \(N \cdot t_{\text{cand}}\) | formula (235.9); the dominant term |
The number of GPUs follows from the work per query. Each query keeps one GPU busy for \(N \cdot t_{\text{cand}}\) seconds. We do not run GPUs at 100%: queues grow without bound near full load and the 95th percentile explodes, so we plan for a target utilization \(u\) (we use 0.6):
- Read the symbols
- \(\lambda\)
- peak queries per second
- \(N\)
- candidates re-ranked per query (the \(K\) of Figure 3)
- \(u\)
- planned average utilization of each GPU at peak
- \(G\)
- number of L4s for the re-ranker
Qwen2.5-VL-3B in BF16, 4 frames at 336×336, \(N = 20\): \(t_{\text{cand}} = 57.7\) ms, re-rank 1,154 ms, total \(90 + 1{,}154 = 1{,}244\) ms. Under the 1.5 s target, with about 250 ms left for queueing and variation.
GPUs at 5 QPS: \(5 \times 1.154 / 0.6 = 9.6\), so 10 L4s, within the cap of 12. At an example price of $0.80 per GPU-hour (prices vary by cloud and contract; use yours), one query costs \(1.154 / 0.6 = 1.92\) GPU-seconds, so $0.43 per 1,000 queries. Keeping 10 GPUs up all month costs \(10 \times 720 \times 0.80 = \$5{,}760\).
Weights: about 3.75 billion parameters in BF16 is 7.5 GB, well within 24 GB, so one L4 holds one re-ranker with room for large batches.
How to read Figure 5
Start from the default (1,244 ms, 10 GPUs). Switch to 8 frames at 448: the bar passes 3.8 s and the GPU count passes 30. Switch to the 7B model: about 2.1 s, over budget. FP8 for the language model cuts its part in half (on the L4, an Ada GPU, FP8 tensor cores exist), giving 831 ms at the default settings. Lowering \(N\) to 10 halves both latency and cost, but read Figure 3 first: it also lowers the recall ceiling. Raise the load: latency per query does not change, but the GPU count grows linearly.
Moving work offline
The numbers suggest another option: run the VLM offline on every segment once, write a caption (“Penalty kick; goalkeeper in yellow dives left and saves”), and search captions as text. The prefill alone for 24 million segments at 3.49 TFLOP each is about 385 GPU-hours at 60.5 TFLOP/s, plus the caption decoding. That is a one-time cost a little smaller than two days of the 10 online re-rankers (480 GPU-hours). Captions make many queries cheap, but they only contain what the caption model chose to say; a query about a detail it skipped (“the ball hits both posts”) still needs the online, query-specific re-ranker. A good design uses both: captions and transcripts as extra first-stage lists in RRF, and the online re-ranker on a small \(N\).
The algorithms and the code
- input: video file, game metadata (teams, date, competition), event feed if available
- decode with the hardware decoder; keep 2 frames per second; store one JPEG per second for the re-ranker
- detect shot boundaries; read the scoreboard clock every second (OCR); flag replay shots
- embed every kept frame with the image tower (TensorRT engine, FP16, batch 64) // formula (235.6) for the cost
- cut segments of \(w\) seconds with stride \(s\), snapped to shot boundaries; mean-pool and normalize // (235.1), (235.3)
- transcribe the commentary; align transcript and event feed to video time with the clock map
- write segment vectors (PQ codes) with metadata to the index version \(v\); write transcripts to the keyword index; store the 2-fps frame embeddings on disk for the per-second scores
- run validation: number of segments, clock coverage, a few known events must be found
- input: query text, optional filters; index version \(v\)
- extract filters from the text (teams, players, season, half) with a dictionary; merge with explicit filters
- \(\mathbf{t}_q \leftarrow\) text embedding of the query (same model version \(v\) as the index)
- list A \(\leftarrow\) IVF-PQ top 200 within the filters, re-scored with exact FP16 vectors
- list B \(\leftarrow\) keyword search over transcripts and event feed within the filters
- candidates \(\leftarrow\) top \(N\) by RRF over lists A and B // (235.4)
- fetch 4 stored frames per candidate; VLM scores \(p(\text{yes})\) for all \(N\) in one batch // (235.8), (235.9)
- for the top results: refine \([\hat a, \hat b]\) from per-second scores; merge replays by game clock // (235.10)
- return moments (game, start, end, clip with pre-roll, score), and log every stage’s time and output
The PyTorch program below runs the whole pipeline on a fake half of football: synthetic frame embeddings with three events and a replay, segments and mean pooling, search, RRF fusion with a fake transcript, threshold refinement, tIoU, and the re-rank budget. It is deterministic; the printed values are shown in the comments.
import math, torch import torch.nn.functional as F torch.manual_seed(0) D, FPS, DUR = 64, 2, 1800 # embedding size, sampled frames per second, a 30-minute half # 1) a fake broadcast: frame embeddings = camera noise + event "concepts" during the events concepts = {e: F.normalize(torch.randn(D), dim=0) for e in ["goal", "corner", "yellow card"]} truth = {"goal": (600, 608), "corner": (300, 306), "yellow card": (1210, 1215)} frames = 0.35 * torch.randn(DUR * FPS, D) + F.normalize(torch.randn(D), dim=0) # shared "pitch" look for e, (s, t) in truth.items(): frames[s * FPS:t * FPS] += 1.2 * concepts[e] frames[700 * FPS:706 * FPS] += 0.8 * concepts["goal"] # a replay of the goal, weaker and shorter frames = F.normalize(frames, dim=-1) # 2) segments: window w, stride s, mean pooling then L2-normalize (formulas 235.1 and 235.3) w, s = 6, 3 starts = torch.arange(0, DUR - w + 1, s) seg = torch.stack([frames[a * FPS:(a + w) * FPS].mean(0) for a in starts.tolist()]) seg = F.normalize(seg, dim=-1) print("segments:", len(starts)) # 599 = (1800 - 6) // 3 + 1 # 3) query: a "text embedding" near the concept (a real system uses the SigLIP text tower) q = F.normalize(concepts["goal"] + 0.6 * torch.randn(D) / math.sqrt(D), dim=0) sim = seg @ q # exact search; IVF-PQ at archive scale top = sim.topk(5).indices print("top segment starts:", starts[top].tolist()) # [600, 603, 702, 699, 597]: 702 and 699 are the replay # 4) fuse with a transcript list by reciprocal rank fusion (formula 235.4) asr_hits = [598, 601, 604, 700] # seconds where commentary says "goal" asr_rank = {} for r, t in enumerate(asr_hits): j = int(torch.argmin((starts - t).abs())) asr_rank.setdefault(j, r + 1) vis_rank = {int(j): r + 1 for r, j in enumerate(sim.argsort(descending=True)[:50])} rrf = {j: sum(1 / (60 + rk[j]) for rk in (vis_rank, asr_rank) if j in rk) for j in set(vis_rank) | set(asr_rank)} fused = sorted(rrf, key=rrf.get, reverse=True)[:5] print("fused starts:", [int(starts[j]) for j in fused]) # [600, 603, 597, 699, 702] # 5) timestamps: per-second scores around the best segment, threshold at alpha * peak (formula 235.10) per_sec = (frames @ q).view(DUR, FPS).mean(1) c = int(starts[fused[0]]); lo, hi = max(0, c - 15), min(DUR, c + 21) a = per_sec[lo:hi]; p = int(a.argmax()); tau = 0.5 * float(a[p]) L = p while L > 0 and a[L - 1] >= tau: L -= 1 R = p while R < len(a) - 1 and a[R + 1] >= tau: R += 1 st, en = lo + L, lo + R + 1 def tiou(a, b): # formula 235.2 inter = max(0, min(a[1], b[1]) - max(a[0], b[0])) return inter / (max(a[1], b[1]) - min(a[0], b[0])) print("moment:", (st, en), "tIoU:", round(tiou((st, en), truth["goal"]), 3)) # (600, 608) tIoU: 1.0 # 6) the re-rank budget (formulas 235.8, 235.9, 235.12) for a 3B VLM on one L4 def rerank_ms(frames=4, res=336, n=20, p_vit=0.667e9, p_llm=2.775e9, text=64, eta=0.5, peak=121e12): tv = frames / 2 * (res / 28) ** 2 # visual tokens after 2x2 merge, 2-frame groups flops = 2 * p_vit * 4 * tv + 2 * p_llm * (tv + text) return n * flops / (eta * peak) * 1e3 ms = rerank_ms() print("re-rank ms:", round(ms), " GPUs for 5 QPS at 60% busy:", math.ceil(5 * ms / 1e3 / 0.6)) # 1154, 10
Notice what the toy already shows: the replay (starting near 700 s) is in the top 5 twice. Without a replay merge, the user would see the same goal three times.
Risks, failure modes, and what to measure
A design review spends as long on what can go wrong as on the happy path. The table lists the main risks for this system, how each one shows up, and what reduces it.
| Risk | How it shows up | Mitigation |
|---|---|---|
| Replays and highlights | the same goal appears 2–4 times in the top 10 | replay detection (shot type, replay graphics), merge by game clock; SoccerNet-v2 even has a replay grounding task [Certain] |
| Celebrations and reactions | close-ups of players celebrating rank above the event itself | re-ranker prompt asks for the action, not the reaction; timestamp step anchors on the action |
| Commentary delay | transcript-based moments start 5–20 s late | use transcripts for candidates only; place the start from frames |
| Ambiguous queries | “best goal”, “crazy moment”: no visual meaning | map to popularity or editor tags; ask a follow-up; do not let the VLM guess |
| Fine-grained identity | “Haaland header” returns any header | player names come from the event feed, jersey numbers or the transcript, not from the frame embedding |
| Domain shift | works on the main league, fails on women’s league, youth games, another sport, night games, low-resolution feeds | slice the test set by competition, camera and resolution; report each slice |
| Model upgrade | new text tower with old segment vectors gives nonsense | versioned indexes; re-embed, compare on the test set, then switch |
| Latency tails | p95 far above the mean when the re-ranker batch is large or GPUs are busy | plan utilization at 0.6; cap \(N\); a timeout that returns first-stage results |
| VLM failure modes | always says “yes”, prefers the first candidate in a batch, or is fooled by on-screen text | calibrate \(p(\text{yes})\) on labeled pairs; score candidates independently; check with blind (text-only) baselines |
| Rights and privacy | video from some competitions may not be stored or shown | rights metadata as an index filter, enforced on the server |
Measure each stage, not only the end
The end-to-end hit rate is roughly a product of stage success rates. For a query to succeed, the right moment must be in the first-stage list, the re-ranker must move it into the top 10, and the timestamp must be good:
With example values 0.93, 0.90 and 0.85, the product is 0.71. Each factor looks fine; the system does not. And when the end number is bad, only per-stage numbers tell you which stage to fix. So we log and measure:
- Offline quality: first-stage recall@\(K\) (the ceiling); re-ranker nDCG@10 and Hit@10 on the candidates; timestamp R1@IoU=0.5 and the mean start error in seconds; duplicate rate in the top 10. All on a fixed test set of real queries with labeled spans, split by query type, competition and camera (session 213).
- Online behavior: click-through and “clip used in an edit” rates; reformulated queries (the user did not find it); zero-result queries.
- System: p50, p95 and p99 latency per stage; GPU utilization; queue length; cost per 1,000 queries; ingest lag after the final whistle; decoder throughput.
Where this is used
| Application | What changes in the design |
|---|---|
| Broadcaster archive search for editors | this page; filters and accuracy matter more than cost per query |
| Fan-facing highlight search in a sports app | much higher QPS: cache popular queries, lower \(N\), more offline captions |
| Automatic highlight reels | no query: score every segment for importance (saliency), then pick and order the best moments (session 207) |
| Coaching and scouting tools | tactical queries (“high press after a goal kick”) need tracking data and player positions, not only frames (session 204) |
| Content moderation and rights checks | near-duplicate search instead of text search (session 209) |
| Video-RAG assistants (“what happened before the red card?”) | the same retrieval feeds moments into a Video-LLM that writes an answer (session 210) |
Common mistakes
- Choosing the model before the requirements. A 7B re-ranker with 8 frames looks great in a notebook and needs 55 L4s at 5 QPS (Figure 5).
- Running the VLM over the whole archive per query. Expensive work belongs offline, or on a small candidate set.
- Re-ranking without measuring first-stage recall. If the right moment is not in the top \(K\), no re-ranker can fix it; tune the first stage and the filters first.
- Stride equal to the window. Events that fall across two windows get tIoU as low as 0.33 for a 6-s event (Figure 2). Overlap by about half.
- Using commentary time as the moment time. Commentary is late; place the start from frames.
- Mixing embedding model versions in one index. Vectors from different models are not comparable.
- Decoding source video online. Store the frames the re-ranker needs at ingest; online, read JPEGs.
- Planning GPUs at 100% utilization. The mean latency looks fine; the 95th percentile does not.
Interview questions
Walk me through your design for sports key-moment search.basic
How do you choose the segment window and stride?core
Why retrieve-then-re-rank, and how do you choose the number of candidates?core
Estimate the VLM re-rank latency for Qwen2.5-VL-3B on one L4.core
The index does not fit on one GPU. What do you do?core
How do you handle replays?core
How do you produce precise timestamps?core
How do you evaluate the system before launch?core
You upgrade the embedding model. What breaks and how do you roll it out?advanced
A product manager now wants live games searchable within one minute. What changes?advanced
Why TensorRT on the L4, and what are the pitfalls?advanced
Should the VLM caption every segment offline instead of re-ranking online?advanced
Exercises
- Your archive is 500 hours. With \(w = 8\) s and \(s = 4\) s, how many segments are there, and how much memory do flat FP16 vectors with \(d = 768\) need?
- With formulas (235.8) and (235.9), compute the per-candidate time for Qwen2.5-VL-3B with 8 frames at 448×448 in BF16.
- Load grows to 12 QPS. With \(N = 10\), 4 frames at 336, 3B in BF16 and \(u = 0.6\), how many L4s do you need?
- First-stage recall@20 is 0.93, the re-ranker puts 90% of found moments in the top 10, and 85% of those have \(\text{tIoU} \ge 0.5\). What is the end-to-end success rate? Which stage would you improve first, and why?
- A 6-s event, a 6-s window. What is the worst-case best tIoU with stride 6 s, and with stride 3 s? Check with Figure 2.
- How many L4-seconds per day does it take to embed 100 new hours of video with SigLIP-B/16 at 2 frames per second? What part of ingest is more likely to be the bottleneck?
Answerscheck
Key takeaways
- Start with numbers: query types, load, latency (1.5 s at p95), accuracy (Hit@10, tIoU), cost (12 L4s) and fixed hardware (L4 + TensorRT).
- Expensive work offline once per video (decode, embed, transcribe, index); online only cheap search plus a VLM re-ranker on a few candidates.
- Window and stride set the best possible tIoU and the index size (235.1, 235.2): 6-s windows with a 3-s stride give 24 million segments for 20,000 hours; PQ fits them in 2.5 GB.
- First-stage recall@K is the ceiling (235.7); metadata filters raise it more cheaply than anything else.
- The re-ranker dominates latency and cost (235.8–235.12): 57.7 ms per candidate for a 3B VLM with 4 frames at 336, so 1.24 s per query and 10 L4s at 5 QPS.
- Replays, late commentary, ambiguous queries and model upgrades are the main risks; measure each stage separately because end-to-end success is their product (235.13).
Key formulas
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
- Luo et al. (2021). CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Johnson et al. (2017). Billion-scale similarity search with GPUs
- Jégou et al. (2011). Product Quantization for Nearest Neighbor Search
- Souček & Lokoč (2020). TransNet V2: An effective deep network architecture for fast shot transition detection
- Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
- NVIDIA (2023). NVIDIA L4 Tensor Core GPU (product page and datasheet)
- NVIDIA (docs). TensorRT documentation
- NVIDIA (code). TensorRT-LLM
- Deliège et al. (2021). SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models