LLM & VLM courseBlog
210Track 8: Video and multimodal systems · session 10 of 21

Video question answering and Video-RAG

A question about a long video is often about a few seconds of it. Video-RAG first finds those seconds in a searchable memory built from captions, speech transcripts, on-screen text and visual embeddings, and then lets a VLM read only them. This session builds that pipeline step by step, shows how to index a video at several levels (shot, scene, whole game), and gives a cost model for choosing between Video-RAG and a long-context video model.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Session 152 (RAG, part 1), session 205 (Video-LLMs), session 206 (long video) and session 208 (text-to-video retrieval)

What you will learn

  • Why uniform frame sampling misses short events, with a formula for the chance of a hit and its token cost
  • The retrieve-then-read pipeline: segment, describe, index offline; decompose, search, fuse, read online
  • How captions, ASR transcripts, OCR and visual embeddings each catch different questions, and how reciprocal rank fusion combines them
  • What the Video-RAG paper (Luo et al.) adds to a Video-LLM, and what it gains on Video-MME
  • How to index at several levels (shot, scene, whole game) and why coarse-to-fine search is about ten times cheaper
  • A cost and accuracy model for choosing Video-RAG or a long-context video model, with a break-even number of questions

Why this matters

Someone uploads a 90-minute football match and asks: “How was the winning goal scored?” The answer is in about 10 seconds of the video. The other 5,390 seconds do not matter for this question. Video question answering (video QA) is the task of answering such questions in text. Video-RAG (retrieval-augmented generation for video) is a way to do it well on long videos: first find the few segments that matter, then let a vision-language model (VLM) read only those segments and answer.

Why not give the whole match to a Video-LLM? Because of tokens. In session 205 we saw that each frame costs a few hundred tokens. At 1 frame per second and 196 tokens per frame, the 90-minute match is \(5{,}400 \times 196 = 1{,}058{,}400\) visual tokens: more than one million tokens for one question. Most open Video-LLMs therefore sample 16 to 128 frames from the whole video. With 64 frames, the gap between two frames is 84 seconds, and a 3-second goal falls between frames about 96% of the time (we derive this below). The model then answers from what it guesses, not from what it saw.

Retrieval fixes this. It is the same idea as text RAG in session 152: keep a searchable memory outside the model, look up the right pieces at question time, and put them in the prompt. For video, the memory is built from what a video contains: captions that describe each shot, the speech transcript, the text on screen, and visual embeddings. This session builds that memory and the pipeline around it.

Real systems work this way. Sports platforms let fans search “all corners that led to a goal”. Lecture platforms answer “where does the teacher explain the proof?”. Security and body-camera teams search hundreds of hours of footage. Google’s Gemini models take the other route and read up to about an hour of video directly in a 1M-token context [Likely]. We will see when each route is the right one. The book chapter on video-language models (Noyan et al., chapter 9) and the Video-RAG paper of Luo et al. (2024) are the main sources.

Video QA: what is asked, and why long videos are hard

Video questions come in a few types. Each type needs different evidence:

  • Local, descriptive: “What color is the goalkeeper’s shirt?” One frame is enough.
  • Local, temporal: “When was number five booked?” You need the exact moment, and the answer is a time.
  • Causal and order: “Why did the referee give a penalty?” You need a few seconds before and after the event.
  • Text in the video: “What was the score at half time?” The answer is on the scoreboard, so you need on-screen text, not motion.
  • Speech: “What did the commentator say about the save?” The answer is in the audio.
  • Global: “How did the second half differ from the first?” You need a summary of everything, not one moment.

Benchmarks reflect this mix. Video-MME (Fu et al., 2024) has 900 videos with a total of 254 hours, from 11 seconds to 1 hour long, and 2,700 multiple-choice questions [Certain]. It reports scores with and without subtitles, which already tells you that the transcript is a strong source of evidence. Other common sets are NExT-QA (causal and temporal questions), EgoSchema (long first-person videos) and MLVU and LongVideoBench (long videos) [Likely]. session 213 covers them in detail.

The cost of seeing everything

Let \(T\) be the video length in seconds, \(f\) the sampling rate in frames per second, and \(n_f\) the number of visual tokens per frame. The number of visual tokens is

\[ N_{\text{vis}} = T \cdot f \cdot n_f \tag{210.1} \]

For the 90-minute match at \(f = 1\) and \(n_f = 196\) (a \(14 \times 14\) grid after pooling, a typical choice; real models use between about 64 and 729 [Likely]), this is 1,058,400 tokens. At 2 frames per second it is 2,116,800. The model also needs the answer to be in the right part of this huge context: session 28 and the “lost in the middle” study (Liu et al., 2023) show that models use the middle of very long contexts less well.

The chance that sampling sees a short event

If we can afford only \(K\) frames, we take them evenly spaced, one every \(T/K\) seconds. An event of length \(d\) seconds is seen if at least one frame lands inside it. When the event starts at a random time, the chance of a hit is

\[ P(\text{hit}) = \min\!\left(1,\ \frac{d\,K}{T}\right) \tag{210.2} \]
Read the symbols
\(T\)
video length in seconds
\(K\)
number of frames the model reads, evenly spaced
\(d\)
length of the event the question is about, in seconds
\(T/K\)
gap between two sampled frames; if \(d \ge T/K\), every event of length \(d\) is hit
Worked example: 64 frames of a football match

\(T = 5{,}400\) s, \(K = 64\), \(d = 3\) s. The gap is \(5{,}400 / 64 = 84.4\) s. The chance of a hit is \(3 \times 64 / 5{,}400 = 0.036\): the model sees the goal in only 3.6% of matches. It costs \(64 \times 196 = 12{,}544\) visual tokens.

To be sure of a hit we need \(T/K \le d\), so \(K = 5{,}400 / 3 = 1{,}800\) frames, or 352,800 tokens. That is 28 times more tokens, and it still covers only one event length. Retrieval instead reads 3 clips of 30 s at 2 frames per second: 180 frames, \(180 \times 196 = 35{,}280\) tokens, and every frame is near the goal.

sampled framethe event the question asks about
Figure 1. Uniform frame sampling on a long video. The top bar is the whole video; the bottom bar zooms into the minute around the event (here placed at 31:50, like the first goal of our example match). Formula (210.2) gives the chance that a randomly placed event of this length is hit.
How to read Figure 1

Start with the defaults: 90 minutes, 64 frames, a 3-second event. In the zoomed bar the event sits between two frames, and the readout says a random event is hit only 3.6% of the time. Raise \(K\) step by step: the hit chance grows linearly, and so do the tokens. You need 2,048 frames before the gap (2.6 s) is shorter than the event, at 401,408 tokens. Now make the event 60 seconds long (a whole attack): 128 frames are enough. Short events are the problem, and short events are exactly what people ask about.

Retrieve, then read

The idea is to split the work into two parts that each do one thing well:

  1. Retrieve: a cheap search over an index finds the \(k\) segments (for example \(k = 3\) to 10) that most likely hold the answer.
  2. Read: a VLM reads only those segments, now at a high frame rate and high resolution, together with their text, and writes the answer with timestamps.

The work also splits in time. Offline, once per video, we cut the video into segments and describe each one. Online, once per question, we search and read. The offline cost is paid once and shared by every question about that video.

Offline: build the memory

First we cut the video into units. A shot is a run of frames from one camera without a cut; a shot-boundary detector such as TransNet V2 or PySceneDetect finds the cuts [Certain]. Shots in sports and TV are often a few seconds long. Very long shots (a fixed camera, a lecture) are cut into fixed windows instead, for example 10 to 30 seconds, often with some overlap so an event is not cut in half. For each segment we store several kinds of evidence, each with its start and end time.

Algorithm 1: build a Video-RAG index (offline, once per video)
  1. input: video \(V\), window length \(w\), overlap \(o\)
  2. cut \(V\) into shots with a shot-boundary detector; split shots longer than \(w\) into windows of length \(w\) with overlap \(o\)
  3. for each segment \(c\) with times \([t_s, t_e]\):
  4.   pick a few keyframes; \(\mathbf{v}_c \leftarrow\) mean of their image embeddings // CLIP or SigLIP, for visual search
  5.   \(\text{cap}_c \leftarrow\) a short caption from a VLM // “header from a corner, ball in the net”
  6.   \(\text{asr}_c \leftarrow\) the speech transcript of \([t_s, t_e]\) // Whisper, with word timestamps
  7.   \(\text{ocr}_c \leftarrow\) text found on screen in the keyframes // scoreboard, slides, captions
  8.   embed each text field with a text encoder; add it to a vector index and a keyword (BM25) index
  9.   store \((t_s, t_e, \text{cap}_c, \text{asr}_c, \text{ocr}_c)\) as metadata
  10. build summaries one level up (scenes, then the whole video) from the segment captions // see the section on levels

Online: answer one question

At question time we turn the question into search requests, search every field, merge the results, and read. The scoring inside one field is the familiar one from session 150: an encoder maps the query \(q\) and each segment field \(c\) to unit vectors, and the score is their cosine similarity:

\[ s_m(q, c) = \cos\big(E_q(q),\ E_m(c)\big) = \frac{E_q(q)^\top E_m(c)}{\lVert E_q(q)\rVert\,\lVert E_m(c)\rVert} \tag{210.3} \]
Read the symbols
\(m\)
one field (modality) of the memory: caption, ASR, OCR or visual
\(E_q, E_m\)
the query encoder and the encoder for field \(m\); for the visual field \(E_m\) is an image encoder and \(E_q\) its paired text encoder (CLIP style)
\(s_m(q, c)\)
how well segment \(c\) matches the query in field \(m\), between −1 and 1

Scores from different fields are on different scales: a cosine of 0.3 from CLIP and a BM25 score of 12 cannot be added. The standard fix is to merge ranks instead of scores. Reciprocal rank fusion (RRF; Cormack et al., 2009) gives each segment

\[ \text{RRF}(c) = \sum_{m \in \mathcal{M}} \frac{1}{\kappa + \text{rank}_m(c)} \tag{210.4} \]
Read the symbols
\(\mathcal{M}\)
the set of fields (and search methods) we search
\(\text{rank}_m(c)\)
the position of segment \(c\) in the result list of field \(m\) (1 = best); a segment missing from a list adds nothing
\(\kappa\)
a constant that softens the gap between rank 1 and rank 2; 60 is the usual value, from the original paper [Likely]

A segment that shows up near the top of several lists wins, even if no single list puts it first. That is what we want: the real goal appears in the transcript (“what a strike”), on the scoreboard (“2-1”) and in the picture (ball in the net).

Algorithm 2: answer a question with Video-RAG (online, once per question)
  1. input: question \(q\), index, number of segments \(k\), read rate \(f_r\)
  2. \(R \leftarrow\) LLM(“write search requests for each field”, \(q\)) // query decomposition; a field may get no request
  3. for each field \(m\) with a request \(r_m\): \(L_m \leftarrow\) top-\(n\) segments for \(r_m\) (dense formula 210.3 and BM25)
  4. \(S \leftarrow\) top-\(k\) segments by \(\text{RRF}\) over all lists \(L_m\) // formula (210.4)
  5. optional: rerank \(S\) with a cross-encoder or a small VLM that looks at the frames
  6. widen each segment by a few seconds on both sides; merge segments that overlap
  7. sample the chosen windows at \(f_r\) (for example 2 fps), in time order
  8. prompt \(\leftarrow\) [frames with timestamps] + [caption, ASR, OCR text of each window] + \(q\)
  9. return VLM(prompt), with the timestamps of the evidence it used

Step 6 matters more than it looks. Causal questions (“why was the penalty given?”) need the seconds before the retrieved moment. Step 7 keeps the windows in time order and labels each frame with its real time, so the reader can say “at 88:45” (see session 205 for how time is given to the model). Figure 2 runs the whole pipeline on a small index.

Figure 2. Algorithm 2 on a toy index of 11 segments from a cup final (the same index as Figure 3). The question is “How was the winning goal scored?”. The bars at the bottom are the fused RRF scores of formula (210.4) with \(\kappa = 60\).
How to read Figure 2

Watch the four searches in step 3. No field is perfect: the caption search puts a saved shot (12:40) first, because its caption contains “shot”; the visual search puts the first goal (31:50) first, because all goals look alike (ball in the net). Only the 88:45 segment appears in all four lists, so RRF gives it 0.0648, twice the next one. The reader then gets three clips, and the two wrong ones do little harm: the VLM sees that they are not the winner, because the scoreboard in them is not 2-1.

Captions, ASR and OCR as searchable memory

Each field of the memory catches a different kind of question. It helps to know what each one is good and bad at.

FieldMade byCatchesMisses
Captionsa VLM that describes each segment in one or two sentencesactions and scenes (“player lies on the grass, medics run on”)exact names, numbers and words; depends on how detailed the captioner is; can describe things that are not there
ASR transcriptspeech recognition such as Whisper, with timestamps per word or phrasewhat people say: names, opinions, explanations; in lectures and meetings this is most of the contenteverything silent; commentary can lag the action by a few seconds; recognition errors on names
OCRa text detector and reader on keyframes (EasyOCR, PaddleOCR, or a VLM)scoreboards, clocks, slides, jersey numbers, signs, burnt-in subtitlestext that is small, moving or at an angle; it repeats a lot (the score is on screen for minutes)
Visual embeddingan image or video encoder (CLIP, SigLIP, InternVideo2) on keyframesappearance and objects that nobody names (“the trophy lift”); the basis of session 208fine differences between similar moments (all goals look alike); counting; text
Objects and tagsan open-vocabulary detector (for example APE) or a classifier“how many players are in the box?”, a list of objects per segmentrelations and actions

The text fields have one big advantage: once the video is turned into text, all the tools of text search work: BM25 for exact words and numbers (a jersey number “14” or a score “2-1” is a terrible target for embeddings), dense retrievers for meaning, rerankers, and plain filters (“only the second half”). This is why many production systems turn video into rich text first and keep the visual embeddings as one more field. [Likely]

Figure 3. Which field finds which answer. Each row is one segment of the toy cup final, with its caption, transcript, on-screen text and visual tags. For the chosen question, a number in a cell is that segment’s rank in that field (only segments that match at least one query word get a rank). The bar on the right is the fused RRF score; the green row is the true answer. To keep the toy transparent, “match” means sharing a query word, and the visual field is shown as words; a real system uses embeddings, formula (210.3).
How to read Figure 3

Turn on one field at a time and read the score line in the readout. Each field alone answers only 2 of the 6 questions: ASR finds the booking (“a booking for number five”), but only OCR finds the substitution (“sub 9 off 14 on”) and the first 2-1, and only the visual field finds the trophy lift. All four fields together answer 6 of 6. Notice also that captions plus ASR plus OCR still miss the trophy: some things nobody says or writes.

A case study: Video-RAG (Luo et al., 2024)

The paper “Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension” turns these ideas into a plug-in for existing Video-LLMs, with no training [Certain]. Its parts:

  • Query decomposition: the Video-LLM first sees only the question (no frames) and writes JSON requests for each tool: what to look for in the speech, which objects to detect, and what kind of information is needed. A request can be empty.
  • Three text sources: Whisper for speech, EasyOCR for on-screen text, and APE (an open-vocabulary detector) for objects, with CLIP to choose the keyframes for detection.
  • Retrieval: Contriever embeds the text chunks, FAISS searches them, and only chunks above a similarity threshold (0.3) are kept.
  • Reading: the kept text is added to the prompt next to the usual sampled frames.

The reported cost is small: about 2,000 extra text tokens per question (the authors compare this to about 14 frames), and for a 7B model about 8 GB more GPU memory and about 5 seconds more per question [Certain, as reported]. The reported gains on Video-MME (overall accuracy, %) [Certain, as reported]:

Model (frames)No subtitlesWith subtitlesWith Video-RAG
Long-LLaVA 7B (64)52.957.862.6
LLaVA-Video 72B (64)70.375.977.4
GPT-4o (384), for reference71.977.2–
Gemini-1.5-Pro (0.5 fps), for reference75.081.3–

Three lessons. First, subtitles alone give the largest jump (+4.9 and +5.6 points): speech is cheap, strong evidence. Second, the retrieved OCR and object text adds more on top (+4.8 for the 7B model, +1.5 for the 72B model); the paper reports an average gain of 2.8 points over seven Video-LLMs. Third, the smaller model gains more, because the retrieved text gives it facts it could not read from 64 frames. Note that this system retrieves text and still reads uniformly sampled frames; the full pipeline of Algorithm 2 also chooses which frames to read.

Two other papers share the name. VideoRAG of Jeong et al. (2025) retrieves whole videos from a large corpus for a question, then reads the retrieved videos with a Video-LLM. VideoRAG of Ren et al. (2025) builds a knowledge graph from many long videos plus visual embeddings, and tests on a benchmark of more than 160 videos and 134 hours [Certain]. VideoAgent (Wang et al., 2024) makes retrieval a loop: an LLM looks at a few frames, decides what it is missing, and asks for more; it reports 54.1% on EgoSchema while reading 8.4 frames on average [Certain, as reported]. This agent style is covered in session 220.

Indexes at several levels: shot, scene, whole game

One flat list of shots is not enough. “Who took the corner at 31:50?” is about one shot. “How did the penalty come about?” is about a scene of a minute or two. “How did the second half differ from the first?” is about the whole game, and no single shot holds the answer: retrieving the top 5 shots for it gives five random moments. So we index at several levels, like the chapters, sections and paragraphs of a book:

  • Shot (seconds): keyframe embeddings, a short caption, ASR and OCR. Answers local questions and gives the exact time.
  • Scene (one to a few minutes): a group of shots about one thing (an attack, a slide in a lecture). A short summary written from the shot captions and transcript.
  • Whole video (the game, the lecture): a summary of the scenes, plus structured facts (score over time, list of goals and cards).

Two uses follow. Global questions go to the top level: the reader gets the scene summaries, which are short text, instead of frames. Local questions can be searched coarse to fine: first find the best \(k\) scenes, then search only the shots inside them. With \(N\) shots grouped into \(S\) scenes of \(N/S\) shots each, the number of comparisons is

\[ C_{\text{flat}} = N, \qquad C_{\text{hier}} = S + k\,\frac{N}{S} \tag{210.5} \]

Setting the derivative of \(C_{\text{hier}}\) with respect to \(S\) to zero gives the best number of scenes and the smallest cost:

\[ S^{*} = \sqrt{k N}, \qquad C_{\text{hier}}^{*} = 2\sqrt{k N} \tag{210.6} \]
Worked example: a 90-minute match

With shots of 4 seconds on average, \(N = 5{,}400/4 = 1{,}350\) shots. With scenes of 2 minutes, \(S = 45\) scenes of 30 shots. Searching \(k = 3\) scenes: \(C_{\text{hier}} = 45 + 3 \times 30 = 135\) comparisons, 10 times fewer than 1,350. The best grouping is \(S^{*} = \sqrt{3 \times 1{,}350} = 63.6\) scenes (about 1.4 minutes each), with cost 127.

For one match, 1,350 comparisons is nothing; a GPU does millions per millisecond. The saving matters for an archive. Ten thousand matches have 13.5 million shots, and the same two-level idea is what vector indexes such as IVF in FAISS do inside (session 151). The bigger benefit of levels is quality: scene summaries answer questions that no single shot can.

searched at this levelsent to the reader
Figure 4. A three-level index of a 90-minute match. Pick a question type to see which level answers it and what the reader receives. The sliders change the grouping, and the readout applies formulas (210.5) and (210.6).
How to read Figure 4

For the local question, all 45 scenes are searched (blue), then only the shots of the top 3 scenes (blue ticks), and one shot is sent to the reader (orange). Move the scene length: very short scenes make the first stage expensive; very long scenes make the second stage expensive; the minimum is near \(\sqrt{kN}\). For the whole-game question nothing is searched at the shot level; the reader gets every scene summary as text, about 2,700 tokens if each summary is 60 tokens.

Video-RAG or a long-context video model?

Since 2024, some models can read a whole long video. Gemini 1.5 has a context of up to millions of tokens, and the Gemini API documentation says it samples video at 1 frame per second, at about 100 tokens per second of video at low resolution or about 300 at high resolution, so a 1M-token context holds about 3 hours or 1 hour of video [Likely; such numbers change between model versions]. session 206 covers open models with long contexts and token compression. So when is retrieval still the right choice? Look at accuracy and at cost.

Accuracy: two ways to fail

A Video-RAG system is right only if retrieval brings the evidence and the reader uses it. Approximately,

\[ P(\text{correct}) \approx \text{Recall@}k \times a_{\text{read}} \tag{210.7} \]
Read the symbols
\(\text{Recall@}k\)
the fraction of questions for which the true evidence is among the \(k\) retrieved segments
\(a_{\text{read}}\)
the accuracy of the reader when it does have the evidence

For example, if Recall@5 is 0.9 and the reader is right 85% of the time with the evidence, the system is right about \(0.9 \times 0.85 = 0.765\) of the time. This split is also the debugging method: label the evidence time for a few hundred questions, measure Recall@\(k\) alone, and measure the reader alone with the true segment given. Then you know which half to fix. A long-context model has no retrieval step, so it cannot lose the evidence that way, but at low frame rate it may never see a short event (formula 210.2), and in a huge context it may not use it.

Cost: pay once, or pay per question

Let \(r_{\text{lc}}\) be the tokens per second of video for the long-context model, \(r_{\text{idx}}\) the tokens per second spent once to build the index (captioning, ASR and OCR; we count it in the same token units, a simplification), \(k\) clips of \(\ell\) seconds read at \(r_{\text{rd}}\) tokens per second, and \(c\) tokens of retrieved text. For \(Q\) questions about one video of length \(T\):

\[ \begin{aligned} \text{Cost}_{\text{lc}} &= Q\, T\, r_{\text{lc}} \\ \text{Cost}_{\text{rag}} &= T\, r_{\text{idx}} + Q\,(k\,\ell\, r_{\text{rd}} + c) \\ Q^{*} &= \frac{T\, r_{\text{idx}}}{T\, r_{\text{lc}} - k\,\ell\, r_{\text{rd}} - c} \end{aligned} \tag{210.8} \]

\(Q^{*}\) is the break-even number of questions: above it, Video-RAG is cheaper. (A long-context model can cache the video’s keys and values and reuse them across questions; that lowers its prefill cost but needs a lot of memory to keep the cache, see session 134.)

Worked example: a 1-hour video

\(T = 3{,}600\) s, \(r_{\text{lc}} = 100\), \(r_{\text{idx}} = 100\), \(k = 5\) clips of \(\ell = 30\) s read at high detail \(r_{\text{rd}} = 300\), and \(c = 2{,}000\) text tokens. Long context: \(3{,}600 \times 100 = 360{,}000\) tokens per question. Video-RAG: \(5 \times 30 \times 300 + 2{,}000 = 47{,}000\) tokens per question, plus 360,000 once. Break-even: \(Q^{*} = 360{,}000 / (360{,}000 - 47{,}000) = 1.15\). From the second question on, Video-RAG is cheaper; for 10 questions it is 830,000 tokens against 3,600,000. And Video-RAG reads its clips at 3 times the detail.

long-context modelVideo-RAG: index, built onceVideo-RAG: per-question reading
Figure 5. The cost model of formula (210.8), with \(r_{\text{idx}} = 100\), \(r_{\text{rd}} = 300\), \(c = 2{,}000\) and a context window of 1M tokens. These rates are assumptions for illustration, close to the Gemini API figures above; plug in your own model’s numbers.
How to read Figure 5

With one question per video, the long-context model costs about the same as building the index, and it is simpler: no index to keep. With many questions, Video-RAG wins by a wide margin. Make the video longer than about 2.8 hours at \(r_{\text{lc}} = 100\): the whole video no longer fits in 1M tokens, so the long-context route must drop frames or split the video, which is a kind of retrieval anyway. Increase \(k\) and \(\ell\) until the RAG reading cost per question passes the long-context cost: then retrieval never pays off.

A decision guide

SituationBetter choiceWhy
One video, one or two questions, under an hourLong-context modelNothing to build; sees everything at once; good for global questions
Many questions per video (a lecture, a match, a meeting)Video-RAGIndex cost is shared; each question reads little (formula 210.8)
An archive of thousands of hoursVideo-RAG (often two-stage: find videos, then segments)No context holds it; must search first
Short events, exact times, scores, namesVideo-RAG with OCR, ASR and high-fps readingUniform sampling misses them (formula 210.2); BM25 finds exact words
Global questions (“summarize”, “how did it change”)Long-context, or RAG over scene summariesTop-\(k\) shots cannot hold a global answer
Live video (a game in progress)Video-RAG with an index updated as segments arriveThe video grows; append to the index rather than re-reading everything
Answers must be checked by a personVideo-RAGEvery answer comes with the timestamps of its evidence

The two are often combined: retrieve the 10 to 20 best minutes of a long video, then let a long-context model read all of them at a good frame rate. [Likely]

Code: a tiny Video-RAG retriever in PyTorch

The code builds a small index of segments, searches three text fields with both dense vectors and BM25, fuses the six result lists with RRF (formula 210.4), and builds the reader’s prompt with timestamps. The text encoder is a hashed bag of words so that the code runs anywhere without downloads; in a real system you would replace embed with Contriever, a sentence-transformer, or the text tower of CLIP, and add a visual field. It is deterministic.

import math, zlib, torch
torch.manual_seed(0)

# 1) the index: one record per 30-second segment (start time, caption, ASR, OCR)
segs = [
    ("23:05", "referee shows a card to a defender", "that is a booking for number five", "0-0 23:05 yellow #5"),
    ("31:50", "header from a corner ball in the net", "goal a header from the corner", "1-0 31:52"),
    ("52:30", "substitution two players swap at the touchline", "a change for the visitors", "sub 9 off 14 on"),
    ("74:20", "player lies on the grass medics run on", "he looks hurt", "1-1 74:20"),
    ("88:45", "long shot ball in the top corner", "what a strike that could be the winner", "2-1 88:47"),
]
fields = ["caption", "asr", "ocr"]

# 2) a stand-in text encoder: hashed bag of words -> unit vector (a real system uses Contriever or similar)
D = 256
STOP = {"a", "an", "the", "of", "on", "in", "to", "as", "is", "that", "for", "when", "did", "come", "what"}
def words(text):
    return [w for w in text.lower().split() if w not in STOP]
def embed(text):
    v = torch.zeros(D)
    for w in words(text):
        v[zlib.crc32(w.encode()) % D] += 1.0
    return v / v.norm().clamp_min(1e-9)
E_ = {f: torch.stack([embed(s[1 + j]) for s in segs]) for j, f in enumerate(fields)}  # (N, D) per field

# 3) BM25 over one field (keyword search; catches names, numbers, jersey digits)
def bm25(query, docs, k1=1.2, b=0.75):
    toks = [words(d) for d in docs]; N = len(toks); avg = sum(map(len, toks)) / N
    out = torch.zeros(N)
    for w in set(words(query)):
        n = sum(w in t for t in toks)
        if n == 0: continue
        idf = math.log(1 + (N - n + 0.5) / (n + 0.5))
        for i, t in enumerate(toks):
            tf = t.count(w)
            out[i] += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(t) / avg))
    return out

# 4) reciprocal rank fusion, formula (210.4): only items with a positive score get a rank
def rrf(score_lists, k=60):
    fused = torch.zeros(len(segs))
    for s in score_lists:
        order = torch.argsort(-s, stable=True)
        for r, i in enumerate(order.tolist(), start=1):
            if s[i] > 0: fused[i] += 1.0 / (k + r)
    return fused

def retrieve(query, top_k=2):
    q = embed(query)
    lists = [E_[f] @ q for f in fields]                                   # dense cosine per field
    lists += [bm25(query, [s[1 + j] for s in segs]) for j in range(3)]    # sparse per field
    fused = rrf(lists)
    return torch.argsort(-fused, stable=True)[:top_k].tolist(), fused

# 5) read: put the retrieved evidence, with timestamps, in front of the question
query = "when did number 14 come on as a substitution"
idx, fused = retrieve(query)
prompt = "".join(f"[{segs[i][0]}] caption: {segs[i][1]} | speech: {segs[i][2]} | screen: {segs[i][3]}\n" for i in idx)
prompt += f"Question: {query}\nAnswer with a timestamp."
print([round(x, 4) for x in fused.tolist()])   # fused score per segment
print(prompt)

# 6) why we retrieve at all: chance that K uniform frames hit an event of d seconds, formula (210.2)
T, d = 90 * 60, 3
for K in (32, 64, 256, 1800):
    print(K, "frames:", round(min(1.0, d * K / T), 4), "hit chance,", K * 196, "visual tokens")

Output: the fused scores are [0.0328, 0.0, 0.0656, 0.0, 0.0]. The substitution at 52:30 gets \(4/61 = 0.0656\): it is first in the dense and the BM25 list of both the caption field (“substitution”) and the OCR field (“14”). The booking at 23:05 gets \(2/61 = 0.0328\) only because its transcript contains the word “number”: a typical distractor, and the reason we give the reader more than one segment and ask it to cite. The last lines print hit chances 0.0178, 0.0356, 0.1422 and 1.0 for 32, 64, 256 and 1,800 frames, at 6,272, 12,544, 50,176 and 352,800 visual tokens.

Where this is used

ApplicationWhat is indexedTypical question
Sports search and highlightsshots with OCR of the scoreboard and clock, commentary ASR, event tags; game-level facts“All corners in the second half that led to a shot”; key-moment detection (session 207)
Lecture and course videoASR (most of the content), OCR of slides, slide-change scenes“Where does she prove the theorem?”, answered with a link to the minute
Meeting recordingsASR with speaker labels, screen-share OCR, a summary per topic“What did we decide about the budget?”
News and media archivesshots, faces and names, ASR, OCR of lower-third captions; near-duplicate groups (session 209)“Every clip where the minister talks about energy, first appearance only”
Security and body-camera footagevisual embeddings, object tags, time and camera id“A red van near gate 3 between 2 and 4 pm”
Egocentric and robotics videodense captions of what the wearer does, object tags“Where did I leave my keys?” (episodic memory)
Customer support and how-to videosteps from ASR and captions, OCR of on-screen menus“How do I reset the router?”, answered with the clip

The production side of these systems (ingestion queues, GPU budgets, keeping indexes fresh) is the topic of session 211. Retrieval over documents and images, the same pattern for PDFs, is in session 219.

Common mistakes

  • Indexing only visual embeddings. They miss scores, names, numbers and anything said. Most questions in lectures and sports are answered by ASR and OCR (Figure 3).
  • Wrong time bases. ASR times relative to an audio chunk, frame indices at a different fps, clip-relative OCR times. Store absolute video time in seconds for every field and test it: ask for a known event and check the cited minute.
  • Reading only the retrieved second. Causal questions need context before the moment. Widen windows and merge overlaps (Algorithm 2, step 6).
  • Top-\(k\) for global questions. “Summarize the match” retrieves 5 random shots. Route such questions to scene or video summaries.
  • Fusing raw scores. Adding a cosine and a BM25 score mixes scales; one field then dominates. Fuse ranks (RRF) or normalize each list.
  • Evaluating only the final answer. You cannot tell whether retrieval or the reader failed. Measure Recall@\(k\) and reader accuracy separately (formula 210.7).
  • Trusting captions as ground truth. A captioner can describe things that did not happen. The reader should look at the frames, and answers should cite times so a person can check.
  • Shuffled evidence. Putting retrieved clips in score order instead of time order breaks “before” and “after” questions. Sort by time and label each frame with its timestamp.

Interview questions

What is Video-RAG, in two sentences?basic
Video-RAG answers questions about long videos by first retrieving the few segments (and their text) that hold the evidence from an index built offline, then letting a VLM read only those segments at high detail. It turns an impossible “read everything” problem into a cheap search plus a short, focused read, and it can cite timestamps.
Why not just sample 64 frames uniformly from a 90-minute video?basic
The gap between frames is \(5{,}400/64 = 84\) s. A 3-second event is hit with probability \(dK/T = 3 \times 64/5400 \approx 3.6\%\) (formula 210.2). Guaranteeing a hit needs \(K = T/d = 1{,}800\) frames, about 353K tokens at 196 tokens per frame. Retrieval reads a few clips around the right moment instead.
What would you put in the index for a sports video? Why each part?core
Shots from a shot-boundary detector, each with: keyframe embeddings (CLIP or SigLIP) for visual queries; a VLM caption for actions; the commentary transcript from ASR with word timestamps; OCR of the scoreboard and clock; event tags (goal, card, corner) from a classifier. One level up, scene summaries, and at the top a game timeline (score over time, events). All with absolute video time. Captions cover actions, ASR covers names and opinions, OCR covers numbers and scores, embeddings cover things nobody names.
How do you combine results from caption, ASR, OCR and visual search?core
Scores are on different scales, so the simple robust choice is reciprocal rank fusion: \(\sum_m 1/(\kappa + \text{rank}_m(c))\) with \(\kappa\) around 60. A segment that appears high in several lists wins. Alternatives: normalize scores per list (z-score or min-max) and take a weighted sum with weights tuned on labeled queries, or retrieve a candidate set with RRF and rerank it with a cross-encoder or a small VLM that looks at the frames. RRF ignores how strong a match is, so remove stopwords and apply per-list thresholds.
Your Video-RAG system gives wrong answers. How do you find out why?core
Split the error (formula 210.7). Label the evidence time span for a sample of questions. Measure Recall@\(k\): is the labeled span among the retrieved segments? Then give the reader the true span directly and measure its accuracy. If recall is low, fix the index (missing fields, bad segmentation, wrong time bases) or the query (decomposition, BM25 for exact terms). If recall is high but answers are wrong, fix the reader: more frames per clip, better ordering, prompts that require citing, or a stronger VLM.
When would you choose a long-context video model over Video-RAG?core
When the video fits in the context (for example under an hour at about 300 tokens per second in 1M tokens), there are only one or a few questions per video, and questions are global (“summarize”, “how did the mood change”). Then there is nothing to build and the model sees everything. Choose Video-RAG for many questions per video (the index cost is shared, formula 210.8), for archives larger than any context, for short events and exact facts, and when you need citations. Often combine: retrieve the best minutes, then read them with a long-context model.
Derive the break-even number of questions between long-context reading and Video-RAG.advanced
Long-context: \(Q T r_{\text{lc}}\). RAG: \(T r_{\text{idx}} + Q(k\ell r_{\text{rd}} + c)\). Equal when \(Q^{*} = T r_{\text{idx}} / (T r_{\text{lc}} - k\ell r_{\text{rd}} - c)\). For a 1-hour video with \(r_{\text{lc}} = r_{\text{idx}} = 100\), 5 clips of 30 s at 300 tokens/s and 2,000 text tokens: \(360{,}000/313{,}000 = 1.15\), so RAG is cheaper from the second question. If the per-question read cost is larger than \(T r_{\text{lc}}\), the denominator is negative and RAG never pays off on tokens. KV-cache reuse can lower the long-context cost for repeated questions, at a large memory cost.
How do you handle “How did the second half differ from the first?”core
Top-\(k\) shot retrieval fails: no shot holds the answer. Use a hierarchical index: scene summaries written offline from shot captions and transcripts, and a game-level summary with structured facts (score timeline, events). A router (a classifier or the LLM itself) sends global questions to the top levels, and the reader gets the scene summaries of each half as text. A long-context model is also a good option here.
Why is coarse-to-fine search cheaper, and what is the best number of scenes?advanced
Flat search compares the query with all \(N\) shots. Two-stage search compares it with \(S\) scenes, then with the \(N/S\) shots in each of the \(k\) best scenes: \(S + kN/S\). The minimum over \(S\) is at \(S^{*} = \sqrt{kN}\), cost \(2\sqrt{kN}\). For \(N = 1{,}350\), \(k = 3\): about 64 scenes, cost 127 instead of 1,350. The price is that a miss in the first stage cannot be recovered, so \(k\) must be large enough; IVF indexes in FAISS make the same trade-off.
What does the Video-RAG paper of Luo et al. add, and what does it gain?advanced
A training-free plug-in: the Video-LLM writes JSON search requests from the question alone; Whisper (speech), EasyOCR (on-screen text) and APE (open-vocabulary detection) turn the video into text; Contriever plus FAISS retrieve the chunks above a 0.3 similarity threshold; the text is added next to the sampled frames. About 2K extra tokens per question. On Video-MME it reports LLaVA-Video 72B going from 75.9 (with subtitles) to 77.4, and Long-LLaVA 7B from 57.8 to 62.6, with an average gain of 2.8 points over seven models. It retrieves text but does not choose which frames to read.
How would you build Video-RAG for a live match that is still being played?advanced
Process the stream in short windows as they arrive: shot detection, ASR and OCR run continuously, and each finished segment is appended to the index (vector indexes support adding items). Scene and game summaries are updated incrementally, for example every few minutes. Questions are answered against the index so far, with recent windows always included because users often ask about what just happened. Latency targets decide model sizes: small captioners and OCR models on the ingest path, a larger VLM only for reading.

Exercises

  1. A 2-hour lecture, \(K = 128\) frames. What is the chance of seeing a slide that is on screen for 20 seconds? How many frames would guarantee it?
  2. In Figure 3, which two fields together answer the most questions? Check your guess, and explain why captions add nothing once ASR, OCR and visual are on.
  3. Compute the RRF score of a segment ranked 1st in one list and 3rd in another, with \(\kappa = 60\). Compare it with a segment ranked 2nd in both.
  4. A match has \(N = 2{,}700\) shots. Find \(S^{*}\) and the minimum cost for \(k = 3\), and compare with flat search.
  5. With the defaults of Figure 5 but a 3-hour video, does the video fit in a 1M-token context at \(r_{\text{lc}} = 100\)? What is \(Q^{*}\)?
  6. Recall@5 is 0.95 and the system answers 0.76 of questions correctly. Estimate the reader accuracy. What would you improve first?
Answerscheck
1. \(T = 7{,}200\) s; \(P = 20 \times 128 / 7{,}200 = 0.356\). To guarantee it: \(K = 7{,}200/20 = 360\) frames. 2. No pair answers more than 4 of 6; four pairs reach 4 (captions + OCR, ASR + OCR, ASR + visual, OCR + visual). ASR + OCR + visual already answers 6 of 6; the two questions that captions answer (medics, first goal) are also answered by visual and by ASR. In a real system captions still help with paraphrases that keyword matching misses. 3. \(1/61 + 1/63 = 0.03227\); \(2/62 = 0.03226\): almost equal, so RRF rewards agreement between lists, not one top rank. 4. \(S^{*} = \sqrt{3 \times 2{,}700} = 90\) scenes; cost \(2 \times 90 = 180\) against 2,700: 15 times fewer. 5. \(10{,}800 \times 100 = 1{,}080{,}000\) tokens: no, just over 1M. \(Q^{*} = 1{,}080{,}000 / (1{,}080{,}000 - 47{,}000) = 1.05\). 6. \(a_{\text{read}} \approx 0.76 / 0.95 = 0.8\). Retrieval is fine; improve the reader (more frames per clip, time-ordered evidence, better prompts or a stronger VLM).

Key takeaways

  • A whole long video costs millions of tokens (210.1), and uniform sampling misses short events: with 64 frames of a 90-minute match, a 3-second goal is seen 3.6% of the time (210.2).
  • Video-RAG retrieves first and reads second: segment and describe offline (Algorithm 1), decompose, search, fuse and read online (Algorithm 2).
  • Captions, ASR, OCR and visual embeddings each catch different questions; fuse their rankings with RRF (210.4) instead of adding raw scores.
  • The Video-RAG paper adds about 2K text tokens per question and reports gains on Video-MME of 1.5 to 4.8 points over subtitles alone for its two headline models.
  • Index at several levels: shots for exact moments, scenes and whole-video summaries for global questions; coarse-to-fine search costs \(2\sqrt{kN}\) at best (210.6).
  • Accuracy is roughly Recall@\(k\) times reader accuracy (210.7): measure both. Video-RAG wins on cost from about two questions per video (210.8), on archives, on short events and when answers must cite times.

Key formulas

\[ P(\text{hit}) = \min\!\left(1, \frac{dK}{T}\right), \quad \text{RRF}(c) = \sum_{m} \frac{1}{\kappa + \text{rank}_m(c)}, \quad S^{*} = \sqrt{kN} \]
\[ P(\text{correct}) \approx \text{Recall@}k \cdot a_{\text{read}}, \qquad Q^{*} = \frac{T\, r_{\text{idx}}}{T\, r_{\text{lc}} - k\ell\, r_{\text{rd}} - c} \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  2. Luo et al. (2024). Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
  3. Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
  4. Izacard et al. (2021). Unsupervised Dense Information Retrieval with Contrastive Learning
  5. Meta (code). FAISS library
  6. Robertson & Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond
  7. Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  8. Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts
  9. Wang et al. (2024). VideoAgent: Long-form Video Understanding with Large Language Model as Agent
  10. Fu et al. (2024). Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
  11. Souček & Lokoč (2020). TransNet V2: An effective deep network architecture for fast shot transition detection
  12. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  13. Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
  14. Jeong et al. (2025). VideoRAG: Retrieval-Augmented Generation over Video Corpus
  15. Ren et al. (2025). VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos
  16. Shen et al. (2023). Aligning and Prompting Everything All at Once for Universal Visual Perception (APE)
  17. JaidedAI (code). EasyOCR
  18. Gemini Team (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
  19. Google (docs). Gemini API: video understanding