LLM & VLM courseBlog
219Track 8: Video and multimodal systems · session 19 of 21

Multimodal RAG

Much of what people ask about lives in charts, scanned pages, photos and videos, not in clean text. Multimodal RAG builds a searchable memory of all of them, finds the right page, image or clip for a text question, and gives the pixels to a VLM to read. This session covers what goes in the index, single vectors against late interaction (ColPali), the token cost of retrieved images, and how to evaluate each modality.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Session 152 (RAG, part 1), session 150 (text embeddings), session 166 (CLIP) and session 210 (Video-RAG)

What you will learn

  • How to choose the unit to index for text, images, document pages and video segments, and the three strategies: everything to text, native embeddings, or hybrid
  • Why scores from different modalities cannot be merged directly (the modality gap), and how per-modality normalization or rank fusion fixes it
  • Late interaction with MaxSim, as in ColBERT and ColPali, and why it finds small regions of busy pages that a single vector misses
  • The storage and compute price of multi-vector indexes (257.5 KiB per page for ColPali) and how token pooling, binary quantization and two-stage search reduce it
  • How many tokens a retrieved page or clip costs a VLM reader, and how to pack evidence into a budget with resolution, cropping and citations
  • How to evaluate retrieval and reading per modality with Recall@k, MRR and nDCG, and why micro averages hide weak modalities

Why this matters

An engineer asks the company assistant: “What was the gross margin in the third quarter?” The answer is in a bar chart on page 14 of a 60-page PDF report. The text around the chart says only “see Figure 3”. A text-only RAG system parses the PDF, gets the words, loses the chart, and answers from the wrong paragraph. Multimodal RAG is retrieval-augmented generation where the memory holds more than text: images, document pages as pictures, tables, charts and video segments. A vision-language model (VLM) then reads what was retrieved, including the pixels.

The problem is real because most company knowledge is not clean text. It is slides, scanned contracts, product photos, manuals with diagrams, dashboards and recorded meetings. In session 152 we built the text pipeline: chunk, embed, search, put the chunks in the prompt. In session 218 we saw how a VLM reads one page. This session joins the two. It answers four questions: what to put in the index for each kind of content, how a text question finds an image or a page, how to feed what was found to a VLM without blowing the token budget, and how to know whether the system works for every kind of content and not only for text.

Real systems work this way. ColPali (Faysse et al., 2024) showed that embedding a picture of the page with a VLM beats a PDF-parsing pipeline on its benchmark, and is about 18 times faster to index [Certain, as reported]. MuRAG (Chen et al., 2022) was an early model that retrieved both images and text passages for open question answering. VisRAG (Yu et al., 2024) and M3DocRAG (Cho et al., 2024) build the full retrieve-then-read loop on page images. For video, session 210 built the same loop on segments. The book (Noyan et al., chapters 8 and 9) and the ColPali paper are the main sources here.

What goes in the index

A text RAG index stores one kind of thing: chunks of text. A multimodal index stores units of several kinds. A unit is the smallest thing we retrieve and later show to the reader. Choosing the unit is the first design decision, and it is different for each kind of content.

ContentUnitHow it is embeddedMetadata to keep
Text (web pages, plain documents)a chunk of 200 to 500 tokensa text encoder (E5, BGE and similar), plus BM25 for exact wordssource, section title, position
Images (photos, diagrams, product shots)one image, or one crop of a large imagean image encoder with a paired text encoder (CLIP, SigLIP); optionally a VLM caption, embedded as textfile, caption, alt text, the page it sits on
Document pages (PDFs, slides, scans)one page, as a picturea VLM retriever on the screenshot: one vector (DSE) or one vector per patch (ColPali); or OCR text plus a text encoderdocument, page number, OCR text for keyword search
Videoa segment (a shot or a 10 to 30 s window)keyframe embeddings (CLIP, SigLIP), plus the caption, speech transcript (ASR) and on-screen text (OCR) as textvideo id, start and end time
Tables and chartsthe table or chart region, or its pageas a page image, or converted to text (a table to markdown, a chart to a short description)the page and the bounding box

There are three ways to make all these units searchable by one question:

  1. Turn everything into text. OCR the pages, caption the images, transcribe the video, then run a normal text RAG. It is simple and uses mature tools. It loses what text cannot say: the shape of a chart, the layout of a form, a photo nobody captioned. Captions can also describe things that are not there.
  2. Embed every unit in its own form into a shared vector space where a text question can be compared with an image directly (CLIP for images, ColPali or DSE for pages). Nothing is lost at indexing time, but each modality now needs its own encoder, and the scores of different encoders are not directly comparable (next section).
  3. Hybrid. Keep both: a visual embedding and the extracted text for each unit, search both, and fuse the results. Most production systems end here, because OCR text is still the best way to find exact names, numbers and part codes. [Likely]
Algorithm 1: build a multimodal index (offline)
  1. input: a collection of documents, images and videos
  2. for each PDF or slide deck: render every page to an image at a fixed DPI // dots per inch; 100 to 150 is common
  3.   page vectors \(\leftarrow\) VLM retriever(page image) // 1 vector (DSE) or about 1,000 (ColPali)
  4.   page text \(\leftarrow\) OCR or the PDF text layer; add it to a BM25 index
  5. for each standalone image: image vector \(\leftarrow\) CLIP or SigLIP image tower; caption \(\leftarrow\) VLM; embed the caption as text
  6. for each video: cut into segments; keyframe vectors, caption, ASR and OCR per segment // as in session 210
  7. for each text chunk: text vector \(\leftarrow\) text encoder; add it to BM25
  8. store every unit with its modality tag and a pointer back to the original pixels (page image, image file, video time range)
  9. build one vector index per encoder // vectors from different encoders do not share a space

Step 8 matters later. At question time we do not show the reader the embedding; we show it the original page or frames. The index must therefore store a pointer to them, and the original pixels must stay available. Figure 1 shows the whole system.

query sideindexed units (keys)what the reader receives (values)
Figure 1. A multimodal RAG system. Top: offline indexing (Algorithm 1), with one index per encoder. Bottom: one question searches every index, the result lists are fused (formula 219.2 or rank fusion), the best units are packed into the VLM's token budget (formula 219.7), and the VLM answers with citations to pages and timestamps.

Cross-modal retrieval: one space, many modalities

Cross-modal retrieval means the query and the result are of different kinds: a text question finds an image, a page or a video segment. It works because of contrastive training (session 166): a text encoder \(E_t\) and an image encoder \(E_v\) are trained together so that a caption and its image land close together. The score between a text query \(q\) and an image \(x\) is the cosine of their vectors:

\[ s(q, x) = \cos\big(E_t(q),\ E_v(x)\big) = \frac{E_t(q)^\top E_v(x)}{\lVert E_t(q)\rVert\ \lVert E_v(x)\rVert} \tag{219.1} \]
Read the symbols
\(E_t, E_v\)
the text encoder and the image encoder of one contrastive model; they share an output space
\(q, x\)
the text query and one indexed image (or page, or keyframe)
\(s(q, x)\)
the similarity score, between \(-1\) and \(1\); higher means a better match

This works well inside one modality pair. The trouble starts when we put text chunks and images in the same result list. In CLIP-style models, images and texts sit in two separate regions of the shared space. This is called the modality gap (Liang et al., 2022) [Certain]. In practice, a text query scores most text chunks higher than even the best-matching image: text-to-text cosines are often around 0.5 to 0.8, while text-to-image cosines of a good match are often only 0.25 to 0.35 [Likely; depends on the model]. If we merge raw scores, images almost never reach the top \(k\).

There are two standard fixes. The first is to search each modality separately and fuse ranks, with reciprocal rank fusion (RRF, Cormack et al., 2009), exactly as in session 210. The second is to normalize scores per modality before merging, for example with a z-score computed over that modality's candidate list:

\[ \tilde{s}(q, x) = \frac{s(q, x) - \mu_m(q)}{\sigma_m(q)}, \qquad m = \text{modality of } x \tag{219.2} \]
Read the symbols
\(\mu_m(q), \sigma_m(q)\)
the mean and standard deviation of the scores of the candidates of modality \(m\) for this query (or fixed values measured on a set of past queries)
\(\tilde{s}\)
the normalized score: “how many standard deviations above the usual score for this modality”

Both fixes remove a constant shift between modalities, so the size of the gap no longer matters. Both have the same weak point: they assume each modality has something relevant. If no image answers the question, the best image still gets a high normalized score and takes a slot in the top \(k\). A per-modality minimum raw score, or a reranker, fixes that. Figure 2 shows the effect.

text chunkimagerelevant
Figure 2. A toy maintenance manual and the query “photo of the cracked pump housing”. Six text chunks and six images. Each image's score is a “gap-free” score minus the gap \(g\); text scores are fixed. The list on the right is the merged top 4 that the reader would see. The scores are made up to show the effect; they are not from a real model.
How to read Figure 2

Start with raw merging and \(g = 0.30\): the top 4 are all text, and the photo the user asked for (I1) is at rank 6. Move \(g\) to 0: with no gap, I1 is second. Now switch to z-score or RRF and move \(g\) again: the ranking does not change at all, and I1 is first, because both methods remove a constant shift per modality. With RRF, I1 and T1 tie at \(1/61\) (both are first in their own list); the tie is broken by name.

Single vectors or late interaction

For document pages, the main choice is how many vectors one page gets. A single-vector retriever (a bi-encoder, as in session 150) compresses the whole page into one vector. DSE (Ma et al., 2024) does this with a VLM on the page screenshot. It is cheap to store and to search with any vector index. But a page is a lot of content: a title, five paragraphs, a small chart, a footnote. One vector has to average all of it, and a small but important region (the chart with the number you want) is diluted by the rest.

Late interaction keeps one vector per piece. ColBERT (Khattab and Zaharia, 2020) did this for text tokens. ColPali does it for image patches: a VLM (PaliGemma-3B) reads the page image, and every one of its 1,024 image patches (a \(32 \times 32\) grid at \(448 \times 448\) pixels) gets an output vector, projected down to \(D = 128\) dimensions [Certain]. The query is also kept as one vector per token. The score is MaxSim: for each query token, find the most similar patch on the page, then add up these maxima:

\[ \text{LI}(q, d) = \sum_{i=1}^{n_q} \max_{j \in \{1, \dots, n_d\}} \ \mathbf{q}_i^\top \mathbf{d}_j \tag{219.3} \]
Read the symbols
\(\mathbf{q}_i\)
the vector of query token \(i\); there are \(n_q\) of them (about 10 to 30 for a question)
\(\mathbf{d}_j\)
the vector of patch \(j\) of page \(d\); there are \(n_d\) of them (1,030 for ColPali: 1,024 patches plus 6 text-prompt tokens)
\(\max_j\)
“which patch of this page matches query token \(i\) best”; the other patches are ignored for that token
\(\text{LI}(q, d)\)
the late-interaction score; with unit vectors it is at most \(n_q\)

The word “late” means the query and the page meet only at the end, in this cheap max-and-sum. The page vectors are still computed offline, once, like a bi-encoder. This is the middle ground between a bi-encoder (meet in one dot product) and a cross-encoder (meet inside the network, which must run for every pair). Because of the max, each query word finds its own region: “revenue” matches the chart title, “2023” matches the axis label, and nothing on the rest of the page can dilute those matches. Figure 3 shows this on a toy page.

Figure 3. Late interaction against one vector per page. Page A is a report page with a title “Revenue”, an axis label “2023” and a small bar chart; the rest is body text. Page B is a policy page about revenue sharing, with two dates. Each patch has a 5-dimensional unit vector over the toy concepts [revenue, 2023, chart, body text, blank]. The query is “revenue 2023 chart”, three token vectors. Cell color is the similarity of each patch to the chosen query token; the outlined cell is the max of formula (219.3).
How to read Figure 3

With a chart of 2 patches, MaxSim gives page A 2.931 (close to the maximum 3) and page B 1.244, so A wins clearly. The single-vector score (mean of all patches, then cosine with the mean query) gives A 0.254 and B 0.321: B wins, because 20 of A's 24 patches are unrelated text. Click the three query tokens to see which patch each one picks. Then make the chart bigger: the single-vector score of A grows and passes B at 4 patches (0.335), while MaxSim does not change at all, because it only looks at the best patch per token.

The price: storage and compute

Late interaction stores many vectors per page. For \(N\) pages with \(n_d\) vectors of \(D\) numbers at \(b\) bytes each, the index needs

\[ M = N \cdot n_d \cdot D \cdot b \ \text{bytes} \tag{219.4} \]

and scoring one query against one page by brute force costs

\[ C_{\text{page}} = n_q \cdot n_d \cdot D \ \text{multiply-adds} \tag{219.5} \]
Worked example: one million pages

ColPali, float16: \(1{,}030 \times 128 \times 2 = 263{,}680\) bytes = 257.5 KiB per page, which is the number the paper reports [Certain]. For \(N = 10^6\) pages: 263.7 GB.

A single vector of 1,536 dimensions, float16: \(1{,}536 \times 2 = 3{,}072\) bytes = 3 KiB per page; 3.1 GB for a million pages. ColPali needs \(263{,}680 / 3{,}072 = 85.8\) times more.

Compute: a 20-token query costs \(20 \times 1{,}030 \times 128 = 2{,}636{,}800\) multiply-adds per page, or \(2.6 \times 10^{12}\) for a million pages. The paper measures about 1 ms per 1,000 pages for the late-interaction step [Certain, as reported], so a brute-force scan of a million pages would take about a second. Fine for 10,000 pages, too slow for a large archive.

Three tricks make late interaction affordable at scale:

  • Token pooling. Merge similar patch vectors of a page (most patches are white space or plain text). The ColPali paper reports that a pool factor of 3 removes 66.7% of the vectors and keeps 97.8% of the retrieval quality [Certain, as reported]. With 344 vectors per page, storage falls to 86 KiB per page.
  • Binary quantization. Keep only the sign of each of the 128 numbers: 16 bytes per vector instead of 256, so \(1{,}030 \times 16 = 16{,}480\) bytes (16.1 KiB) per page. The dot product becomes a fast bit count. Vector databases that support multi-vector search offer this, usually with a full-precision rerank of the top candidates [Likely].
  • Two stages. Use a cheap first stage (a single pooled vector per page, BM25 on OCR text, or an approximate multi-vector index such as ColBERT's PLAID) to get a few hundred candidates, then compute exact MaxSim only on them.
Algorithm 2: two-stage page retrieval with MaxSim reranking
  1. input: query \(q\), page index, candidates \(n_c\) (for example 200), final \(k\) (for example 5)
  2. \(\mathbf{Q} \leftarrow\) query token vectors \((n_q \times D)\) from the retriever's text side
  3. \(C_1 \leftarrow\) top \(n_c\) pages by the mean-pooled page vector // one ANN search, any vector index
  4. \(C_2 \leftarrow\) top \(n_c\) pages by BM25 on the page OCR text // catches exact numbers and codes
  5. for each page \(d\) in \(C_1 \cup C_2\): load its patch vectors \(\mathbf{D}_d\) \((n_d \times D)\)
  6.   \(\text{LI}(q, d) \leftarrow \sum_i \max_j (\mathbf{Q}\mathbf{D}_d^\top)_{ij}\) // formula (219.3)
  7. return the top \(k\) pages by LI, with the arg-max patch of each query token // used later to crop

The cost of stage 2 is \(|C_1 \cup C_2| \cdot n_q n_d D\), independent of the collection size. The price is that a page missed by both first stages cannot be recovered, so measure the recall of the first stage on its own (see the evaluation section).

Feeding retrieved images and frames to a VLM

Retrieval gives us a list of units. The reader is a VLM, and every image we give it costs tokens. Text chunks are cheap (a few hundred tokens). A page image is not. How many tokens it costs depends on the VLM's image encoder; see session 180 for the designs. A common modern design is the one of Qwen2-VL (Wang et al., 2024): \(14 \times 14\)-pixel patches, and every \(2 \times 2\) group of patches merged into one token, so one token covers \(28 \times 28\) pixels. The image is resized so both sides are multiples of 28, within a minimum of 4 and a maximum of 16,384 tokens (the default min_pixels and max_pixels of the released model) [Certain]. For an image of \(H \times W\) pixels:

\[ N_{\text{img}}(H, W) \approx \Big\lfloor \frac{H}{28} \Big\rceil \cdot \Big\lfloor \frac{W}{28} \Big\rceil, \qquad N_{\text{clip}} = \frac{F}{2}\, N_{\text{img}}(H, W) \tag{219.6} \]
Read the symbols
\(\lfloor \cdot \rceil\)
round to the nearest integer (each side is rounded to a multiple of 28 pixels)
\(N_{\text{img}}\)
visual tokens for one image, capped at 16,384 for Qwen2-VL
\(F\)
frames in a video clip; Qwen2-VL merges every 2 consecutive frames into one set of tokens (its temporal patch size is 2) [Certain]

A page rendered at \(r\) DPI (dots per inch) has \(8.27 r \times 11.69 r\) pixels for A4. All retrieved items must fit into the part of the context we give to evidence, the evidence budget \(B\):

\[ \sum_{u \in \text{pages}} N_{\text{img}}(u) \;+\; \sum_{v \in \text{clips}} N_{\text{clip}}(v) \;+\; \sum_{c \in \text{chunks}} N_{\text{text}}(c) \;\le\; B \tag{219.7} \]
Worked example: five A4 pages

At 150 DPI an A4 page is \(1{,}240 \times 1{,}754\) pixels. It is resized to \(1{,}232 \times 1{,}764\) (\(44 \times 63\) tokens): 2,772 tokens. Five pages cost 13,860 tokens. At 100 DPI the page is \(827 \times 1{,}169\), resized to \(840 \times 1{,}176\): 1,260 tokens, so five pages cost 6,300. At 72 DPI: 630 tokens.

But resolution decides what the VLM can read. A 10-point letter is \(10/72\) inch tall, so it is \(10/72 \times r\) pixels tall: 10 px at 72 DPI, 13.9 px at 100 DPI, 20.8 px at 150 DPI. One ViT patch is 14 px. Below about one patch per letter, small print and table digits become hard to read [Likely]. A 16-frame video clip at \(448 \times 448\) costs \(8 \times 16 \times 16 = 2{,}048\) tokens. Five pages at 150 DPI plus one clip: 15,908 tokens, just under a 16,384 budget.

So there is a real trade-off: more pages at low resolution (better recall: the answer is more likely among them) or fewer pages at high resolution (better reading). Two tricks help. Cropping: late interaction tells us which patches matched the query (the arg-max in formula 219.3), so we can send only the region around them, for example a quarter page at 150 DPI for 682 tokens instead of 2,772. Mixing: send the top 1 or 2 pages at high resolution and the next few at low resolution or as OCR text. Figure 4 lets you try budgets.

page tokensvideo tokensover budget
Figure 4. The evidence budget of formula (219.7) for A4 pages and video clips, with the Qwen2-VL token rule of formula (219.6). The letter height is for 10-point text. The budget choices are 8,192, 16,384, 32,768, 65,536 and 131,072 tokens.
How to read Figure 4

The defaults (5 pages at 150 DPI, one clip, 16,384 budget) use 15,908 tokens: it just fits. Add a sixth page and it does not. Lower the DPI to 100: each page drops to 1,260 tokens and 11 pages would fit, but 10-point letters are now only 13.9 px tall, about one patch. Turn on cropping at 150 DPI: each crop is 682 tokens, so 10 crops fit at full sharpness. That is why the arg-max patches of late interaction are worth keeping.

Packing the prompt

How the evidence is laid out matters as much as how much there is. The reader must know where each item came from, so it can cite it, and items should not be shuffled in a way that breaks their meaning.

Algorithm 3: pack retrieved units into a VLM prompt
  1. input: ranked units \(U\) (pages, images, clips, chunks), budget \(B\), high and low resolution \(r_{\text{hi}}, r_{\text{lo}}\)
  2. used \(\leftarrow\) 0; prompt \(\leftarrow\) [system instruction: “answer only from the evidence; cite [doc, page] or [video, time]”]
  3. for rank \(i\), unit \(u\) in \(U\):
  4.   \(r \leftarrow r_{\text{hi}}\) if \(i \le 2\) else \(r_{\text{lo}}\) // sharp top pages, cheaper tail
  5.   \(x \leftarrow\) crop of \(u\) around its matched patches, if the crop is enough; else the whole unit at \(r\)
  6.   cost \(\leftarrow N(x)\) by formula (219.6); if used + cost \(\gt B\): add \(u\)'s OCR text instead if it fits, else skip
  7.   append a text label “[report.pdf, page 14]” then \(x\); used \(\leftarrow\) used + cost
  8. keep pages of one document in page order and clips in time order // “the next page”, “before”, “after”
  9. append the question; return prompt

Evaluation across modalities

A multimodal RAG system can fail in two places: retrieval does not bring the right unit, or the reader does not use it well. As in session 210, measure both, and measure them per modality. The usual retrieval metrics, for one query with a set of relevant units \(\mathcal{R}\) and a ranked result list:

\[ \text{Recall@}k = \frac{|\mathcal{R} \cap \text{top}_k|}{|\mathcal{R}|}, \qquad \text{MRR} = \frac{1}{\text{rank of the first relevant unit}} \tag{219.8} \]
\[ \text{nDCG@}k = \frac{\text{DCG@}k}{\text{IDCG@}k}, \qquad \text{DCG@}k = \sum_{i=1}^{k} \frac{\text{rel}_i}{\log_2(i + 1)} \tag{219.9} \]
Read the symbols
\(\text{top}_k\)
the first \(k\) results
\(\text{rel}_i\)
the relevance grade of the result at rank \(i\) (0 = not relevant; 1 or 2 = relevant); ViDoRe uses one relevant page per query
\(\text{IDCG@}k\)
the DCG of the best possible order, so nDCG is between 0 and 1
\(\log_2(i+1)\)
the discount: a hit at rank 1 counts fully, at rank 3 it counts half
Worked example: nDCG@5

The top 5 results have grades \([0, 2, 0, 1, 0]\). \(\text{DCG} = 2/\log_2 3 + 1/\log_2 5 = 1.2619 + 0.4307 = 1.6925\). The best order is \([2, 1, 0, 0, 0]\): \(\text{IDCG} = 2/1 + 1/\log_2 3 = 2.6309\). So \(\text{nDCG@5} = 0.6433\). Recall@5 is 1 (both relevant units are in the top 5), and MRR is \(1/2\).

The end-to-end accuracy then splits by modality. If a fraction \(w_m\) of the questions need evidence of modality \(m\) (text, table, chart, photo, video), and retrieval and reading have recall \(R_m\) and reader accuracy \(a_m\) for those questions,

\[ \text{Acc} \approx \sum_{m} w_m \, R_m \, a_m \tag{219.10} \]

This one line explains the most common evaluation trap. If the test set is 80% text questions, a system that fails on every chart still scores high overall. Report each modality on its own, and report the macro average (mean over modalities) next to the micro average (mean over questions). Figure 5 shows a toy example.

Figure 5. A toy test set of 12 questions: 4 answered by text, 3 by a table, 3 by a chart and 2 by a photo. Each cell shows the rank of the first relevant unit for one question (“–” = not in the top 10). The bars on the right are Recall@\(k\) per evidence type. These ranks are invented to illustrate the metrics; they are not benchmark results.
How to read Figure 5

For the OCR + text system at \(k = 5\), text questions are perfect (Recall 1.0), but chart and photo questions are at 0. The micro average over the 12 questions is 0.500 and the macro average over the four types is 0.417. The page-image retriever misses one text question (a dense page of small print) but finds every chart: micro 0.833. The hybrid gets 0.917. Set \(k = 1\): the hybrid gets 7 of 12 at rank 1, the others 4 and 5.

What to measure, per modality

StageMetricNotes
Retrieval, pages and imagesRecall@\(k\), nDCG@5 or @10, MRRViDoRe reports nDCG@5 per dataset; label the relevant page per question
Retrieval, video segmentsRecall@\(k\) where a hit means temporal overlap (IoU above a threshold) with the labeled spansee session 213
First stage of a two-stage retrieverRecall@\(n_c\) (for example 200)an upper bound for everything after it
Reader, given the true evidenceexact match, ANLS for document answers, or a judge modelANLS (from DocVQA) gives partial credit for near-correct strings and 0 when the edit distance is too large [Certain]
Groundingcitation accuracy: does the cited page or time contain the answer?a correct answer with a wrong citation is a failure in most products
Costtokens per question, latency, index sizeformulas (219.4) to (219.7)

Public test sets help for a first comparison: ViDoRe for page retrieval (academic sets built from DocVQA, InfoVQA, TAT-DQA, arXivQA and TabFQuAD, plus practical sets on energy, government, healthcare, AI and the Shift Project) [Certain]; M3DocVQA with more than 3,000 PDFs and 40,000 pages for open-domain document QA [Certain]; MMLongBench-Doc and MP-DocVQA for long and multi-page documents; WebQA and MultimodalQA for mixed text and images [Likely]. Your own data will differ. A few hundred labeled questions from real users, tagged by evidence type, are worth more than any public score.

Code: late interaction, budgets and metrics in PyTorch

The code builds the two toy pages of Figure 3, scores them with MaxSim and with single vectors, runs batched MaxSim over 1,000 random pages of realistic size, and computes the storage, token and nDCG numbers used above. It needs only PyTorch and is deterministic.

import math, torch
import torch.nn.functional as F
torch.manual_seed(0)

# 1) toy multi-vector pages: 5 concept axes [revenue, 2023, chart, body text, blank]
def u(*v): return F.normalize(torch.tensor(v, dtype=torch.float32), dim=-1)
TITLE, YEAR, BAR = u(.85, .2, .1, .35, .1), u(.1, .9, .25, .1, .15), u(.1, .1, .95, 0, .2)
BODY, BLANK      = u(.05, 0, 0, 1, .1), u(0, 0, 0, .1, 1)
POLICY, DATE     = u(.4, .05, 0, .9, .05), u(.1, .6, 0, .8, .1)
page_a = torch.stack([TITLE, YEAR] + [BAR] * 2 + [BODY] * 20)    # report page, small chart
page_b = torch.stack([POLICY] * 14 + [DATE] * 2 + [BLANK] * 8)   # revenue-sharing policy text
pages = torch.stack([page_a, page_b])                            # (N=2, n_d=24, D=5)
query = torch.stack([u(1, .1, .15, .2, 0), u(.1, 1, .15, 0, 0), u(.15, .1, 1, 0, .05)])  # "revenue", "2023", "chart"

# 2) late interaction (MaxSim), formula (219.3): for each query token, best patch; then sum
def maxsim(q, P):                             # q: (n_q, D), P: (N, n_d, D)
    sim = torch.einsum("qd,npd->nqp", q, P)     # (N, n_q, n_d) all dot products
    return sim.max(dim=2).values.sum(dim=1)    # (N,)

# 3) single vector: mean-pool patches and query tokens, then cosine
def single(q, P):
    return F.normalize(P.mean(1), dim=-1) @ F.normalize(q.mean(0), dim=-1)

print("MaxSim  :", [round(x, 4) for x in maxsim(query, pages).tolist()])
print("single  :", [round(x, 4) for x in single(query, pages).tolist()])

# 4) the same scoring at a realistic size: 1,000 random pages of 1,030 vectors, D = 128
P = F.normalize(torch.randn(1000, 1030, 128), dim=-1).half().float()   # stored as float16
q = F.normalize(torch.randn(20, 128), dim=-1)
s = maxsim(q, P)
print("scores:", tuple(s.shape), "multiply-adds per page:", 20 * 1030 * 128)   # formula (219.5)

# 5) storage per page, formula (219.4)
for name, n, D, nbytes in [("single fp16", 1, 1536, 2), ("ColPali fp16", 1030, 128, 2), ("ColPali binary", 1030, 128, 1 / 8)]:
    print(f"{name:15s} {n * D * nbytes / 1024:8.2f} KiB per page")

# 6) visual tokens of a page for a Qwen2-VL-style encoder (28 x 28 pixels per token), formula (219.6)
def page_tokens(h, w, f=28, max_tok=16384):
    hb, wb = max(f, round(h / f) * f), max(f, round(w / f) * f)
    if hb * wb > max_tok * f * f:                     # too big: shrink, keep the aspect ratio
        beta = math.sqrt(h * w / (max_tok * f * f))
        hb, wb = math.floor(h / beta / f) * f, math.floor(w / beta / f) * f
    return (hb // f) * (wb // f)
for dpi in (72, 100, 150, 200):
    print("A4 at", dpi, "dpi:", page_tokens(round(11.69 * dpi), round(8.27 * dpi)), "tokens")

# 7) nDCG@k, formula (219.9), with graded relevance
def ndcg(rels, k):
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(rels[:k]))
    ideal = sorted(rels, reverse=True)
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal[:k]))
    return dcg / idcg if idcg > 0 else 0.0
print("nDCG@5:", round(ndcg([0, 2, 0, 1, 0], 5), 4))

Output: MaxSim gives [2.9311, 1.2443] and the single vector gives [0.2543, 0.3212], the same as Figure 3: the two methods disagree about which page is better. The batched scorer returns 1,000 scores at 2,636,800 multiply-adds per page. Storage is 3.00, 257.50 and 16.09 KiB per page. A4 pages cost 630, 1,260, 2,772 and 4,956 tokens at 72, 100, 150 and 200 DPI. The nDCG@5 is 0.6433, as in the worked example. In a real system, query and pages come from a ColPali-style model (for example through the colpali-engine package), and the pages are stored in a multi-vector index.

Where this is used

ApplicationWhat is indexedTypical question
Financial and annual reportspage images (charts, tables) with late interaction, OCR text for numbers“What was the gross margin in Q3?”, answered from a chart, with the page cited
Technical manuals and maintenancepages with diagrams, part photos, part numbers in BM25“Which bolt holds the pump housing?”, answered with the exploded-view diagram
Slide decks and internal wikisone unit per slide as an image, plus speaker notes as text“Which deck showed the 2023 roadmap?”
E-commerce and product catalogsproduct photos (SigLIP), titles and specs as text“A blue backpack with a laptop pocket”, a mixed text and image query
Insurance claims and formsscanned forms as page images, photos of the damage“Is the dent in the photo on the panel listed in the claim?”
Lecture and meeting videovideo segments with ASR, slide OCR and keyframes, next to the slide PDFs“Where does the teacher explain this formula?”, answered with a timestamp and the slide page (Video-RAG, session 210)
Sports and media archivesshots with keyframe embeddings, scoreboard OCR, commentary; match reports as documents“The corner that led to the second goal”: text-to-video retrieval (session 208) plus key-moment detection
Scientific literaturepaper pages with figures and equations as images“Which paper has a plot of loss against compute with a kink?”

In session 221 you build one of these systems end to end and measure it with the metrics above. session 220 extends the idea: the model decides itself when to search again, which page to zoom into, and when it has enough evidence.

Common mistakes

  • Merging raw scores from different modalities. The modality gap pushes all images below all text (Figure 2). Search per modality and fuse ranks, or normalize scores per modality.
  • One vector index for several encoders. A query vector from the text encoder is compared with vectors from an unrelated image encoder. The scores are meaningless.
  • Feeding the retriever's low-resolution image to the reader. ColPali matches on \(448 \times 448\) pixels; the reader needs the page at 100 to 150 DPI to read numbers. Keep the source and render again.
  • Ignoring the token cost of images. Ten pages at 150 DPI are 27,720 tokens with a Qwen2-VL-style encoder. Set an evidence budget, crop, and mix resolutions (Algorithm 3).
  • Dropping OCR text after moving to page images. Exact strings (part numbers, IDs, names) are still found best by BM25. Keep a hybrid index.
  • Losing the pointer back to the source. Without the document, page and time of each unit, the reader cannot cite, and users cannot check answers.
  • Only reporting an overall score. A test set full of text questions hides a system that never finds charts. Report per evidence type and the macro average (Figure 5).
  • Underestimating index size. 257.5 KiB per page is 264 GB per million pages for ColPali in float16. Plan token pooling, binary quantization or two stages before you index.

Interview questions

What is multimodal RAG, and why not just OCR everything into text?basic
Multimodal RAG retrieves units of several kinds (text chunks, images, page images, video segments) and lets a VLM read the retrieved pixels as well as text. OCR-to-text is a valid baseline and still the best way to find exact strings, but it loses charts, diagrams, layout and photos, it is slow (the ColPali paper reports 7.22 s per page for a parsing pipeline against 0.39 s for page embedding), and captions can invent things. On ViDoRe the ColPali paper reports 81.3 nDCG@5 for page-image retrieval against 65.5 for OCR plus BM25. In practice you keep both and fuse them.
Explain late interaction and the MaxSim score.basic
Both the query and the document are kept as sets of vectors: one per query token, one per document token or image patch. The score is \(\sum_i \max_j \mathbf{q}_i^\top \mathbf{d}_j\): each query token picks its best-matching patch, and the maxima are added. Document vectors are computed offline like a bi-encoder, so it is cheap at query time, but each query word can match its own region, which a single averaged vector cannot. It sits between a bi-encoder (one dot product) and a cross-encoder (full joint attention per pair).
You put text chunks and CLIP image embeddings into one result list, and images never show up. Why, and how do you fix it?core
The modality gap: in contrastive models, image and text embeddings occupy different regions, so text-to-text cosines are systematically higher than text-to-image cosines, even for the best image. Raw-score merging then puts every text chunk above every image. Fixes: search each modality separately and fuse with reciprocal rank fusion; or z-normalize scores per modality (formula 219.2) using statistics from that query's candidates or from past queries; or rerank the merged candidates with one model that sees both (a VLM reranker). Add a minimum raw score per modality so an irrelevant image does not get a slot just because it is the best of a bad set.
How much storage does ColPali need for a million pages, and how do you reduce it?core
1,030 vectors of 128 dimensions in float16: 263,680 bytes, 257.5 KiB per page, about 264 GB per million pages, roughly 86 times a single 1,536-dimensional float16 vector. Reductions: token pooling (pool factor 3 removes 66.7% of vectors and keeps 97.8% of quality, as reported), binary quantization (16 bytes per vector, 16.1 KiB per page, with a full-precision rerank of the top candidates), smaller projection dimensions, and two-stage search so only the candidates' vectors must be in fast memory.
How many tokens does a retrieved page cost the reader, and how do you fit the top 10 into the prompt?core
It depends on the reader's image encoder. With Qwen2-VL (one token per \(28 \times 28\) pixels), an A4 page at 150 DPI is resized to \(1{,}232 \times 1{,}764\), \(44 \times 63\) = 2,772 tokens; 10 pages are 27,720 tokens. Options: lower DPI for lower-ranked pages (1,260 tokens at 100 DPI), crop around the patches that matched (a quarter page is 682 tokens), replace low-ranked pages by their OCR text, and set a hard evidence budget (formula 219.7). Watch readability: a 10-point letter is 10/72 × DPI pixels tall, so below about 100 DPI it is smaller than one 14-pixel patch.
Design multimodal RAG for 100,000 PDF reports full of charts and tables.core
Say 50 pages each: 5 million pages. Offline: render pages at about 150 DPI and keep them; compute ColPali-style patch vectors with token pooling and binary quantization (about 5M × 16 KiB = 82 GB binary, or less with pooling); extract the text layer or OCR into BM25; store document, page and bounding boxes. Online: first stage from BM25 plus a pooled single vector (a few hundred candidates); MaxSim rerank on the candidates; fuse with RRF; pack the top 3 to 5 pages into the VLM with crops and page labels; require citations. Evaluate with labeled questions tagged text, table, chart; measure first-stage recall, reranked nDCG@5, reader accuracy given the true page, and citation accuracy.
When does a single vector per page fail, and when is it good enough?advanced
It fails when the evidence is a small region of a busy page: the mean of all patch vectors is dominated by unrelated content, so a page that is “about” the query words in general can win over the page that has the exact chart (Figure 3: with a 2-patch chart the single vector prefers the wrong page, 0.321 against 0.254, while MaxSim prefers the right one, 2.931 against 1.244). It is good enough when pages or images are about one thing (product photos, slides with one message), when storage or latency is tight, or as the first stage before a MaxSim rerank. On ViDoRe the paper reports 58.8 nDCG@5 for the single-vector BiPali against 81.3 for ColPali.
How do you evaluate a multimodal RAG system so that weak modalities are not hidden?advanced
Build a labeled set where each question has its evidence unit (page, image, time span) and an evidence-type tag. Report retrieval (Recall@k, nDCG@5, MRR) and reader accuracy given the true evidence separately, per type, with micro and macro averages. Accuracy is approximately \(\sum_m w_m R_m a_m\) (formula 219.10), so a test set with mostly text questions hides a system that fails on charts. Add citation accuracy and cost. For video, count a retrieval hit by temporal IoU with the labeled span. For free-form answers, use ANLS or exact match where possible and a judge model with spot checks where not.
Two-stage retrieval: what do you gain and what can go wrong?advanced
Stage 1 (pooled single vectors, BM25, or an approximate multi-vector index) finds \(n_c\) candidates in time that grows slowly with the collection. Stage 2 computes exact MaxSim only on them: cost \(n_c \cdot n_q n_d D\), for example \(200 \times 2.6\)M = 527M multiply-adds, independent of collection size. Only candidate vectors need to be loaded. The risk: anything missed by stage 1 is lost forever, so measure Recall@\(n_c\) of stage 1 alone, use two different first stages (dense and BM25) to cover each other's misses, and raise \(n_c\) if recall is low.
How do you put video segments into the same system as documents?core
Treat a segment (a shot or a 10 to 30 s window) as a unit with its own indexes: keyframe embeddings from an image encoder, and caption, ASR and OCR text in the text indexes. Search per index, fuse ranks with the document results, and keep time ranges as metadata. In the prompt, a clip costs \(F/2 \times\) tokens per frame with Qwen2-VL (16 frames at \(448 \times 448\): 2,048 tokens), so it competes with pages for the same budget. Label frames with timestamps so the reader can cite times. Session 210 covers the video side in depth.
The reader gives a right answer but cites the wrong page. Why does this happen and how do you reduce it?advanced
Causes: page labels are far from the images in the prompt or missing; several retrieved pages contain similar content (a number repeated in a summary table); the model answers from its own knowledge and then attaches a plausible citation. Fixes: put a short text label right before each image, keep documents in page order, ask for the citation first and the answer second, check citations automatically (does the cited page's OCR text or crop contain the answer string?), and train or prompt the reader to say “not found” when no page supports the answer. Track citation accuracy as its own metric.

Exercises

  1. How much storage do 10 million pages need with ColPali in float16, with token pooling (factor 3, 344 vectors per page) and with binary quantization?
  2. How many Qwen2-VL tokens does a US Letter page (\(8.5 \times 11\) inches) cost at 150 DPI?
  3. A query has 2 tokens and a page has 3 patches. The dot products are \(\begin{pmatrix} 0.2 & 0.8 & 0.1 \\ 0.5 & 0.3 & 0.6 \end{pmatrix}\) (rows = query tokens). Compute the MaxSim score.
  4. Binary relevance, the top 3 results are \([1, 0, 1]\), and there are exactly 2 relevant pages. Compute nDCG@3.
  5. The evidence budget is 32,768 tokens. You must include 2 video clips of 16 frames at \(448 \times 448\). How many A4 pages at 150 DPI fit?
  6. 60% of questions need text (Recall 0.95, reader 0.9) and 40% need a chart (Recall 0.6, reader 0.7). Estimate the overall accuracy with formula (219.10). Which part would you improve first?
Answerscheck
1. \(10^7 \times 263{,}680 = 2.64\) TB; pooled: \(344 \times 128 \times 2 = 88{,}064\) bytes per page, 881 GB; binary: \(1{,}030 \times 16 = 16{,}480\) bytes per page, 165 GB. 2. \(1{,}275 \times 1{,}650\) pixels; \(1{,}275/28 = 45.5 \to 46\), \(1{,}650/28 = 58.9 \to 59\); \(46 \times 59 = 2{,}714\) tokens. 3. Row maxima 0.8 and 0.6; MaxSim = 1.4. 4. DCG = \(1 + 1/\log_2 4 = 1.5\); IDCG = \(1 + 1/\log_2 3 = 1.6309\); nDCG@3 = 0.9197. 5. Clips: \(2 \times 2{,}048 = 4{,}096\); left: 28,672; \(28{,}672 / 2{,}772 = 10.3\), so 10 pages. 6. \(0.6 \times 0.95 \times 0.9 + 0.4 \times 0.6 \times 0.7 = 0.513 + 0.168 = 0.681\). Chart retrieval: raising chart recall from 0.6 to 0.9 adds \(0.4 \times 0.3 \times 0.7 = 0.084\), almost as much as making text retrieval and reading perfect would add (\(0.6 \times (1 - 0.855) = 0.087\)), and it is far more realistic to get.

Key takeaways

  • A multimodal index stores units of several kinds (chunks, images, page images, video segments), one index per encoder, each with a pointer back to the original pixels.
  • Cross-modal scores are not comparable across modalities because of the modality gap; search per modality and fuse ranks, or normalize per modality (219.2).
  • Late interaction (MaxSim, 219.3) lets each query token find its own patch, so small regions of busy pages are not diluted; ColPali reports 81.3 nDCG@5 on ViDoRe against 58.8 for its single-vector version.
  • The price is storage, 257.5 KiB per page in float16 (219.4); token pooling, binary quantization and two-stage search make it affordable.
  • Every retrieved image costs the reader tokens: 2,772 for an A4 page at 150 DPI with a Qwen2-VL-style encoder (219.6). Budget, crop and mix resolutions (219.7).
  • Evaluate retrieval and reading separately, per evidence type, with micro and macro averages; overall accuracy is about \(\sum_m w_m R_m a_m\) (219.10).

Key formulas

\[ \text{LI}(q, d) = \sum_{i} \max_{j} \mathbf{q}_i^\top \mathbf{d}_j, \qquad M = N\, n_d\, D\, b, \qquad \tilde{s} = \frac{s - \mu_m}{\sigma_m} \]
\[ N_{\text{img}} \approx \Big\lfloor \tfrac{H}{28} \Big\rceil \Big\lfloor \tfrac{W}{28} \Big\rceil, \qquad \text{nDCG@}k = \frac{\sum_{i \le k} \text{rel}_i / \log_2(i+1)}{\text{IDCG@}k}, \qquad \text{Acc} \approx \sum_m w_m R_m a_m \]

References for this session

  1. Faysse et al. (2024). ColPali: Efficient Document Retrieval with Vision Language Models
  2. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 8: Document AI
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
  4. Khattab & Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
  5. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  6. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  7. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  8. Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images
  9. Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  10. Robertson & Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond
  11. Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
  12. Liang et al. (2022). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
  13. Ma et al. (2024). Unifying Multimodal Retrieval via Document Screenshot Embedding
  14. Yu et al. (2024). VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
  15. Cho et al. (2024). M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
  16. Chen et al. (2022). MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
  17. Santhanam et al. (2022). PLAID: An Efficient Engine for Late Interaction Retrieval