LLM & VLM courseBlog
221Track 8: Video and multimodal systems · session 21 of 21

multimodal RAG over PDFs

In this lab you build question answering over PDF files twice: once from the text layer of each page, and once from a picture of each page, embedded ColPali-style. A vision-language model reads the retrieved pages and cites them. Then you measure, per question type, why text-only RAG misses charts and scans, why tables are mostly a reading problem, and what page resolution costs in tokens.

Time: 3 to 4 hours (mostly hands-on; the real version needs a GPU)Level: Intermediate · hands-on labBefore this: Session 219 (Multimodal RAG), session 152 (RAG, part 1), session 218 (Document AI), Python and PyTorch

What you will learn

  • How to render PDF pages at a chosen DPI and read their text layer, and why charts stored as pictures and scanned pages have no text layer
  • How to build a BM25 index on page text and a late-interaction (MaxSim) index on page images, with a page pointer on every unit
  • How reciprocal rank fusion combines the two lists, and the case where it pushes down a page that only one index can see
  • How to give retrieved pages to a VLM such as Qwen2.5-VL with labels and citations, and what each page costs in visual tokens
  • How to score retrieval recall and answer accuracy separately for text, table and chart questions, and read the gap between them
  • The real pipeline with pypdfium2, colpali-engine (ColQwen2) and Qwen2.5-VL, next to a toy version that runs anywhere

Why this matters

In this lab you build a small but complete multimodal RAG system over PDF files. You index every page twice: once as text (the text layer of the PDF) and once as an image (a picture of the page, embedded ColPali-style). A vision-language model (VLM) then answers questions from the retrieved pages and cites the page it used. Finally you measure which pipeline wins on plain text, on tables and on charts, and why.

Why build it yourself? Because the most common failure of document assistants is invisible from the outside. The system answers, the answer sounds good, and it is wrong because the number was in a chart that the text pipeline never saw. Two simple facts cause most of the trouble:

  • A PDF's text layer is not the page. Charts that are stored as pictures, scanned pages and text inside figures have no text layer. You will make a real two-page PDF in Step 1 and see that its chart page gives only the words "Quarterly results" and "Figure 2. See chart.", without a single number from the chart.
  • Retrieval and reading fail in different places. For tables, text retrieval usually finds the right page, but a text reader can misread a flattened table. For charts, text retrieval often does not find the page at all. You only see this if you score the two stages apart and per question type.

Session 219 explained the ideas: units, late interaction (MaxSim), token budgets and evaluation per modality. Session 218 showed how a VLM reads one page. This lab turns them into code and numbers. You will (1) render pages and read the text layer, (2) build a BM25 text index, (3) build a page-image index with late interaction, (4) retrieve with text, images and a fusion of both, (5) answer with a VLM and citations, and (6) compare the pipelines per question type.

Setup

There are two versions of this lab. Both use the same steps.

  • The toy version runs anywhere. It uses a 10-page toy annual report written as word lists, a real BM25, a toy late-interaction scorer and a rule-based reader. It needs only PyTorch, runs in under a second, and every figure on this page runs the same code in your browser. Its numbers show the mechanisms. They are not benchmark results.
  • The real version needs a GPU. It renders real PDFs with pypdfium2, embeds pages with ColQwen2 (a ColPali-style retriever) from the colpali-engine package, and answers with Qwen2.5-VL. A GPU with 24 GB of memory is enough if you load the retriever, index, free it, and then load the reader [Likely].
pip install torch pypdfium2 pillow rank_bm25          # toy version + Step 1
pip install colpali-engine transformers accelerate qwen-vl-utils   # real version

For real documents, use your own PDFs (annual reports, manuals, slide decks), or one of the ViDoRe test sets that the ColPali authors published on Hugging Face. Write 30 to 100 questions yourself and tag each one with its gold page (the page that holds the answer) and its type: text, table or chart. This small labeled set is the most valuable part of the lab.

Our toy report, “Northwind Tools, Annual Report 2023”, has 10 pages chosen to contain every hard case: two text pages, a scanned auditor’s letter with no text layer, a page of small print, three tables (one with a two-level header), and three charts that are stored as pictures. There are 12 questions, 4 of each type. Figure 1 shows the system you are building.

query sideindexeswhat the reader receives
Figure 1. The lab pipeline. Offline (top): every PDF page becomes an image (Step 1) and a text string (Step 1); text goes into a BM25 index (Step 2), the image into a late-interaction index (Step 3). Online (bottom): the question searches both indexes, the two ranked lists are fused (Step 4), and the VLM reads the top \(k\) pages at a chosen resolution and cites them (Step 5). Step 6 scores every stage.

Step 1: render pages and read the text layer

A PDF page has a size in points (1 point = 1/72 inch). An A4 page is \(595.3 \times 841.9\) points. To get an image, we render (draw) the page at some resolution \(r\) in dots per inch (DPI). Each side then has

\[ W_{\text{px}} = \Big\lfloor w_{\text{pt}} \cdot \frac{r}{72} \Big\rceil, \qquad H_{\text{px}} = \Big\lfloor h_{\text{pt}} \cdot \frac{r}{72} \Big\rceil \tag{221.1} \]
Read the symbols
\(w_{\text{pt}}, h_{\text{pt}}\)
the page width and height in points, from the PDF itself
\(r\)
the render resolution in DPI; pypdfium2 calls \(r/72\) the scale, so scale=1 means 72 DPI [Certain]
\(\lfloor\cdot\rceil\)
round to the nearest whole pixel

So A4 at 144 DPI (scale=2) is \(1{,}191 \times 1{,}684\) pixels. The same library also gives the text layer: the characters that the program which made the PDF wrote into the file, with their positions. Born-digital PDFs (exported from Word, LaTeX or a reporting tool) have a good text layer. Scanned PDFs have none, unless someone ran OCR (optical character recognition) on them. Charts and photos inserted as pictures have no text layer either.

Let us check this on a real file. The script below builds a two-page A4 PDF with reportlab: page 1 has a title and a small revenue table written as text; page 2 has a title, a bar chart inserted as a PNG picture (with the values 38, 41, 40 and 44 drawn on the bars), and a caption. Then it renders each page and prints the text layer.

import pypdfium2 as pdfium

pdf = pdfium.PdfDocument("report.pdf")
for i in range(len(pdf)):
    page = pdf[i]
    w_pt, h_pt = page.get_size()                        # points, 1/72 inch
    img = page.render(scale=144 / 72).to_pil()           # formula (221.1): 144 DPI
    img.save(f"page_{i+1}.png")
    text = page.get_textpage().get_text_range().replace("\r\n", "\n")
    print(i + 1, (round(w_pt, 1), round(h_pt, 1)), img.size, repr(text))

Output (we ran it):

1 (595.3, 841.9) (1191, 1684) 'Revenue by region\nRegion 2022 2023\nNorth 41 44\nSouth 28 31\nEast 19 24\nWest 12 15'
2 (595.3, 841.9) (1191, 1684) 'Quarterly results\nFigure 2. See chart.'

Page 1 is fine: the table comes back as text, row by row (the column structure is gone, but the numbers are there). Page 2 is the problem: the text layer has none of the four margin values and not even the chart title, because they are pixels inside a picture. A text-only RAG system cannot find this page for “gross margin in Q4”, and even if it did, it could not read the answer.

Figure 2 shows the same thing for all 10 pages of the toy report: on the left, what the page shows; on the right, what the text layer gives. Words that you can see on the page but that are missing from the text layer are marked in red.

What the page shows (pixels)

What the text layer gives

Figure 2. The 10 pages of the toy annual report. The drawing on the left lists the words a reader can see; the box on the right is the page’s text layer, followed by the visible words it is missing (red, dashed). The page texts are the same word lists that the code in this lab uses.
How to read Figure 2

Start with page p1 (a letter): 0 words are missing; text-only RAG is fine here. Go to p4, the line chart: the text layer has 5 words (“financial highlights see figure 1”), and 12 visible words are missing, including “revenue” and all five years. Page p9 is a scanned letter: the text layer is empty and all 13 visible words are missing, including the auditor’s name. Page p6 is a table: no word is missing, but look at how the two-level header turns into a flat list of numbers. That is a reading problem, not a retrieval problem.

Algorithm 1: ingest a folder of PDFs
  1. input: PDF files, retriever resolution \(r_{\text{idx}}\) (for example 144 DPI), reader resolution \(r_{\text{read}}\)
  2. for each PDF, for each page \(i\):
  3.   image \(\leftarrow\) render(page, scale \(= r_{\text{idx}}/72\)); save it as doc_i.png // formula (221.1)
  4.   text \(\leftarrow\) the text layer of the page
  5.   if text has fewer than ~20 characters but the image is not blank: mark the page “needs OCR” // scans, picture-only slides
  6.   store a record {doc, page \(i\), image path, text, page size in points}
  7. return the list of page records // every later step points back to (doc, page)

The last line is the important one. Every index you build later stores only a page ID. At question time you go back to the record, open the original PDF, and render the page again at the reader’s resolution. You never show the reader the retriever’s small image.

Step 2: the text index (BM25 on the text layer)

The text pipeline treats each page’s text layer as one document. (In a long-text system you would cut pages into chunks; one page per unit keeps the comparison with page images fair, because both pipelines then return pages.) We score with BM25, the classic keyword ranking function used by search engines such as Lucene and Elasticsearch. For a query with words \(t\) and a page \(d\):

\[ \text{BM25}(q, d) = \sum_{t \in q} \text{idf}(t) \cdot \frac{f(t,d)\,(k_1 + 1)}{f(t,d) + k_1 \big(1 - b + b\, \tfrac{|d|}{\text{avgdl}}\big)}, \qquad \text{idf}(t) = \ln\!\Big(1 + \frac{N - n(t) + 0.5}{n(t) + 0.5}\Big) \tag{221.2} \]
Read the symbols
\(f(t,d)\)
how many times word \(t\) appears in page \(d\)’s text layer
\(|d|, \text{avgdl}\)
the length of page \(d\) in words, and the average length over all pages
\(N, n(t)\)
the number of pages, and the number of pages that contain \(t\); rare words get a large idf (inverse document frequency)
\(k_1, b\)
two constants: \(k_1 = 1.2\) limits how much repeating a word helps, \(b = 0.75\) controls how much long pages are penalized
Worked example: BM25 picks the wrong page for a chart question

Question q10: “What was the gross margin in Q4?”, query words gross margin q4. The answer is on p5, the bar chart. But p5’s text layer is only “quarterly results figure 2”, so p5 scores 0. The words “gross” and “margin” are in no text layer at all. Only “q4” matches anything: the cost table p6 (27 words; average page length 17.3 words) has it once.

\(\text{idf}(\text{q4}) = \ln(1 + 9.5/1.5) = 1.9924\). The length part is \(1 + 1.2 \times (0.25 + 0.75 \times 27/17.3) = 2.7046\), so the score is \(1.9924 \times 2.2 / 2.7046 = 1.621\). BM25 returns exactly one page, p6, with score 1.621. The gold page p5 is not in the list at all. No reader can recover from this.

BM25 has real strengths that we will see in the results: it matches exact strings such as codes, part numbers and names, it needs no GPU, and it explains itself (you can see which word matched). In the toy, the only question where text search beats the image index is q4, which asks for an exact policy code in small print.

In real code you can use rank_bm25: build BM25Okapi(pages) from the tokenized pages, then call get_scores(words). Its defaults differ a little from formula (221.2): \(k_1 = 1.5\) and an idf without the “1 +”, with a floor for very common words. Scores change; rankings usually stay close [Likely]. Lower-case the text and split it into words the same way for pages and queries.

Step 3: the page-image index (late interaction)

The second index embeds the picture of each page. A ColPali-style retriever is a VLM whose output for every image patch is projected to a short vector (128 numbers). ColPali (Faysse et al., 2024) uses PaliGemma-3B and gives each page 1,024 patch vectors (a \(32 \times 32\) grid at \(448 \times 448\) pixels) [Certain]. ColQwen2 uses Qwen2-VL-2B with dynamic resolution; the README lists 768 patches per page for it [Certain, as listed]. The question is also embedded as one vector per token. The score is MaxSim, from session 219:

\[ \text{LI}(q, d) = \sum_{i=1}^{n_q} \max_{j \le n_d} \ \mathbf{q}_i^\top \mathbf{d}_j \tag{221.3} \]
Read the symbols
\(\mathbf{q}_i\)
the vector of query token \(i\), out of \(n_q\)
\(\mathbf{d}_j\)
the vector of patch \(j\) of page \(d\), out of \(n_d\)
\(\max_j\)
each query token keeps only its best-matching patch; the rest of the page cannot dilute it

Because the retriever looks at pixels, a word inside a chart, a scanned signature or a slide title counts as much as text in the text layer. This is the whole reason page-image retrieval helps on charts and scans.

The toy scorer. We cannot run a 2-billion-parameter model in your browser, so the toy replaces the patch vectors by the words a reader can see on the page, and replaces \(\mathbf{q}_i^\top \mathbf{d}_j\) by a word similarity: 1 for the same word, 0.7 for a few related pairs (“plant” and “factory”, “emissions” and “co2”), 0 otherwise. Each visible word also has a legibility weight: 1 for normal print, 0.8 for the scanned page, and 0.35 for the small print on p8, because at the retriever’s \(448 \times 448\) input such letters are only a few pixels tall. The toy score is \(\sum_i \max_j \text{sim}(q_i, w_j) \cdot \ell_j\). It keeps the two properties that matter: each query word finds its own best region, and the retriever sees what the pixels show, not what the text layer says.

Algorithm 2: build and search the page-image index
  1. input: page records from Algorithm 1, a ColPali-style model and processor
  2. for each page: \(\mathbf{D} \leftarrow\) model(processor.process_images([image])) // shape \((n_d, 128)\)
  3.   store \(\mathbf{D}\) in float16 (or binary) under the page ID // 257.5 KiB per page for ColPali in float16
  4. at question time: \(\mathbf{Q} \leftarrow\) model(processor.process_queries([question])) // shape \((n_q, 128)\)
  5. for each page (or each candidate from a cheap first stage): \(s_d \leftarrow \sum_i \max_j (\mathbf{Q}\mathbf{D}^\top)_{ij}\) // formula (221.3)
  6. return pages sorted by \(s_d\), and for each the arg-max patch of every query token // for crops and for checking
Worked example: how big is the index?

One ColPali page is \(1{,}030 \times 128\) numbers (1,024 patches plus 6 prompt tokens) at 2 bytes: 263,680 bytes, 257.5 KiB [Certain, as reported]. A lab collection of 500 pages needs \(500 \times 263{,}680 = 131{,}840{,}000\) bytes, about 125.7 MiB. That fits in memory, so a brute-force MaxSim over all pages is fine. For millions of pages, see the storage tricks (token pooling, binary vectors, two stages) in session 219.

Step 4: retrieve with text, images, or both

Now each question gives two ranked lists: BM25 over text layers, and MaxSim over page images. The hybrid pipeline fuses them with reciprocal rank fusion (RRF, Cormack et al., 2009). RRF ignores the raw scores, which live on different scales (BM25 1.6 against MaxSim 3.0 means nothing), and uses only ranks:

\[ \text{RRF}(d) = \sum_{L \in \{\text{text},\ \text{image}\}} \frac{\mathbb{1}[d \in L]}{\kappa + \text{rank}_L(d)}, \qquad \kappa = 60 \tag{221.4} \]
Read the symbols
\(L\)
one ranked list of pages; a page with score 0 is not in the list
\(\text{rank}_L(d)\)
the position of page \(d\) in list \(L\), starting at 1
\(\kappa\)
a constant that flattens the difference between rank 1 and rank 2; 60 is the value from the original paper

A page that is near the top of both lists beats a page that is first in only one. That is usually what you want, but not always. Figure 3 lets you check every question.

BM25, text layerMaxSim, page imageRRF fusiongold page (outlined)
Figure 3. Retrieval for the 12 toy questions, computed live with formulas (221.2), (221.3) and (221.4) on the page word lists of Figure 2. Each column is one ranked list (top 6 shown); bars are scores, the gold page is green. Pages with score 0 are not returned.
How to read Figure 3

Start with q10 (gross margin in Q4): BM25 returns only p6, the cost table, because “q4” is its only match; the chart page p5 is first for MaxSim with 3.000; RRF puts p6 first (\(1/61 + 1/62 = 0.03252\)) and p5 second (\(1/61 = 0.01639\)), because p6 is in both lists. With \(k = 1\) the hybrid would fail. Now try q9 (revenue from 2019 to 2023): the image index has the gold page p4 at rank 1, but RRF drops it to rank 4, behind three pages that both lists liked a little. Then try q3 (the scanned auditor’s letter): BM25 returns nothing, the image index finds p9. Finally q4 (policy code in small print): BM25 is first with 8.093, while the image index ranks p8 second (1.400) behind the climate-policy page p7 (2.000), because the small print is hard to see.

Step 5: answer with a VLM and cite the page

The reader gets the top \(k\) pages and the question. In the text-only pipeline the reader is an LLM and gets the text layers. In the page-image pipeline it is a VLM and gets the page images, rendered again from the PDF at the reader’s resolution \(r_{\text{read}}\). In the hybrid pipeline it gets both: each page image followed by its text layer, which helps with small print and exact codes.

Images cost tokens. Qwen2-VL and Qwen2.5-VL use one visual token per \(28 \times 28\) pixels, after rounding each side to a multiple of 28, between 4 and 16,384 tokens per image by default [Certain]:

\[ N_{\text{page}}(r) = \Big\lfloor \frac{H_{\text{px}}(r)}{28} \Big\rceil \cdot \Big\lfloor \frac{W_{\text{px}}(r)}{28} \Big\rceil \tag{221.5} \]
Worked example: what three A4 pages cost

At 144 DPI an A4 page is \(1{,}191 \times 1{,}684\) pixels. \(1{,}684/28 = 60.1 \to 60\) and \(1{,}191/28 = 42.5 \to 43\): \(60 \times 43 = 2{,}580\) tokens. With the same rule, A4 costs 630 tokens at 72 DPI, 1,260 at 100 DPI and 2,772 at 150 DPI. Three pages at 100 DPI cost 3,780 tokens; at 150 DPI they cost 8,316. The text layer of a dense page is usually far fewer tokens than its image at 150 DPI [Likely].

The prompt must make citation easy. Put a short label right before each page image, keep pages of one document in page order, and ask for the page first.

Algorithm 3: read the retrieved pages and answer with a citation
  1. input: question, ranked page IDs, \(k\), reader resolution \(r_{\text{read}}\)
  2. content \(\leftarrow\) [ ]
  3. for each of the top \(k\) pages, in document and page order:
  4.   append text “[report.pdf, page 5]” // the label the model must cite
  5.   append the page image rendered at \(r_{\text{read}}\) // costs formula (221.5) tokens
  6.   (hybrid only) append the page’s text layer
  7. append: “Answer from these pages only. First write the page label you used, then the answer. If no page contains the answer, write NOT FOUND.”
  8. answer \(\leftarrow\) VLM.generate(content); check that the cited page is one of the \(k\) pages // a cheap guard against invented citations

The toy reader. We do not run a VLM in the browser either. The toy uses rules that say, for each question, whether a reader can answer it if the gold page is in the top \(k\). The text reader can answer 8 of the 12 questions from text layers: it cannot read charts or the scan, and it misreads the two-level table on p6. One chart question (q12, the share of power tools) it can still answer, by computing 71 / 114 from the segment table. The VLM reader can answer every question if the page is sharp enough: 72 DPI is enough for normal text and simple tables, chart labels, the two-level table and the scan need 100 DPI, and the small print on p8 needs 150 DPI. These rules are our assumptions to show the trade-off; with a real VLM you measure them.

Step 6: compare the pipelines on text, tables and charts

For each question \(j\) with gold page \(g_j\), a pipeline is correct only if retrieval and reading both work. For a question type \(m\) (text, table or chart) with the set of questions \(\mathcal{Q}_m\):

\[ \text{R@}k_m = \frac{1}{|\mathcal{Q}_m|} \sum_{j \in \mathcal{Q}_m} \mathbb{1}\big[g_j \in \text{top}_k(j)\big], \qquad \text{Acc}_m = \frac{1}{|\mathcal{Q}_m|} \sum_{j \in \mathcal{Q}_m} \mathbb{1}\big[g_j \in \text{top}_k(j)\big]\, \mathbb{1}\big[\text{read}_j\big] \tag{221.6} \]
Read the symbols
\(\text{top}_k(j)\)
the first \(k\) pages the pipeline returns for question \(j\)
\(\text{read}_j\)
1 if the reader gives the right answer when the gold page is in its input
\(\text{R@}k_m, \text{Acc}_m\)
retrieval recall and end-to-end accuracy for type \(m\); \(\text{Acc}_m \le \text{R@}k_m\) always

Report both numbers for every type, and the macro average \(\frac{1}{3}\sum_m \text{Acc}_m\), the mean over types. (Our toy set has 4 questions per type, so here the macro and micro averages are equal; on a real set with 80% text questions they are not.) The gap between \(\text{R@}k_m\) and \(\text{Acc}_m\) tells you whether to fix the retriever or the reader. Figure 4 runs the whole toy lab.

answeredgold page retrieved, reader failedgold page not in top \(k\)
Figure 4. End-to-end results of the toy lab, formula (221.6). Retrieval is computed live as in Figure 3; reading uses the toy reader rules of Step 5. The reader resolution changes only the page-image and hybrid pipelines. Image tokens use formula (221.5) for A4. These are toy numbers that show the mechanisms; they are not benchmark results.
How to read Figure 4

The defaults (text-only, \(k = 3\)) give chart accuracy 0.25: q9 and q10 are red (BM25 never finds the chart pages), q11 is amber (BM25 finds the sustainability page because its paragraph says “emissions”, but the value is only in the chart), and only q12 is green. Table accuracy is 0.75: all 4 table pages are retrieved, but the reader misreads the two-level table (q6). Macro accuracy 0.583. Switch to page-image RAG: 0.917 at 100 DPI, using 3,780 image tokens per question. Lower the reader to 72 DPI: it drops to 0.417, below text-only, because charts, the scan and the two-level table become unreadable. Raise it to 150 DPI: 1.000, for 8,316 tokens. The hybrid at 100 DPI and \(k = 3\) also gets 0.917, but in a different way: it reads q4’s small print from the text layer, and it loses q9 because fusion pushed p4 to rank 4. Set \(k = 5\) and the hybrid reaches 1.000 already at 100 DPI.

Three lessons from the toy carry over to real documents, and match what the ColPali paper reports on its benchmark:

  • Charts and scans need the page image. On ViDoRe (nDCG@5, average of ten sets), ColPali scores 81.3, a parsing pipeline with captions and BGE-M3 67.0, and OCR plus BM25 65.5 [Certain, as reported]. The ColPali README lists ColQwen2 (v1.0) at 89.3 on the same benchmark [Certain, as listed].
  • Tables are mostly a reading problem. Born-digital tables are in the text layer, so text retrieval finds them; flattening loses the row and column structure, and complex headers then cause misreadings [Likely]. Check this on your own set by giving the reader the gold page directly.
  • Resolution is a budget decision. The page-image pipeline only wins if the reader sees the page sharply enough, and sharpness costs tokens (formula 221.5).

The full program

The toy lab in one file: the 10 pages, the 12 questions, BM25, MaxSim, RRF, the toy reader and the evaluation of formula (221.6). It needs only PyTorch and is deterministic (no randomness at all). Figures 3 and 4 run the same computation in JavaScript.

import math, torch

# 1) the toy report: (id, kind, text layer, extra words visible only in the pixels)
PAGES = [
    ("p1", "text", "letter from the ceo 2023 was a year of growth revenue grew and we opened a new factory in porto our team delivered record orders under a new code of conduct", ""),
    ("p2", "text", "company overview northwind was founded in 1987 in lyon we make power tools and garden tools we employ 1240 people in five countries", ""),
    ("p3", "table", "revenue by region table 1 region 2022 2023 north 41 44 south 28 31 east 19 24 west 12 15 total 100 114 eur million", ""),
    ("p4", "chart", "financial highlights see figure 1", "revenue 2019 2020 2021 2022 2023 eur million line chart trend growth"),
    ("p5", "chart", "quarterly results figure 2", "gross margin q1 q2 q3 q4 percent bar chart"),
    ("p6", "table", "cost breakdown table 2 operating cost q1 q2 q3 q4 materials 14 15 15 16 logistics 4 5 6 5 staff 9 9 10 10 eur million", ""),
    ("p7", "chart", "sustainability we cut emissions at every plant and moved porto to solar power under our climate risk policy see figure 3", "co2 emissions 2021 2022 2023 tonnes bar chart"),
    ("p8", "small", "risk factors currency risk policy code fx 17 interest rate risk policy code ir 4 supply risk single supplier for batteries", ""),
    ("p9", "scan", "", "independent auditor report signed by kessler partners lyon march 2024 opinion accounts 2023"),
    ("p10", "table", "product segments table 3 segment revenue power tools 71 garden tools 32 accessories 11 eur million", "share of revenue pie chart percent"),
]
# (id, type, query words, gold page, text reader can answer?, min DPI the VLM reader needs)
QUESTIONS = [
    ("q1", "text", "northwind founded", "p2", True, 72),       ("q2", "text", "company open new factory", "p1", True, 72),
    ("q3", "text", "firm signed auditor report", "p9", False, 100), ("q4", "text", "policy code currency risk", "p8", True, 150),
    ("q5", "table", "revenue south region 2023", "p3", True, 72), ("q6", "table", "logistics cost q3", "p6", False, 100),
    ("q7", "table", "revenue garden tools", "p10", True, 72),     ("q8", "table", "total revenue 2022 regions", "p3", True, 72),
    ("q9", "chart", "revenue change 2019 2023", "p4", False, 100), ("q10", "chart", "gross margin q4", "p5", False, 100),
    ("q11", "chart", "co2 emissions 2022", "p7", False, 100),     ("q12", "chart", "share revenue power tools", "p10", True, 100),
]
ids = [p[0] for p in PAGES]
text_layer = [p[2].split() for p in PAGES]
LEGIBLE = {"small": 0.35, "scan": 0.8}          # how well the retriever sees the words at its 448 px input
visible = [[(w, LEGIBLE.get(k, 1.0)) for w in (t + " " + extra).split()] for _, k, t, extra in PAGES]

# 2) toy "embedding" similarity: 1 for the same word, a few related pairs, else 0
vocab = sorted({w for d in visible for w, _ in d} | {w for q in QUESTIONS for w in q[2].split()})
V = {w: i for i, w in enumerate(vocab)}
S = torch.eye(len(vocab), dtype=torch.float64)
for a, b, s in [("sales", "revenue", .7), ("employees", "employ", .7), ("plant", "factory", .7), ("emissions", "co2", .7), ("costs", "cost", .9)]:
    if a in V and b in V: S[V[a], V[b]] = S[V[b], V[a]] = s

# 3) BM25 on the text layer, formula (221.2)
N, k1, b = len(PAGES), 1.2, 0.75
avgdl = sum(map(len, text_layer)) / N
df = {}
for d in text_layer:
    for w in set(d): df[w] = df.get(w, 0) + 1
def bm25(q):
    out = []
    for d in text_layer:
        s = 0.0
        for w in q:
            f = d.count(w)
            if f:
                idf = math.log(1 + (N - df[w] + 0.5) / (df[w] + 0.5))
                s += idf * f * (k1 + 1) / (f + k1 * (1 - b + b * len(d) / avgdl))
        out.append(s)
    return torch.tensor(out, dtype=torch.float64)

# 4) late interaction (MaxSim) on what the page shows, formula (221.3)
def maxsim(q):
    qi = torch.tensor([V[w] for w in q])
    out = []
    for d in visible:
        pj = torch.tensor([V[w] for w, _ in d]); wt = torch.tensor([x for _, x in d], dtype=torch.float64)
        out.append((S[qi][:, pj] * wt).max(dim=1).values.sum())   # best patch per query word, then sum
    return torch.stack(out)

def ranking(scores):                       # pages with score 0 are not returned
    order = sorted(range(N), key=lambda i: (-scores[i].item(), i))
    return [i for i in order if scores[i] > 0]
def rrf(*lists, kappa=60):                 # formula (221.4)
    s = torch.zeros(N, dtype=torch.float64)
    for L in lists:
        for r, i in enumerate(L): s[i] += 1 / (kappa + r + 1)
    return ranking(s)

# 5) retrieve, read, score per question type, formula (221.6)
def run(k=3, dpi=100):
    acc = {"text": {}, "image": {}, "hybrid": {}}
    for qid, typ, words, gold, text_ok, need in QUESTIONS:
        q = words.split(); g = ids.index(gold)
        Lt, Li = ranking(bm25(q)), ranking(maxsim(q))
        lists = {"text": Lt, "image": Li, "hybrid": rrf(Lt, Li)}
        reads = {"text": text_ok, "image": dpi >= need, "hybrid": text_ok or dpi >= need}
        for sname, L in lists.items():
            hit = g in L[:k]
            acc[sname].setdefault(typ, []).append(hit and reads[sname])
        if k == 3: print(qid, typ, "rank of gold page  text:", Lt.index(g) + 1 if g in Lt else "-",
                         " image:", Li.index(g) + 1 if g in Li else "-", " hybrid:", lists["hybrid"].index(g) + 1)
    for sname, d in acc.items():
        per = {t: sum(v) / len(v) for t, v in d.items()}
        print(f"k={k} dpi={dpi} {sname:6s}", {t: round(x, 2) for t, x in per.items()}, "macro", round(sum(per.values()) / 3, 3))
run(k=3, dpi=100)
run(k=1, dpi=72)

Output (we ran it with PyTorch 2.8): the rank of the gold page for text, image and hybrid is 1, 1, 1 for q1, q2 and q5 to q8; “–”, 1, 1 for q3; 1, 2, 2 for q4; “–”, 1, 4 for q9; “–”, 1, 2 for q10; and 1, 1, 1 for q11 and q12. At \(k = 3\) and 100 DPI the accuracies are text-only {text 0.75, table 0.75, chart 0.25}, macro 0.583; page-image {0.75, 1.0, 1.0}, macro 0.917; hybrid {1.0, 1.0, 0.75}, macro 0.917. At \(k = 1\) and 72 DPI: text-only 0.583, page-image 0.417, hybrid 0.5.

The real version

The same steps with real models. We did not run this block for this page (it needs a GPU and model downloads); the calls follow the official READMEs and model cards as of November 2026, so check them against the versions you install.

import glob, pathlib, torch, pypdfium2 as pdfium
from PIL import Image
from rank_bm25 import BM25Okapi
from colpali_engine.models import ColQwen2, ColQwen2Processor

# Step 1: render every page and keep its text layer (Algorithm 1)
pages = []
for path in sorted(glob.glob("pdfs/*.pdf")):
    pdf = pdfium.PdfDocument(path)
    for i in range(len(pdf)):
        png = f"pages/{pathlib.Path(path).stem}_p{i+1}.png"
        pdf[i].render(scale=144 / 72).to_pil().save(png)
        text = pdf[i].get_textpage().get_text_range().replace("\r\n", "\n")
        pages.append({"doc": path, "page": i + 1, "png": png, "text": text})

# Step 2: BM25 on the text layers
tok = lambda s: s.lower().split()
bm25 = BM25Okapi([tok(p["text"]) for p in pages])

# Step 3: page-image index, one page at a time (no padding issues)
name = "vidore/colqwen2-v1.0"
model = ColQwen2.from_pretrained(name, torch_dtype=torch.bfloat16, device_map="cuda:0").eval()
proc = ColQwen2Processor.from_pretrained(name)
page_vecs = []
with torch.no_grad():
    for p in pages:
        batch = proc.process_images([Image.open(p["png"])]).to(model.device)
        page_vecs.append(model(**batch)[0].to("cpu", torch.float16))   # (n_d, 128)

# Step 4: retrieve with both and fuse ranks, formula (221.4)
def retrieve(question, k=3, kappa=60):
    s_txt = bm25.get_scores(tok(question))
    with torch.no_grad():
        q = model(**proc.process_queries([question]).to(model.device)).to("cpu", torch.float16)
    s_img = proc.score_multi_vector(q, page_vecs)[0]          # MaxSim, formula (221.3)
    fused = {}
    for scores in (list(s_txt), s_img.tolist()):
        order = [i for i in sorted(range(len(pages)), key=lambda i: -scores[i]) if scores[i] > 0]
        for r, i in enumerate(order):
            fused[i] = fused.get(i, 0.0) + 1 / (kappa + r + 1)
    return sorted(fused, key=lambda i: -fused[i])[:k]

Step 5 with Qwen2.5-VL as the reader. Free the retriever first (del model; torch.cuda.empty_cache()) if memory is tight. The page images are rendered again at the reader’s resolution, and every image gets a label.

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

rname = "Qwen/Qwen2.5-VL-7B-Instruct"
reader = Qwen2_5_VLForConditionalGeneration.from_pretrained(rname, torch_dtype="auto", device_map="auto")
rproc = AutoProcessor.from_pretrained(rname, min_pixels=256 * 28 * 28, max_pixels=2880 * 28 * 28)

def answer(question, top, dpi=150):
    content = []
    for i in sorted(top, key=lambda i: (pages[i]["doc"], pages[i]["page"])):
        p = pages[i]
        png = p["png"].replace(".png", f"_{dpi}.png")
        pdfium.PdfDocument(p["doc"])[p["page"] - 1].render(scale=dpi / 72).to_pil().save(png)
        content += [{"type": "text", "text": f"[{pathlib.Path(p['doc']).name}, page {p['page']}]"},
                    {"type": "image", "image": "file://" + str(pathlib.Path(png).resolve())}]
    content.append({"type": "text", "text": "Answer from these pages only. First write the page label you used, "
                    "then the answer. If no page contains the answer, write NOT FOUND.\nQuestion: " + question})
    messages = [{"role": "user", "content": content}]
    text = rproc.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    images, videos = process_vision_info(messages)
    inputs = rproc(text=[text], images=images, videos=videos, padding=True, return_tensors="pt").to(reader.device)
    out = reader.generate(**inputs, max_new_tokens=128, do_sample=False)
    return rproc.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]

Note max_pixels: we raised it to \(2{,}880 \times 28 \times 28\) pixels so that an A4 page at 150 DPI (2,772 tokens by formula 221.5) is not shrunk. With the model card’s example value of \(1{,}280 \times 28 \times 28\) the processor would scale every page down to about 1,280 tokens, which is close to 100 DPI. Step 6 is then a loop over your labeled questions: call retrieve with BM25 only, ColQwen2 only and both; record whether the gold page is in the top \(k\); call answer; compare the answer with the gold answer (exact match after normalizing numbers, or ANLS for strings); and fill the table of formula (221.6) per type. Also run answer with the gold page alone: that measures the reader without retrieval errors.

Where this is used

ApplicationHard pagesWhat the lab teaches for it
Financial and annual reportscharts of revenue and margins, tables with two-level headerspage-image retrieval for charts; check table reading with the gold page alone
Technical manualsexploded-view diagrams, part numbers in small printhybrid: BM25 for exact part numbers, page images for diagrams
Slide decksslides that are mostly pictures, text inside imagesone unit per slide image; the text layer is often almost empty
Scanned contracts, invoices, formsno text layer at allflag pages without text in ingestion; page images or OCR
Scientific papersplots, equations, tablesfigures are often vector graphics with partial text; still search the image
Lecture videos with slidesslide frames inside a videotreat sampled keyframes as “pages”, keep the timestamp as the citation; the same pipeline becomes Video-RAG (see session 210)
Meeting recordings and screen sharesdashboards and documents shown on screenkeyframe deduplication (as in near-duplicate detection) before indexing, then page-image retrieval on the frames

Session 220 extends the reader of Step 5: instead of reading a fixed top \(k\), the model can search again, zoom into a region, and stop when it has enough evidence.

Common mistakes

  • Trusting the text layer. Charts stored as pictures and scanned pages have no text (Step 1). Count, at ingestion, how many pages have almost no text but are not blank.
  • Showing the reader the retriever’s image. The retriever’s small input is fine for matching and too small for reading numbers. Render again from the PDF at the reader’s DPI.
  • Letting the processor shrink pages silently. A max_pixels limit lower than your page size scales the page down. Compute the tokens per page with formula (221.5) and set the limit on purpose.
  • Fusing raw scores. BM25 and MaxSim scores are on different scales. Fuse ranks (RRF) or normalize per list, and remember that RRF punishes pages only one index can see (q9).
  • Losing the page pointer. Without (document, page) on every unit, the reader cannot cite and you cannot check answers.
  • One overall score. A test set of mostly text questions hides a pipeline that never answers charts. Report retrieval and accuracy per type, and the macro average.
  • Not separating retrieval from reading. Run the reader on the gold page alone. If accuracy is still low there, a better retriever will not help.
  • Dropping BM25 after moving to images. Exact codes and small print (q4) are still found best by keywords.

Interview questions

Why does text-only RAG fail on PDFs with charts, even with a perfect text embedding model?basic
Because the information is not in the text it searches. A chart inserted as a picture has no text layer: in our real two-page PDF, the chart page returned only “Quarterly results” and “Figure 2. See chart.”, with none of the four values and not even the chart title. No embedding model can retrieve or read words that are not there. The same holds for scanned pages and text inside images. Fixes: embed the page image (ColPali-style), or run OCR and chart-to-text conversion at ingestion, or both.
Walk through the pipeline you would build for QA over a folder of PDF reports.basic
Ingest: render each page (for example 144 DPI) and extract the text layer; flag pages with no text. Index: BM25 (or a text embedding) on the text layer, and a late-interaction page-image index (ColPali or ColQwen2) on the images; store (document, page) for every unit. Retrieve: search both, fuse ranks with RRF or a weighted variant, take the top \(k\). Read: render those pages again at the reader’s resolution, label each one, give them to a VLM with an instruction to cite the page first and say NOT FOUND when needed. Evaluate: labeled questions with gold pages and a type tag; retrieval recall and accuracy per type; the reader on the gold page alone.
Explain MaxSim and why it helps for small regions of a page.core
The page is kept as one vector per image patch, the query as one vector per token. For each query token we take the maximum dot product over all patches, then sum over tokens: \(\sum_i \max_j \mathbf{q}_i^\top \mathbf{d}_j\). Each query word finds its own best region (the chart title, an axis label), and the rest of the page cannot dilute that match, unlike a single averaged vector. Page vectors are computed offline, so the query cost is only the max-and-sum. The price is storage: 1,030 vectors of 128 numbers per page for ColPali, 257.5 KiB in float16.
Text retrieval finds the right table page, but answers about that table are often wrong. Where is the problem and how do you confirm it?core
The problem is reading, not retrieval. The text layer gives the table as a flat sequence of words and numbers, so row and column structure, merged cells and multi-level headers are lost, and the reader assigns a number to the wrong column. Confirm it by giving the reader the gold page alone: if accuracy stays low, the retriever is not the cause. Fixes: give the VLM the page image (it sees the grid), or extract the table with a layout model into markdown or HTML before reading, and measure each change per type.
How does reciprocal rank fusion work, and when can it hurt?core
Each list contributes \(1/(\kappa + \text{rank})\) for every page it returns, with \(\kappa = 60\); the contributions are added. It needs no score calibration and rewards pages that several lists agree on. It hurts when only one list can see the right page: in the toy, the line chart p4 is first for the image index and absent from BM25, so its fused score is \(1/61 = 0.0164\), less than three pages that both lists ranked lower (for example p3 with \(1/61 + 1/63\)); it falls to rank 4. Doubling the image list’s weight does not fix this case, because the pages above p4 are in the image list too. Keeping a few slots for each list (the union of each list’s top 2 includes p4) or reranking the fused candidates with a VLM does.
How many tokens does one retrieved page cost a Qwen2.5-VL reader, and how do you choose the resolution?core
One token per \(28 \times 28\) pixels after rounding each side to a multiple of 28. A4 at 144 DPI is \(1{,}191 \times 1{,}684\) pixels, so \(43 \times 60 = 2{,}580\) tokens; at 100 DPI 1,260 and at 150 DPI 2,772. Choose by measuring: run the reader on gold pages at 72, 100 and 150 DPI per question type and pick the lowest resolution that keeps accuracy, possibly higher for the top 1 or 2 pages and lower for the rest. Also check the processor’s max_pixels, which can silently shrink pages.
Your page-image RAG beats text RAG on charts but loses on questions about part numbers. Why?advanced
Part numbers and codes are exact strings, often in small print. A page-image retriever sees them at its own input resolution (\(448 \times 448\) for ColPali), where small letters are a few pixels tall, and its embeddings match meaning more than exact characters. BM25 matches the exact token and gives rare tokens a high idf. In the toy, q4 (“policy code currency risk”) is first for BM25 (8.093) and only second for the image index, behind a page with “risk policy” in large print. The answer is a hybrid: keep a keyword index on the text layer or OCR, and fuse.
How do you evaluate this system so you know whether to fix the retriever or the reader?advanced
Label each question with its gold page and type. For each pipeline and \(k\), measure retrieval recall \(\text{R@}k_m\) and end-to-end accuracy \(\text{Acc}_m\) per type (formula 221.6), plus the reader’s accuracy on the gold page alone. If \(\text{R@}k_m\) is low, fix retrieval (better index, fusion, larger \(k\)). If recall is high but accuracy is low, fix the reader (resolution, table extraction, prompt). Report the macro average over types, citation accuracy, and image tokens per question. In the toy, text-only charts have recall 0.5 and accuracy 0.25: two retrieval misses (q9, q10) and one reading miss (q11).
The scanned pages in your collection are never retrieved by the text pipeline. What are your options and their trade-offs?advanced
Scans have no text layer, so BM25 sees an empty page. Options: (1) run OCR at ingestion and index its text: cheap to search, works with BM25 and exact matches, but OCR errors on stamps, handwriting and low-quality scans, and it adds a slow step; (2) index the page image with a ColPali-style retriever: no OCR, sees layout and signatures, but less precise on exact strings and needs GPU indexing and more storage; (3) both, fused. In all cases the reader should see the image, rendered sharply, because OCR mistakes would otherwise become answer mistakes.
How would you make this work for a collection of a million pages?advanced
Brute-force MaxSim over all pages is too slow and the float16 index is about 264 GB for ColPali. Use two stages: a cheap first stage (BM25 on text and OCR, plus one pooled vector per page in an approximate nearest-neighbor index) that returns a few hundred candidates, then exact MaxSim only on those. Shrink storage with token pooling (the ColPali authors report that pool factor 3 removes 66.7% of vectors and keeps 97.8% of quality) and binary quantization. Measure the first stage’s recall on its own, because pages it misses are lost. Batch the indexing on GPUs and keep the original PDFs for rendering at read time.

Exercises

  1. A US Letter page is \(612 \times 792\) points. What is its pixel size at 144 DPI, and how many Qwen2.5-VL tokens does it cost by formula (221.5)?
  2. Page A is rank 2 in BM25 and rank 5 in the image list. Page B is rank 1 in the image list and absent from BM25. Which one does RRF with \(\kappa = 60\) put first?
  3. In a 10-page collection, word X appears in 1 page and word Y in 5 pages. Compute their idf by formula (221.2).
  4. In Figure 4, text-only RAG at \(k = 3\) has chart accuracy 0.25. For each of the 4 chart questions, say whether retrieval or reading failed.
  5. How much memory does a ColPali index for 2,000 pages need in float16, and with binary vectors (1 bit per number)?
  6. Change the toy program so that the image list counts twice in RRF (each image term is \(2/(\kappa + r)\)). Does q9’s gold page p4 get into the top 3? Then try another fix: give the reader the union of the top 2 pages of each list. Does that include p4?
Answerscheck
1. \(612 \times 2 = 1{,}224\) by \(792 \times 2 = 1{,}584\) pixels. \(1{,}584/28 = 56.6 \to 57\) and \(1{,}224/28 = 43.7 \to 44\): \(57 \times 44 = 2{,}508\) tokens. 2. A: \(1/62 + 1/65 = 0.03151\). B: \(1/61 = 0.01639\). A wins, even though only the image index finds B at the top: this is the q9 effect. 3. \(\text{idf}(X) = \ln(1 + 9.5/1.5) = 1.9924\); \(\text{idf}(Y) = \ln(1 + 5.5/5.5) = \ln 2 = 0.6931\). 4. q9 (revenue trend, p4) and q10 (gross margin, p5): retrieval failed, BM25 never returns the chart page. q11 (CO2 in 2022, p7): retrieval worked through the word “emissions” in the paragraph, reading failed because the value is only in the chart. q12 (share of power tools, p10): both worked, the reader computes 71/114 from the segment table. 5. \(2{,}000 \times 263{,}680 = 527{,}360{,}000\) bytes, about 502.9 MiB. Binary: 16 bytes per vector, \(2{,}000 \times 1{,}030 \times 16 = 32{,}960{,}000\) bytes, about 31.4 MiB. 6. Weighting does not help q9: p4 gets \(2/61 = 0.03279\), but p1 (\(1/62 + 2/62 = 0.04839\)), p3 (\(1/61 + 2/63 = 0.04814\)) and p10 (\(1/63 + 2/65 = 0.04664\)) also get image credit, so p4 stays at rank 4. (q4 also stays at rank 2: p7 0.04892, p8 0.04865.) The union works: BM25’s top 2 are p3 and p1, the image top 2 are p4 and p1, so the reader gets {p1, p3, p4} and p4 is in. The lesson: when one index sees a page the other cannot, keep a few slots for each index instead of hoping a weight will fix the fused order.

Key takeaways

  • A PDF’s text layer is not the page: charts stored as pictures and scans have none. Check it at ingestion (Step 1).
  • Index every page twice: BM25 on the text layer (221.2) for exact strings, and a ColPali-style late-interaction index on the page image (221.3) for charts, scans and layout. Keep (document, page) on every unit.
  • Fuse ranks, not scores (221.4), and know that RRF can push down a page only one index can see.
  • The reader needs the page re-rendered at its own resolution; each A4 page costs 1,260 tokens at 100 DPI and 2,772 at 150 DPI with Qwen2.5-VL (221.5).
  • Score retrieval and reading separately, per question type, with the macro average (221.6). For charts, text RAG usually fails at retrieval; for tables, it usually fails at reading.

Key formulas

\[ W_{\text{px}} = \Big\lfloor w_{\text{pt}} \tfrac{r}{72} \Big\rceil, \qquad \text{LI}(q,d) = \sum_i \max_j \mathbf{q}_i^\top \mathbf{d}_j, \qquad \text{RRF}(d) = \sum_L \frac{1}{60 + \text{rank}_L(d)} \]
\[ \text{idf}(t) = \ln\!\Big(1 + \tfrac{N - n(t) + 0.5}{n(t) + 0.5}\Big), \qquad N_{\text{page}} = \Big\lfloor \tfrac{H_{\text{px}}}{28} \Big\rceil \Big\lfloor \tfrac{W_{\text{px}}}{28} \Big\rceil, \qquad \text{Acc}_m = \tfrac{1}{|\mathcal{Q}_m|} \sum_{j} \mathbb{1}[g_j \in \text{top}_k]\, \mathbb{1}[\text{read}_j] \]

References for this session

  1. Faysse et al. (2024). ColPali: Efficient Document Retrieval with Vision Language Models
  2. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 8: Document AI
  3. Khattab & Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
  4. Robertson & Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond
  5. Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
  6. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  7. Bai et al. (2025). Qwen2.5-VL Technical Report
  8. Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images
  9. Yu et al. (2024). VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
  10. Cho et al. (2024). M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
  11. Illuin Technology (2024). colpali-engine: ColPali and ColQwen2 retrievers and the ViDoRe benchmark (code and model list)
  12. pypdfium2 team (2024). pypdfium2: Python bindings to PDFium (documentation)
  13. D. Brown (2020). rank_bm25: a collection of BM25 algorithms in Python (code)