multimodal RAG over PDFs
In this lab you build question answering over PDF files twice: once from the text layer of each page, and once from a picture of each page, embedded ColPali-style. A vision-language model reads the retrieved pages and cites them. Then you measure, per question type, why text-only RAG misses charts and scans, why tables are mostly a reading problem, and what page resolution costs in tokens.
What you will learn
- How to render PDF pages at a chosen DPI and read their text layer, and why charts stored as pictures and scanned pages have no text layer
- How to build a BM25 index on page text and a late-interaction (MaxSim) index on page images, with a page pointer on every unit
- How reciprocal rank fusion combines the two lists, and the case where it pushes down a page that only one index can see
- How to give retrieved pages to a VLM such as Qwen2.5-VL with labels and citations, and what each page costs in visual tokens
- How to score retrieval recall and answer accuracy separately for text, table and chart questions, and read the gap between them
- The real pipeline with pypdfium2, colpali-engine (ColQwen2) and Qwen2.5-VL, next to a toy version that runs anywhere
Why this matters
In this lab you build a small but complete multimodal RAG system over PDF files. You index every page twice: once as text (the text layer of the PDF) and once as an image (a picture of the page, embedded ColPali-style). A vision-language model (VLM) then answers questions from the retrieved pages and cites the page it used. Finally you measure which pipeline wins on plain text, on tables and on charts, and why.
Why build it yourself? Because the most common failure of document assistants is invisible from the outside. The system answers, the answer sounds good, and it is wrong because the number was in a chart that the text pipeline never saw. Two simple facts cause most of the trouble:
- A PDF's text layer is not the page. Charts that are stored as pictures, scanned pages and text inside figures have no text layer. You will make a real two-page PDF in Step 1 and see that its chart page gives only the words
"Quarterly results"and"Figure 2. See chart.", without a single number from the chart. - Retrieval and reading fail in different places. For tables, text retrieval usually finds the right page, but a text reader can misread a flattened table. For charts, text retrieval often does not find the page at all. You only see this if you score the two stages apart and per question type.
Session 219 explained the ideas: units, late interaction (MaxSim), token budgets and evaluation per modality. Session 218 showed how a VLM reads one page. This lab turns them into code and numbers. You will (1) render pages and read the text layer, (2) build a BM25 text index, (3) build a page-image index with late interaction, (4) retrieve with text, images and a fusion of both, (5) answer with a VLM and citations, and (6) compare the pipelines per question type.
Setup
There are two versions of this lab. Both use the same steps.
- The toy version runs anywhere. It uses a 10-page toy annual report written as word lists, a real BM25, a toy late-interaction scorer and a rule-based reader. It needs only PyTorch, runs in under a second, and every figure on this page runs the same code in your browser. Its numbers show the mechanisms. They are not benchmark results.
- The real version needs a GPU. It renders real PDFs with
pypdfium2, embeds pages with ColQwen2 (a ColPali-style retriever) from thecolpali-enginepackage, and answers with Qwen2.5-VL. A GPU with 24 GB of memory is enough if you load the retriever, index, free it, and then load the reader [Likely].
pip install torch pypdfium2 pillow rank_bm25 # toy version + Step 1 pip install colpali-engine transformers accelerate qwen-vl-utils # real version
For real documents, use your own PDFs (annual reports, manuals, slide decks), or one of the ViDoRe test sets that the ColPali authors published on Hugging Face. Write 30 to 100 questions yourself and tag each one with its gold page (the page that holds the answer) and its type: text, table or chart. This small labeled set is the most valuable part of the lab.
Our toy report, “Northwind Tools, Annual Report 2023”, has 10 pages chosen to contain every hard case: two text pages, a scanned auditor’s letter with no text layer, a page of small print, three tables (one with a two-level header), and three charts that are stored as pictures. There are 12 questions, 4 of each type. Figure 1 shows the system you are building.
Step 1: render pages and read the text layer
A PDF page has a size in points (1 point = 1/72 inch). An A4 page is \(595.3 \times 841.9\) points. To get an image, we render (draw) the page at some resolution \(r\) in dots per inch (DPI). Each side then has
- Read the symbols
- \(w_{\text{pt}}, h_{\text{pt}}\)
- the page width and height in points, from the PDF itself
- \(r\)
- the render resolution in DPI;
pypdfium2calls \(r/72\) thescale, soscale=1means 72 DPI [Certain] - \(\lfloor\cdot\rceil\)
- round to the nearest whole pixel
So A4 at 144 DPI (scale=2) is \(1{,}191 \times 1{,}684\) pixels. The same library also gives the text layer: the characters that the program which made the PDF wrote into the file, with their positions. Born-digital PDFs (exported from Word, LaTeX or a reporting tool) have a good text layer. Scanned PDFs have none, unless someone ran OCR (optical character recognition) on them. Charts and photos inserted as pictures have no text layer either.
Let us check this on a real file. The script below builds a two-page A4 PDF with reportlab: page 1 has a title and a small revenue table written as text; page 2 has a title, a bar chart inserted as a PNG picture (with the values 38, 41, 40 and 44 drawn on the bars), and a caption. Then it renders each page and prints the text layer.
import pypdfium2 as pdfium pdf = pdfium.PdfDocument("report.pdf") for i in range(len(pdf)): page = pdf[i] w_pt, h_pt = page.get_size() # points, 1/72 inch img = page.render(scale=144 / 72).to_pil() # formula (221.1): 144 DPI img.save(f"page_{i+1}.png") text = page.get_textpage().get_text_range().replace("\r\n", "\n") print(i + 1, (round(w_pt, 1), round(h_pt, 1)), img.size, repr(text))
Output (we ran it):
1 (595.3, 841.9) (1191, 1684) 'Revenue by region\nRegion 2022 2023\nNorth 41 44\nSouth 28 31\nEast 19 24\nWest 12 15' 2 (595.3, 841.9) (1191, 1684) 'Quarterly results\nFigure 2. See chart.'
Page 1 is fine: the table comes back as text, row by row (the column structure is gone, but the numbers are there). Page 2 is the problem: the text layer has none of the four margin values and not even the chart title, because they are pixels inside a picture. A text-only RAG system cannot find this page for “gross margin in Q4”, and even if it did, it could not read the answer.
Figure 2 shows the same thing for all 10 pages of the toy report: on the left, what the page shows; on the right, what the text layer gives. Words that you can see on the page but that are missing from the text layer are marked in red.
What the page shows (pixels)
What the text layer gives
How to read Figure 2
Start with page p1 (a letter): 0 words are missing; text-only RAG is fine here. Go to p4, the line chart: the text layer has 5 words (“financial highlights see figure 1”), and 12 visible words are missing, including “revenue” and all five years. Page p9 is a scanned letter: the text layer is empty and all 13 visible words are missing, including the auditor’s name. Page p6 is a table: no word is missing, but look at how the two-level header turns into a flat list of numbers. That is a reading problem, not a retrieval problem.
- input: PDF files, retriever resolution \(r_{\text{idx}}\) (for example 144 DPI), reader resolution \(r_{\text{read}}\)
- for each PDF, for each page \(i\):
- image \(\leftarrow\) render(page, scale \(= r_{\text{idx}}/72\)); save it as
doc_i.png// formula (221.1) - text \(\leftarrow\) the text layer of the page
- if text has fewer than ~20 characters but the image is not blank: mark the page “needs OCR” // scans, picture-only slides
- store a record {doc, page \(i\), image path, text, page size in points}
- return the list of page records // every later step points back to (doc, page)
The last line is the important one. Every index you build later stores only a page ID. At question time you go back to the record, open the original PDF, and render the page again at the reader’s resolution. You never show the reader the retriever’s small image.
Step 2: the text index (BM25 on the text layer)
The text pipeline treats each page’s text layer as one document. (In a long-text system you would cut pages into chunks; one page per unit keeps the comparison with page images fair, because both pipelines then return pages.) We score with BM25, the classic keyword ranking function used by search engines such as Lucene and Elasticsearch. For a query with words \(t\) and a page \(d\):
- Read the symbols
- \(f(t,d)\)
- how many times word \(t\) appears in page \(d\)’s text layer
- \(|d|, \text{avgdl}\)
- the length of page \(d\) in words, and the average length over all pages
- \(N, n(t)\)
- the number of pages, and the number of pages that contain \(t\); rare words get a large idf (inverse document frequency)
- \(k_1, b\)
- two constants: \(k_1 = 1.2\) limits how much repeating a word helps, \(b = 0.75\) controls how much long pages are penalized
Question q10: “What was the gross margin in Q4?”, query words gross margin q4. The answer is on p5, the bar chart. But p5’s text layer is only “quarterly results figure 2”, so p5 scores 0. The words “gross” and “margin” are in no text layer at all. Only “q4” matches anything: the cost table p6 (27 words; average page length 17.3 words) has it once.
\(\text{idf}(\text{q4}) = \ln(1 + 9.5/1.5) = 1.9924\). The length part is \(1 + 1.2 \times (0.25 + 0.75 \times 27/17.3) = 2.7046\), so the score is \(1.9924 \times 2.2 / 2.7046 = 1.621\). BM25 returns exactly one page, p6, with score 1.621. The gold page p5 is not in the list at all. No reader can recover from this.
BM25 has real strengths that we will see in the results: it matches exact strings such as codes, part numbers and names, it needs no GPU, and it explains itself (you can see which word matched). In the toy, the only question where text search beats the image index is q4, which asks for an exact policy code in small print.
In real code you can use rank_bm25: build BM25Okapi(pages) from the tokenized pages, then call get_scores(words). Its defaults differ a little from formula (221.2): \(k_1 = 1.5\) and an idf without the “1 +”, with a floor for very common words. Scores change; rankings usually stay close [Likely]. Lower-case the text and split it into words the same way for pages and queries.
Step 3: the page-image index (late interaction)
The second index embeds the picture of each page. A ColPali-style retriever is a VLM whose output for every image patch is projected to a short vector (128 numbers). ColPali (Faysse et al., 2024) uses PaliGemma-3B and gives each page 1,024 patch vectors (a \(32 \times 32\) grid at \(448 \times 448\) pixels) [Certain]. ColQwen2 uses Qwen2-VL-2B with dynamic resolution; the README lists 768 patches per page for it [Certain, as listed]. The question is also embedded as one vector per token. The score is MaxSim, from session 219:
- Read the symbols
- \(\mathbf{q}_i\)
- the vector of query token \(i\), out of \(n_q\)
- \(\mathbf{d}_j\)
- the vector of patch \(j\) of page \(d\), out of \(n_d\)
- \(\max_j\)
- each query token keeps only its best-matching patch; the rest of the page cannot dilute it
Because the retriever looks at pixels, a word inside a chart, a scanned signature or a slide title counts as much as text in the text layer. This is the whole reason page-image retrieval helps on charts and scans.
The toy scorer. We cannot run a 2-billion-parameter model in your browser, so the toy replaces the patch vectors by the words a reader can see on the page, and replaces \(\mathbf{q}_i^\top \mathbf{d}_j\) by a word similarity: 1 for the same word, 0.7 for a few related pairs (“plant” and “factory”, “emissions” and “co2”), 0 otherwise. Each visible word also has a legibility weight: 1 for normal print, 0.8 for the scanned page, and 0.35 for the small print on p8, because at the retriever’s \(448 \times 448\) input such letters are only a few pixels tall. The toy score is \(\sum_i \max_j \text{sim}(q_i, w_j) \cdot \ell_j\). It keeps the two properties that matter: each query word finds its own best region, and the retriever sees what the pixels show, not what the text layer says.
- input: page records from Algorithm 1, a ColPali-style model and processor
- for each page: \(\mathbf{D} \leftarrow\) model(processor.process_images([image])) // shape \((n_d, 128)\)
- store \(\mathbf{D}\) in float16 (or binary) under the page ID // 257.5 KiB per page for ColPali in float16
- at question time: \(\mathbf{Q} \leftarrow\) model(processor.process_queries([question])) // shape \((n_q, 128)\)
- for each page (or each candidate from a cheap first stage): \(s_d \leftarrow \sum_i \max_j (\mathbf{Q}\mathbf{D}^\top)_{ij}\) // formula (221.3)
- return pages sorted by \(s_d\), and for each the arg-max patch of every query token // for crops and for checking
One ColPali page is \(1{,}030 \times 128\) numbers (1,024 patches plus 6 prompt tokens) at 2 bytes: 263,680 bytes, 257.5 KiB [Certain, as reported]. A lab collection of 500 pages needs \(500 \times 263{,}680 = 131{,}840{,}000\) bytes, about 125.7 MiB. That fits in memory, so a brute-force MaxSim over all pages is fine. For millions of pages, see the storage tricks (token pooling, binary vectors, two stages) in session 219.
Step 4: retrieve with text, images, or both
Now each question gives two ranked lists: BM25 over text layers, and MaxSim over page images. The hybrid pipeline fuses them with reciprocal rank fusion (RRF, Cormack et al., 2009). RRF ignores the raw scores, which live on different scales (BM25 1.6 against MaxSim 3.0 means nothing), and uses only ranks:
- Read the symbols
- \(L\)
- one ranked list of pages; a page with score 0 is not in the list
- \(\text{rank}_L(d)\)
- the position of page \(d\) in list \(L\), starting at 1
- \(\kappa\)
- a constant that flattens the difference between rank 1 and rank 2; 60 is the value from the original paper
A page that is near the top of both lists beats a page that is first in only one. That is usually what you want, but not always. Figure 3 lets you check every question.
How to read Figure 3
Start with q10 (gross margin in Q4): BM25 returns only p6, the cost table, because “q4” is its only match; the chart page p5 is first for MaxSim with 3.000; RRF puts p6 first (\(1/61 + 1/62 = 0.03252\)) and p5 second (\(1/61 = 0.01639\)), because p6 is in both lists. With \(k = 1\) the hybrid would fail. Now try q9 (revenue from 2019 to 2023): the image index has the gold page p4 at rank 1, but RRF drops it to rank 4, behind three pages that both lists liked a little. Then try q3 (the scanned auditor’s letter): BM25 returns nothing, the image index finds p9. Finally q4 (policy code in small print): BM25 is first with 8.093, while the image index ranks p8 second (1.400) behind the climate-policy page p7 (2.000), because the small print is hard to see.
Step 5: answer with a VLM and cite the page
The reader gets the top \(k\) pages and the question. In the text-only pipeline the reader is an LLM and gets the text layers. In the page-image pipeline it is a VLM and gets the page images, rendered again from the PDF at the reader’s resolution \(r_{\text{read}}\). In the hybrid pipeline it gets both: each page image followed by its text layer, which helps with small print and exact codes.
Images cost tokens. Qwen2-VL and Qwen2.5-VL use one visual token per \(28 \times 28\) pixels, after rounding each side to a multiple of 28, between 4 and 16,384 tokens per image by default [Certain]:
At 144 DPI an A4 page is \(1{,}191 \times 1{,}684\) pixels. \(1{,}684/28 = 60.1 \to 60\) and \(1{,}191/28 = 42.5 \to 43\): \(60 \times 43 = 2{,}580\) tokens. With the same rule, A4 costs 630 tokens at 72 DPI, 1,260 at 100 DPI and 2,772 at 150 DPI. Three pages at 100 DPI cost 3,780 tokens; at 150 DPI they cost 8,316. The text layer of a dense page is usually far fewer tokens than its image at 150 DPI [Likely].
The prompt must make citation easy. Put a short label right before each page image, keep pages of one document in page order, and ask for the page first.
- input: question, ranked page IDs, \(k\), reader resolution \(r_{\text{read}}\)
- content \(\leftarrow\) [ ]
- for each of the top \(k\) pages, in document and page order:
- append text “[report.pdf, page 5]” // the label the model must cite
- append the page image rendered at \(r_{\text{read}}\) // costs formula (221.5) tokens
- (hybrid only) append the page’s text layer
- append: “Answer from these pages only. First write the page label you used, then the answer. If no page contains the answer, write NOT FOUND.”
- answer \(\leftarrow\) VLM.generate(content); check that the cited page is one of the \(k\) pages // a cheap guard against invented citations
The toy reader. We do not run a VLM in the browser either. The toy uses rules that say, for each question, whether a reader can answer it if the gold page is in the top \(k\). The text reader can answer 8 of the 12 questions from text layers: it cannot read charts or the scan, and it misreads the two-level table on p6. One chart question (q12, the share of power tools) it can still answer, by computing 71 / 114 from the segment table. The VLM reader can answer every question if the page is sharp enough: 72 DPI is enough for normal text and simple tables, chart labels, the two-level table and the scan need 100 DPI, and the small print on p8 needs 150 DPI. These rules are our assumptions to show the trade-off; with a real VLM you measure them.
Step 6: compare the pipelines on text, tables and charts
For each question \(j\) with gold page \(g_j\), a pipeline is correct only if retrieval and reading both work. For a question type \(m\) (text, table or chart) with the set of questions \(\mathcal{Q}_m\):
- Read the symbols
- \(\text{top}_k(j)\)
- the first \(k\) pages the pipeline returns for question \(j\)
- \(\text{read}_j\)
- 1 if the reader gives the right answer when the gold page is in its input
- \(\text{R@}k_m, \text{Acc}_m\)
- retrieval recall and end-to-end accuracy for type \(m\); \(\text{Acc}_m \le \text{R@}k_m\) always
Report both numbers for every type, and the macro average \(\frac{1}{3}\sum_m \text{Acc}_m\), the mean over types. (Our toy set has 4 questions per type, so here the macro and micro averages are equal; on a real set with 80% text questions they are not.) The gap between \(\text{R@}k_m\) and \(\text{Acc}_m\) tells you whether to fix the retriever or the reader. Figure 4 runs the whole toy lab.
How to read Figure 4
The defaults (text-only, \(k = 3\)) give chart accuracy 0.25: q9 and q10 are red (BM25 never finds the chart pages), q11 is amber (BM25 finds the sustainability page because its paragraph says “emissions”, but the value is only in the chart), and only q12 is green. Table accuracy is 0.75: all 4 table pages are retrieved, but the reader misreads the two-level table (q6). Macro accuracy 0.583. Switch to page-image RAG: 0.917 at 100 DPI, using 3,780 image tokens per question. Lower the reader to 72 DPI: it drops to 0.417, below text-only, because charts, the scan and the two-level table become unreadable. Raise it to 150 DPI: 1.000, for 8,316 tokens. The hybrid at 100 DPI and \(k = 3\) also gets 0.917, but in a different way: it reads q4’s small print from the text layer, and it loses q9 because fusion pushed p4 to rank 4. Set \(k = 5\) and the hybrid reaches 1.000 already at 100 DPI.
Three lessons from the toy carry over to real documents, and match what the ColPali paper reports on its benchmark:
- Charts and scans need the page image. On ViDoRe (nDCG@5, average of ten sets), ColPali scores 81.3, a parsing pipeline with captions and BGE-M3 67.0, and OCR plus BM25 65.5 [Certain, as reported]. The ColPali README lists ColQwen2 (v1.0) at 89.3 on the same benchmark [Certain, as listed].
- Tables are mostly a reading problem. Born-digital tables are in the text layer, so text retrieval finds them; flattening loses the row and column structure, and complex headers then cause misreadings [Likely]. Check this on your own set by giving the reader the gold page directly.
- Resolution is a budget decision. The page-image pipeline only wins if the reader sees the page sharply enough, and sharpness costs tokens (formula 221.5).
The full program
The toy lab in one file: the 10 pages, the 12 questions, BM25, MaxSim, RRF, the toy reader and the evaluation of formula (221.6). It needs only PyTorch and is deterministic (no randomness at all). Figures 3 and 4 run the same computation in JavaScript.
import math, torch # 1) the toy report: (id, kind, text layer, extra words visible only in the pixels) PAGES = [ ("p1", "text", "letter from the ceo 2023 was a year of growth revenue grew and we opened a new factory in porto our team delivered record orders under a new code of conduct", ""), ("p2", "text", "company overview northwind was founded in 1987 in lyon we make power tools and garden tools we employ 1240 people in five countries", ""), ("p3", "table", "revenue by region table 1 region 2022 2023 north 41 44 south 28 31 east 19 24 west 12 15 total 100 114 eur million", ""), ("p4", "chart", "financial highlights see figure 1", "revenue 2019 2020 2021 2022 2023 eur million line chart trend growth"), ("p5", "chart", "quarterly results figure 2", "gross margin q1 q2 q3 q4 percent bar chart"), ("p6", "table", "cost breakdown table 2 operating cost q1 q2 q3 q4 materials 14 15 15 16 logistics 4 5 6 5 staff 9 9 10 10 eur million", ""), ("p7", "chart", "sustainability we cut emissions at every plant and moved porto to solar power under our climate risk policy see figure 3", "co2 emissions 2021 2022 2023 tonnes bar chart"), ("p8", "small", "risk factors currency risk policy code fx 17 interest rate risk policy code ir 4 supply risk single supplier for batteries", ""), ("p9", "scan", "", "independent auditor report signed by kessler partners lyon march 2024 opinion accounts 2023"), ("p10", "table", "product segments table 3 segment revenue power tools 71 garden tools 32 accessories 11 eur million", "share of revenue pie chart percent"), ] # (id, type, query words, gold page, text reader can answer?, min DPI the VLM reader needs) QUESTIONS = [ ("q1", "text", "northwind founded", "p2", True, 72), ("q2", "text", "company open new factory", "p1", True, 72), ("q3", "text", "firm signed auditor report", "p9", False, 100), ("q4", "text", "policy code currency risk", "p8", True, 150), ("q5", "table", "revenue south region 2023", "p3", True, 72), ("q6", "table", "logistics cost q3", "p6", False, 100), ("q7", "table", "revenue garden tools", "p10", True, 72), ("q8", "table", "total revenue 2022 regions", "p3", True, 72), ("q9", "chart", "revenue change 2019 2023", "p4", False, 100), ("q10", "chart", "gross margin q4", "p5", False, 100), ("q11", "chart", "co2 emissions 2022", "p7", False, 100), ("q12", "chart", "share revenue power tools", "p10", True, 100), ] ids = [p[0] for p in PAGES] text_layer = [p[2].split() for p in PAGES] LEGIBLE = {"small": 0.35, "scan": 0.8} # how well the retriever sees the words at its 448 px input visible = [[(w, LEGIBLE.get(k, 1.0)) for w in (t + " " + extra).split()] for _, k, t, extra in PAGES] # 2) toy "embedding" similarity: 1 for the same word, a few related pairs, else 0 vocab = sorted({w for d in visible for w, _ in d} | {w for q in QUESTIONS for w in q[2].split()}) V = {w: i for i, w in enumerate(vocab)} S = torch.eye(len(vocab), dtype=torch.float64) for a, b, s in [("sales", "revenue", .7), ("employees", "employ", .7), ("plant", "factory", .7), ("emissions", "co2", .7), ("costs", "cost", .9)]: if a in V and b in V: S[V[a], V[b]] = S[V[b], V[a]] = s # 3) BM25 on the text layer, formula (221.2) N, k1, b = len(PAGES), 1.2, 0.75 avgdl = sum(map(len, text_layer)) / N df = {} for d in text_layer: for w in set(d): df[w] = df.get(w, 0) + 1 def bm25(q): out = [] for d in text_layer: s = 0.0 for w in q: f = d.count(w) if f: idf = math.log(1 + (N - df[w] + 0.5) / (df[w] + 0.5)) s += idf * f * (k1 + 1) / (f + k1 * (1 - b + b * len(d) / avgdl)) out.append(s) return torch.tensor(out, dtype=torch.float64) # 4) late interaction (MaxSim) on what the page shows, formula (221.3) def maxsim(q): qi = torch.tensor([V[w] for w in q]) out = [] for d in visible: pj = torch.tensor([V[w] for w, _ in d]); wt = torch.tensor([x for _, x in d], dtype=torch.float64) out.append((S[qi][:, pj] * wt).max(dim=1).values.sum()) # best patch per query word, then sum return torch.stack(out) def ranking(scores): # pages with score 0 are not returned order = sorted(range(N), key=lambda i: (-scores[i].item(), i)) return [i for i in order if scores[i] > 0] def rrf(*lists, kappa=60): # formula (221.4) s = torch.zeros(N, dtype=torch.float64) for L in lists: for r, i in enumerate(L): s[i] += 1 / (kappa + r + 1) return ranking(s) # 5) retrieve, read, score per question type, formula (221.6) def run(k=3, dpi=100): acc = {"text": {}, "image": {}, "hybrid": {}} for qid, typ, words, gold, text_ok, need in QUESTIONS: q = words.split(); g = ids.index(gold) Lt, Li = ranking(bm25(q)), ranking(maxsim(q)) lists = {"text": Lt, "image": Li, "hybrid": rrf(Lt, Li)} reads = {"text": text_ok, "image": dpi >= need, "hybrid": text_ok or dpi >= need} for sname, L in lists.items(): hit = g in L[:k] acc[sname].setdefault(typ, []).append(hit and reads[sname]) if k == 3: print(qid, typ, "rank of gold page text:", Lt.index(g) + 1 if g in Lt else "-", " image:", Li.index(g) + 1 if g in Li else "-", " hybrid:", lists["hybrid"].index(g) + 1) for sname, d in acc.items(): per = {t: sum(v) / len(v) for t, v in d.items()} print(f"k={k} dpi={dpi} {sname:6s}", {t: round(x, 2) for t, x in per.items()}, "macro", round(sum(per.values()) / 3, 3)) run(k=3, dpi=100) run(k=1, dpi=72)
Output (we ran it with PyTorch 2.8): the rank of the gold page for text, image and hybrid is 1, 1, 1 for q1, q2 and q5 to q8; “–”, 1, 1 for q3; 1, 2, 2 for q4; “–”, 1, 4 for q9; “–”, 1, 2 for q10; and 1, 1, 1 for q11 and q12. At \(k = 3\) and 100 DPI the accuracies are text-only {text 0.75, table 0.75, chart 0.25}, macro 0.583; page-image {0.75, 1.0, 1.0}, macro 0.917; hybrid {1.0, 1.0, 0.75}, macro 0.917. At \(k = 1\) and 72 DPI: text-only 0.583, page-image 0.417, hybrid 0.5.
The real version
The same steps with real models. We did not run this block for this page (it needs a GPU and model downloads); the calls follow the official READMEs and model cards as of November 2026, so check them against the versions you install.
import glob, pathlib, torch, pypdfium2 as pdfium from PIL import Image from rank_bm25 import BM25Okapi from colpali_engine.models import ColQwen2, ColQwen2Processor # Step 1: render every page and keep its text layer (Algorithm 1) pages = [] for path in sorted(glob.glob("pdfs/*.pdf")): pdf = pdfium.PdfDocument(path) for i in range(len(pdf)): png = f"pages/{pathlib.Path(path).stem}_p{i+1}.png" pdf[i].render(scale=144 / 72).to_pil().save(png) text = pdf[i].get_textpage().get_text_range().replace("\r\n", "\n") pages.append({"doc": path, "page": i + 1, "png": png, "text": text}) # Step 2: BM25 on the text layers tok = lambda s: s.lower().split() bm25 = BM25Okapi([tok(p["text"]) for p in pages]) # Step 3: page-image index, one page at a time (no padding issues) name = "vidore/colqwen2-v1.0" model = ColQwen2.from_pretrained(name, torch_dtype=torch.bfloat16, device_map="cuda:0").eval() proc = ColQwen2Processor.from_pretrained(name) page_vecs = [] with torch.no_grad(): for p in pages: batch = proc.process_images([Image.open(p["png"])]).to(model.device) page_vecs.append(model(**batch)[0].to("cpu", torch.float16)) # (n_d, 128) # Step 4: retrieve with both and fuse ranks, formula (221.4) def retrieve(question, k=3, kappa=60): s_txt = bm25.get_scores(tok(question)) with torch.no_grad(): q = model(**proc.process_queries([question]).to(model.device)).to("cpu", torch.float16) s_img = proc.score_multi_vector(q, page_vecs)[0] # MaxSim, formula (221.3) fused = {} for scores in (list(s_txt), s_img.tolist()): order = [i for i in sorted(range(len(pages)), key=lambda i: -scores[i]) if scores[i] > 0] for r, i in enumerate(order): fused[i] = fused.get(i, 0.0) + 1 / (kappa + r + 1) return sorted(fused, key=lambda i: -fused[i])[:k]
Step 5 with Qwen2.5-VL as the reader. Free the retriever first (del model; torch.cuda.empty_cache()) if memory is tight. The page images are rendered again at the reader’s resolution, and every image gets a label.
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor from qwen_vl_utils import process_vision_info rname = "Qwen/Qwen2.5-VL-7B-Instruct" reader = Qwen2_5_VLForConditionalGeneration.from_pretrained(rname, torch_dtype="auto", device_map="auto") rproc = AutoProcessor.from_pretrained(rname, min_pixels=256 * 28 * 28, max_pixels=2880 * 28 * 28) def answer(question, top, dpi=150): content = [] for i in sorted(top, key=lambda i: (pages[i]["doc"], pages[i]["page"])): p = pages[i] png = p["png"].replace(".png", f"_{dpi}.png") pdfium.PdfDocument(p["doc"])[p["page"] - 1].render(scale=dpi / 72).to_pil().save(png) content += [{"type": "text", "text": f"[{pathlib.Path(p['doc']).name}, page {p['page']}]"}, {"type": "image", "image": "file://" + str(pathlib.Path(png).resolve())}] content.append({"type": "text", "text": "Answer from these pages only. First write the page label you used, " "then the answer. If no page contains the answer, write NOT FOUND.\nQuestion: " + question}) messages = [{"role": "user", "content": content}] text = rproc.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) images, videos = process_vision_info(messages) inputs = rproc(text=[text], images=images, videos=videos, padding=True, return_tensors="pt").to(reader.device) out = reader.generate(**inputs, max_new_tokens=128, do_sample=False) return rproc.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
Note max_pixels: we raised it to \(2{,}880 \times 28 \times 28\) pixels so that an A4 page at 150 DPI (2,772 tokens by formula 221.5) is not shrunk. With the model card’s example value of \(1{,}280 \times 28 \times 28\) the processor would scale every page down to about 1,280 tokens, which is close to 100 DPI. Step 6 is then a loop over your labeled questions: call retrieve with BM25 only, ColQwen2 only and both; record whether the gold page is in the top \(k\); call answer; compare the answer with the gold answer (exact match after normalizing numbers, or ANLS for strings); and fill the table of formula (221.6) per type. Also run answer with the gold page alone: that measures the reader without retrieval errors.
Where this is used
| Application | Hard pages | What the lab teaches for it |
|---|---|---|
| Financial and annual reports | charts of revenue and margins, tables with two-level headers | page-image retrieval for charts; check table reading with the gold page alone |
| Technical manuals | exploded-view diagrams, part numbers in small print | hybrid: BM25 for exact part numbers, page images for diagrams |
| Slide decks | slides that are mostly pictures, text inside images | one unit per slide image; the text layer is often almost empty |
| Scanned contracts, invoices, forms | no text layer at all | flag pages without text in ingestion; page images or OCR |
| Scientific papers | plots, equations, tables | figures are often vector graphics with partial text; still search the image |
| Lecture videos with slides | slide frames inside a video | treat sampled keyframes as “pages”, keep the timestamp as the citation; the same pipeline becomes Video-RAG (see session 210) |
| Meeting recordings and screen shares | dashboards and documents shown on screen | keyframe deduplication (as in near-duplicate detection) before indexing, then page-image retrieval on the frames |
Session 220 extends the reader of Step 5: instead of reading a fixed top \(k\), the model can search again, zoom into a region, and stop when it has enough evidence.
Common mistakes
- Trusting the text layer. Charts stored as pictures and scanned pages have no text (Step 1). Count, at ingestion, how many pages have almost no text but are not blank.
- Showing the reader the retriever’s image. The retriever’s small input is fine for matching and too small for reading numbers. Render again from the PDF at the reader’s DPI.
- Letting the processor shrink pages silently. A
max_pixelslimit lower than your page size scales the page down. Compute the tokens per page with formula (221.5) and set the limit on purpose. - Fusing raw scores. BM25 and MaxSim scores are on different scales. Fuse ranks (RRF) or normalize per list, and remember that RRF punishes pages only one index can see (q9).
- Losing the page pointer. Without (document, page) on every unit, the reader cannot cite and you cannot check answers.
- One overall score. A test set of mostly text questions hides a pipeline that never answers charts. Report retrieval and accuracy per type, and the macro average.
- Not separating retrieval from reading. Run the reader on the gold page alone. If accuracy is still low there, a better retriever will not help.
- Dropping BM25 after moving to images. Exact codes and small print (q4) are still found best by keywords.
Interview questions
Why does text-only RAG fail on PDFs with charts, even with a perfect text embedding model?basic
Walk through the pipeline you would build for QA over a folder of PDF reports.basic
Explain MaxSim and why it helps for small regions of a page.core
Text retrieval finds the right table page, but answers about that table are often wrong. Where is the problem and how do you confirm it?core
How does reciprocal rank fusion work, and when can it hurt?core
How many tokens does one retrieved page cost a Qwen2.5-VL reader, and how do you choose the resolution?core
max_pixels, which can silently shrink pages.Your page-image RAG beats text RAG on charts but loses on questions about part numbers. Why?advanced
How do you evaluate this system so you know whether to fix the retriever or the reader?advanced
The scanned pages in your collection are never retrieved by the text pipeline. What are your options and their trade-offs?advanced
How would you make this work for a collection of a million pages?advanced
Exercises
- A US Letter page is \(612 \times 792\) points. What is its pixel size at 144 DPI, and how many Qwen2.5-VL tokens does it cost by formula (221.5)?
- Page A is rank 2 in BM25 and rank 5 in the image list. Page B is rank 1 in the image list and absent from BM25. Which one does RRF with \(\kappa = 60\) put first?
- In a 10-page collection, word X appears in 1 page and word Y in 5 pages. Compute their idf by formula (221.2).
- In Figure 4, text-only RAG at \(k = 3\) has chart accuracy 0.25. For each of the 4 chart questions, say whether retrieval or reading failed.
- How much memory does a ColPali index for 2,000 pages need in float16, and with binary vectors (1 bit per number)?
- Change the toy program so that the image list counts twice in RRF (each image term is \(2/(\kappa + r)\)). Does q9’s gold page p4 get into the top 3? Then try another fix: give the reader the union of the top 2 pages of each list. Does that include p4?
Answerscheck
Key takeaways
- A PDF’s text layer is not the page: charts stored as pictures and scans have none. Check it at ingestion (Step 1).
- Index every page twice: BM25 on the text layer (221.2) for exact strings, and a ColPali-style late-interaction index on the page image (221.3) for charts, scans and layout. Keep (document, page) on every unit.
- Fuse ranks, not scores (221.4), and know that RRF can push down a page only one index can see.
- The reader needs the page re-rendered at its own resolution; each A4 page costs 1,260 tokens at 100 DPI and 2,772 at 150 DPI with Qwen2.5-VL (221.5).
- Score retrieval and reading separately, per question type, with the macro average (221.6). For charts, text RAG usually fails at retrieval; for tables, it usually fails at reading.
Key formulas
References for this session
- Faysse et al. (2024). ColPali: Efficient Document Retrieval with Vision Language Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 8: Document AI
- Khattab & Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Robertson & Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond
- Cormack, Clarke & Buettcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images
- Yu et al. (2024). VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
- Cho et al. (2024). M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
- Illuin Technology (2024). colpali-engine: ColPali and ColQwen2 retrievers and the ViDoRe benchmark (code and model list)
- pypdfium2 team (2024). pypdfium2: Python bindings to PDFium (documentation)
- D. Brown (2020). rank_bm25: a collection of BM25 algorithms in Python (code)