LLM & VLM courseBlog
218Track 8: Video and multimodal systems · session 18 of 21

Document AI

Documents carry meaning in their text and in where the text sits. This session compares OCR pipelines with OCR-free models such as Donut and modern VLMs, shows how layout, reading order, tables and charts are handled, how DocVQA and extraction are scored, and how ColPali finds the right page by embedding the page image as many small vectors.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on text embeddings and semantic search (150), RAG (152), high resolution in VLMs (180) and VLM case studies (182); basic PyTorch

What you will learn

  • The main document tasks (extraction, DocVQA, parsing to Markdown, classification, page retrieval) and how each one is scored
  • How an OCR pipeline works stage by stage, why its errors multiply, and how LayoutLM adds box positions to text tokens
  • Reading order with recursive XY-cut, table recognition with TEDS, and chart questions with relaxed accuracy
  • How OCR-free models (Donut, Pix2Struct, Qwen2-VL-style VLMs) read pages, and how many visual tokens a page costs at a given resolution
  • ANLS for DocVQA, field-level F1 for extraction, and where these metrics mislead
  • ColPali: late-interaction scoring over 1,030 patch vectors per page, its hardest-negative loss, and the storage cost of multi-vector indexes
  • How to choose a document system for a given workload, and how to make its outputs checkable

Why this matters

A large part of the world’s useful information lives in documents: invoices, contracts, bank statements, forms, scientific papers, slide decks, manuals, medical reports. They are made for human eyes. The meaning is in the text, but also in where the text sits: which number is under which column, which label belongs to which box, which caption goes with which chart.

Document AI (also called visually-rich document understanding) is the set of methods that let a machine read such pages. The common tasks are:

  • Information extraction: fill a fixed set of fields from a page, for example vendor, date, total and line items from an invoice. The output is JSON.
  • Document question answering (DocVQA): answer a free question about a page, such as “What is the due date?”.
  • Document parsing: turn a whole page into clean, structured text (usually Markdown or HTML) with headings, lists, tables and formulas in the right order, so that an LLM or a search index can use it.
  • Classification: is this page an invoice, a letter or a form?
  • Retrieval: find the right page among millions, for a question.

For a long time all of these started with OCR (optical character recognition: turning pixels of text into characters). Today there are two families. OCR pipelines chain specialized models: find text, read it, find the layout, put it in reading order, then extract. OCR-free models such as Donut and modern vision-language models (VLMs) read the page image directly and write the answer. And for search, ColPali showed that you can embed the page image itself, with one vector per image patch, and skip text extraction for retrieval.

The previous session, session 217, worked on pixels that move. This session works on pixels that hold text. The two meet more often than you may think: lecture videos are slide decks over time, news videos carry tickers, sports videos show scoreboards. The next session, session 219, builds on the page retrieval part of this one. This session follows chapter 8 of the VLM book (Document AI).

What makes documents different

A photo of a dog has one main object. A page has hundreds of small objects (words), and their positions carry meaning. Three examples:

  • In a table, “1,240” means nothing until you know its row (“Q3”) and column (“Revenue, $k”).
  • In a form, the value “Keller” belongs to the label on its left, “Last name”, not to the label above it.
  • In a two-column paper, line 1 of the left column continues on line 2 of the left column, not on line 1 of the right column.

There are also two very different kinds of input. A born-digital PDF (made by Word, LaTeX, a web page) already contains the characters and their coordinates; we can read them without any OCR, although the reading order and the table structure are still lost. A scanned or photographed page is only pixels: skewed, blurred, stamped, handwritten. Many real archives mix both.

TaskOutputUsual metricExample dataset
Key information extractionJSON with fixed fieldsfield-level F1, tree-edit accuracyCORD (receipts), FUNSD (forms)
Document VQAshort text answerANLSDocVQA, InfographicVQA
Chart QAnumber or short textrelaxed accuracy (5% tolerance)ChartQA
Table recognitionHTML or Markdown tableTEDSPubTabNet
Layout analysisboxes with classesmAP (as in detection)DocLayNet
Page parsingMarkdown / HTML of the pageedit distance, TEDS for tables, reading-order scoresacademic PDFs (Nougat)
Page retrievalranked list of pagesnDCG@5, recall@kViDoRe

Each row has its own section below. DocLayNet, for scale, has 80,863 human-annotated pages with 11 layout classes (for example text, title, table, picture, caption, page-header) [Certain].

The OCR pipeline

The classic system is a chain of specialized models. Each stage is small, fast and can be checked on its own.

Algorithm 1: a classic OCR pipeline, page image to fields
  1. input: page image \(I\) (or a PDF page), list of target fields
  2. preprocess: deskew, denoise, convert to gray, render the PDF at about 300 dpi
  3. text detection: boxes around every word or line // a detector such as DB or CRAFT
  4. text recognition: read each box into characters, with a confidence // a CRNN or TrOCR-style model
  5. layout analysis: boxes for title, paragraph, table, figure, caption, header, footer // a detector trained on DocLayNet-style data
  6. reading order: sort the blocks and lines into the order a human reads them // XY-cut or a learned model
  7. structure: run a table-structure model on table regions; formula and chart models on their regions
  8. extract: feed words + boxes to a layout-aware model (LayoutLM), or the serialized text to an LLM with the field list
  9. return fields, each with its source box and confidence

The strengths are real: each output word has an exact box (good for highlighting and audit), confidence scores are available per word, stages can be improved one at a time, and the models are small. The weakness is that errors compound. If each stage is right with probability \(p_s\), and we assume the errors are independent, a field survives the whole chain with probability

\[ P_{\text{field}} \approx \prod_{s=1}^{S} p_s, \qquad P_{\text{all } K \text{ fields}} \approx P_{\text{field}}^{\,K}, \qquad P_{\text{exact string}} \approx c^{\,L} \tag{218.1} \]
Read the symbols
\(S\)
number of stages in the chain (detection, recognition, layout and order, extraction)
\(p_s\)
probability that stage \(s\) does not break this field
\(K\)
number of fields we need from one page
\(c\), \(L\)
per-character accuracy and the length of the string in characters
Worked example: 99% is not good enough

An invoice number has \(L = 12\) characters. With a character accuracy of \(c = 0.99\), the whole number is exactly right with probability \(0.99^{12} = 0.886\). At \(c = 0.98\) it drops to \(0.785\); at \(c = 0.999\) it is \(0.988\).

Now the chain: detection 0.99, recognition 0.97, layout and order 0.98, extraction 0.96 gives \(P_{\text{field}} = 0.903\). A page with \(K = 10\) fields is fully correct only with probability \(0.903^{10} = 0.362\). An end-to-end model that gets each field right with probability 0.95 gives \(0.95^{10} = 0.599\). The independence assumption is rough (a bad scan hurts every stage at once), but the lesson holds: in a chain, small losses multiply.

Figure 1. Two ways from a page to JSON. Top: the OCR pipeline, a chain of four models. Bottom: one OCR-free model. The bars show the chance that one field, and that all \(K\) fields of a page, come out right, using formula (218.1).
How to read Figure 1

Move one stage slider down to 0.9 and watch the page-level number fall much faster than the field-level number: the power \(K\) amplifies every loss. Then set \(K = 1\): for a single field, a good pipeline and a good end-to-end model are close. The pipeline is not “worse”; it needs every stage to be very good. This is why production pipelines spend most of their effort on the weakest stage, and why they keep per-word confidences to send uncertain pages to a human.

Adding layout to a text model: LayoutLM

After OCR we have words and boxes. A plain text model (BERT) sees only the words in a 1-D order, so it loses the table and form structure. LayoutLM (Xu et al., 2019) adds the box to every token. First the box corners are scaled to a virtual page of size 1000 × 1000, so that page size does not matter:

\[ \tilde{x} = \left\lfloor 1000 \cdot \frac{x}{W} \right\rfloor, \qquad \tilde{y} = \left\lfloor 1000 \cdot \frac{y}{H} \right\rfloor \tag{218.2} \]

Then the input vector of token \(i\) is the sum of its text embedding, its 1-D position embedding and four 2-D position embeddings, one for each box coordinate:

\[ \mathbf{h}_i^{(0)} = \mathbf{E}_{\text{tok}}[w_i] + \mathbf{E}_{\text{pos}}[i] + \mathbf{X}[\tilde{x}_{0,i}] + \mathbf{Y}[\tilde{y}_{0,i}] + \mathbf{X}[\tilde{x}_{1,i}] + \mathbf{Y}[\tilde{y}_{1,i}] \tag{218.3} \]
Read the symbols
\(W, H\)
page width and height in pixels or PDF points
\((\tilde{x}_0, \tilde{y}_0), (\tilde{x}_1, \tilde{y}_1)\)
the top-left and bottom-right corners of the word’s box, on the 0 to 1000 scale
\(\mathbf{X}, \mathbf{Y}\)
learned tables: one shared for x-coordinates, one shared for y-coordinates
\(\mathbf{E}_{\text{tok}}, \mathbf{E}_{\text{pos}}\)
the usual BERT token and position embeddings (session 11)

The model is then pre-trained like BERT with masked words, but it keeps the boxes of masked words, so it must use the layout to guess them. On the FUNSD form-understanding benchmark, LayoutLM-base reached an F1 of 0.787, against about 0.603 for BERT-base with the same OCR text [Certain]. That jump is the value of knowing where each word is. LayoutLMv2 and v3 later added image features of the page too. The limit stays: everything depends on the OCR output given to the model.

Layout, reading order, tables and charts

Reading order

To give a page to an LLM, we must turn a 2-D page into a 1-D sequence. The order matters: an LLM reads tokens in order, and a sentence broken across two columns becomes nonsense if we interleave them. The naive rule, “sort lines from top to bottom, then left to right” (raster order), fails on every multi-column page.

A classic fix is the recursive XY-cut (Nagy and Seth, 1984): look for a straight band of white space that cuts the page fully from side to side. Cut there. Repeat inside each part. If both a horizontal and a vertical cut exist, take the wider gap, because a column gutter is usually wider than the space between two lines.

Algorithm 2: recursive XY-cut for reading order
  1. function XYCUT(boxes)
  2.   if one box or none: return boxes
  3.   \(g_y \leftarrow\) widest horizontal empty band that crosses all boxes // projection of boxes on the y-axis
  4.   \(g_x \leftarrow\) widest vertical empty band // projection on the x-axis
  5.   if neither exists: return boxes sorted by (top, left)
  6.   if \(g_x\) is wider than \(g_y\): split into left and right, return XYCUT(left) + XYCUT(right)
  7.   else: split into top and bottom, return XYCUT(top) + XYCUT(bottom)
Serialized text an LLM would read
Figure 2. A two-column page with a title, a figure with caption, a table, a closing paragraph and a footer. Each line or block is numbered in the order the algorithm reads it. Lines read out of order are marked in red in the text on the right.
How to read Figure 2

In raster order, line 1 of the left column is followed by line 1 of the right column, so two different sentences are woven together. The readout counts inverted pairs: pairs of lines that come in the wrong order compared with how a person reads the page (20 items give 190 pairs). The dashed lines are the cuts XY-cut makes with a gap of 10 points or more: first the footer, then the table, then the title, then the column gutter. The table and the figure are treated as single blocks: a layout model found them first, and a separate table model reads the table cells.

XY-cut fails on pages with no clean white-space bands (magazine layouts, text wrapped around pictures, forms). Modern pipelines use learned reading-order models, or let a VLM write the page in order directly. Page furniture (headers, footers, page numbers) should usually be dropped or tagged, so it does not break a sentence that continues on the next page.

Tables

A table is a tree: table, rows, cells, cell text. Table recognition must find the cell grid, including merged cells that span rows or columns, and the text in each cell. The output is usually HTML (which can express spanning cells) or Markdown (which cannot). The standard metric is TEDS (tree-edit-distance-based similarity, from the PubTabNet paper): write both tables as HTML trees and count the edits that turn one into the other:

\[ \text{TEDS}(T_a, T_b) = 1 - \frac{\text{EditDist}(T_a, T_b)}{\max(|T_a|, |T_b|)} \tag{218.4} \]

Here \(|T|\) is the number of nodes in a tree, and an edit is inserting, deleting or renaming one node (renaming a cell node costs its normalized text distance). A predicted table with 3 edits against a reference of 20 nodes scores \(1 - 3/20 = 0.85\). TEDS punishes a wrong structure (a shifted column) much more than a typo inside a cell, which is what users care about: a value in the wrong column is a wrong value.

Charts

A chart stores numbers as lengths, angles and colors. Reading it needs fine visual detail and often arithmetic (“how much higher is 2023 than 2021?”). The ChartQA benchmark uses relaxed accuracy for numeric answers: a predicted number \(\hat{a}\) counts as correct if it is within 5% of the true number \(a\):

\[ \text{correct} \iff \frac{|\hat{a} - a|}{|a|} \le 0.05 \tag{218.5} \]

For \(a = 43\), the accepted range is \(40.85\) to \(45.15\): \(41.2\) (error 4.2%) is accepted, \(40.5\) (5.8%) is not. Text answers still need an exact match. Some systems first convert a chart into its data table (“chart derendering”), then answer from the table; this makes the numbers inspectable.

Parsing a page to Markdown

For RAG and for LLM training data, the most common need is “give me this PDF as clean Markdown”. Three routes exist today:

  1. Pipeline parsers: the stages of Algorithm 1, then a writer that emits headings, lists, Markdown tables and LaTeX formulas in reading order. Open-source tools in this family combine a layout detector, OCR (skipped for born-digital text), a table model and a formula model.
  2. Specialized end-to-end models: Nougat (Blecher et al., 2023) is a Swin encoder with an mBART-style decoder that reads an academic page image and writes Markdown with LaTeX math [Likely for the exact decoder]. It was trained on arXiv papers, where the source gives the target text for free.
  3. General VLMs with a prompt: “Convert this page to Markdown. Keep tables as HTML.” Strong VLMs now do this well on clean pages. The risks are the generation risks of any LLM: skipped lines, repeated lines (a decoder stuck in a loop), and, most dangerous, plausible text that is not on the page.

OCR-free models: Donut, Pix2Struct and VLMs

Donut (Document Understanding Transformer, Kim et al., 2021) removed OCR from the loop. It is an encoder-decoder: a Swin Transformer encoder reads the page image (2560 × 1920 pixels in pre-training), and a 4-layer BART decoder writes tokens [Certain]. Everything is one sequence task, trained with the usual next-token cross-entropy:

\[ \mathcal{L}(\theta) = -\sum_{t=1}^{T} \log p_\theta\!\left(y_t \mid y_{\lt t},\ \text{Enc}_\theta(I)\right) \tag{218.6} \]
Read the symbols
\(I\)
the page image; \(\text{Enc}_\theta(I)\) is the grid of encoder output vectors that the decoder attends to with cross-attention
\(y_1, \dots, y_T\)
the target token sequence: all page text in pre-training, a structured string in fine-tuning

Pre-training teaches the model to read: the target is all the text on the page, from top-left to bottom-right. Donut used 11 million scanned pages (IIT-CDIP) and about 2 million synthetic pages from its generator SynthDoG, in four languages [Certain]. Fine-tuning teaches it to structure: the target becomes a tagged string that maps one-to-one to JSON. The paper writes the tags as [START_name] … [END_name]; the released code uses tokens like <s_total> [Likely]. For a receipt:

<s_menu><s_nm>Latte</s_nm><s_price>4.50</s_price></s_menu><s_total>12.50</s_total>
->  {"menu": {"nm": "Latte", "price": "4.50"}, "total": "12.50"}

Classification and question answering use the same trick: the answer is just another tagged string. On the CORD receipt benchmark, Donut (143M parameters) reached 84.1 field F1 at 1.2 s per image, against 78.9 F1 at 1.7 s for LayoutLMv2 with OCR, on the paper’s P40 GPU setup [Certain]. On DocVQA it was weaker (67.5 ANLS against 78.1 for LayoutLMv2) [Certain]: free questions about dense text still needed better reading than a 2021 encoder had.

Pix2Struct (Lee et al., 2022) pre-trained on a different free signal: predict a simplified HTML of a web page from its masked screenshot. It also introduced variable-resolution input: scale the image to fit a patch budget while keeping the aspect ratio, instead of squashing it to a square [Certain]. Both ideas are now standard in VLMs.

Modern VLMs (Qwen2-VL and Qwen2.5-VL, InternVL, and others; see session 182) are OCR-free by design. They were trained on huge mixtures of OCR, chart, table and document data, and they take instructions, so one model does extraction, QA and parsing. Qwen2-VL-72B reports 96.5 ANLS on the DocVQA test set (GPT-4o: 92.8) [Certain, as reported in the Qwen2-VL paper]. For comparison, humans score 98.1 [Certain, from the DocVQA paper].

Resolution is the whole game

Text is small. A 10-point letter is \(10/72\) of an inch tall (the “em” size). If the model sees the page at too low a resolution, the letters are a few pixels tall and no model can read them. But higher resolution means more visual tokens, and attention cost grows with the number of tokens. In Qwen2-VL, the vision encoder uses 14 × 14 pixel patches, and an MLP merges each 2 × 2 group into one token, so one token covers 28 × 28 pixels [Certain]. The image is first resized to multiples of 28 (and shrunk if it exceeds a pixel budget):

\[ N_{\text{tok}} = \frac{\bar{H}}{28} \cdot \frac{\bar{W}}{28}, \qquad \bar{H} = 28 \cdot \operatorname{round}\!\left(\frac{H}{28}\right), \quad \bar{W} = 28 \cdot \operatorname{round}\!\left(\frac{W}{28}\right) \tag{218.7} \]

If \(\bar{H}\bar{W}\) exceeds the budget \(P_{\max}\), both sides are first divided by \(\beta = \sqrt{HW / P_{\max}}\) and rounded down to multiples of 28. The default budget in the released Qwen2-VL processor is 12,845,056 pixels, which is 16,384 tokens [Certain, from its preprocessor_config.json].

Worked example: the token price of one A4 page

A4 is 210 × 297 mm, or 8.27 × 11.69 inches. Rendered at 150 dpi, it is 1240 × 1754 pixels. Rounded to multiples of 28: 1232 × 1764, so \(N_{\text{tok}} = 44 \times 63 = 2{,}772\) tokens. The em size of 10-point text is \(150 \times 10/72 = 20.8\) pixels (21.0 after the small resize), so a letter spans less than one token.

At 300 dpi (what Tesseract’s documentation recommends for OCR [Certain]) the page is 2480 × 3508 and costs 11,125 tokens. A 100-page report at 150 dpi costs 277,200 tokens: too much for most context windows and slow even when it fits. This is why page retrieval comes before reading, the topic of the next section.

Squash the same page to a fixed 448 × 448 square (as in PaliGemma-448, the base of ColPali) and the vertical resolution falls to 38 dpi: the em of 10-point text is about 5.3 pixels. Fine print is not readable at that size, although large titles and the general look of the page still are.

Figure 3. What an A4 page costs, and what text looks like, at a given resolution. The sample line is drawn at the true pixel size the model gets, then enlarged without smoothing so you see each pixel. Native mode follows formula (218.7) with a token budget (\(2^8 = 256\) to \(2^{14} = 16{,}384\) tokens).
How to read Figure 3

Start at 150 dpi and 10 pt: 2,772 tokens and an em of about 21 pixels, clearly readable. Drop the budget to 1,024 tokens: the page is shrunk to 728 × 1064 (988 tokens) and the em falls to about 12.6 pixels; the digits get visibly coarser. Switch to the fixed square: about the same token count (1,024), but now the page is squeezed, the em is about 5 pixels, and the text is a gray smear. Same token count, very different legibility: keeping the aspect ratio matters. The numbers are exact; the picture depends a little on your browser’s font.

Other ways to pay for resolution exist: cutting the page into tiles plus a small overview image (InternVL-style dynamic tiling), and compressing visual tokens after the encoder (session 180, session 183).

DocVQA and information extraction

DocVQA

DocVQA (Mathew et al., 2020) has 50,000 questions on 12,767 document images taken from the UCSF Industry Documents Library, split 80/10/10 into train, validation and test [Certain]. The answer is usually a span of text on the page: a date, a name, an amount. Questions cover forms, tables and lists, layout, running text, figures, handwriting and yes/no.

Exact match is too harsh for this: a model that reads “Dr Maria Keller” for “Dr. Maria Keller” has understood the page. But plain string similarity is too soft: a wrong answer that happens to share letters should get nothing. The metric ANLS (Average Normalized Levenshtein Similarity, introduced for scene-text VQA by Biten et al., 2019) takes the middle road. The Levenshtein distance \(\text{Lev}(a, b)\) is the minimum number of single-character insertions, deletions and substitutions that turn \(a\) into \(b\). Normalize it and cut it off at a threshold \(\tau\):

\[ \text{NL}(g, o) = \frac{\text{Lev}(g, o)}{\max(|g|, |o|)}, \qquad s(g, o) = \begin{cases} 1 - \text{NL}(g, o) & \text{if } \text{NL}(g, o) \lt \tau \\ 0 & \text{otherwise} \end{cases} \tag{218.8} \]
\[ \text{ANLS} = \frac{1}{N} \sum_{i=1}^{N} \max_{j} \; s\!\left(g_{ij}, o_i\right), \qquad \tau = 0.5 \tag{218.9} \]
Read the symbols
\(g_{ij}\)
the \(j\)-th accepted answer for question \(i\) (several are allowed, for example “$1,234.56” and “1234.56”)
\(o_i\)
the model’s answer to question \(i\); both are lower-cased and their spaces cleaned before comparison
\(|g|\)
length in characters
\(\tau\)
the cut-off: answers more than half wrong get 0, not partial credit
#Accepted answersModel answerScore
Figure 4. An ANLS calculator for six DocVQA-style questions about one invoice. Click a row to select it, then edit its answer. The score of each row is formula (218.8), maximized over the accepted answers; the readout gives ANLS (218.9) and exact match.
How to read Figure 4

Row 1 shows the weak side of ANLS: “INV-2023-0041” is a different invoice number, but it is one character away, so it earns 0.923. Row 4 shows the other side: “03/03/2024” is the right date in another format, and it earns 0. Lower \(\tau\) to 0.1 and only near-perfect strings score; raise it to 1 and every answer gets partial credit, even a wrong company. For IDs, amounts and dates in production, use exact match after normalization (parse the date, parse the number); keep ANLS for comparing with published results.

Worked example: three rows of Figure 4 by hand

Row 2: gold “$1,234.56” (9 characters), model “1,234.56”: one deletion, \(\text{NL} = 1/9 = 0.111\), score \(0.889\). The second gold “1234.56” is also one edit away, but over 8 characters: \(1 - 1/8 = 0.875\). The max is 0.889.

Row 3: gold “Dr. Maria Keller” (16 characters), model “Maria Keller”: 4 deletions, \(\text{NL} = 0.25\), score \(0.75\).

Row 4: gold “March 3, 2024” (13 characters), model “03/03/2024”: 8 edits, \(\text{NL} = 0.615 \gt 0.5\), score 0. The six rows average to ANLS = 0.594, while exact match is 1/6 = 0.167.

Information extraction

Extraction has a fixed schema, so we can score each field. A predicted field counts as correct only if its key and its full value match the reference. With \(\text{TP}\) correct predicted fields:

\[ P = \frac{\text{TP}}{\#\text{predicted fields}}, \qquad R = \frac{\text{TP}}{\#\text{reference fields}}, \qquad F_1 = \frac{2PR}{P + R} \tag{218.10} \]

Example: the model outputs 9 fields, 8 are right, the reference has 10: \(P = 0.889\), \(R = 0.8\), \(F_1 = 0.842\). Field F1 is strict (one wrong character fails the field) and ignores nesting; Donut also reports a tree-edit-distance accuracy, \(\max(0, 1 - \text{TED}(\text{pred}, \text{gt}) / \text{TED}(\varnothing, \text{gt}))\), which gives credit for a mostly-right nested structure such as a list of line items [Certain].

With VLMs, extraction is a prompt plus a schema: “Return JSON with keys vendor, date (YYYY-MM-DD), total (number), line_items[]”. Two practices make it reliable. Constrained decoding forces the output to be valid JSON for the schema (session 146). Grounding asks for the box or the quoted source text of each value, so a cheap check can confirm the value is really on the page (for born-digital PDFs, look it up in the PDF text layer).

Page retrieval with multi-vector embeddings: ColPali

Before you can read the right page, you must find it. The text route is: parse every PDF (OCR, layout, tables, maybe captions for figures), chunk the text, embed the chunks, search (session 152). Every parsing error becomes a retrieval error, and charts or diagrams are mostly lost.

ColPali (Faysse et al., 2024) skips parsing for retrieval. It feeds the page image to PaliGemma-3B, a VLM at 448 × 448 input, and keeps the output vector of every image patch: 1,024 patches (a 32 × 32 grid of 14-pixel patches) plus 6 prompt tokens, each projected to 128 dimensions [Certain]. A query is encoded by the same model into one 128-d vector per query token. The score is the late interaction of ColBERT (session 150): every query token finds its best-matching patch, and these best matches are summed:

\[ \text{LI}(q, d) = \sum_{i=1}^{N_q} \max_{j \in \{1, \dots, N_d\}} \left\langle \mathbf{E}_q^{(i)}, \mathbf{E}_d^{(j)} \right\rangle \tag{218.11} \]
Read the symbols
\(\mathbf{E}_q^{(i)}\)
the 128-d vector of query token \(i\) (\(N_q\) tokens in the query)
\(\mathbf{E}_d^{(j)}\)
the 128-d vector of page patch \(j\) (\(N_d = 1{,}030\) vectors per page in ColPali)
\(\langle \cdot, \cdot \rangle\)
dot product (the vectors are normalized, so it is a cosine)

Why this beats one vector per page: a page is mostly irrelevant to any given query. Average-pooling all patches into one vector (the “BiPali” baseline in the paper) mixes the small relevant region with everything else. MaxSim lets each query word look only at its best spot. The model is trained on query-page pairs with an in-batch contrastive loss that compares each positive pair with the hardest negative page in the batch:

\[ \mathcal{L} = \frac{1}{b} \sum_{k=1}^{b} \log\!\left(1 + e^{\,s_k^- - s_k^+}\right), \qquad s_k^+ = \text{LI}(q_k, d_k), \quad s_k^- = \max_{l \ne k} \text{LI}(q_k, d_l) \tag{218.12} \]

The function \(\log(1 + e^{z})\) is the softplus: near 0 when the positive beats the hardest negative by a wide margin, and growing linearly when it loses. With \(s^+ = 2.3\) and \(s^- = 1.4\) the loss is \(0.341\); with \(s^+ = 1.2\) and \(s^- = 1.5\) it is \(0.854\). Training used about 118,695 query-page pairs, 37% of them synthetic queries written by an LLM, with LoRA on the language model [Certain].

Query tokens (click one to see its similarity map):
similarity to the selected token (darker = higher)best patch for that token (the max in formula 218.11)
Figure 5. Late interaction on a toy index of three pages. Each page is a 4 × 5 grid of patches; each patch vector is built from the concepts written in it plus noise (32 dimensions, fixed seed). The readout compares the MaxSim score (218.11) with a single mean-pooled vector per page.
How to read Figure 5

For “revenue chart 2023”, click each token: “revenue” lights up the title patch of page A, “chart” the chart patches, “2023” the chart patch with the year. The page score is the sum of the three outlined maxima. Now pick “staff growth”: MaxSim finds the two small patches on page C (the letter) and ranks it first. The mean-pooled vectors rank page B (a table) first, because on page C the two useful patches are averaged with 18 patches of ordinary text. That is the dilution problem late interaction solves. Note also that page scores are never near 0: a max over many patches always finds something a little similar.

The paper’s numbers: on its ViDoRe benchmark (academic sets such as DocVQA, InfoVQA and TAT-DQA, and practical sets on energy, government, healthcare and AI documents), ColPali reached an average nDCG@5 of 81.3, against 67.0 for the best text pipeline (Unstructured parsing with figure captioning and BGE-M3 embeddings) and 58.8 for BiPali [Certain]. Indexing took about 0.39 s per page against 7.22 s for the parsing pipeline, on the authors’ hardware [Certain]. A Qwen2-VL-based variant, ColQwen2, added 5.3 points [Certain, as reported in the ColPali paper’s later version].

The cost: storage

Multi-vector indexes are large. The size of an index with \(M\) pages is

\[ \text{bytes} = M \cdot N_d \cdot D \cdot b \tag{218.13} \]

with \(N_d\) vectors per page, \(D\) dimensions and \(b\) bytes per number.

Worked example: one million pages

ColPali in float16: \(1030 \times 128 \times 2 = 263{,}680\) bytes = 257.5 KiB per page, matching the paper. One million pages: 263.7 GB. One 1024-d float16 vector per page would be 2,048 bytes: 129 times smaller.

Three ways to shrink it: token pooling (cluster similar patch vectors and average them; the paper reports that a pool factor of 3, which removes 66.7% of vectors, keeps about 97.8% of retrieval quality [Certain]): \(\lceil 1030/3 \rceil = 344\) vectors, 86 KiB per page. Binary quantization: 128 bits = 16 bytes per vector, 16.1 KiB per page, 16.5 GB for a million pages. Two-stage search: a cheap single-vector or text search finds a few hundred candidates, and MaxSim reranks only those (session 151).

Algorithm 3: visual page retrieval and answering (index once, query many times)
  1. index: for each PDF page: render to an image; \(\mathbf{E}_d \leftarrow\) ColPali(image) // about \(1{,}030 \times 128\) per page
  2.   optional: pool or binarize \(\mathbf{E}_d\); store with (document id, page number)
  3. query: \(\mathbf{E}_q \leftarrow\) ColPali(query text) // one vector per query token
  4. candidates \(\leftarrow\) top 200 pages by a cheap first stage // pooled vectors or BM25 on any text layer
  5. rank candidates by \(\text{LI}(q, d)\), formula (218.11); keep the top \(k\) (say 3)
  6. answer: send the \(k\) page images plus the question to a VLM; ask it to cite the page

This is document RAG without a text parser, and it is the core of the next session, session 219.

Choosing a system

There is no single best design. Chapter 8 of the VLM book has its own section on choosing a model; the table below is a simple rule of thumb: start from the documents and from the cost of an error.

SituationGood defaultWhy
Born-digital PDFs, need text for searchRead the PDF text layer; layout model only for order and tablesNo OCR errors at all; cheap
Scans, fixed forms, millions per dayOCR pipeline + small layout-aware extractor, or a fine-tuned small OCR-free modelCheap per page, boxes and confidences for audit
Varied layouts, free questions, chartsA strong VLM at native resolutionReads layout, charts and handwriting in one model
Search over many visually rich pagesColPali-style page retrieval, then a VLM on the top pagesNo parsing step; keeps figures and tables
High-stakes values (amounts, IDs)Any of the above + grounding + validation rules + human review on low confidenceFluent wrong outputs are the main risk

Code: tokens per page, ANLS, late interaction and the ColPali loss

A small, deterministic script. It needs only PyTorch. The comments show the printed output.

import math, torch
import torch.nn.functional as F
torch.manual_seed(0)

# 1) How many visual tokens does a page cost? (Qwen2-VL resize rule, formula 218.7)
def page_tokens(h, w, factor=28, min_px=3136, max_px=12845056):
    hb, wb = round(h / factor) * factor, round(w / factor) * factor
    if hb * wb > max_px:                      # too big: shrink, keep aspect ratio
        beta = math.sqrt(h * w / max_px)
        hb, wb = math.floor(h / beta / factor) * factor, math.floor(w / beta / factor) * factor
    elif hb * wb < min_px:                    # too small: grow
        beta = math.sqrt(min_px / (h * w))
        hb, wb = math.ceil(h * beta / factor) * factor, math.ceil(w * beta / factor) * factor
    return hb, wb, (hb // factor) * (wb // factor)

print(page_tokens(1754, 1240))               # A4 at 150 dpi -> (1764, 1232, 2772)

# 2) ANLS, formulas 218.8 and 218.9
def lev(a, b):
    d = list(range(len(b) + 1))
    for i, ca in enumerate(a, 1):
        prev, d[0] = d[:], i
        for j, cb in enumerate(b, 1):
            d[j] = min(prev[j] + 1, d[j - 1] + 1, prev[j - 1] + (ca != cb))
    return d[-1]

def anls(golds_list, preds, tau=0.5):
    norm = lambda s: " ".join(s.lower().split())
    total = 0.0
    for golds, p in zip(golds_list, preds):
        best = 0.0
        for g in golds:
            g2, p2 = norm(g), norm(p)
            nl = lev(g2, p2) / max(len(g2), len(p2), 1)
            best = max(best, 1 - nl if nl < tau else 0.0)
        total += best
    return total / len(preds)

print(round(anls([["INV-2023-0047"], ["March 3, 2024"]], ["INV-2023-0041", "03/03/2024"]), 4))  # 0.4615

# 3) Late interaction (ColPali scoring, formula 218.11) for a batch of queries and pages
def late_interaction(q, d):
    # q: (B, Nq, D) query token vectors, d: (C, Nd, D) page patch vectors, all L2-normalized
    sim = torch.einsum("bnd,cmd->bcnm", q, d)   # every query token vs every patch
    return sim.max(dim=3).values.sum(dim=2)     # max over patches, sum over query tokens -> (B, C)

B, Nq, Nd, D = 4, 6, 1030, 128               # 1030 vectors of 128 dims per page, as in ColPali
q = F.normalize(torch.randn(B, Nq, D), dim=-1)
d = F.normalize(torch.randn(B, Nd, D), dim=-1)
d[:, :Nq] = q + 0.05 * torch.randn(B, Nq, D)  # plant each query's tokens on its own page
d = F.normalize(d, dim=-1)
S = late_interaction(q, d)                   # (4, 4); diagonal = matching pairs
print(S.argmax(dim=1).tolist(), [round(x, 2) for x in S.diag().tolist()])
# [0, 1, 2, 3] [5.19, 5.2, 5.2, 5.26]; off-diagonal scores are about 1.7, not 0

# 4) ColPali training loss, formula 218.12: softplus(hardest in-batch negative - positive)
pos = S.diag()
neg = S.masked_fill(torch.eye(B, dtype=torch.bool), float("-inf")).max(dim=1).values
loss = F.softplus(neg - pos).mean()
print(round(loss.item(), 4))                 # 0.0308

# 5) Storage per page in float16, formula 218.13
print(Nd * D * 2, "bytes =", Nd * D * 2 / 1024, "KiB")  # 263680 bytes = 257.5 KiB

Two things to notice in the output. The matching pairs score about 5.2 while the non-matching pairs score about 1.7, not 0: with 1,030 patches, each of the 6 query tokens always finds some patch with a cosine near 0.29 by chance. Scores of different queries are therefore not comparable on an absolute scale; only the ranking for one query is. And the loss is small (0.031) because every positive beats its hardest negative by more than 3.

Where this is used

ApplicationTask and method
Accounts payable, expense appsInvoice and receipt extraction to JSON; OCR pipeline or small OCR-free model, validation rules on totals
Banking, insurance, KYC onboardingForm and ID-document extraction; classification of document type first
Enterprise search and RAGParse PDFs to Markdown, or ColPali page retrieval plus a VLM reader
Scientific literaturePDF to Markdown with LaTeX (Nougat-style), table and figure extraction
LLM pre-training dataConverting billions of PDF pages into clean text; order and tables decide data quality
Lecture and meeting videoTreat key frames of slides as pages: OCR or a VLM reads them, ColPali indexes them for Video-RAG (session 210)
Sports and news videoRead scoreboards, clocks and tickers frame by frame; text changes become events for key-moment detection
Video retrieval and moderationOn-screen text as an extra signal for search and for policy checks (phone numbers, scam text)

Common mistakes

  1. Running OCR on born-digital PDFs. The text layer is already exact. OCR can only add errors. Check for a text layer first (but watch for PDFs whose text layer is itself bad OCR).
  2. Too low resolution. Squashing a page to a 448-pixel square or a tight token budget makes small print unreadable (Figure 3). Keep the aspect ratio and check the pixel height of your smallest important text.
  3. Ignoring reading order. Raster order on multi-column pages feeds the LLM mixed sentences (Figure 2). Headers and footers break sentences across pages.
  4. Reporting ANLS for business fields. ANLS gives 0.92 to a wrong invoice number (Figure 4). Use exact match after normalization for IDs, dates and amounts.
  5. Trusting fluent VLM output. A VLM that cannot read a value may invent one. Ask for the source box or quote, check it against the PDF text or a second reader, and route low-confidence pages to people.
  6. Flattening tables to plain text. Losing rows and columns turns a table into a list of numbers with no meaning. Keep tables as HTML or Markdown, and evaluate them with TEDS.
  7. Forgetting the cost of multi-vector indexes. 257.5 KiB per page in float16 is 263.7 GB for a million pages. Plan pooling, quantization or a two-stage search before you build.

Interview questions

OCR pipeline or end-to-end VLM: how do you choose?core
Look at the documents, the volume and the cost of an error. A pipeline gives per-word boxes and confidences, is cheap per page, and each stage can be fixed alone; but errors multiply across stages (formula 218.1) and it struggles with charts and complex layouts. A VLM handles varied layouts, charts and free questions in one model, but costs more per page, needs high resolution, and can produce fluent wrong text. For fixed forms at high volume, a pipeline or a small fine-tuned OCR-free model; for varied documents and QA, a VLM; for high-stakes fields, either one plus grounding, validation and human review. For born-digital PDFs, use the text layer and skip OCR entirely.
What does LayoutLM add to BERT, and why does it help?basic
It adds 2-D position embeddings for each token’s bounding box: the corners are scaled to a 0 to 1000 grid, and embeddings of \(x_0, y_0, x_1, y_1\) are added to the token and 1-D position embeddings (formula 218.3). The model then knows which words are in the same row, column or form field. On FUNSD this raised F1 from about 0.60 (BERT) to 0.79. It still depends on external OCR for the words and boxes.
How does Donut produce JSON without OCR?core
It is a Swin encoder plus a BART decoder trained with next-token cross-entropy (218.6). Pre-training: read all the text of the page in order (synthetic SynthDoG pages and 11M real scans). Fine-tuning: the target is a tagged sequence where each field has open and close tags; this sequence converts one-to-one into JSON. Classification and QA use the same format. No OCR is called at training or inference time.
Define ANLS. What are its weaknesses?core
For each question, take the best over accepted answers of \(1 - \text{Lev}/\max(|g|, |o|)\), set to 0 if the normalized distance is at least \(\tau = 0.5\); average over questions (218.8, 218.9). It forgives small OCR slips, but it also gives high credit to wrong IDs and numbers that differ by one character (0.923 for a wrong invoice number), and it gives 0 to correct answers written in a different format (dates). For production, normalize values and use exact match per field.
How many visual tokens does an A4 page cost in Qwen2-VL, and what does that imply?core
One token covers 28 × 28 pixels (14-pixel patches, 2 × 2 merge). At 150 dpi the page is 1240 × 1754, resized to 1232 × 1764, so \(44 \times 63 = 2{,}772\) tokens; at 300 dpi about 11,125. So a 100-page document is around 277k tokens at 150 dpi: retrieve the relevant pages first, or lower resolution only where the text is large. Below roughly the point where letters are a few pixels tall, accuracy collapses no matter how good the model is.
Explain ColPali’s scoring and why it beats a single vector per page.core
The page image goes through PaliGemma-3B; each of the 1,030 output vectors (1,024 patches + 6 prompt tokens) is projected to 128 dims. The query is encoded into one vector per token. Score = sum over query tokens of the max dot product with any page vector (218.11). Relevant content is usually a small region; a single pooled vector averages it with the rest of the page (dilution), while MaxSim lets each query token match its best region. On ViDoRe, ColPali got 81.3 nDCG@5 against 67.0 for the best text-parsing pipeline and 58.8 for the mean-pooled BiPali.
Write the ColPali training loss and explain the hardest negative.advanced
\(\mathcal{L} = \frac{1}{b}\sum_k \text{softplus}(s_k^- - s_k^+)\), with \(s_k^+\) the late-interaction score of query \(k\) with its own page and \(s_k^-\) the maximum over the other pages in the batch (218.12). This is a two-way softmax cross-entropy between the positive and the single hardest in-batch negative. Using the hardest one focuses the gradient on the page that is most confusing now, instead of averaging over many easy negatives.
Your multi-vector index for 10 million pages is too large. What do you do?advanced
Compute it first: \(10^{7} \times 1030 \times 128 \times 2\) bytes \(= 2.64\) TB in float16. Options: token pooling (pool factor 3 cuts vectors by two thirds with about 2% quality loss in the paper), binary quantization (16 bytes per vector, 16x smaller than float16, usually with a float rerank of the top results), lower vector dimension, and a two-stage design where a single-vector or BM25 index proposes candidates and MaxSim reranks only a few hundred. Also drop vectors of blank patches. Measure recall@k after each step.
How would you get a correct reading order for a two-column paper?basic
Detect lines or blocks, then use recursive XY-cut: split along the widest white-space band, preferring the column gutter over line gaps; recurse. Treat tables and figures as blocks found by a layout model. Drop or tag headers, footers and page numbers. For complex layouts (wrapped text, sidebars), use a learned reading-order model or ask a strong VLM to transcribe the page in order, then check that no text was skipped.
How do you evaluate table extraction?basic
Represent tables as HTML trees and use TEDS = 1 − edit distance / size of the larger tree (218.4). It scores structure (rows, columns, spans) and content together, so a shifted column costs a lot while a typo inside a cell costs little. Also check downstream: can you answer questions from the extracted table?
Design: extract 20 fields from 1 million mixed invoices per day.advanced
Step 1: route by type: born-digital PDFs use the text layer; scans go to OCR or a VLM. Step 2: classify the vendor or template; known templates go to a cheap, fine-tuned small model; unknown ones to a larger VLM with a JSON schema and constrained decoding. Step 3: ask for a source box or quote per field; verify against the text layer or OCR. Step 4: business rules (line items sum to the total, dates valid, tax rate plausible). Step 5: send pages that fail checks or have low confidence to people, and use their corrections as training data. Track exact-match per field, the human-review rate and cost per page. With 20 fields, even 99% per field gives only 0.99^20 ≈ 82% fully correct pages, so the review path is required.

Exercises

  1. A 12-character ID is read at 99.5% character accuracy. What is the probability the whole ID is right? What about a page with 20 fields at 99% per field?
  2. Compute the score of answer “12.5O” (letter O) against gold “12.50”, and of “125.0”. Which is more wrong for a business user, and which gets the higher ANLS?
  3. A US Letter page (8.5 × 11 in) is rendered at 200 dpi. How many Qwen2-VL tokens does it cost with the default budget?
  4. A query has 2 tokens and a page has 3 patches. The dot products are: token 1 with patches \((0.2, 0.9, 0.1)\), token 2 with patches \((0.7, 0.3, 0.4)\). Compute LI.
  5. How much storage does a ColPali index of 50,000 pages need with pool factor 3 in float16?
  6. For ChartQA with gold answer 43, which predictions are accepted: 41.2, 41, 45?
Answerscheck
1. \(0.995^{12} = 0.942\). \(0.99^{20} = 0.818\). 2. The gold has 5 characters. “12.5O” is one substitution away: \(\text{NL} = 0.2\), score 0.8; “125.0” needs 2 edits (move the dot: delete and insert), \(\text{NL} = 0.4\), score 0.6. “125.0” is ten times the true value, a far worse error, yet it still gets 0.6. 3. 1700 × 2200 pixels, rounded to 1708 × 2212, so \(61 \times 79 = 4{,}819\) tokens, under the 16,384 budget. 4. \(\max(0.2, 0.9, 0.1) + \max(0.7, 0.3, 0.4) = 0.9 + 0.7 = 1.6\). 5. \(\lceil 1030/3 \rceil = 344\) vectors, \(344 \times 128 \times 2 = 88{,}064\) bytes per page; \(88{,}064 \times 50{,}000\) bytes \(= 4.40\) GB. 6. The limit is \(\pm 5\%\): 40.85 to 45.15. So 41.2 (4.2%), 41 (4.7%) and 45 (4.7%) are all accepted.

Key takeaways

  • Document AI = extraction, DocVQA, parsing to Markdown, classification and retrieval over pages where position carries meaning.
  • OCR pipelines are cheap, auditable and modular, but errors multiply across stages (218.1); LayoutLM adds box positions to text tokens (218.2, 218.3).
  • Reading order (XY-cut), tables (TEDS, 218.4) and charts (relaxed accuracy, 218.5) are where naive text extraction breaks.
  • OCR-free models (Donut, Pix2Struct, VLMs) map image to tokens with one cross-entropy loss (218.6); resolution decides what they can read, at 28 × 28 pixels per Qwen2-VL token (218.7).
  • DocVQA uses ANLS (218.8, 218.9), which forgives small slips but also rewards near-miss wrong IDs; extraction uses field F1 (218.10).
  • ColPali embeds the page image as 1,030 vectors of 128 dims and scores with MaxSim (218.11), trained with a hardest-negative softplus loss (218.12); the price is storage (218.13).

Key formulas

\[ P_{\text{all}} \approx \Big(\prod_s p_s\Big)^{K}, \qquad N_{\text{tok}} = \frac{\bar H}{28}\cdot\frac{\bar W}{28}, \qquad s = 1 - \frac{\text{Lev}(g,o)}{\max(|g|,|o|)} \ \text{if} \ \lt \tau \]
\[ \text{LI}(q,d) = \sum_i \max_j \langle \mathbf{E}_q^{(i)}, \mathbf{E}_d^{(j)} \rangle, \qquad \mathcal{L} = \text{softplus}(s^- - s^+), \qquad \text{TEDS} = 1 - \frac{\text{EditDist}}{\max(|T_a|,|T_b|)} \]

References for this session

  1. Kim et al. (2021). OCR-free Document Understanding Transformer
  2. Faysse et al. (2024). ColPali: Efficient Document Retrieval with Vision Language Models
  3. Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 8: Document AI
  5. Kim et al. (2021). OCR-free Document Understanding Transformer
  6. Faysse et al. (2024). ColPali: Efficient Document Retrieval with Vision Language Models
  7. Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images
  8. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 8: Document AI
  9. Khattab & Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
  10. Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
  11. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  12. Xu, Li, Cui, Huang, Wei & Zhou (2019). LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  13. Xu et al. (2020). LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
  14. Lee et al. (2022). Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
  15. Blecher, Cucurull, Scialom & Stojnic (2023). Nougat: Neural Optical Understanding for Academic Documents
  16. Pfitzmann, Auer, Dolfi, Nassar & Staar (2022). DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis
  17. Zhong, ShafieiBavani & Jimeno Yepes (2019). Image-based table recognition: data, model, and evaluation
  18. Masry, Long, Tan, Joty & Hoque (2022). ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
  19. Biten et al. (2019). Scene Text Visual Question Answering
  20. Nagy & Seth (1984). Hierarchical representation of optically scanned documents