Document AI
Documents carry meaning in their text and in where the text sits. This session compares OCR pipelines with OCR-free models such as Donut and modern VLMs, shows how layout, reading order, tables and charts are handled, how DocVQA and extraction are scored, and how ColPali finds the right page by embedding the page image as many small vectors.
What you will learn
- The main document tasks (extraction, DocVQA, parsing to Markdown, classification, page retrieval) and how each one is scored
- How an OCR pipeline works stage by stage, why its errors multiply, and how LayoutLM adds box positions to text tokens
- Reading order with recursive XY-cut, table recognition with TEDS, and chart questions with relaxed accuracy
- How OCR-free models (Donut, Pix2Struct, Qwen2-VL-style VLMs) read pages, and how many visual tokens a page costs at a given resolution
- ANLS for DocVQA, field-level F1 for extraction, and where these metrics mislead
- ColPali: late-interaction scoring over 1,030 patch vectors per page, its hardest-negative loss, and the storage cost of multi-vector indexes
- How to choose a document system for a given workload, and how to make its outputs checkable
Why this matters
A large part of the world’s useful information lives in documents: invoices, contracts, bank statements, forms, scientific papers, slide decks, manuals, medical reports. They are made for human eyes. The meaning is in the text, but also in where the text sits: which number is under which column, which label belongs to which box, which caption goes with which chart.
Document AI (also called visually-rich document understanding) is the set of methods that let a machine read such pages. The common tasks are:
- Information extraction: fill a fixed set of fields from a page, for example vendor, date, total and line items from an invoice. The output is JSON.
- Document question answering (DocVQA): answer a free question about a page, such as “What is the due date?”.
- Document parsing: turn a whole page into clean, structured text (usually Markdown or HTML) with headings, lists, tables and formulas in the right order, so that an LLM or a search index can use it.
- Classification: is this page an invoice, a letter or a form?
- Retrieval: find the right page among millions, for a question.
For a long time all of these started with OCR (optical character recognition: turning pixels of text into characters). Today there are two families. OCR pipelines chain specialized models: find text, read it, find the layout, put it in reading order, then extract. OCR-free models such as Donut and modern vision-language models (VLMs) read the page image directly and write the answer. And for search, ColPali showed that you can embed the page image itself, with one vector per image patch, and skip text extraction for retrieval.
The previous session, session 217, worked on pixels that move. This session works on pixels that hold text. The two meet more often than you may think: lecture videos are slide decks over time, news videos carry tickers, sports videos show scoreboards. The next session, session 219, builds on the page retrieval part of this one. This session follows chapter 8 of the VLM book (Document AI).
What makes documents different
A photo of a dog has one main object. A page has hundreds of small objects (words), and their positions carry meaning. Three examples:
- In a table, “1,240” means nothing until you know its row (“Q3”) and column (“Revenue, $k”).
- In a form, the value “Keller” belongs to the label on its left, “Last name”, not to the label above it.
- In a two-column paper, line 1 of the left column continues on line 2 of the left column, not on line 1 of the right column.
There are also two very different kinds of input. A born-digital PDF (made by Word, LaTeX, a web page) already contains the characters and their coordinates; we can read them without any OCR, although the reading order and the table structure are still lost. A scanned or photographed page is only pixels: skewed, blurred, stamped, handwritten. Many real archives mix both.
| Task | Output | Usual metric | Example dataset |
|---|---|---|---|
| Key information extraction | JSON with fixed fields | field-level F1, tree-edit accuracy | CORD (receipts), FUNSD (forms) |
| Document VQA | short text answer | ANLS | DocVQA, InfographicVQA |
| Chart QA | number or short text | relaxed accuracy (5% tolerance) | ChartQA |
| Table recognition | HTML or Markdown table | TEDS | PubTabNet |
| Layout analysis | boxes with classes | mAP (as in detection) | DocLayNet |
| Page parsing | Markdown / HTML of the page | edit distance, TEDS for tables, reading-order scores | academic PDFs (Nougat) |
| Page retrieval | ranked list of pages | nDCG@5, recall@k | ViDoRe |
Each row has its own section below. DocLayNet, for scale, has 80,863 human-annotated pages with 11 layout classes (for example text, title, table, picture, caption, page-header) [Certain].
The OCR pipeline
The classic system is a chain of specialized models. Each stage is small, fast and can be checked on its own.
- input: page image \(I\) (or a PDF page), list of target fields
- preprocess: deskew, denoise, convert to gray, render the PDF at about 300 dpi
- text detection: boxes around every word or line // a detector such as DB or CRAFT
- text recognition: read each box into characters, with a confidence // a CRNN or TrOCR-style model
- layout analysis: boxes for title, paragraph, table, figure, caption, header, footer // a detector trained on DocLayNet-style data
- reading order: sort the blocks and lines into the order a human reads them // XY-cut or a learned model
- structure: run a table-structure model on table regions; formula and chart models on their regions
- extract: feed words + boxes to a layout-aware model (LayoutLM), or the serialized text to an LLM with the field list
- return fields, each with its source box and confidence
The strengths are real: each output word has an exact box (good for highlighting and audit), confidence scores are available per word, stages can be improved one at a time, and the models are small. The weakness is that errors compound. If each stage is right with probability \(p_s\), and we assume the errors are independent, a field survives the whole chain with probability
- Read the symbols
- \(S\)
- number of stages in the chain (detection, recognition, layout and order, extraction)
- \(p_s\)
- probability that stage \(s\) does not break this field
- \(K\)
- number of fields we need from one page
- \(c\), \(L\)
- per-character accuracy and the length of the string in characters
An invoice number has \(L = 12\) characters. With a character accuracy of \(c = 0.99\), the whole number is exactly right with probability \(0.99^{12} = 0.886\). At \(c = 0.98\) it drops to \(0.785\); at \(c = 0.999\) it is \(0.988\).
Now the chain: detection 0.99, recognition 0.97, layout and order 0.98, extraction 0.96 gives \(P_{\text{field}} = 0.903\). A page with \(K = 10\) fields is fully correct only with probability \(0.903^{10} = 0.362\). An end-to-end model that gets each field right with probability 0.95 gives \(0.95^{10} = 0.599\). The independence assumption is rough (a bad scan hurts every stage at once), but the lesson holds: in a chain, small losses multiply.
How to read Figure 1
Move one stage slider down to 0.9 and watch the page-level number fall much faster than the field-level number: the power \(K\) amplifies every loss. Then set \(K = 1\): for a single field, a good pipeline and a good end-to-end model are close. The pipeline is not “worse”; it needs every stage to be very good. This is why production pipelines spend most of their effort on the weakest stage, and why they keep per-word confidences to send uncertain pages to a human.
Adding layout to a text model: LayoutLM
After OCR we have words and boxes. A plain text model (BERT) sees only the words in a 1-D order, so it loses the table and form structure. LayoutLM (Xu et al., 2019) adds the box to every token. First the box corners are scaled to a virtual page of size 1000 × 1000, so that page size does not matter:
Then the input vector of token \(i\) is the sum of its text embedding, its 1-D position embedding and four 2-D position embeddings, one for each box coordinate:
- Read the symbols
- \(W, H\)
- page width and height in pixels or PDF points
- \((\tilde{x}_0, \tilde{y}_0), (\tilde{x}_1, \tilde{y}_1)\)
- the top-left and bottom-right corners of the word’s box, on the 0 to 1000 scale
- \(\mathbf{X}, \mathbf{Y}\)
- learned tables: one shared for x-coordinates, one shared for y-coordinates
- \(\mathbf{E}_{\text{tok}}, \mathbf{E}_{\text{pos}}\)
- the usual BERT token and position embeddings (session 11)
The model is then pre-trained like BERT with masked words, but it keeps the boxes of masked words, so it must use the layout to guess them. On the FUNSD form-understanding benchmark, LayoutLM-base reached an F1 of 0.787, against about 0.603 for BERT-base with the same OCR text [Certain]. That jump is the value of knowing where each word is. LayoutLMv2 and v3 later added image features of the page too. The limit stays: everything depends on the OCR output given to the model.
Layout, reading order, tables and charts
Reading order
To give a page to an LLM, we must turn a 2-D page into a 1-D sequence. The order matters: an LLM reads tokens in order, and a sentence broken across two columns becomes nonsense if we interleave them. The naive rule, “sort lines from top to bottom, then left to right” (raster order), fails on every multi-column page.
A classic fix is the recursive XY-cut (Nagy and Seth, 1984): look for a straight band of white space that cuts the page fully from side to side. Cut there. Repeat inside each part. If both a horizontal and a vertical cut exist, take the wider gap, because a column gutter is usually wider than the space between two lines.
- function XYCUT(boxes)
- if one box or none: return boxes
- \(g_y \leftarrow\) widest horizontal empty band that crosses all boxes // projection of boxes on the y-axis
- \(g_x \leftarrow\) widest vertical empty band // projection on the x-axis
- if neither exists: return boxes sorted by (top, left)
- if \(g_x\) is wider than \(g_y\): split into left and right, return XYCUT(left) + XYCUT(right)
- else: split into top and bottom, return XYCUT(top) + XYCUT(bottom)
How to read Figure 2
In raster order, line 1 of the left column is followed by line 1 of the right column, so two different sentences are woven together. The readout counts inverted pairs: pairs of lines that come in the wrong order compared with how a person reads the page (20 items give 190 pairs). The dashed lines are the cuts XY-cut makes with a gap of 10 points or more: first the footer, then the table, then the title, then the column gutter. The table and the figure are treated as single blocks: a layout model found them first, and a separate table model reads the table cells.
XY-cut fails on pages with no clean white-space bands (magazine layouts, text wrapped around pictures, forms). Modern pipelines use learned reading-order models, or let a VLM write the page in order directly. Page furniture (headers, footers, page numbers) should usually be dropped or tagged, so it does not break a sentence that continues on the next page.
Tables
A table is a tree: table, rows, cells, cell text. Table recognition must find the cell grid, including merged cells that span rows or columns, and the text in each cell. The output is usually HTML (which can express spanning cells) or Markdown (which cannot). The standard metric is TEDS (tree-edit-distance-based similarity, from the PubTabNet paper): write both tables as HTML trees and count the edits that turn one into the other:
Here \(|T|\) is the number of nodes in a tree, and an edit is inserting, deleting or renaming one node (renaming a cell node costs its normalized text distance). A predicted table with 3 edits against a reference of 20 nodes scores \(1 - 3/20 = 0.85\). TEDS punishes a wrong structure (a shifted column) much more than a typo inside a cell, which is what users care about: a value in the wrong column is a wrong value.
Charts
A chart stores numbers as lengths, angles and colors. Reading it needs fine visual detail and often arithmetic (“how much higher is 2023 than 2021?”). The ChartQA benchmark uses relaxed accuracy for numeric answers: a predicted number \(\hat{a}\) counts as correct if it is within 5% of the true number \(a\):
For \(a = 43\), the accepted range is \(40.85\) to \(45.15\): \(41.2\) (error 4.2%) is accepted, \(40.5\) (5.8%) is not. Text answers still need an exact match. Some systems first convert a chart into its data table (“chart derendering”), then answer from the table; this makes the numbers inspectable.
Parsing a page to Markdown
For RAG and for LLM training data, the most common need is “give me this PDF as clean Markdown”. Three routes exist today:
- Pipeline parsers: the stages of Algorithm 1, then a writer that emits headings, lists, Markdown tables and LaTeX formulas in reading order. Open-source tools in this family combine a layout detector, OCR (skipped for born-digital text), a table model and a formula model.
- Specialized end-to-end models: Nougat (Blecher et al., 2023) is a Swin encoder with an mBART-style decoder that reads an academic page image and writes Markdown with LaTeX math [Likely for the exact decoder]. It was trained on arXiv papers, where the source gives the target text for free.
- General VLMs with a prompt: “Convert this page to Markdown. Keep tables as HTML.” Strong VLMs now do this well on clean pages. The risks are the generation risks of any LLM: skipped lines, repeated lines (a decoder stuck in a loop), and, most dangerous, plausible text that is not on the page.
OCR-free models: Donut, Pix2Struct and VLMs
Donut (Document Understanding Transformer, Kim et al., 2021) removed OCR from the loop. It is an encoder-decoder: a Swin Transformer encoder reads the page image (2560 × 1920 pixels in pre-training), and a 4-layer BART decoder writes tokens [Certain]. Everything is one sequence task, trained with the usual next-token cross-entropy:
- Read the symbols
- \(I\)
- the page image; \(\text{Enc}_\theta(I)\) is the grid of encoder output vectors that the decoder attends to with cross-attention
- \(y_1, \dots, y_T\)
- the target token sequence: all page text in pre-training, a structured string in fine-tuning
Pre-training teaches the model to read: the target is all the text on the page, from top-left to bottom-right. Donut used 11 million scanned pages (IIT-CDIP) and about 2 million synthetic pages from its generator SynthDoG, in four languages [Certain]. Fine-tuning teaches it to structure: the target becomes a tagged string that maps one-to-one to JSON. The paper writes the tags as [START_name] … [END_name]; the released code uses tokens like <s_total> [Likely]. For a receipt:
<s_menu><s_nm>Latte</s_nm><s_price>4.50</s_price></s_menu><s_total>12.50</s_total>
-> {"menu": {"nm": "Latte", "price": "4.50"}, "total": "12.50"}
Classification and question answering use the same trick: the answer is just another tagged string. On the CORD receipt benchmark, Donut (143M parameters) reached 84.1 field F1 at 1.2 s per image, against 78.9 F1 at 1.7 s for LayoutLMv2 with OCR, on the paper’s P40 GPU setup [Certain]. On DocVQA it was weaker (67.5 ANLS against 78.1 for LayoutLMv2) [Certain]: free questions about dense text still needed better reading than a 2021 encoder had.
Pix2Struct (Lee et al., 2022) pre-trained on a different free signal: predict a simplified HTML of a web page from its masked screenshot. It also introduced variable-resolution input: scale the image to fit a patch budget while keeping the aspect ratio, instead of squashing it to a square [Certain]. Both ideas are now standard in VLMs.
Modern VLMs (Qwen2-VL and Qwen2.5-VL, InternVL, and others; see session 182) are OCR-free by design. They were trained on huge mixtures of OCR, chart, table and document data, and they take instructions, so one model does extraction, QA and parsing. Qwen2-VL-72B reports 96.5 ANLS on the DocVQA test set (GPT-4o: 92.8) [Certain, as reported in the Qwen2-VL paper]. For comparison, humans score 98.1 [Certain, from the DocVQA paper].
Resolution is the whole game
Text is small. A 10-point letter is \(10/72\) of an inch tall (the “em” size). If the model sees the page at too low a resolution, the letters are a few pixels tall and no model can read them. But higher resolution means more visual tokens, and attention cost grows with the number of tokens. In Qwen2-VL, the vision encoder uses 14 × 14 pixel patches, and an MLP merges each 2 × 2 group into one token, so one token covers 28 × 28 pixels [Certain]. The image is first resized to multiples of 28 (and shrunk if it exceeds a pixel budget):
If \(\bar{H}\bar{W}\) exceeds the budget \(P_{\max}\), both sides are first divided by \(\beta = \sqrt{HW / P_{\max}}\) and rounded down to multiples of 28. The default budget in the released Qwen2-VL processor is 12,845,056 pixels, which is 16,384 tokens [Certain, from its preprocessor_config.json].
A4 is 210 × 297 mm, or 8.27 × 11.69 inches. Rendered at 150 dpi, it is 1240 × 1754 pixels. Rounded to multiples of 28: 1232 × 1764, so \(N_{\text{tok}} = 44 \times 63 = 2{,}772\) tokens. The em size of 10-point text is \(150 \times 10/72 = 20.8\) pixels (21.0 after the small resize), so a letter spans less than one token.
At 300 dpi (what Tesseract’s documentation recommends for OCR [Certain]) the page is 2480 × 3508 and costs 11,125 tokens. A 100-page report at 150 dpi costs 277,200 tokens: too much for most context windows and slow even when it fits. This is why page retrieval comes before reading, the topic of the next section.
Squash the same page to a fixed 448 × 448 square (as in PaliGemma-448, the base of ColPali) and the vertical resolution falls to 38 dpi: the em of 10-point text is about 5.3 pixels. Fine print is not readable at that size, although large titles and the general look of the page still are.
How to read Figure 3
Start at 150 dpi and 10 pt: 2,772 tokens and an em of about 21 pixels, clearly readable. Drop the budget to 1,024 tokens: the page is shrunk to 728 × 1064 (988 tokens) and the em falls to about 12.6 pixels; the digits get visibly coarser. Switch to the fixed square: about the same token count (1,024), but now the page is squeezed, the em is about 5 pixels, and the text is a gray smear. Same token count, very different legibility: keeping the aspect ratio matters. The numbers are exact; the picture depends a little on your browser’s font.
Other ways to pay for resolution exist: cutting the page into tiles plus a small overview image (InternVL-style dynamic tiling), and compressing visual tokens after the encoder (session 180, session 183).
DocVQA and information extraction
DocVQA
DocVQA (Mathew et al., 2020) has 50,000 questions on 12,767 document images taken from the UCSF Industry Documents Library, split 80/10/10 into train, validation and test [Certain]. The answer is usually a span of text on the page: a date, a name, an amount. Questions cover forms, tables and lists, layout, running text, figures, handwriting and yes/no.
Exact match is too harsh for this: a model that reads “Dr Maria Keller” for “Dr. Maria Keller” has understood the page. But plain string similarity is too soft: a wrong answer that happens to share letters should get nothing. The metric ANLS (Average Normalized Levenshtein Similarity, introduced for scene-text VQA by Biten et al., 2019) takes the middle road. The Levenshtein distance \(\text{Lev}(a, b)\) is the minimum number of single-character insertions, deletions and substitutions that turn \(a\) into \(b\). Normalize it and cut it off at a threshold \(\tau\):
- Read the symbols
- \(g_{ij}\)
- the \(j\)-th accepted answer for question \(i\) (several are allowed, for example “$1,234.56” and “1234.56”)
- \(o_i\)
- the model’s answer to question \(i\); both are lower-cased and their spaces cleaned before comparison
- \(|g|\)
- length in characters
- \(\tau\)
- the cut-off: answers more than half wrong get 0, not partial credit
| # | Accepted answers | Model answer | Score |
|---|
How to read Figure 4
Row 1 shows the weak side of ANLS: “INV-2023-0041” is a different invoice number, but it is one character away, so it earns 0.923. Row 4 shows the other side: “03/03/2024” is the right date in another format, and it earns 0. Lower \(\tau\) to 0.1 and only near-perfect strings score; raise it to 1 and every answer gets partial credit, even a wrong company. For IDs, amounts and dates in production, use exact match after normalization (parse the date, parse the number); keep ANLS for comparing with published results.
Row 2: gold “$1,234.56” (9 characters), model “1,234.56”: one deletion, \(\text{NL} = 1/9 = 0.111\), score \(0.889\). The second gold “1234.56” is also one edit away, but over 8 characters: \(1 - 1/8 = 0.875\). The max is 0.889.
Row 3: gold “Dr. Maria Keller” (16 characters), model “Maria Keller”: 4 deletions, \(\text{NL} = 0.25\), score \(0.75\).
Row 4: gold “March 3, 2024” (13 characters), model “03/03/2024”: 8 edits, \(\text{NL} = 0.615 \gt 0.5\), score 0. The six rows average to ANLS = 0.594, while exact match is 1/6 = 0.167.
Information extraction
Extraction has a fixed schema, so we can score each field. A predicted field counts as correct only if its key and its full value match the reference. With \(\text{TP}\) correct predicted fields:
Example: the model outputs 9 fields, 8 are right, the reference has 10: \(P = 0.889\), \(R = 0.8\), \(F_1 = 0.842\). Field F1 is strict (one wrong character fails the field) and ignores nesting; Donut also reports a tree-edit-distance accuracy, \(\max(0, 1 - \text{TED}(\text{pred}, \text{gt}) / \text{TED}(\varnothing, \text{gt}))\), which gives credit for a mostly-right nested structure such as a list of line items [Certain].
With VLMs, extraction is a prompt plus a schema: “Return JSON with keys vendor, date (YYYY-MM-DD), total (number), line_items[]”. Two practices make it reliable. Constrained decoding forces the output to be valid JSON for the schema (session 146). Grounding asks for the box or the quoted source text of each value, so a cheap check can confirm the value is really on the page (for born-digital PDFs, look it up in the PDF text layer).
Page retrieval with multi-vector embeddings: ColPali
Before you can read the right page, you must find it. The text route is: parse every PDF (OCR, layout, tables, maybe captions for figures), chunk the text, embed the chunks, search (session 152). Every parsing error becomes a retrieval error, and charts or diagrams are mostly lost.
ColPali (Faysse et al., 2024) skips parsing for retrieval. It feeds the page image to PaliGemma-3B, a VLM at 448 × 448 input, and keeps the output vector of every image patch: 1,024 patches (a 32 × 32 grid of 14-pixel patches) plus 6 prompt tokens, each projected to 128 dimensions [Certain]. A query is encoded by the same model into one 128-d vector per query token. The score is the late interaction of ColBERT (session 150): every query token finds its best-matching patch, and these best matches are summed:
- Read the symbols
- \(\mathbf{E}_q^{(i)}\)
- the 128-d vector of query token \(i\) (\(N_q\) tokens in the query)
- \(\mathbf{E}_d^{(j)}\)
- the 128-d vector of page patch \(j\) (\(N_d = 1{,}030\) vectors per page in ColPali)
- \(\langle \cdot, \cdot \rangle\)
- dot product (the vectors are normalized, so it is a cosine)
Why this beats one vector per page: a page is mostly irrelevant to any given query. Average-pooling all patches into one vector (the “BiPali” baseline in the paper) mixes the small relevant region with everything else. MaxSim lets each query word look only at its best spot. The model is trained on query-page pairs with an in-batch contrastive loss that compares each positive pair with the hardest negative page in the batch:
The function \(\log(1 + e^{z})\) is the softplus: near 0 when the positive beats the hardest negative by a wide margin, and growing linearly when it loses. With \(s^+ = 2.3\) and \(s^- = 1.4\) the loss is \(0.341\); with \(s^+ = 1.2\) and \(s^- = 1.5\) it is \(0.854\). Training used about 118,695 query-page pairs, 37% of them synthetic queries written by an LLM, with LoRA on the language model [Certain].
How to read Figure 5
For “revenue chart 2023”, click each token: “revenue” lights up the title patch of page A, “chart” the chart patches, “2023” the chart patch with the year. The page score is the sum of the three outlined maxima. Now pick “staff growth”: MaxSim finds the two small patches on page C (the letter) and ranks it first. The mean-pooled vectors rank page B (a table) first, because on page C the two useful patches are averaged with 18 patches of ordinary text. That is the dilution problem late interaction solves. Note also that page scores are never near 0: a max over many patches always finds something a little similar.
The paper’s numbers: on its ViDoRe benchmark (academic sets such as DocVQA, InfoVQA and TAT-DQA, and practical sets on energy, government, healthcare and AI documents), ColPali reached an average nDCG@5 of 81.3, against 67.0 for the best text pipeline (Unstructured parsing with figure captioning and BGE-M3 embeddings) and 58.8 for BiPali [Certain]. Indexing took about 0.39 s per page against 7.22 s for the parsing pipeline, on the authors’ hardware [Certain]. A Qwen2-VL-based variant, ColQwen2, added 5.3 points [Certain, as reported in the ColPali paper’s later version].
The cost: storage
Multi-vector indexes are large. The size of an index with \(M\) pages is
with \(N_d\) vectors per page, \(D\) dimensions and \(b\) bytes per number.
ColPali in float16: \(1030 \times 128 \times 2 = 263{,}680\) bytes = 257.5 KiB per page, matching the paper. One million pages: 263.7 GB. One 1024-d float16 vector per page would be 2,048 bytes: 129 times smaller.
Three ways to shrink it: token pooling (cluster similar patch vectors and average them; the paper reports that a pool factor of 3, which removes 66.7% of vectors, keeps about 97.8% of retrieval quality [Certain]): \(\lceil 1030/3 \rceil = 344\) vectors, 86 KiB per page. Binary quantization: 128 bits = 16 bytes per vector, 16.1 KiB per page, 16.5 GB for a million pages. Two-stage search: a cheap single-vector or text search finds a few hundred candidates, and MaxSim reranks only those (session 151).
- index: for each PDF page: render to an image; \(\mathbf{E}_d \leftarrow\) ColPali(image) // about \(1{,}030 \times 128\) per page
- optional: pool or binarize \(\mathbf{E}_d\); store with (document id, page number)
- query: \(\mathbf{E}_q \leftarrow\) ColPali(query text) // one vector per query token
- candidates \(\leftarrow\) top 200 pages by a cheap first stage // pooled vectors or BM25 on any text layer
- rank candidates by \(\text{LI}(q, d)\), formula (218.11); keep the top \(k\) (say 3)
- answer: send the \(k\) page images plus the question to a VLM; ask it to cite the page
This is document RAG without a text parser, and it is the core of the next session, session 219.
Choosing a system
There is no single best design. Chapter 8 of the VLM book has its own section on choosing a model; the table below is a simple rule of thumb: start from the documents and from the cost of an error.
| Situation | Good default | Why |
|---|---|---|
| Born-digital PDFs, need text for search | Read the PDF text layer; layout model only for order and tables | No OCR errors at all; cheap |
| Scans, fixed forms, millions per day | OCR pipeline + small layout-aware extractor, or a fine-tuned small OCR-free model | Cheap per page, boxes and confidences for audit |
| Varied layouts, free questions, charts | A strong VLM at native resolution | Reads layout, charts and handwriting in one model |
| Search over many visually rich pages | ColPali-style page retrieval, then a VLM on the top pages | No parsing step; keeps figures and tables |
| High-stakes values (amounts, IDs) | Any of the above + grounding + validation rules + human review on low confidence | Fluent wrong outputs are the main risk |
Code: tokens per page, ANLS, late interaction and the ColPali loss
A small, deterministic script. It needs only PyTorch. The comments show the printed output.
import math, torch import torch.nn.functional as F torch.manual_seed(0) # 1) How many visual tokens does a page cost? (Qwen2-VL resize rule, formula 218.7) def page_tokens(h, w, factor=28, min_px=3136, max_px=12845056): hb, wb = round(h / factor) * factor, round(w / factor) * factor if hb * wb > max_px: # too big: shrink, keep aspect ratio beta = math.sqrt(h * w / max_px) hb, wb = math.floor(h / beta / factor) * factor, math.floor(w / beta / factor) * factor elif hb * wb < min_px: # too small: grow beta = math.sqrt(min_px / (h * w)) hb, wb = math.ceil(h * beta / factor) * factor, math.ceil(w * beta / factor) * factor return hb, wb, (hb // factor) * (wb // factor) print(page_tokens(1754, 1240)) # A4 at 150 dpi -> (1764, 1232, 2772) # 2) ANLS, formulas 218.8 and 218.9 def lev(a, b): d = list(range(len(b) + 1)) for i, ca in enumerate(a, 1): prev, d[0] = d[:], i for j, cb in enumerate(b, 1): d[j] = min(prev[j] + 1, d[j - 1] + 1, prev[j - 1] + (ca != cb)) return d[-1] def anls(golds_list, preds, tau=0.5): norm = lambda s: " ".join(s.lower().split()) total = 0.0 for golds, p in zip(golds_list, preds): best = 0.0 for g in golds: g2, p2 = norm(g), norm(p) nl = lev(g2, p2) / max(len(g2), len(p2), 1) best = max(best, 1 - nl if nl < tau else 0.0) total += best return total / len(preds) print(round(anls([["INV-2023-0047"], ["March 3, 2024"]], ["INV-2023-0041", "03/03/2024"]), 4)) # 0.4615 # 3) Late interaction (ColPali scoring, formula 218.11) for a batch of queries and pages def late_interaction(q, d): # q: (B, Nq, D) query token vectors, d: (C, Nd, D) page patch vectors, all L2-normalized sim = torch.einsum("bnd,cmd->bcnm", q, d) # every query token vs every patch return sim.max(dim=3).values.sum(dim=2) # max over patches, sum over query tokens -> (B, C) B, Nq, Nd, D = 4, 6, 1030, 128 # 1030 vectors of 128 dims per page, as in ColPali q = F.normalize(torch.randn(B, Nq, D), dim=-1) d = F.normalize(torch.randn(B, Nd, D), dim=-1) d[:, :Nq] = q + 0.05 * torch.randn(B, Nq, D) # plant each query's tokens on its own page d = F.normalize(d, dim=-1) S = late_interaction(q, d) # (4, 4); diagonal = matching pairs print(S.argmax(dim=1).tolist(), [round(x, 2) for x in S.diag().tolist()]) # [0, 1, 2, 3] [5.19, 5.2, 5.2, 5.26]; off-diagonal scores are about 1.7, not 0 # 4) ColPali training loss, formula 218.12: softplus(hardest in-batch negative - positive) pos = S.diag() neg = S.masked_fill(torch.eye(B, dtype=torch.bool), float("-inf")).max(dim=1).values loss = F.softplus(neg - pos).mean() print(round(loss.item(), 4)) # 0.0308 # 5) Storage per page in float16, formula 218.13 print(Nd * D * 2, "bytes =", Nd * D * 2 / 1024, "KiB") # 263680 bytes = 257.5 KiB
Two things to notice in the output. The matching pairs score about 5.2 while the non-matching pairs score about 1.7, not 0: with 1,030 patches, each of the 6 query tokens always finds some patch with a cosine near 0.29 by chance. Scores of different queries are therefore not comparable on an absolute scale; only the ranking for one query is. And the loss is small (0.031) because every positive beats its hardest negative by more than 3.
Where this is used
| Application | Task and method |
|---|---|
| Accounts payable, expense apps | Invoice and receipt extraction to JSON; OCR pipeline or small OCR-free model, validation rules on totals |
| Banking, insurance, KYC onboarding | Form and ID-document extraction; classification of document type first |
| Enterprise search and RAG | Parse PDFs to Markdown, or ColPali page retrieval plus a VLM reader |
| Scientific literature | PDF to Markdown with LaTeX (Nougat-style), table and figure extraction |
| LLM pre-training data | Converting billions of PDF pages into clean text; order and tables decide data quality |
| Lecture and meeting video | Treat key frames of slides as pages: OCR or a VLM reads them, ColPali indexes them for Video-RAG (session 210) |
| Sports and news video | Read scoreboards, clocks and tickers frame by frame; text changes become events for key-moment detection |
| Video retrieval and moderation | On-screen text as an extra signal for search and for policy checks (phone numbers, scam text) |
Common mistakes
- Running OCR on born-digital PDFs. The text layer is already exact. OCR can only add errors. Check for a text layer first (but watch for PDFs whose text layer is itself bad OCR).
- Too low resolution. Squashing a page to a 448-pixel square or a tight token budget makes small print unreadable (Figure 3). Keep the aspect ratio and check the pixel height of your smallest important text.
- Ignoring reading order. Raster order on multi-column pages feeds the LLM mixed sentences (Figure 2). Headers and footers break sentences across pages.
- Reporting ANLS for business fields. ANLS gives 0.92 to a wrong invoice number (Figure 4). Use exact match after normalization for IDs, dates and amounts.
- Trusting fluent VLM output. A VLM that cannot read a value may invent one. Ask for the source box or quote, check it against the PDF text or a second reader, and route low-confidence pages to people.
- Flattening tables to plain text. Losing rows and columns turns a table into a list of numbers with no meaning. Keep tables as HTML or Markdown, and evaluate them with TEDS.
- Forgetting the cost of multi-vector indexes. 257.5 KiB per page in float16 is 263.7 GB for a million pages. Plan pooling, quantization or a two-stage search before you build.
Interview questions
OCR pipeline or end-to-end VLM: how do you choose?core
What does LayoutLM add to BERT, and why does it help?basic
How does Donut produce JSON without OCR?core
Define ANLS. What are its weaknesses?core
How many visual tokens does an A4 page cost in Qwen2-VL, and what does that imply?core
Explain ColPali’s scoring and why it beats a single vector per page.core
Write the ColPali training loss and explain the hardest negative.advanced
Your multi-vector index for 10 million pages is too large. What do you do?advanced
How would you get a correct reading order for a two-column paper?basic
How do you evaluate table extraction?basic
Design: extract 20 fields from 1 million mixed invoices per day.advanced
Exercises
- A 12-character ID is read at 99.5% character accuracy. What is the probability the whole ID is right? What about a page with 20 fields at 99% per field?
- Compute the score of answer “12.5O” (letter O) against gold “12.50”, and of “125.0”. Which is more wrong for a business user, and which gets the higher ANLS?
- A US Letter page (8.5 × 11 in) is rendered at 200 dpi. How many Qwen2-VL tokens does it cost with the default budget?
- A query has 2 tokens and a page has 3 patches. The dot products are: token 1 with patches \((0.2, 0.9, 0.1)\), token 2 with patches \((0.7, 0.3, 0.4)\). Compute LI.
- How much storage does a ColPali index of 50,000 pages need with pool factor 3 in float16?
- For ChartQA with gold answer 43, which predictions are accepted: 41.2, 41, 45?
Answerscheck
Key takeaways
- Document AI = extraction, DocVQA, parsing to Markdown, classification and retrieval over pages where position carries meaning.
- OCR pipelines are cheap, auditable and modular, but errors multiply across stages (218.1); LayoutLM adds box positions to text tokens (218.2, 218.3).
- Reading order (XY-cut), tables (TEDS, 218.4) and charts (relaxed accuracy, 218.5) are where naive text extraction breaks.
- OCR-free models (Donut, Pix2Struct, VLMs) map image to tokens with one cross-entropy loss (218.6); resolution decides what they can read, at 28 × 28 pixels per Qwen2-VL token (218.7).
- DocVQA uses ANLS (218.8, 218.9), which forgives small slips but also rewards near-miss wrong IDs; extraction uses field F1 (218.10).
- ColPali embeds the page image as 1,030 vectors of 128 dims and scores with MaxSim (218.11), trained with a hardest-negative softplus loss (218.12); the price is storage (218.13).
Key formulas
References for this session
- Kim et al. (2021). OCR-free Document Understanding Transformer
- Faysse et al. (2024). ColPali: Efficient Document Retrieval with Vision Language Models
- Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 8: Document AI
- Kim et al. (2021). OCR-free Document Understanding Transformer
- Faysse et al. (2024). ColPali: Efficient Document Retrieval with Vision Language Models
- Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 8: Document AI
- Khattab & Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Xu, Li, Cui, Huang, Wei & Zhou (2019). LayoutLM: Pre-training of Text and Layout for Document Image Understanding
- Xu et al. (2020). LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
- Lee et al. (2022). Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
- Blecher, Cucurull, Scialom & Stojnic (2023). Nougat: Neural Optical Understanding for Academic Documents
- Pfitzmann, Auer, Dolfi, Nassar & Staar (2022). DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis
- Zhong, ShafieiBavani & Jimeno Yepes (2019). Image-based table recognition: data, model, and evaluation
- Masry, Long, Tan, Joty & Hoque (2022). ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- Biten et al. (2019). Scene Text Visual Question Answering
- Nagy & Seth (1984). Hierarchical representation of optically scanned documents