LLM & VLM courseBlog
191Track 7: Vision and VLMs · session 29 of 38

VLM tasks and how models answer them

A vision-language model answers every task the same way: it writes text. A caption, a one-word answer, a bounding box, a list of points, the text of a receipt and a chart value are all token sequences. This session shows how each task is asked and answered: the prompt formats, how coordinates become text and back, why counting works better by pointing, what resolution reading needs, and how to get a clean answer out of a model that likes to talk.

Time: 2.5 to 3.5 hours (reading 1.5 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Session 177 (anatomy of a VLM) and session 180 (high resolution); session 79 (decoding strategies) helps

What you will learn

  • Why one formula, the probability of the answer text given the image and the prompt, covers captioning, VQA, reasoning, grounding, counting and OCR
  • The real prompt formats: PaliGemma task prefixes, LLaVA-1.5 format instructions, Florence-2 task tokens, and chat prompts with a JSON schema (Qwen2.5-VL)
  • How boxes and points become tokens: bins and location tokens (Kosmos-2, PaliGemma, Florence-2), numbers in [0, 1000) (Qwen2-VL), absolute pixels (Qwen2.5-VL) and 0 to 100 points (Molmo), with the error each one adds
  • The resize trap: Qwen2.5-VL coordinates live in the resized image, and how to map them back
  • Counting by pointing first (Molmo), and what it costs in output tokens
  • OCR, charts and documents: how many visual tokens a page needs, and what the pixel budget does to small text
  • Getting the answer out: free generation with a parser, format instructions, and scoring the options by log-probability, including the length-normalization trap

Why this matters

Show a modern vision-language model (VLM) a photo of a desk and ask five things: “Describe it.”, “What color is the mug?”, “Where is the mug?”, “How many apples are there?”, “What does the note say?”. One model with one set of weights answers all five. It does not have a caption head, a detection head and an OCR engine. It has one output: the next token. The box, the count and the note’s text are all written as text.

This is a big change from the computer vision of ten years ago, where every task had its own model, its own output layer and its own loss. It is also the source of most practical problems with VLMs. If the box is text, then someone chose a text format for boxes, and your code must write the prompt and read the answer in exactly that format. If the count is text, the model can write “3” without looking carefully, or write a whole paragraph when you wanted a number. In this session we go through the main task families and, for each one, answer three questions:

  • How is the task asked? (the prompt format the model was trained on)
  • How is the answer written? (free text, a short phrase, a letter, coordinates, a JSON list)
  • How do you get the answer back out, and what error does the format add?

The tasks are: captioning, visual question answering (VQA) and visual reasoning; grounding and localization (boxes and points as text); counting; reading text (OCR), charts and documents. We use real formats from PaliGemma, Qwen2-VL, Qwen2.5-VL, Molmo, Kosmos-2 and Florence-2. The book discusses the main VLM tasks and their data in chapter 2 (Noyan et al.); here we focus on how the answer is written and parsed. The next session, session 192, turns these answers into scores.

These questions come from real systems. Product teams use VLMs to read receipts and forms, to find UI elements for agents that click on screens, to count items on shelves, and to find the moment in a video where something happens. In every case the model writes text, and a small parser turns it into a number, a box or a timestamp. Most production bugs live in that parser and in the prompt that feeds it.

One recipe for every task: text in, text out

A VLM in the unified-sequence style (session 177) reads the image as visual tokens, then the prompt, and then writes the answer one token at a time. Whatever the task, the model computes the same thing:

\[ p_\theta(\mathbf{y} \mid I, \mathbf{x}) = \prod_{t=1}^{T} p_\theta\big(y_t \mid y_{\lt t},\, I,\, \mathbf{x}\big) \tag{191.1} \]
Read the symbols
\(I\)
the image (or video frames), already turned into visual tokens by the encoder and the connector
\(\mathbf{x}\)
the prompt: the instruction, the question, and any format rule such as “answer with one word”
\(\mathbf{y} = (y_1, \dots, y_T)\)
the answer as tokens: words, digits, special location tokens, JSON brackets, the end-of-turn token
\(p_\theta(y_t \mid \dots)\)
the probability of the next token, from the softmax over the vocabulary (see session 79)

So the only things that differ between tasks are \(\mathbf{x}\) (how you ask) and the shape of \(\mathbf{y}\) (how the answer is written). We can sort tasks by the shape of the answer:

Answer shapeTasksHow you read it out
Free text, longDetailed captions, explanations, reasoning tracesUse as is; judge quality by humans or another model
Short textVQA, short captions, OCR of one fieldNormalize (lower case, strip punctuation) and compare
A choiceMultiple-choice questions, yes/noRead the letter, or score each option by its probability
A numberCounting, chart values, clock readingParse the number; compare with a tolerance
CoordinatesGrounding, detection, pointing, UI clicksParse with a regex or JSON; map back to image pixels
Structured textDocuments to Markdown or HTML, tables, forms, key-value extractionParse the structure; check the schema

Training on (191.1) is the same for all tasks too: next-token cross-entropy on the answer tokens only (session 188). This is why a VLM can learn a new task from data alone, with no new layer: you only need a format for the answer and examples written in that format. It is also why the format matters so much. A model trained to write boxes as (646,379),(793,625) will not write them as JSON just because you ask nicely, and a parser written for one model is wrong for the next.

Figure 1. One 640 × 480 image, five tasks, four output formats. The output strings are written by hand in the format each model was trained on (they are not real model runs); the wording of the chat instructions is ours. Grounding and counting answers are parsed back into pixels of the original image and drawn: solid = decoded answer, dashed = the true box. Token counts use the Qwen2 tokenizer (also used by Molmo-7B-D); PaliGemma uses the Gemma tokenizer, so we do not count its tokens.
How to read Figure 1

Choose a task, then a model. Caption with PaliGemma: the prompt is just the prefix caption en. Switch to Molmo: the same task gives a long, dense caption (52 tokens against 15), because Molmo was trained on long spoken descriptions. Choose “Ground the mug”: PaliGemma writes four location tokens in the order ymin, xmin, ymax, xmax; Qwen2-VL writes numbers in [0, 1000); Qwen2.5-VL writes JSON with pixel coordinates of its resized 644 × 476 input; Molmo writes a point, not a box. All three boxes decode to an IoU of 0.98 to 0.99 with the true box. Choose “Count apples” with Molmo: it points to each apple, then gives the total, which costs 70 tokens; Qwen2.5-VL’s detect-then-count answer costs 111; PaliGemma and Qwen2-VL just write “3”.

Captioning, VQA and visual reasoning

Captioning: the length is part of the task

Captioning means writing a description of the image. The hard part is not the words; it is the length and style. A COCO-style caption is one sentence (“A blue mug and three apples on a table.”). A dense caption is a paragraph that lists objects, colors, positions and text. Both are correct answers to “describe the image”, so the model needs a signal for which one you want. Three kinds of signal are used in practice:

  • Task prefixes. PaliGemma is trained with a fixed prefix for each task, for example caption en (with a language code) or ocr, “to avoid conflicting learning signals across skills” [Certain, from the paper]. Florence-2 uses task tokens such as <CAPTION>, <DETAILED_CAPTION> and <MORE_DETAILED_CAPTION> [Certain, from its model card].
  • Format instructions in natural language. LLaVA-1.5 adds a sentence to each academic training sample, such as “Provide a one-sentence caption for the provided image.” [Certain, from the paper].
  • The training data itself. Molmo’s captions (PixMo-Cap) come from annotators who spoke about each image for at least 60 seconds; the set has 712k images with 1.3M transcripts and captions [Certain, from the paper]. A model trained on such data writes long captions by default.

VQA: short answers need a format rule

Visual question answering means answering a question about the image: “What color is the mug?” → “Blue.” Academic VQA datasets (VQAv2, GQA, OK-VQA) have answers of one to three words, because they are scored by exact match with human answers. Chat models like to answer in full sentences. LLaVA-1.5 found that a vague prompt format (such as “Q: … A:”) made the model either too short in chat or too long on benchmarks, and fixed it with an explicit instruction: “Answer the question using a single word or phrase.” [Certain, from the paper]. For multiple choice it uses “Answer with the option’s letter from the given choices directly.” [Certain]. These two sentences are now the standard prompts for evaluating many open VLMs [Likely].

The lesson is general: a format instruction is part of the task. The same question with and without “one word” is two different tasks, and you should evaluate the one your product needs.

Visual reasoning: think first, then answer

Visual reasoning tasks need several steps: “Is the mug to the left of the note, and is it taller than the apples?”, a geometry problem from a figure, or a science question with a diagram (MMMU, session 192). Two answer styles exist:

  1. Direct answer: one letter or one number. Fast, easy to parse, and the whole computation must happen inside the forward passes before the first answer token.
  2. Reasoning, then answer: the model first writes its steps (a chain of thought, session 72) and then a final line such as “Answer: C”. The written steps give the model more computation per question. The parser must then find the final answer inside a long text.

Reasoning-first answers are what GRPO-style training rewards in session 190: the reward checks only the final answer, so the format of that final line must be strict. When you use such a model, ask for a fixed final line (for example “End with: Answer: <letter>”) and parse only that line.

Grounding: boxes and points as text

Grounding (or localization) means linking words to places in the image. Referring expression comprehension gives a phrase (“the mug on the right”) and asks for its box. Detection asks for all boxes of one or more classes. Pointing asks for one point per object. Grounded captioning writes a caption where each phrase carries its box. In a VLM all of these produce coordinates as tokens. The first idea came from Pix2Seq (2021), which trained a model to write detections as sequences of discrete tokens: quantized coordinates followed by a class token [Certain, from the paper]. VLMs then adopted it.

Quantization: from pixels to bins

A coordinate is a real number, but tokens are discrete. The simplest map cuts the image width \(W\) into \(N\) equal bins and writes the bin number:

\[ k = \min\!\Big(N - 1,\ \Big\lfloor \frac{N x}{W} \Big\rfloor\Big), \qquad \hat x = \Big(k + \tfrac{1}{2}\Big)\frac{W}{N} \tag{191.2} \]
Read the symbols
\(x\)
the true coordinate in pixels (the same rule is used for \(y\) with the height \(H\))
\(N\)
the number of bins: 32 in Kosmos-2, 1,000 in Florence-2 and Qwen2-VL, 1,024 in PaliGemma
\(k\)
the bin index that the model writes, as a special token (<loc0388>) or as digits (388)
\(\hat x\)
the decoded coordinate; using the center of the bin halves the worst error compared with using its left edge \(k W / N\)

Decoding to the bin center, the error of each coordinate is at most half a bin. If each of the four edges of a box can move by up to that much, the worst case is the box that shrinks on all sides (we checked this with a brute-force search over all edge shifts), so

\[ |x - \hat x| \le \delta_x = \frac{W}{2N}, \qquad \text{IoU} \ \ge\ \Big(1 - \frac{2\delta_x}{w}\Big)\Big(1 - \frac{2\delta_y}{h}\Big) \tag{191.3} \]
Read the symbols
\(\delta_x, \delta_y\)
half a bin in pixels, horizontally (\(W/2N\)) and vertically (\(H/2N\))
\(w, h\)
the width and height of the object’s box in pixels
IoU
intersection over union of the true box and the decoded box: 1 is perfect, and detection benchmarks usually count IoU \(\ge 0.5\) as a hit

The bound depends on the ratio of the bin size to the object size. Quantization is harmless for big objects and decides everything for small ones.

Worked example: a 40 × 30 px cup in a 1080p frame

The frame is 1920 × 1080; the cup’s box is \(x\) from 100 to 140, \(y\) from 500 to 530.

  • \(N = 32\) (Kosmos-2’s grid): a bin is 60 × 33.75 px, larger than the cup. The box becomes bins 1–2 and 14–15, decoded as 90–150 × 489.4–523.1: IoU 0.40. The worst case is 0.
  • \(N = 100\): bins of 19.2 × 10.8 px. IoU 0.64 here; the worst case is \((1 - 19.2/40)(1 - 10.8/30) = 0.33\).
  • \(N = 1000\) (Qwen2-VL, Florence-2): \(\delta_x = 0.96\), \(\delta_y = 0.54\) px. IoU 0.94; worst case \((1 - 1.92/40)(1 - 1.08/30) = 0.918\).
  • \(N = 1024\) (PaliGemma): IoU 0.97; worst case 0.920.

With about 1,000 bins, quantization costs at most 8% of IoU even for a 30-pixel object in a 1080p frame. With a 32 × 32 grid, small objects cannot be localized at all.

Five real formats

Different model families made different choices. They differ in the number of bins, in whether coordinates are special tokens or digits, in the order of the four numbers, and in what the numbers are relative to.

ModelWhat the model writes for one boxCoordinates
Kosmos-2<p> It </p><box><loc44><loc863></box>A 32 × 32 grid (1,024 location tokens, 7 × 7 px cells of a 224 px image); one token for the top-left cell, one for the bottom-right cell [Certain]
PaliGemma<loc0388><loc0662><loc0640><loc0812> mug1,024 location tokens <loc0000> to <loc1023>, normalized image coordinates, order ymin, xmin, ymax, xmax; several objects separated by ; [Certain]
Florence-2Location tokens for x0, y0, x1, y1; quad boxes and polygons as more corners, clockwise1,000 bins [Certain]; tokens are named <loc_0> to <loc_999> [Likely]
Qwen2-VL<|object_ref_start|>the mug<|object_ref_end|><|box_start|>(646,379),(793,625)<|box_end|>Plain digits, normalized to [0, 1000), top-left then bottom-right [Certain]
Qwen2.5-VL{"bbox_2d": [417, 180, 511, 298], "label": "mug"}Absolute pixels of the resized input image, in JSON [Certain, from the paper and the official cookbook]
Molmo<point x="72.0" y="50.2" alt="mug">mug</point>Points, not boxes; plain text from 0.0 to 100.0 with one decimal, in percent of width and height [Certain]

Two design questions hide in this table.

Special tokens or digits? A location token (<loc0388>) is one token per coordinate, so a box costs 4 tokens; but the model must learn 1,024 new embeddings from scratch, with no link between neighboring bins. Digits reuse the language model’s number tokens. The Qwen2 tokenizer splits numbers into single digits (its pre-tokenizer pattern has \p{N} alone) [Certain, from tokenizer.json], so (646,379),(793,625) is 17 tokens and the whole grounded phrase above is 23. Molmo’s authors tried both and report that special point tokens performed “substantially worse” than the plain-text representation [Certain, from the paper’s ablation].

Normalized or absolute? Normalized coordinates (bins of the full image) do not depend on the resolution the model saw. Absolute coordinates tell the model the real size of the image and of the objects; Qwen2.5-VL uses them for this reason [Certain that it uses absolute coordinates; that this helps is the authors’ claim, Likely]. But absolute coordinates are only useful if you know which image they are absolute to.

The resize trap

Qwen2-VL and Qwen2.5-VL resize every image so that both sides are multiples of 28 (14-pixel patches, then a 2 × 2 merge) and the pixel count stays inside a budget (session 180). The resize function (smart_resize in qwen_vl_utils) is

\[ H' = 28 \cdot \operatorname{round}\!\Big(\frac{H}{28}\Big),\quad W' = 28 \cdot \operatorname{round}\!\Big(\frac{W}{28}\Big); \quad \text{if } H'W' \gt P_{\max}:\ \ \beta = \sqrt{\frac{HW}{P_{\max}}},\ \ H' = 28 \Big\lfloor \frac{H}{28\beta} \Big\rfloor,\ \ W' = 28 \Big\lfloor \frac{W}{28\beta} \Big\rfloor \tag{191.4} \]
Read the symbols
\(H, W\)
the size of the original image in pixels
\(H', W'\)
the size the vision encoder actually sees; the model gets \((H'/28)(W'/28)\) visual tokens
\(P_{\max}\)
the pixel budget, max_pixels: \(16{,}384 \times 28^2 = 12{,}845{,}056\) by default; examples often use \(1{,}280 \times 28^2 = 1{,}003{,}520\). A minimum budget works the same way upward
\(\beta\)
the downscale factor when the image is over the budget

Qwen2.5-VL writes box coordinates in pixels of the \(W' \times H'\) image. The official cookbook computes input_height = image_grid_thw[1] * 14 and divides the model’s numbers by it before drawing on the original [Certain]. To map back:

\[ x_{\text{orig}} = x_{\text{model}} \cdot \frac{W}{W'}, \qquad y_{\text{orig}} = y_{\text{model}} \cdot \frac{H}{H'} \tag{191.5} \]
Worked example: the same cup, two budgets

With the default budget, the 1920 × 1080 frame becomes \(W' \times H' = 1932 \times 1092\) (round 68.57 up to 69 and 38.57 up to 39 patches of 28 px): 2,691 visual tokens. A point the model writes at \(x = 1900\) is at \(1900 \cdot 1920/1932 = 1888.2\) in the original: 11.8 px off if you forget (191.5). Annoying, not fatal.

With max_pixels \(= 1{,}003{,}520\): \(\beta = \sqrt{1920 \cdot 1080 / 1{,}003{,}520} = 1.4375\), so \(W' = 28 \lfloor 47.70 \rfloor = 1316\) and \(H' = 28 \lfloor 26.83 \rfloor = 728\): 1,222 visual tokens. The cup at (100, 500)–(140, 530) is written as [69, 337, 96, 357]. Drawn on the original frame without (191.5), this box does not touch the cup at all: IoU 0. Every box shrinks toward the top-left corner by a factor of 0.685.

The rule for every model: coordinates are relative to something (the full image, a resized image, a crop, a tile). Find out what, from the processor code, not from the prompt. When the image is tiled (session 180), models are trained to give coordinates of the full image, but check this for your model before you trust it.

Algorithm 1: ask for a box and read it back in original pixels
  1. input: image of size \(W \times H\); a phrase; the model family
  2. run the model’s own processor; record the size it actually fed to the encoder (\(W', H'\)) // from image_grid_thw, or the resize setting
  3. write the prompt in the trained format: task prefix (detect mug), instruction plus format (“output its bbox coordinates using JSON format”), or reference tokens
  4. decode greedily, with skip_special_tokens=False if coordinates are special tokens
  5. parse: regex for <locNNNN> or (x,y); JSON parser for JSON (strip the code fence first); XML-like parser for points
  6. map to the original image: bins with (191.2), [0, 1000) with \(x W / 1000\), absolute with (191.5), percent with \(x W / 100\)
  7. clip to \([0, W] \times [0, H]\); swap corners if \(x_1 \gt x_2\); drop boxes with zero area
  8. return boxes with labels; log every answer that failed to parse
true boxdecoded box (this position)worst case, formula 191.3bin edges
Figure 2. A square object of side \(s\) pixels, with its top-left corner at 37% of the width and 41% of the height, quantized with \(N\) bins per side by formula (191.2) and decoded to bin centers. The view is zoomed to the object. The worst case is the bound (191.3): every edge moved inward by half a bin.
How to read Figure 2

Start at 1,000 bins, 1920 × 1080 and a 40 px object: half a bin is 0.96 × 0.54 px, the decoded box at this position has IoU 0.937 and the worst case is 0.926. Choose 32 bins: one bin (60 × 33.75 px) is bigger than the object, the IoU at this position falls to 0.422 and the worst case is 0. Move to 224 × 224 (Kosmos-2’s own input size): a 32-bin cell is 7 px and a 40 px object keeps a worst case of 0.681. Go to 3840 × 2160 with 1,000 bins and shrink the object to 10 px: the worst case falls to 0.483, below the usual 0.5 hit threshold, so a 4K screenshot with small UI icons needs more bins, a crop, or absolute pixel coordinates.

Counting: point first, then count

Counting looks like VQA with a number as the answer: “How many apples are there?” → “3”. It is one of the weakest skills of VLMs [Likely; it is the reason Molmo and PaliGemma built dedicated counting data and benchmarks]. Writing one digit gives the model no place to keep track: the count must be computed inside the forward pass, over hundreds of visual tokens, before the first answer token.

Molmo’s fix is to make the model point first. Asked to count, it writes one point per object, ordered top-down and left to right, with each point numbered, and then the total [Certain]. The authors note that numbering makes counting easy, because the total is the number of the last point, and they report that this “chain-of-thought point-then-count performs significantly better than generating a count only” [Certain, from the paper]. Their pointing data, PixMo-Points, has 2.3M question-point pairs on 223k images [Certain]. Qwen2.5-VL does the same with boxes, a “detect then count” prompt, and reports 93.6 on CountBench for the 72B model [Certain, from the paper]. CountBenchQA itself is a 540-image benchmark introduced with PaliGemma [Certain].

Pointing works for three reasons. Each point is a small, local decision that the model can check against the image. The written points stay in the context, so the model can see which objects are already counted. And the count becomes a by-product of a task that is easy to supervise and to check: a human (or a reward function, session 190) can see a missing or duplicate point.

The price is output length. With the Qwen2 tokenizer, Molmo’s multi-point string costs

\[ T(K) = \underbrace{8 + 2L}_{\text{tag, label twice}} + \sum_{k=1}^{K} \underbrace{\big(6 + 2\,d(k) + c(x_k) + c(y_k)\big)}_{\texttt{ xk="…" yk="…"}} \tag{191.6} \]
Read the symbols
\(K\)
the number of points (for \(K = 1\) Molmo uses a shorter <point x=… y=…> tag)
\(L\)
the number of tokens of the label (“person” is 1, “apples” is 2: app + les)
\(d(k)\)
the number of digits of the point index \(k\) (1 for points 1–9, 2 for 10–99)
\(c(x_k)\)
the characters of the coordinate, each one a token: 3 for “5.0”, 4 for “45.8”, 5 for “100.0”

We checked (191.6) against the real tokenizer on 400 random point lists: it matches exactly. So each object costs about 16 tokens (18 from the tenth point on). Counting 30 people by pointing is about 500 output tokens, while the bare answer “30” is 2 tokens. Decoding is the slow phase of inference (session 196), so for a latency-critical service you may point during training (to teach counting) and still ask for a bare number when the count is small, or point only when the model’s first guess is large.

Worked example: three apples, three formats

For the desk image of Figure 1 (Qwen2 tokenizer, exact counts):

  • bare answer “3”: 1 token;
  • Molmo, Counting the <points x1="45.8" y1="67.9" x2="20.0" y2="69.0" x3="32.7" y3="72.1" alt="apples">apples</points> shows a total of 3.: 70 tokens (the points tag alone is 60, as (191.6) gives: “apples” is 2 tokens, so \(8 + 2 \cdot 2 + 3 \times 16\)). The wording around the tag follows Molmo’s demo outputs [Likely];
  • Qwen2.5-VL, three JSON boxes in a code fence and “There are 3 apples.”: 111 tokens.
Figure 3. Counting by pointing. A fixed (seeded) image of 640 × 400 with 14 apples and 3 leaves. “Point to the next apple” adds points in Molmo’s order (top to bottom by center; ties left to right). You can also click anywhere to add a point yourself, or click a point again to remove it. The text is the answer in Molmo’s format; the token count uses formula (191.6), which matches the Qwen2 tokenizer exactly.
How to read Figure 3

Press “Point to all”: 14 points, 14 distinct apples hit, the count is 14 and the points string costs 246 tokens, against 2 tokens for the bare answer “14”. Each point is about 16 tokens, and 18 from point 10 on, because the index “10” is two digits in both x10 and y10. Now reset and play the model yourself: click one apple twice in different places (a duplicate), or click a leaf (a false point). The count, which is just the number of points, is now wrong, and you can see exactly why. That is the main practical gain of pointing: a wrong count comes with its evidence.

OCR, charts and documents

OCR (optical character recognition) means reading the text in an image. In a VLM it appears in several tasks: transcribe all text (PaliGemma’s ocr prefix, Florence-2’s <OCR>), transcribe with positions (<OCR_WITH_REGION>, or Qwen2.5-VL’s document parsing), answer a question that needs reading (TextVQA, DocVQA), and read values from a chart (ChartQA). Older pipelines ran a separate OCR engine and passed its text to the model. OCR-free models such as Donut showed that a model can learn to read directly from pixels [Certain]; today’s VLMs do the same, and their main limit is resolution.

Reading needs pixels

A letter must cover enough pixels for the encoder to tell “8” from “B”. After the resize (191.4), the height of the text in pixels is

\[ e = s \cdot \frac{p}{72} \cdot \text{dpi}, \qquad s = \frac{W'}{W}, \qquad v = \frac{H'}{28} \cdot \frac{W'}{28} \tag{191.7} \]
Read the symbols
\(p\)
the font size in points (1 point = 1/72 inch); body text is usually 9 to 12 pt
dpi
dots per inch of the scan or render: an A4 page at 150 dpi is 1240 × 1754 px
\(s\)
the resize factor applied by the processor; below 1 when the page is over the pixel budget
\(e\)
the font size (the em, roughly the line height without spacing) in pixels of the image the encoder sees
\(v\)
the number of visual tokens for the page; each covers a 28 × 28 px square
Worked example: one A4 page at 150 dpi, 10 pt text

At 150 dpi, 10 pt is \(10/72 \cdot 150 = 20.8\) px. With (191.4) and (191.7):

  • default budget: the page becomes 1232 × 1764, \(v = 2{,}772\) tokens, \(e = 20.7\) px;
  • \(P_{\max} = 1{,}280 \cdot 28^2\): 840 × 1176, \(v = 1{,}260\), \(e = 14.1\) px;
  • \(P_{\max} = 768 \cdot 28^2\): 644 × 896, \(v = 736\), \(e = 10.8\) px;
  • \(P_{\max} = 256 \cdot 28^2\): 364 × 532, \(v = 247\), \(e = 6.1\) px, so one 28 px token covers almost four lines of text (with 1.2 line spacing).

A page full of text needs thousands of visual tokens to stay readable. This is why document models use tiles or native resolution (session 180) and why token reduction (session 183) is risky for OCR: dropping tokens that look “empty” in a photo can delete half a word in a scan.

How small can text get? There is no single threshold: it depends on the encoder, the font and the training data. Measure it: render your own documents at several budgets and plot accuracy against \(e\). Figure 4 shows what the encoder receives.

Figure 4. An A4 page scanned at 150 dpi (1240 × 1754 px) goes through Qwen2.5-VL’s resize (191.4) with the chosen budget. A line of the page is drawn at the pixel size the encoder receives and then enlarged without smoothing, so each square you see is one pixel. Grid lines mark the 28 × 28 px squares that become one visual token each.
How to read Figure 4

At a budget of 1,280 and 10 pt, the page is 840 × 1176 px (1,260 tokens) and the text is 14.1 px high: the digits are clear. Drop the budget to 256: 247 tokens, 6.1 px text, and “1,284.50” turns into blobs where 3 and 8 look the same. Raise the font to 14 pt at the same budget: the text is 8.6 px high, and the digits begin to separate again. The budget and the font size trade against each other through (191.7). Small print in contracts and invoices is exactly where a low budget fails first.

Charts and documents: structure in, structure out

Charts add arithmetic to reading: “What is the difference between the tallest and shortest bars?” needs the values (read from the axis), then a computation. Errors come from both steps. A good prompt asks the model to first write the values it reads, then compute; this is a chain of thought with a checkable middle step. Chart answers are scored with a numeric tolerance, not exact match (session 192).

Documents (forms, invoices, papers, slides) have layout: headings, tables, key-value pairs. Two output styles are common. Question answering (DocVQA: “What is the invoice total?”) returns one span. Parsing returns the whole document as structured text. Qwen2.5-VL’s document parsing writes HTML in which each element carries its box, as an attribute data-bbox="x1 y1 x2 y2" [Certain, from the paper]; others write Markdown. For extraction into your own schema, ask for JSON with fixed keys and validate it (session 146).

Much of the training data for these skills is synthetic, because real annotated documents are rare and private. Molmo’s PixMo-Docs prompted a language model to write code that renders 255k text- and figure-heavy images (charts, tables, documents), then generated 2.3M question-answer pairs from the code, which gives exact answers for free [Certain, from the paper].

Prompt formats for each task

We can now put the prompts in one place. There are three styles, and a model is trained on one of them (or a mix):

  1. Task prefixes or task tokens (PaliGemma, Florence-2): short, fixed strings. Very reliable for the trained tasks, and not meant for free chat. Using a different wording than the trained prefix usually lowers quality.
  2. Instruction plus a format rule (LLaVA-1.5 and most chat VLMs): a natural question followed by a sentence that fixes the shape of the answer.
  3. Chat with a schema (Qwen2.5-VL and newer chat models): ask for JSON (or XML, or HTML) with named keys. The schema doubles as documentation for your parser.
TaskPrefix style (PaliGemma)Chat style (examples)Parse
Short captioncaption en“Provide a one-sentence caption for the provided image.” (LLaVA-1.5)none
Detailed caption—“Describe this image in detail.”; Florence-2: <MORE_DETAILED_CAPTION>none
Short VQAanswer en what color is the mug?question + “Answer the question using a single word or phrase.”normalize text
Multiple choice—question, options as “A. … B. …”, + “Answer with the option’s letter from the given choices directly.”first letter, or scoring (next section)
Reasoning—question + “Think step by step, then end with ‘Answer: X’.”regex on the last line
Detectiondetect mug ; apple“Locate every apple, output bbox coordinates in JSON format.” (Qwen2.5-VL)regex or JSON, then map back
Referring segmentationsegment mug (output: 4 location tokens + 16 of the 128 <segNNN> mask tokens [Likely for the 16])usually a box, then a separate mask model (session 168)decode mask tokens with the VQ-VAE decoder
Pointing and countinganswer en how many apples are there?“Point to the apples.” / “How many apples are there?” (Molmo points first)count the points
OCRocr“Read all the text in the image.”; Florence-2: <OCR>, <OCR_WITH_REGION>none, or boxes per line
Document parsing—“Parse this document into HTML.” (Qwen2.5-VL writes data-bbox attributes)HTML or Markdown parser
Video moment—“When does the person open the door? Give start and end in seconds.”parse two numbers; map to frames

Two more details matter. First, where the image goes: the chat template puts the image placeholder at a fixed place in the user turn, and the processor expands it into the visual tokens (session 9, session 188). Always build the prompt with the model’s own apply_chat_template; a hand-written template with the image after the question, or a missing system prompt, is a different input from the one the model was trained on. Second, visual prompts: instead of writing a box in the prompt, you can draw it on the image. The PaliGemma paper reports that a red box drawn on the image works about as well as giving the box coordinates in the prompt [Likely, from its appendix F].

Getting the answer out: generate, constrain or score

Once the model is trained, there are three ways to turn (191.1) into an answer:

  1. Generate and parse. Decode greedily (temperature 0) and parse the text. Simple and general. It fails when the model ignores the format: “The answer is C because…”, a box in the wrong syntax, JSON with a trailing comma.
  2. Constrain the decoding. At each step, allow only the tokens that keep the output valid: only “A”–“D”, only digits, only JSON that fits a schema (grammar-constrained decoding, session 146). The output always parses. Be careful: a constraint the model did not learn can push it into answers it would not have chosen.
  3. Score the options. For a closed set of answers, do not generate at all. Run one forward pass per option and compare their log-probabilities. This is how many benchmarks evaluate base models (session 40).

Scoring needs care, because options have different lengths. A longer option is a product of more probabilities below 1, so its raw log-probability is lower. The usual fix divides by a power of the length:

\[ \hat c = \argmax_{c \in \mathcal{C}} \ \frac{1}{|c|^{\alpha}} \sum_{t=1}^{|c|} \log p_\theta\big(c_t \mid c_{\lt t},\, I,\, \mathbf{x}\big) \tag{191.8} \]
Read the symbols
\(\mathcal{C}\)
the set of options (for example the four answers of a multiple-choice question)
\(|c|\)
the number of tokens of option \(c\)
\(\alpha\)
the length normalization: 0 = raw sum (favors short options), 1 = mean log-probability per token (favors long fluent options)

The other closed-set method scores only the letter: show the options in the prompt, and read the probability of the next token being “A”, “B”, “C” or “D”:

\[ \hat \ell = \argmax_{\ell \in \{\text{A}, \text{B}, \text{C}, \text{D}\}} \ p_\theta\big(\ell \mid I,\, \mathbf{x},\, \text{options}\big) \tag{191.9} \]

Letter scoring needs only one forward pass and has no length problem, but it needs a model that has learned the letter format (an instruction-tuned model), and it is known to be sensitive to the order of the options [Likely]. A cheap test is to shuffle the options and check that the answer follows the content, not the letter.

Worked example: the length trap (toy numbers)

Question: “What is the man holding?” The image shows a tennis racket. The per-token log-probabilities below are invented for the example: A “kite” \([-2.2]\); B “umbrella” \([-3.0]\); C “a tennis racket” \([-2.6, -0.5, -0.2]\); D “a bat” \([-1.5, -1.3]\).

  • \(\alpha = 0\): scores \(-2.2, -3.0, -3.3, -2.8\). The one-token “kite” wins, wrongly. The right answer has the lowest score, because its first token “a” is not very likely.
  • \(\alpha = 0.37\): D wins (\(-2.167\) against \(-2.198\) for C and \(-2.2\) for A). D overtakes A at \(\alpha = 0.348\) and C overtakes D at \(\alpha = 0.405\).
  • \(\alpha = 1\): means \(-2.2, -3.0, -1.1, -1.4\). C wins.
  • Letter scoring with next-token logits 0.2, −1.0, 1.6, 0.9 for A–D: softmax gives 0.136, 0.041, 0.550, 0.273. C wins.

The same model gives three different answers depending on one hyperparameter of the evaluation. Report the method with the number.

Figure 5. The four options of the worked example (toy log-probabilities). In “score full options” mode, each bar is the score (191.8); the longest bar to the right wins. In “score the letter” mode, the bars are the letter probabilities (191.9). The correct answer is C.
How to read Figure 5

At \(\alpha = 0\) the one-token “kite” wins with −2.2 and the correct “a tennis racket” is last with −3.3. Move \(\alpha\) slowly: at 0.35 “a bat” takes the lead, and from 0.41 “a tennis racket” wins. At \(\alpha = 1\) it wins clearly (−1.1 against −1.4). Switch to the letter mode: the slider no longer matters, and C wins with probability 0.550. When you compare two models on a multiple-choice benchmark, they must be scored with the same rule.

Algorithm 2: choose how to ask and how to read, per task
  1. input: a task, a model, the answer type your product needs
  2. find the trained format: prefix, format instruction or schema // model card, paper, processor code
  3. if closed set (letters, yes/no, labels): use letter scoring (191.9) for instruction-tuned models, option scoring (191.8) with a fixed \(\alpha\) for base models; shuffle options once to test order bias
  4. if number or count: allow reasoning or pointing first, then parse the last number; check its range
  5. if coordinates: Algorithm 1; test the round trip on a synthetic image with known boxes
  6. if structured output: ask for the schema, decode with a grammar if available, validate, and retry once on failure
  7. log the raw text of every answer that fails to parse and track the failure rate as a metric
  8. return the parsed answer, and the raw text for debugging

Code: write and read every answer format

This file has no model in it. It writes a box in the PaliGemma, Qwen2-VL and Qwen2.5-VL formats and points in Molmo’s format, parses them back with the same code a production parser would use, measures the IoU of the round trip, shows the resize trap, checks the worst-case bound (191.3), and scores multiple-choice options with (191.8) and (191.9). It runs on a CPU in well under a second and is deterministic. Do not save it as code.py: that name hides Python’s own code module, which import torch needs.

import math, re, json, torch
torch.manual_seed(0)

# ---------- 1) coordinates as text: four real formats ----------
def smart_resize(h, w, f=28, max_pixels=16384 * 28 * 28):    # Qwen2.5-VL resize (qwen_vl_utils)
    hb, wb = round(h / f) * f, round(w / f) * f
    if hb * wb > max_pixels:
        b = math.sqrt(h * w / max_pixels)
        hb, wb = math.floor(h / b / f) * f, math.floor(w / b / f) * f
    return hb, wb

def to_paligemma(b, W, H, label):            # 1024 bins, order ymin xmin ymax xmax
    k = lambda v, S: min(1023, math.floor(1024 * v / S))
    return "".join(f"<loc{k(v, S):04d}>" for v, S in [(b[1], H), (b[0], W), (b[3], H), (b[2], W)]) + " " + label

def to_qwen2vl(b, W, H, label):              # normalized to [0, 1000)
    x1, y1, x2, y2 = (math.floor(1000 * v / S) for v, S in zip(b, (W, H, W, H)))
    return f"<|object_ref_start|>{label}<|object_ref_end|><|box_start|>({x1},{y1}),({x2},{y2})<|box_end|>"

def to_qwen25vl(b, W, H, label, max_pixels=16384 * 28 * 28):   # absolute pixels of the RESIZED image
    Hm, Wm = smart_resize(H, W, max_pixels=max_pixels)
    xy = [round(v * S2 / S) for v, S, S2 in zip(b, (W, H, W, H), (Wm, Hm, Wm, Hm))]
    return json.dumps([{"bbox_2d": xy, "label": label}])

def to_molmo(pts, W, H, label):              # plain-text points, 0-100 with one decimal
    a = " ".join(f'x{i+1}="{100*x/W:.1f}" y{i+1}="{100*y/H:.1f}"' for i, (x, y) in enumerate(pts))
    return f'<points {a} alt="{label}">{label}</points>'

# ---------- 2) parse the text back into pixels of the ORIGINAL image ----------
def parse(fmt, s, W, H, max_pixels=16384 * 28 * 28):
    if fmt == "paligemma":
        y1, x1, y2, x2 = [int(k) for k in re.findall(r"<loc(\d{4})>", s)[:4]]
        return [(x1 + .5) * W / 1024, (y1 + .5) * H / 1024, (x2 + .5) * W / 1024, (y2 + .5) * H / 1024]
    if fmt == "qwen2vl":
        x1, y1, x2, y2 = map(int, re.findall(r"\((\d+),(\d+)\)", s)[0] + re.findall(r"\((\d+),(\d+)\)", s)[1])
        return [x1 * W / 1000, y1 * H / 1000, x2 * W / 1000, y2 * H / 1000]
    if fmt == "qwen25vl":
        Hm, Wm = smart_resize(H, W, max_pixels=max_pixels)
        x1, y1, x2, y2 = json.loads(s)[0]["bbox_2d"]
        return [x1 * W / Wm, y1 * H / Hm, x2 * W / Wm, y2 * H / Hm]
    if fmt == "molmo":
        xs = re.findall(r'x\d+="([\d.]+)"', s); ys = re.findall(r'y\d+="([\d.]+)"', s)
        return [(float(x) * W / 100, float(y) * H / 100) for x, y in zip(xs, ys)]

def iou(a, b):
    ix = max(0, min(a[2], b[2]) - max(a[0], b[0])); iy = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = ix * iy
    return inter / ((a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter)

W, H, mug = 640, 480, [414, 182, 508, 300]
for fmt, enc in [("paligemma", to_paligemma), ("qwen2vl", to_qwen2vl), ("qwen25vl", to_qwen25vl)]:
    s = enc(mug, W, H, "mug"); print(f"{fmt:9s} {s}  IoU={iou(mug, parse(fmt, s, W, H)):.4f}")
# paligemma <loc0388><loc0662><loc0640><loc0812> mug          IoU=0.9944
# qwen2vl   <|object_ref_start|>mug...(646,379),(793,625)...   IoU=0.9883
# qwen25vl  [{"bbox_2d": [417, 180, 511, 298], "label": "mug"}] IoU=0.9856

# the resize trap: a 1080p frame with a smaller pixel budget
s = to_qwen25vl([100, 500, 140, 530], 1920, 1080, "cup", max_pixels=1280 * 28 * 28)
print(s, smart_resize(1080, 1920, max_pixels=1280 * 28 * 28))   # [{"bbox_2d": [69, 337, 96, 357] ...}] (728, 1316)
print(f"{iou([100, 500, 140, 530], json.loads(s)[0]['bbox_2d']):.3f}")   # 0.000: used as original pixels, the box misses

# ---------- 3) point, then count ----------
apples = [(128, 331), (209, 346), (293, 326)]
pts = sorted(apples, key=lambda p: (p[1], p[0]))                # top to bottom, as Molmo orders points
s = to_molmo(pts, W, H, "apples"); print(s, "count =", len(parse("molmo", s, W, H)))
# <points x1="45.8" y1="67.9" x2="20.0" y2="69.0" x3="32.7" y3="72.1" alt="apples">apples</points> count = 3

# ---------- 4) the worst-case IoU of a quantized box (formula 191.3) ----------
def worst_iou(w, h, W, H, N):
    dx, dy = W / (2 * N), H / (2 * N)
    return max(0, 1 - 2 * dx / w) * max(0, 1 - 2 * dy / h)
for N in [32, 100, 1000, 1024]:
    print(N, f"{worst_iou(40, 30, 1920, 1080, N):.3f}")           # 32 0.000 | 100 0.333 | 1000 0.918 | 1024 0.920

# ---------- 5) multiple choice: score each option by its log-probability ----------
opt_logp = {"A kite": torch.tensor([-2.2]), "B umbrella": torch.tensor([-3.0]),
            "C a tennis racket": torch.tensor([-2.6, -0.5, -0.2]), "D a bat": torch.tensor([-1.5, -1.3])}
def choose(alpha):                                               # formula 191.8
    sc = {k: (v.sum() / len(v) ** alpha).item() for k, v in opt_logp.items()}
    return max(sc, key=sc.get)
print([choose(a) for a in (0.0, 0.37, 0.5, 1.0)])               # ['A kite', 'D a bat', 'C a tennis racket', 'C a tennis racket']
letter_logits = torch.tensor([0.2, -1.0, 1.6, 0.9])              # next-token logits of " A", " B", " C", " D"
p = torch.softmax(letter_logits, 0); print([f"{v:.3f}" for v in p], "ABCD"[int(p.argmax())])   # C with 0.550

Two lines in the output are the main lessons. The three box formats all survive the round trip with IoU above 0.98, so the formats themselves are fine. The resize trap gives IoU 0.000: the model’s answer was correct, and the parser made it wrong. In a real system, add one test like this per model: draw known boxes on a blank image, run the model, and check the IoU of the parsed answers.

Where this is used

ApplicationTask and answer formatWhat to watch
Alt text and image search indexingCaptioning, short or denseLength and style instruction; hallucinated details in dense captions (session 193)
Receipts, invoices, formsOCR plus key-value extraction into JSONPixel budget for small print (191.7); schema validation
Screen agents (click the “Submit” button)Pointing or boxes on screenshotsAbsolute vs normalized coordinates; resize trap on high-resolution screens (session 154)
Retail shelves, inventory, crowd estimatesCounting by pointing or detect-then-countOutput length grows with the count (191.6); duplicates and misses
Dashboards and reportsChart QA: read values, then computeAsk for the values first; numeric tolerance
Video key-moment detection, temporal groundingStart and end times as textFrame rate and absolute-time alignment; timestamps are coordinates in time
Video QA and Video-RAGVQA over sampled frames or retrieved clipsWhich frames the model saw; short-answer format for evaluation (session 210)
Action recognition and video classificationClosed-set choice over action labelsScore labels with (191.8) or letters (191.9), not free text
Video object tracking and segmentationA VLM gives a box or point in one frame; a tracker such as SAM 2 follows itCoordinate frame of the first box (session 204)
Content moderationYes/no or label questions, sometimes with reasonsLetter or label scoring with calibrated thresholds, not parsed sentences

Common mistakes

  • Using another model’s output format. Asking Qwen2-VL for JSON boxes, or parsing PaliGemma’s <loc> tokens in the x-first order. Read the model card and the processor code; test the round trip.
  • Forgetting the coordinate frame. Qwen2.5-VL’s pixels are pixels of the resized image (191.5); Qwen2-VL’s numbers are in [0, 1000); Molmo’s are percent. A wrong frame gives boxes that look plausible and are wrong.
  • Decoding with skip_special_tokens=True. For models whose coordinates are special tokens, this deletes the coordinates. Florence-2’s model card decodes with skip_special_tokens=False for this reason.
  • Asking for a bare count of many objects. Let the model point or detect first, then count; or at least check large counts with pointing.
  • Too small a pixel budget for documents. At \(256 \times 28^2\) pixels, 10 pt text on an A4 scan is about 6 px high. Raise the budget, tile, or crop the region of interest.
  • Free generation for closed-set answers. The model writes “It looks like a racket.” and the parser fails. Use letter scoring, option scoring or constrained decoding.
  • Comparing scores with different length normalization. \(\alpha = 0\) and \(\alpha = 1\) can pick different answers from the same probabilities (Figure 5). Fix the rule and report it.
  • Hand-written chat templates. The image placeholder in the wrong place, or a missing system prompt, changes the input. Use the processor’s apply_chat_template.

Interview questions

How does a VLM output a bounding box when it can only produce tokens?basic
It writes the coordinates as tokens. Either the image is cut into \(N\) bins per side and the model writes bin indices as special location tokens (Kosmos-2: a 32 × 32 grid; PaliGemma: 1,024 tokens <loc0000>–<loc1023> in the order ymin, xmin, ymax, xmax; Florence-2: 1,000 bins), or it writes the numbers as ordinary digits (Qwen2-VL: normalized to [0, 1000); Qwen2.5-VL: absolute pixels of the resized image in JSON; Molmo: points in percent with one decimal). The model learns it with ordinary next-token cross-entropy on examples written in that format; the parser then maps the numbers back to pixels.
How much localization error does quantizing coordinates add?core
With \(N\) bins and decoding to the bin center, each coordinate is off by at most \(\delta = W/(2N)\) (191.2, 191.3). The worst IoU is when all edges move inward: \((1 - 2\delta_x/w)(1 - 2\delta_y/h)\). For a 40 × 30 px object in a 1920 × 1080 frame this is 0.918 with 1,000 bins and 0 with 32 bins (a 32-bin cell is 60 px wide, larger than the object). So about 1,000 bins is enough for most objects in HD images, but small objects in 4K screenshots need crops, more bins or absolute coordinates.
Your Qwen2.5-VL boxes are all shifted toward the top-left and too small. What happened?core
Qwen2.5-VL writes coordinates in pixels of the image after its resize (smart_resize: sides rounded to multiples of 28 and scaled down to fit max_pixels). If you draw them on the original image, every coordinate is multiplied by \(W'/W \lt 1\). Example: a 1080p frame with max_pixels \(= 1{,}280 \times 28^2\) becomes 1316 × 728, so boxes shrink by 0.685 toward the origin and a small object’s box can miss it completely (IoU 0). Fix: multiply by \(W/W'\) and \(H/H'\), with \(W', H'\) read from image_grid_thw \(\times\, 14\) (191.5).
Location tokens or plain digits: which is better for coordinates?advanced
Location tokens cost one token per coordinate (4 per box) and cannot produce malformed numbers, but they add hundreds of new embeddings with no built-in order: the model must learn that <loc0388> is next to <loc0389>. Digits reuse the number knowledge of the language model and need no new vocabulary, but cost more tokens (the Qwen2 tokenizer splits numbers into single digits, so a box is about 17 tokens) and can be malformed. Molmo’s ablation found special point tokens “substantially worse” than plain text. In practice: digits for chat models that also need numbers elsewhere; location tokens when output length matters a lot and the model is trained heavily on grounding.
Why does “point, then count” work better than answering with a number?core
A bare number must be computed inside the forward pass before the first answer token, with no memory of which objects are counted. Pointing turns counting into a sequence of local decisions: each point is checked against the image, the written points stay in the context so the model knows what it has covered, and numbering the points makes the count the index of the last point. Molmo reports that point-then-count is significantly better than count only. The cost is output length: about 16 tokens per object with Molmo’s format and the Qwen2 tokenizer (191.6), so 30 objects take about 500 tokens instead of 2.
How many visual tokens does a page of text need, and why is OCR sensitive to token reduction?core
For an A4 page at 150 dpi (1240 × 1754) with Qwen2.5-VL’s 28 px tokens, keeping the full resolution takes about 2,772 tokens and 10 pt text stays about 21 px high. At \(1{,}280 \times 28^2\) pixels the page is 1,260 tokens and the text 14 px; at \(256 \times 28^2\) it is 247 tokens and 6 px, about four text lines per token (191.7). Text is dense and every part of it matters, so pruning or merging “unimportant” tokens, which works for photos, can delete characters. Documents need high resolution, tiles or crops.
The model must answer A, B, C or D but often writes a sentence. What do you do?basic
Use the trained format instruction (“Answer with the option’s letter from the given choices directly.”). Better, do not parse free text: read the next-token probabilities of “A”–“D” and take the argmax (191.9), or constrain decoding to those four tokens. For a base model, score each option’s log-probability with a fixed length normalization (191.8). Also shuffle the options to check that the choice follows the content, not the position.
What is the length-normalization problem in option scoring?core
The log-probability of an option is a sum over its tokens, and every term is negative, so long options score lower even when they are right. Dividing by \(|c|^\alpha\) (191.8) fixes this partly, but the choice of \(\alpha\) changes the answer: in the toy example, \(\alpha = 0\) picks the one-token “kite”, \(\alpha = 0.37\) picks “a bat” and \(\alpha \ge 0.41\) picks “a tennis racket”. Benchmarks must state the rule; comparing models scored with different rules is meaningless.
How would you design the prompt and parser for a receipt-extraction service?advanced
Choose a pixel budget from the smallest font you must read (191.7) and test accuracy against text height on real receipts; crop to the receipt first. Ask for JSON with fixed keys (merchant, date, total, currency, line items), give one example in the prompt or fine-tune on the schema, and decode with a JSON grammar if your server supports it. Validate types (the total is a number, the date parses), check that line items sum to the total, and retry once with the error message on failure. Log raw outputs of failures. If you need positions (to highlight fields), ask for boxes per field in the model’s trained coordinate format and map them back with the processor’s resize factors.
Why do PaliGemma and Florence-2 use task prefixes instead of natural instructions?basic
They are trained as multi-task transfer models on many academic tasks with very different answer styles (one word, a sentence, location tokens, mask tokens). A fixed prefix per task (caption en, detect mug, ocr; <OD>, <OCR>) tells the model exactly which output format to use and, in PaliGemma’s words, avoids conflicting learning signals across skills. The price is that they are not chat models: other wordings work less well, and new tasks are added by fine-tuning with a new prefix.
How does a VLM answer “when does the door open?” for a video?advanced
The same way as a box, but in time: it writes start and end times as text, which a parser turns into frame indices. The model must know the time of each frame. Qwen2-VL samples at 2 frames per second; Qwen2.5-VL aligns the temporal position ids with absolute time, so it can answer in seconds even at other frame rates. The traps match the spatial ones: the time frame of reference (seconds of the original video, or index of the sampled frame), and the sampling rate assumed by the processor. Evaluate with temporal IoU between the predicted and true intervals.

Exercises

  1. An image is 896 × 896 and the model uses PaliGemma’s 1,024 bins. What is the largest coordinate error after decoding to bin centers, and the worst IoU for a 20 × 20 px object?
  2. Write the PaliGemma location tokens for the box x from 414 to 508, y from 182 to 300 in a 640 × 480 image (floor rule, 1,024 bins). In what order do they appear?
  3. A 3000 × 2000 photo goes to Qwen2.5-VL with max_pixels \(= 1{,}280 \times 28^2\). What are \(W'\), \(H'\) and the number of visual tokens? The model answers [700, 400, 760, 450]. What is the box in original pixels?
  4. With formula (191.6), how many tokens does Molmo’s points string for 12 objects with label “person” take, if every coordinate has two digits before the decimal point?
  5. In the multiple-choice example, find the \(\alpha\) at which option C overtakes option A, if option D did not exist.
  6. A scan at 200 dpi has 8 pt text. With a resize factor \(s = 0.5\), how many pixels high is the text?
Answerscheck
1. A bin is \(896/1024 = 0.875\) px, so \(\delta = 0.4375\) px. Worst IoU \((1 - 0.875/20)^2 = 0.956^2 = 0.914\). 2. ymin \(\lfloor 1024 \cdot 182/480 \rfloor = 388\), xmin \(\lfloor 1024 \cdot 414/640 \rfloor = 662\), ymax \(\lfloor 1024 \cdot 300/480 \rfloor = 640\), xmax \(\lfloor 1024 \cdot 508/640 \rfloor = 812\): <loc0388><loc0662><loc0640><loc0812>, in the order ymin, xmin, ymax, xmax. 3. \(\beta = \sqrt{6{,}000{,}000/1{,}003{,}520} = 2.445\); \(W' = 28 \lfloor 3000/(28 \cdot 2.445) \rfloor = 28 \cdot 43 = 1204\), \(H' = 28 \lfloor 2000/(28 \cdot 2.445) \rfloor = 28 \cdot 29 = 812\); \(43 \times 29 = 1{,}247\) tokens. Original box: \(x \cdot 3000/1204\), \(y \cdot 2000/812\): about (1744.2, 985.2)–(1893.7, 1108.4). 4. \(8 + 2 \cdot 1 = 10\) for the tag; points 1–9 cost \(6 + 2 + 4 + 4 = 16\) each (144), points 10–12 cost \(6 + 4 + 4 + 4 = 18\) each (54): total 208. 5. C beats A when \(3.3/3^\alpha \lt 2.2\), so \(3^\alpha \gt 1.5\), \(\alpha \gt \ln 1.5/\ln 3 = 0.369\). 6. \(e = 0.5 \cdot 8/72 \cdot 200 = 11.1\) px.

Key takeaways

  • Every VLM task is \(p(\mathbf{y} \mid I, \mathbf{x})\) (191.1): only the prompt and the shape of the answer change. The format of the answer is part of the task.
  • Captions and VQA need format signals (prefixes, “single word or phrase”, letter instructions); reasoning needs a strict final-answer line.
  • Coordinates are text: bins (191.2) add at most half a bin of error, and the IoU bound (191.3) shows that small objects need many bins.
  • Know the coordinate frame: [0, 1000), percent, or pixels of the resized image (191.4, 191.5). The resize trap turns correct answers into wrong boxes.
  • Count by pointing first: more accurate and checkable, at about 16 output tokens per object (191.6).
  • Reading needs pixels: the text height after the resize (191.7) sets what a document model can read for a given token budget.
  • For closed-set answers, score instead of parse: options with a stated length normalization (191.8) or letters (191.9).

Key formulas

\[ p_\theta(\mathbf{y} \mid I, \mathbf{x}) = \prod_t p_\theta(y_t \mid y_{\lt t}, I, \mathbf{x}), \qquad k = \Big\lfloor \frac{N x}{W} \Big\rfloor, \quad \hat x = \Big(k + \tfrac12\Big)\frac{W}{N} \]
\[ \text{IoU} \ge \Big(1 - \frac{W}{N w}\Big)\Big(1 - \frac{H}{N h}\Big), \qquad x_{\text{orig}} = x_{\text{model}} \frac{W}{W'} \]
\[ e = s \cdot \frac{p}{72} \cdot \text{dpi}, \qquad \hat c = \argmax_c \frac{1}{|c|^\alpha} \sum_t \log p_\theta(c_t \mid c_{\lt t}, I, \mathbf{x}) \]

Next, session 192 turns these parsed answers into scores: VQA accuracy, CIDEr for captions, ANLS for documents, and benchmarks such as MMMU, DocVQA and POPE.

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
  2. Deitke et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
  4. Deitke et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models
  5. Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
  6. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  7. Bai et al. (2025). Qwen2.5-VL Technical Report
  8. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  9. Peng, Wang, Dong, Hao, Huang, Ma and Wei (2023). Kosmos-2: Grounding Multimodal Large Language Models to the World
  10. Chen, Saxena, Li, Fleet and Hinton (2021). Pix2seq: A Language Modeling Framework for Object Detection
  11. Xiao et al. (2023). Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
  12. Qwen Team (2025). Qwen2.5-VL cookbook: spatial understanding (notebook, January 2025 version)
  13. Kim et al. (2021). OCR-free Document Understanding Transformer
  14. Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images