VLM tasks and how models answer them
A vision-language model answers every task the same way: it writes text. A caption, a one-word answer, a bounding box, a list of points, the text of a receipt and a chart value are all token sequences. This session shows how each task is asked and answered: the prompt formats, how coordinates become text and back, why counting works better by pointing, what resolution reading needs, and how to get a clean answer out of a model that likes to talk.
What you will learn
- Why one formula, the probability of the answer text given the image and the prompt, covers captioning, VQA, reasoning, grounding, counting and OCR
- The real prompt formats: PaliGemma task prefixes, LLaVA-1.5 format instructions, Florence-2 task tokens, and chat prompts with a JSON schema (Qwen2.5-VL)
- How boxes and points become tokens: bins and location tokens (Kosmos-2, PaliGemma, Florence-2), numbers in [0, 1000) (Qwen2-VL), absolute pixels (Qwen2.5-VL) and 0 to 100 points (Molmo), with the error each one adds
- The resize trap: Qwen2.5-VL coordinates live in the resized image, and how to map them back
- Counting by pointing first (Molmo), and what it costs in output tokens
- OCR, charts and documents: how many visual tokens a page needs, and what the pixel budget does to small text
- Getting the answer out: free generation with a parser, format instructions, and scoring the options by log-probability, including the length-normalization trap
Why this matters
Show a modern vision-language model (VLM) a photo of a desk and ask five things: “Describe it.”, “What color is the mug?”, “Where is the mug?”, “How many apples are there?”, “What does the note say?”. One model with one set of weights answers all five. It does not have a caption head, a detection head and an OCR engine. It has one output: the next token. The box, the count and the note’s text are all written as text.
This is a big change from the computer vision of ten years ago, where every task had its own model, its own output layer and its own loss. It is also the source of most practical problems with VLMs. If the box is text, then someone chose a text format for boxes, and your code must write the prompt and read the answer in exactly that format. If the count is text, the model can write “3” without looking carefully, or write a whole paragraph when you wanted a number. In this session we go through the main task families and, for each one, answer three questions:
- How is the task asked? (the prompt format the model was trained on)
- How is the answer written? (free text, a short phrase, a letter, coordinates, a JSON list)
- How do you get the answer back out, and what error does the format add?
The tasks are: captioning, visual question answering (VQA) and visual reasoning; grounding and localization (boxes and points as text); counting; reading text (OCR), charts and documents. We use real formats from PaliGemma, Qwen2-VL, Qwen2.5-VL, Molmo, Kosmos-2 and Florence-2. The book discusses the main VLM tasks and their data in chapter 2 (Noyan et al.); here we focus on how the answer is written and parsed. The next session, session 192, turns these answers into scores.
These questions come from real systems. Product teams use VLMs to read receipts and forms, to find UI elements for agents that click on screens, to count items on shelves, and to find the moment in a video where something happens. In every case the model writes text, and a small parser turns it into a number, a box or a timestamp. Most production bugs live in that parser and in the prompt that feeds it.
One recipe for every task: text in, text out
A VLM in the unified-sequence style (session 177) reads the image as visual tokens, then the prompt, and then writes the answer one token at a time. Whatever the task, the model computes the same thing:
- Read the symbols
- \(I\)
- the image (or video frames), already turned into visual tokens by the encoder and the connector
- \(\mathbf{x}\)
- the prompt: the instruction, the question, and any format rule such as “answer with one word”
- \(\mathbf{y} = (y_1, \dots, y_T)\)
- the answer as tokens: words, digits, special location tokens, JSON brackets, the end-of-turn token
- \(p_\theta(y_t \mid \dots)\)
- the probability of the next token, from the softmax over the vocabulary (see session 79)
So the only things that differ between tasks are \(\mathbf{x}\) (how you ask) and the shape of \(\mathbf{y}\) (how the answer is written). We can sort tasks by the shape of the answer:
| Answer shape | Tasks | How you read it out |
|---|---|---|
| Free text, long | Detailed captions, explanations, reasoning traces | Use as is; judge quality by humans or another model |
| Short text | VQA, short captions, OCR of one field | Normalize (lower case, strip punctuation) and compare |
| A choice | Multiple-choice questions, yes/no | Read the letter, or score each option by its probability |
| A number | Counting, chart values, clock reading | Parse the number; compare with a tolerance |
| Coordinates | Grounding, detection, pointing, UI clicks | Parse with a regex or JSON; map back to image pixels |
| Structured text | Documents to Markdown or HTML, tables, forms, key-value extraction | Parse the structure; check the schema |
Training on (191.1) is the same for all tasks too: next-token cross-entropy on the answer tokens only (session 188). This is why a VLM can learn a new task from data alone, with no new layer: you only need a format for the answer and examples written in that format. It is also why the format matters so much. A model trained to write boxes as (646,379),(793,625) will not write them as JSON just because you ask nicely, and a parser written for one model is wrong for the next.
How to read Figure 1
Choose a task, then a model. Caption with PaliGemma: the prompt is just the prefix caption en. Switch to Molmo: the same task gives a long, dense caption (52 tokens against 15), because Molmo was trained on long spoken descriptions. Choose “Ground the mug”: PaliGemma writes four location tokens in the order ymin, xmin, ymax, xmax; Qwen2-VL writes numbers in [0, 1000); Qwen2.5-VL writes JSON with pixel coordinates of its resized 644 × 476 input; Molmo writes a point, not a box. All three boxes decode to an IoU of 0.98 to 0.99 with the true box. Choose “Count apples” with Molmo: it points to each apple, then gives the total, which costs 70 tokens; Qwen2.5-VL’s detect-then-count answer costs 111; PaliGemma and Qwen2-VL just write “3”.
Captioning, VQA and visual reasoning
Captioning: the length is part of the task
Captioning means writing a description of the image. The hard part is not the words; it is the length and style. A COCO-style caption is one sentence (“A blue mug and three apples on a table.”). A dense caption is a paragraph that lists objects, colors, positions and text. Both are correct answers to “describe the image”, so the model needs a signal for which one you want. Three kinds of signal are used in practice:
- Task prefixes. PaliGemma is trained with a fixed prefix for each task, for example
caption en(with a language code) orocr, “to avoid conflicting learning signals across skills” [Certain, from the paper]. Florence-2 uses task tokens such as<CAPTION>,<DETAILED_CAPTION>and<MORE_DETAILED_CAPTION>[Certain, from its model card]. - Format instructions in natural language. LLaVA-1.5 adds a sentence to each academic training sample, such as “Provide a one-sentence caption for the provided image.” [Certain, from the paper].
- The training data itself. Molmo’s captions (PixMo-Cap) come from annotators who spoke about each image for at least 60 seconds; the set has 712k images with 1.3M transcripts and captions [Certain, from the paper]. A model trained on such data writes long captions by default.
VQA: short answers need a format rule
Visual question answering means answering a question about the image: “What color is the mug?” → “Blue.” Academic VQA datasets (VQAv2, GQA, OK-VQA) have answers of one to three words, because they are scored by exact match with human answers. Chat models like to answer in full sentences. LLaVA-1.5 found that a vague prompt format (such as “Q: … A:”) made the model either too short in chat or too long on benchmarks, and fixed it with an explicit instruction: “Answer the question using a single word or phrase.” [Certain, from the paper]. For multiple choice it uses “Answer with the option’s letter from the given choices directly.” [Certain]. These two sentences are now the standard prompts for evaluating many open VLMs [Likely].
The lesson is general: a format instruction is part of the task. The same question with and without “one word” is two different tasks, and you should evaluate the one your product needs.
Visual reasoning: think first, then answer
Visual reasoning tasks need several steps: “Is the mug to the left of the note, and is it taller than the apples?”, a geometry problem from a figure, or a science question with a diagram (MMMU, session 192). Two answer styles exist:
- Direct answer: one letter or one number. Fast, easy to parse, and the whole computation must happen inside the forward passes before the first answer token.
- Reasoning, then answer: the model first writes its steps (a chain of thought, session 72) and then a final line such as “Answer: C”. The written steps give the model more computation per question. The parser must then find the final answer inside a long text.
Reasoning-first answers are what GRPO-style training rewards in session 190: the reward checks only the final answer, so the format of that final line must be strict. When you use such a model, ask for a fixed final line (for example “End with: Answer: <letter>”) and parse only that line.
Grounding: boxes and points as text
Grounding (or localization) means linking words to places in the image. Referring expression comprehension gives a phrase (“the mug on the right”) and asks for its box. Detection asks for all boxes of one or more classes. Pointing asks for one point per object. Grounded captioning writes a caption where each phrase carries its box. In a VLM all of these produce coordinates as tokens. The first idea came from Pix2Seq (2021), which trained a model to write detections as sequences of discrete tokens: quantized coordinates followed by a class token [Certain, from the paper]. VLMs then adopted it.
Quantization: from pixels to bins
A coordinate is a real number, but tokens are discrete. The simplest map cuts the image width \(W\) into \(N\) equal bins and writes the bin number:
- Read the symbols
- \(x\)
- the true coordinate in pixels (the same rule is used for \(y\) with the height \(H\))
- \(N\)
- the number of bins: 32 in Kosmos-2, 1,000 in Florence-2 and Qwen2-VL, 1,024 in PaliGemma
- \(k\)
- the bin index that the model writes, as a special token (
<loc0388>) or as digits (388) - \(\hat x\)
- the decoded coordinate; using the center of the bin halves the worst error compared with using its left edge \(k W / N\)
Decoding to the bin center, the error of each coordinate is at most half a bin. If each of the four edges of a box can move by up to that much, the worst case is the box that shrinks on all sides (we checked this with a brute-force search over all edge shifts), so
- Read the symbols
- \(\delta_x, \delta_y\)
- half a bin in pixels, horizontally (\(W/2N\)) and vertically (\(H/2N\))
- \(w, h\)
- the width and height of the object’s box in pixels
- IoU
- intersection over union of the true box and the decoded box: 1 is perfect, and detection benchmarks usually count IoU \(\ge 0.5\) as a hit
The bound depends on the ratio of the bin size to the object size. Quantization is harmless for big objects and decides everything for small ones.
The frame is 1920 × 1080; the cup’s box is \(x\) from 100 to 140, \(y\) from 500 to 530.
- \(N = 32\) (Kosmos-2’s grid): a bin is 60 × 33.75 px, larger than the cup. The box becomes bins 1–2 and 14–15, decoded as 90–150 × 489.4–523.1: IoU 0.40. The worst case is 0.
- \(N = 100\): bins of 19.2 × 10.8 px. IoU 0.64 here; the worst case is \((1 - 19.2/40)(1 - 10.8/30) = 0.33\).
- \(N = 1000\) (Qwen2-VL, Florence-2): \(\delta_x = 0.96\), \(\delta_y = 0.54\) px. IoU 0.94; worst case \((1 - 1.92/40)(1 - 1.08/30) = 0.918\).
- \(N = 1024\) (PaliGemma): IoU 0.97; worst case 0.920.
With about 1,000 bins, quantization costs at most 8% of IoU even for a 30-pixel object in a 1080p frame. With a 32 × 32 grid, small objects cannot be localized at all.
Five real formats
Different model families made different choices. They differ in the number of bins, in whether coordinates are special tokens or digits, in the order of the four numbers, and in what the numbers are relative to.
| Model | What the model writes for one box | Coordinates |
|---|---|---|
| Kosmos-2 | <p> It </p><box><loc44><loc863></box> | A 32 × 32 grid (1,024 location tokens, 7 × 7 px cells of a 224 px image); one token for the top-left cell, one for the bottom-right cell [Certain] |
| PaliGemma | <loc0388><loc0662><loc0640><loc0812> mug | 1,024 location tokens <loc0000> to <loc1023>, normalized image coordinates, order ymin, xmin, ymax, xmax; several objects separated by ; [Certain] |
| Florence-2 | Location tokens for x0, y0, x1, y1; quad boxes and polygons as more corners, clockwise | 1,000 bins [Certain]; tokens are named <loc_0> to <loc_999> [Likely] |
| Qwen2-VL | <|object_ref_start|>the mug<|object_ref_end|><|box_start|>(646,379),(793,625)<|box_end|> | Plain digits, normalized to [0, 1000), top-left then bottom-right [Certain] |
| Qwen2.5-VL | {"bbox_2d": [417, 180, 511, 298], "label": "mug"} | Absolute pixels of the resized input image, in JSON [Certain, from the paper and the official cookbook] |
| Molmo | <point x="72.0" y="50.2" alt="mug">mug</point> | Points, not boxes; plain text from 0.0 to 100.0 with one decimal, in percent of width and height [Certain] |
Two design questions hide in this table.
Special tokens or digits? A location token (<loc0388>) is one token per coordinate, so a box costs 4 tokens; but the model must learn 1,024 new embeddings from scratch, with no link between neighboring bins. Digits reuse the language model’s number tokens. The Qwen2 tokenizer splits numbers into single digits (its pre-tokenizer pattern has \p{N} alone) [Certain, from tokenizer.json], so (646,379),(793,625) is 17 tokens and the whole grounded phrase above is 23. Molmo’s authors tried both and report that special point tokens performed “substantially worse” than the plain-text representation [Certain, from the paper’s ablation].
Normalized or absolute? Normalized coordinates (bins of the full image) do not depend on the resolution the model saw. Absolute coordinates tell the model the real size of the image and of the objects; Qwen2.5-VL uses them for this reason [Certain that it uses absolute coordinates; that this helps is the authors’ claim, Likely]. But absolute coordinates are only useful if you know which image they are absolute to.
The resize trap
Qwen2-VL and Qwen2.5-VL resize every image so that both sides are multiples of 28 (14-pixel patches, then a 2 × 2 merge) and the pixel count stays inside a budget (session 180). The resize function (smart_resize in qwen_vl_utils) is
- Read the symbols
- \(H, W\)
- the size of the original image in pixels
- \(H', W'\)
- the size the vision encoder actually sees; the model gets \((H'/28)(W'/28)\) visual tokens
- \(P_{\max}\)
- the pixel budget,
max_pixels: \(16{,}384 \times 28^2 = 12{,}845{,}056\) by default; examples often use \(1{,}280 \times 28^2 = 1{,}003{,}520\). A minimum budget works the same way upward - \(\beta\)
- the downscale factor when the image is over the budget
Qwen2.5-VL writes box coordinates in pixels of the \(W' \times H'\) image. The official cookbook computes input_height = image_grid_thw[1] * 14 and divides the model’s numbers by it before drawing on the original [Certain]. To map back:
With the default budget, the 1920 × 1080 frame becomes \(W' \times H' = 1932 \times 1092\) (round 68.57 up to 69 and 38.57 up to 39 patches of 28 px): 2,691 visual tokens. A point the model writes at \(x = 1900\) is at \(1900 \cdot 1920/1932 = 1888.2\) in the original: 11.8 px off if you forget (191.5). Annoying, not fatal.
With max_pixels \(= 1{,}003{,}520\): \(\beta = \sqrt{1920 \cdot 1080 / 1{,}003{,}520} = 1.4375\), so \(W' = 28 \lfloor 47.70 \rfloor = 1316\) and \(H' = 28 \lfloor 26.83 \rfloor = 728\): 1,222 visual tokens. The cup at (100, 500)–(140, 530) is written as [69, 337, 96, 357]. Drawn on the original frame without (191.5), this box does not touch the cup at all: IoU 0. Every box shrinks toward the top-left corner by a factor of 0.685.
The rule for every model: coordinates are relative to something (the full image, a resized image, a crop, a tile). Find out what, from the processor code, not from the prompt. When the image is tiled (session 180), models are trained to give coordinates of the full image, but check this for your model before you trust it.
- input: image of size \(W \times H\); a phrase; the model family
- run the model’s own processor; record the size it actually fed to the encoder (\(W', H'\)) // from image_grid_thw, or the resize setting
- write the prompt in the trained format: task prefix (
detect mug), instruction plus format (“output its bbox coordinates using JSON format”), or reference tokens - decode greedily, with
skip_special_tokens=Falseif coordinates are special tokens - parse: regex for
<locNNNN>or(x,y); JSON parser for JSON (strip the code fence first); XML-like parser for points - map to the original image: bins with (191.2), [0, 1000) with \(x W / 1000\), absolute with (191.5), percent with \(x W / 100\)
- clip to \([0, W] \times [0, H]\); swap corners if \(x_1 \gt x_2\); drop boxes with zero area
- return boxes with labels; log every answer that failed to parse
How to read Figure 2
Start at 1,000 bins, 1920 × 1080 and a 40 px object: half a bin is 0.96 × 0.54 px, the decoded box at this position has IoU 0.937 and the worst case is 0.926. Choose 32 bins: one bin (60 × 33.75 px) is bigger than the object, the IoU at this position falls to 0.422 and the worst case is 0. Move to 224 × 224 (Kosmos-2’s own input size): a 32-bin cell is 7 px and a 40 px object keeps a worst case of 0.681. Go to 3840 × 2160 with 1,000 bins and shrink the object to 10 px: the worst case falls to 0.483, below the usual 0.5 hit threshold, so a 4K screenshot with small UI icons needs more bins, a crop, or absolute pixel coordinates.
Counting: point first, then count
Counting looks like VQA with a number as the answer: “How many apples are there?” → “3”. It is one of the weakest skills of VLMs [Likely; it is the reason Molmo and PaliGemma built dedicated counting data and benchmarks]. Writing one digit gives the model no place to keep track: the count must be computed inside the forward pass, over hundreds of visual tokens, before the first answer token.
Molmo’s fix is to make the model point first. Asked to count, it writes one point per object, ordered top-down and left to right, with each point numbered, and then the total [Certain]. The authors note that numbering makes counting easy, because the total is the number of the last point, and they report that this “chain-of-thought point-then-count performs significantly better than generating a count only” [Certain, from the paper]. Their pointing data, PixMo-Points, has 2.3M question-point pairs on 223k images [Certain]. Qwen2.5-VL does the same with boxes, a “detect then count” prompt, and reports 93.6 on CountBench for the 72B model [Certain, from the paper]. CountBenchQA itself is a 540-image benchmark introduced with PaliGemma [Certain].
Pointing works for three reasons. Each point is a small, local decision that the model can check against the image. The written points stay in the context, so the model can see which objects are already counted. And the count becomes a by-product of a task that is easy to supervise and to check: a human (or a reward function, session 190) can see a missing or duplicate point.
The price is output length. With the Qwen2 tokenizer, Molmo’s multi-point string costs
- Read the symbols
- \(K\)
- the number of points (for \(K = 1\) Molmo uses a shorter
<point x=… y=…>tag) - \(L\)
- the number of tokens of the label (“person” is 1, “apples” is 2:
app+les) - \(d(k)\)
- the number of digits of the point index \(k\) (1 for points 1–9, 2 for 10–99)
- \(c(x_k)\)
- the characters of the coordinate, each one a token: 3 for “5.0”, 4 for “45.8”, 5 for “100.0”
We checked (191.6) against the real tokenizer on 400 random point lists: it matches exactly. So each object costs about 16 tokens (18 from the tenth point on). Counting 30 people by pointing is about 500 output tokens, while the bare answer “30” is 2 tokens. Decoding is the slow phase of inference (session 196), so for a latency-critical service you may point during training (to teach counting) and still ask for a bare number when the count is small, or point only when the model’s first guess is large.
For the desk image of Figure 1 (Qwen2 tokenizer, exact counts):
- bare answer “3”: 1 token;
- Molmo,
Counting the <points x1="45.8" y1="67.9" x2="20.0" y2="69.0" x3="32.7" y3="72.1" alt="apples">apples</points> shows a total of 3.: 70 tokens (the points tag alone is 60, as (191.6) gives: “apples” is 2 tokens, so \(8 + 2 \cdot 2 + 3 \times 16\)). The wording around the tag follows Molmo’s demo outputs [Likely]; - Qwen2.5-VL, three JSON boxes in a code fence and “There are 3 apples.”: 111 tokens.
How to read Figure 3
Press “Point to all”: 14 points, 14 distinct apples hit, the count is 14 and the points string costs 246 tokens, against 2 tokens for the bare answer “14”. Each point is about 16 tokens, and 18 from point 10 on, because the index “10” is two digits in both x10 and y10. Now reset and play the model yourself: click one apple twice in different places (a duplicate), or click a leaf (a false point). The count, which is just the number of points, is now wrong, and you can see exactly why. That is the main practical gain of pointing: a wrong count comes with its evidence.
OCR, charts and documents
OCR (optical character recognition) means reading the text in an image. In a VLM it appears in several tasks: transcribe all text (PaliGemma’s ocr prefix, Florence-2’s <OCR>), transcribe with positions (<OCR_WITH_REGION>, or Qwen2.5-VL’s document parsing), answer a question that needs reading (TextVQA, DocVQA), and read values from a chart (ChartQA). Older pipelines ran a separate OCR engine and passed its text to the model. OCR-free models such as Donut showed that a model can learn to read directly from pixels [Certain]; today’s VLMs do the same, and their main limit is resolution.
Reading needs pixels
A letter must cover enough pixels for the encoder to tell “8” from “B”. After the resize (191.4), the height of the text in pixels is
- Read the symbols
- \(p\)
- the font size in points (1 point = 1/72 inch); body text is usually 9 to 12 pt
- dpi
- dots per inch of the scan or render: an A4 page at 150 dpi is 1240 × 1754 px
- \(s\)
- the resize factor applied by the processor; below 1 when the page is over the pixel budget
- \(e\)
- the font size (the em, roughly the line height without spacing) in pixels of the image the encoder sees
- \(v\)
- the number of visual tokens for the page; each covers a 28 × 28 px square
At 150 dpi, 10 pt is \(10/72 \cdot 150 = 20.8\) px. With (191.4) and (191.7):
- default budget: the page becomes 1232 × 1764, \(v = 2{,}772\) tokens, \(e = 20.7\) px;
- \(P_{\max} = 1{,}280 \cdot 28^2\): 840 × 1176, \(v = 1{,}260\), \(e = 14.1\) px;
- \(P_{\max} = 768 \cdot 28^2\): 644 × 896, \(v = 736\), \(e = 10.8\) px;
- \(P_{\max} = 256 \cdot 28^2\): 364 × 532, \(v = 247\), \(e = 6.1\) px, so one 28 px token covers almost four lines of text (with 1.2 line spacing).
A page full of text needs thousands of visual tokens to stay readable. This is why document models use tiles or native resolution (session 180) and why token reduction (session 183) is risky for OCR: dropping tokens that look “empty” in a photo can delete half a word in a scan.
How small can text get? There is no single threshold: it depends on the encoder, the font and the training data. Measure it: render your own documents at several budgets and plot accuracy against \(e\). Figure 4 shows what the encoder receives.
How to read Figure 4
At a budget of 1,280 and 10 pt, the page is 840 × 1176 px (1,260 tokens) and the text is 14.1 px high: the digits are clear. Drop the budget to 256: 247 tokens, 6.1 px text, and “1,284.50” turns into blobs where 3 and 8 look the same. Raise the font to 14 pt at the same budget: the text is 8.6 px high, and the digits begin to separate again. The budget and the font size trade against each other through (191.7). Small print in contracts and invoices is exactly where a low budget fails first.
Charts and documents: structure in, structure out
Charts add arithmetic to reading: “What is the difference between the tallest and shortest bars?” needs the values (read from the axis), then a computation. Errors come from both steps. A good prompt asks the model to first write the values it reads, then compute; this is a chain of thought with a checkable middle step. Chart answers are scored with a numeric tolerance, not exact match (session 192).
Documents (forms, invoices, papers, slides) have layout: headings, tables, key-value pairs. Two output styles are common. Question answering (DocVQA: “What is the invoice total?”) returns one span. Parsing returns the whole document as structured text. Qwen2.5-VL’s document parsing writes HTML in which each element carries its box, as an attribute data-bbox="x1 y1 x2 y2" [Certain, from the paper]; others write Markdown. For extraction into your own schema, ask for JSON with fixed keys and validate it (session 146).
Much of the training data for these skills is synthetic, because real annotated documents are rare and private. Molmo’s PixMo-Docs prompted a language model to write code that renders 255k text- and figure-heavy images (charts, tables, documents), then generated 2.3M question-answer pairs from the code, which gives exact answers for free [Certain, from the paper].
Prompt formats for each task
We can now put the prompts in one place. There are three styles, and a model is trained on one of them (or a mix):
- Task prefixes or task tokens (PaliGemma, Florence-2): short, fixed strings. Very reliable for the trained tasks, and not meant for free chat. Using a different wording than the trained prefix usually lowers quality.
- Instruction plus a format rule (LLaVA-1.5 and most chat VLMs): a natural question followed by a sentence that fixes the shape of the answer.
- Chat with a schema (Qwen2.5-VL and newer chat models): ask for JSON (or XML, or HTML) with named keys. The schema doubles as documentation for your parser.
| Task | Prefix style (PaliGemma) | Chat style (examples) | Parse |
|---|---|---|---|
| Short caption | caption en | “Provide a one-sentence caption for the provided image.” (LLaVA-1.5) | none |
| Detailed caption | — | “Describe this image in detail.”; Florence-2: <MORE_DETAILED_CAPTION> | none |
| Short VQA | answer en what color is the mug? | question + “Answer the question using a single word or phrase.” | normalize text |
| Multiple choice | — | question, options as “A. … B. …”, + “Answer with the option’s letter from the given choices directly.” | first letter, or scoring (next section) |
| Reasoning | — | question + “Think step by step, then end with ‘Answer: X’.” | regex on the last line |
| Detection | detect mug ; apple | “Locate every apple, output bbox coordinates in JSON format.” (Qwen2.5-VL) | regex or JSON, then map back |
| Referring segmentation | segment mug (output: 4 location tokens + 16 of the 128 <segNNN> mask tokens [Likely for the 16]) | usually a box, then a separate mask model (session 168) | decode mask tokens with the VQ-VAE decoder |
| Pointing and counting | answer en how many apples are there? | “Point to the apples.” / “How many apples are there?” (Molmo points first) | count the points |
| OCR | ocr | “Read all the text in the image.”; Florence-2: <OCR>, <OCR_WITH_REGION> | none, or boxes per line |
| Document parsing | — | “Parse this document into HTML.” (Qwen2.5-VL writes data-bbox attributes) | HTML or Markdown parser |
| Video moment | — | “When does the person open the door? Give start and end in seconds.” | parse two numbers; map to frames |
Two more details matter. First, where the image goes: the chat template puts the image placeholder at a fixed place in the user turn, and the processor expands it into the visual tokens (session 9, session 188). Always build the prompt with the model’s own apply_chat_template; a hand-written template with the image after the question, or a missing system prompt, is a different input from the one the model was trained on. Second, visual prompts: instead of writing a box in the prompt, you can draw it on the image. The PaliGemma paper reports that a red box drawn on the image works about as well as giving the box coordinates in the prompt [Likely, from its appendix F].
Getting the answer out: generate, constrain or score
Once the model is trained, there are three ways to turn (191.1) into an answer:
- Generate and parse. Decode greedily (temperature 0) and parse the text. Simple and general. It fails when the model ignores the format: “The answer is C because…”, a box in the wrong syntax, JSON with a trailing comma.
- Constrain the decoding. At each step, allow only the tokens that keep the output valid: only “A”–“D”, only digits, only JSON that fits a schema (grammar-constrained decoding, session 146). The output always parses. Be careful: a constraint the model did not learn can push it into answers it would not have chosen.
- Score the options. For a closed set of answers, do not generate at all. Run one forward pass per option and compare their log-probabilities. This is how many benchmarks evaluate base models (session 40).
Scoring needs care, because options have different lengths. A longer option is a product of more probabilities below 1, so its raw log-probability is lower. The usual fix divides by a power of the length:
- Read the symbols
- \(\mathcal{C}\)
- the set of options (for example the four answers of a multiple-choice question)
- \(|c|\)
- the number of tokens of option \(c\)
- \(\alpha\)
- the length normalization: 0 = raw sum (favors short options), 1 = mean log-probability per token (favors long fluent options)
The other closed-set method scores only the letter: show the options in the prompt, and read the probability of the next token being “A”, “B”, “C” or “D”:
Letter scoring needs only one forward pass and has no length problem, but it needs a model that has learned the letter format (an instruction-tuned model), and it is known to be sensitive to the order of the options [Likely]. A cheap test is to shuffle the options and check that the answer follows the content, not the letter.
Question: “What is the man holding?” The image shows a tennis racket. The per-token log-probabilities below are invented for the example: A “kite” \([-2.2]\); B “umbrella” \([-3.0]\); C “a tennis racket” \([-2.6, -0.5, -0.2]\); D “a bat” \([-1.5, -1.3]\).
- \(\alpha = 0\): scores \(-2.2, -3.0, -3.3, -2.8\). The one-token “kite” wins, wrongly. The right answer has the lowest score, because its first token “a” is not very likely.
- \(\alpha = 0.37\): D wins (\(-2.167\) against \(-2.198\) for C and \(-2.2\) for A). D overtakes A at \(\alpha = 0.348\) and C overtakes D at \(\alpha = 0.405\).
- \(\alpha = 1\): means \(-2.2, -3.0, -1.1, -1.4\). C wins.
- Letter scoring with next-token logits 0.2, −1.0, 1.6, 0.9 for A–D: softmax gives 0.136, 0.041, 0.550, 0.273. C wins.
The same model gives three different answers depending on one hyperparameter of the evaluation. Report the method with the number.
How to read Figure 5
At \(\alpha = 0\) the one-token “kite” wins with −2.2 and the correct “a tennis racket” is last with −3.3. Move \(\alpha\) slowly: at 0.35 “a bat” takes the lead, and from 0.41 “a tennis racket” wins. At \(\alpha = 1\) it wins clearly (−1.1 against −1.4). Switch to the letter mode: the slider no longer matters, and C wins with probability 0.550. When you compare two models on a multiple-choice benchmark, they must be scored with the same rule.
- input: a task, a model, the answer type your product needs
- find the trained format: prefix, format instruction or schema // model card, paper, processor code
- if closed set (letters, yes/no, labels): use letter scoring (191.9) for instruction-tuned models, option scoring (191.8) with a fixed \(\alpha\) for base models; shuffle options once to test order bias
- if number or count: allow reasoning or pointing first, then parse the last number; check its range
- if coordinates: Algorithm 1; test the round trip on a synthetic image with known boxes
- if structured output: ask for the schema, decode with a grammar if available, validate, and retry once on failure
- log the raw text of every answer that fails to parse and track the failure rate as a metric
- return the parsed answer, and the raw text for debugging
Code: write and read every answer format
This file has no model in it. It writes a box in the PaliGemma, Qwen2-VL and Qwen2.5-VL formats and points in Molmo’s format, parses them back with the same code a production parser would use, measures the IoU of the round trip, shows the resize trap, checks the worst-case bound (191.3), and scores multiple-choice options with (191.8) and (191.9). It runs on a CPU in well under a second and is deterministic. Do not save it as code.py: that name hides Python’s own code module, which import torch needs.
import math, re, json, torch torch.manual_seed(0) # ---------- 1) coordinates as text: four real formats ---------- def smart_resize(h, w, f=28, max_pixels=16384 * 28 * 28): # Qwen2.5-VL resize (qwen_vl_utils) hb, wb = round(h / f) * f, round(w / f) * f if hb * wb > max_pixels: b = math.sqrt(h * w / max_pixels) hb, wb = math.floor(h / b / f) * f, math.floor(w / b / f) * f return hb, wb def to_paligemma(b, W, H, label): # 1024 bins, order ymin xmin ymax xmax k = lambda v, S: min(1023, math.floor(1024 * v / S)) return "".join(f"<loc{k(v, S):04d}>" for v, S in [(b[1], H), (b[0], W), (b[3], H), (b[2], W)]) + " " + label def to_qwen2vl(b, W, H, label): # normalized to [0, 1000) x1, y1, x2, y2 = (math.floor(1000 * v / S) for v, S in zip(b, (W, H, W, H))) return f"<|object_ref_start|>{label}<|object_ref_end|><|box_start|>({x1},{y1}),({x2},{y2})<|box_end|>" def to_qwen25vl(b, W, H, label, max_pixels=16384 * 28 * 28): # absolute pixels of the RESIZED image Hm, Wm = smart_resize(H, W, max_pixels=max_pixels) xy = [round(v * S2 / S) for v, S, S2 in zip(b, (W, H, W, H), (Wm, Hm, Wm, Hm))] return json.dumps([{"bbox_2d": xy, "label": label}]) def to_molmo(pts, W, H, label): # plain-text points, 0-100 with one decimal a = " ".join(f'x{i+1}="{100*x/W:.1f}" y{i+1}="{100*y/H:.1f}"' for i, (x, y) in enumerate(pts)) return f'<points {a} alt="{label}">{label}</points>' # ---------- 2) parse the text back into pixels of the ORIGINAL image ---------- def parse(fmt, s, W, H, max_pixels=16384 * 28 * 28): if fmt == "paligemma": y1, x1, y2, x2 = [int(k) for k in re.findall(r"<loc(\d{4})>", s)[:4]] return [(x1 + .5) * W / 1024, (y1 + .5) * H / 1024, (x2 + .5) * W / 1024, (y2 + .5) * H / 1024] if fmt == "qwen2vl": x1, y1, x2, y2 = map(int, re.findall(r"\((\d+),(\d+)\)", s)[0] + re.findall(r"\((\d+),(\d+)\)", s)[1]) return [x1 * W / 1000, y1 * H / 1000, x2 * W / 1000, y2 * H / 1000] if fmt == "qwen25vl": Hm, Wm = smart_resize(H, W, max_pixels=max_pixels) x1, y1, x2, y2 = json.loads(s)[0]["bbox_2d"] return [x1 * W / Wm, y1 * H / Hm, x2 * W / Wm, y2 * H / Hm] if fmt == "molmo": xs = re.findall(r'x\d+="([\d.]+)"', s); ys = re.findall(r'y\d+="([\d.]+)"', s) return [(float(x) * W / 100, float(y) * H / 100) for x, y in zip(xs, ys)] def iou(a, b): ix = max(0, min(a[2], b[2]) - max(a[0], b[0])); iy = max(0, min(a[3], b[3]) - max(a[1], b[1])) inter = ix * iy return inter / ((a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter) W, H, mug = 640, 480, [414, 182, 508, 300] for fmt, enc in [("paligemma", to_paligemma), ("qwen2vl", to_qwen2vl), ("qwen25vl", to_qwen25vl)]: s = enc(mug, W, H, "mug"); print(f"{fmt:9s} {s} IoU={iou(mug, parse(fmt, s, W, H)):.4f}") # paligemma <loc0388><loc0662><loc0640><loc0812> mug IoU=0.9944 # qwen2vl <|object_ref_start|>mug...(646,379),(793,625)... IoU=0.9883 # qwen25vl [{"bbox_2d": [417, 180, 511, 298], "label": "mug"}] IoU=0.9856 # the resize trap: a 1080p frame with a smaller pixel budget s = to_qwen25vl([100, 500, 140, 530], 1920, 1080, "cup", max_pixels=1280 * 28 * 28) print(s, smart_resize(1080, 1920, max_pixels=1280 * 28 * 28)) # [{"bbox_2d": [69, 337, 96, 357] ...}] (728, 1316) print(f"{iou([100, 500, 140, 530], json.loads(s)[0]['bbox_2d']):.3f}") # 0.000: used as original pixels, the box misses # ---------- 3) point, then count ---------- apples = [(128, 331), (209, 346), (293, 326)] pts = sorted(apples, key=lambda p: (p[1], p[0])) # top to bottom, as Molmo orders points s = to_molmo(pts, W, H, "apples"); print(s, "count =", len(parse("molmo", s, W, H))) # <points x1="45.8" y1="67.9" x2="20.0" y2="69.0" x3="32.7" y3="72.1" alt="apples">apples</points> count = 3 # ---------- 4) the worst-case IoU of a quantized box (formula 191.3) ---------- def worst_iou(w, h, W, H, N): dx, dy = W / (2 * N), H / (2 * N) return max(0, 1 - 2 * dx / w) * max(0, 1 - 2 * dy / h) for N in [32, 100, 1000, 1024]: print(N, f"{worst_iou(40, 30, 1920, 1080, N):.3f}") # 32 0.000 | 100 0.333 | 1000 0.918 | 1024 0.920 # ---------- 5) multiple choice: score each option by its log-probability ---------- opt_logp = {"A kite": torch.tensor([-2.2]), "B umbrella": torch.tensor([-3.0]), "C a tennis racket": torch.tensor([-2.6, -0.5, -0.2]), "D a bat": torch.tensor([-1.5, -1.3])} def choose(alpha): # formula 191.8 sc = {k: (v.sum() / len(v) ** alpha).item() for k, v in opt_logp.items()} return max(sc, key=sc.get) print([choose(a) for a in (0.0, 0.37, 0.5, 1.0)]) # ['A kite', 'D a bat', 'C a tennis racket', 'C a tennis racket'] letter_logits = torch.tensor([0.2, -1.0, 1.6, 0.9]) # next-token logits of " A", " B", " C", " D" p = torch.softmax(letter_logits, 0); print([f"{v:.3f}" for v in p], "ABCD"[int(p.argmax())]) # C with 0.550
Two lines in the output are the main lessons. The three box formats all survive the round trip with IoU above 0.98, so the formats themselves are fine. The resize trap gives IoU 0.000: the model’s answer was correct, and the parser made it wrong. In a real system, add one test like this per model: draw known boxes on a blank image, run the model, and check the IoU of the parsed answers.
Where this is used
| Application | Task and answer format | What to watch |
|---|---|---|
| Alt text and image search indexing | Captioning, short or dense | Length and style instruction; hallucinated details in dense captions (session 193) |
| Receipts, invoices, forms | OCR plus key-value extraction into JSON | Pixel budget for small print (191.7); schema validation |
| Screen agents (click the “Submit” button) | Pointing or boxes on screenshots | Absolute vs normalized coordinates; resize trap on high-resolution screens (session 154) |
| Retail shelves, inventory, crowd estimates | Counting by pointing or detect-then-count | Output length grows with the count (191.6); duplicates and misses |
| Dashboards and reports | Chart QA: read values, then compute | Ask for the values first; numeric tolerance |
| Video key-moment detection, temporal grounding | Start and end times as text | Frame rate and absolute-time alignment; timestamps are coordinates in time |
| Video QA and Video-RAG | VQA over sampled frames or retrieved clips | Which frames the model saw; short-answer format for evaluation (session 210) |
| Action recognition and video classification | Closed-set choice over action labels | Score labels with (191.8) or letters (191.9), not free text |
| Video object tracking and segmentation | A VLM gives a box or point in one frame; a tracker such as SAM 2 follows it | Coordinate frame of the first box (session 204) |
| Content moderation | Yes/no or label questions, sometimes with reasons | Letter or label scoring with calibrated thresholds, not parsed sentences |
Common mistakes
- Using another model’s output format. Asking Qwen2-VL for JSON boxes, or parsing PaliGemma’s
<loc>tokens in the x-first order. Read the model card and the processor code; test the round trip. - Forgetting the coordinate frame. Qwen2.5-VL’s pixels are pixels of the resized image (191.5); Qwen2-VL’s numbers are in [0, 1000); Molmo’s are percent. A wrong frame gives boxes that look plausible and are wrong.
- Decoding with
skip_special_tokens=True. For models whose coordinates are special tokens, this deletes the coordinates. Florence-2’s model card decodes withskip_special_tokens=Falsefor this reason. - Asking for a bare count of many objects. Let the model point or detect first, then count; or at least check large counts with pointing.
- Too small a pixel budget for documents. At \(256 \times 28^2\) pixels, 10 pt text on an A4 scan is about 6 px high. Raise the budget, tile, or crop the region of interest.
- Free generation for closed-set answers. The model writes “It looks like a racket.” and the parser fails. Use letter scoring, option scoring or constrained decoding.
- Comparing scores with different length normalization. \(\alpha = 0\) and \(\alpha = 1\) can pick different answers from the same probabilities (Figure 5). Fix the rule and report it.
- Hand-written chat templates. The image placeholder in the wrong place, or a missing system prompt, changes the input. Use the processor’s
apply_chat_template.
Interview questions
How does a VLM output a bounding box when it can only produce tokens?basic
<loc0000>–<loc1023> in the order ymin, xmin, ymax, xmax; Florence-2: 1,000 bins), or it writes the numbers as ordinary digits (Qwen2-VL: normalized to [0, 1000); Qwen2.5-VL: absolute pixels of the resized image in JSON; Molmo: points in percent with one decimal). The model learns it with ordinary next-token cross-entropy on examples written in that format; the parser then maps the numbers back to pixels.How much localization error does quantizing coordinates add?core
Your Qwen2.5-VL boxes are all shifted toward the top-left and too small. What happened?core
smart_resize: sides rounded to multiples of 28 and scaled down to fit max_pixels). If you draw them on the original image, every coordinate is multiplied by \(W'/W \lt 1\). Example: a 1080p frame with max_pixels \(= 1{,}280 \times 28^2\) becomes 1316 × 728, so boxes shrink by 0.685 toward the origin and a small object’s box can miss it completely (IoU 0). Fix: multiply by \(W/W'\) and \(H/H'\), with \(W', H'\) read from image_grid_thw \(\times\, 14\) (191.5).Location tokens or plain digits: which is better for coordinates?advanced
<loc0388> is next to <loc0389>. Digits reuse the number knowledge of the language model and need no new vocabulary, but cost more tokens (the Qwen2 tokenizer splits numbers into single digits, so a box is about 17 tokens) and can be malformed. Molmo’s ablation found special point tokens “substantially worse” than plain text. In practice: digits for chat models that also need numbers elsewhere; location tokens when output length matters a lot and the model is trained heavily on grounding.Why does “point, then count” work better than answering with a number?core
How many visual tokens does a page of text need, and why is OCR sensitive to token reduction?core
The model must answer A, B, C or D but often writes a sentence. What do you do?basic
What is the length-normalization problem in option scoring?core
How would you design the prompt and parser for a receipt-extraction service?advanced
Why do PaliGemma and Florence-2 use task prefixes instead of natural instructions?basic
caption en, detect mug, ocr; <OD>, <OCR>) tells the model exactly which output format to use and, in PaliGemma’s words, avoids conflicting learning signals across skills. The price is that they are not chat models: other wordings work less well, and new tasks are added by fine-tuning with a new prefix.How does a VLM answer “when does the door open?” for a video?advanced
Exercises
- An image is 896 × 896 and the model uses PaliGemma’s 1,024 bins. What is the largest coordinate error after decoding to bin centers, and the worst IoU for a 20 × 20 px object?
- Write the PaliGemma location tokens for the box x from 414 to 508, y from 182 to 300 in a 640 × 480 image (floor rule, 1,024 bins). In what order do they appear?
- A 3000 × 2000 photo goes to Qwen2.5-VL with
max_pixels\(= 1{,}280 \times 28^2\). What are \(W'\), \(H'\) and the number of visual tokens? The model answers[700, 400, 760, 450]. What is the box in original pixels? - With formula (191.6), how many tokens does Molmo’s points string for 12 objects with label “person” take, if every coordinate has two digits before the decimal point?
- In the multiple-choice example, find the \(\alpha\) at which option C overtakes option A, if option D did not exist.
- A scan at 200 dpi has 8 pt text. With a resize factor \(s = 0.5\), how many pixels high is the text?
Answerscheck
<loc0388><loc0662><loc0640><loc0812>, in the order ymin, xmin, ymax, xmax. 3. \(\beta = \sqrt{6{,}000{,}000/1{,}003{,}520} = 2.445\); \(W' = 28 \lfloor 3000/(28 \cdot 2.445) \rfloor = 28 \cdot 43 = 1204\), \(H' = 28 \lfloor 2000/(28 \cdot 2.445) \rfloor = 28 \cdot 29 = 812\); \(43 \times 29 = 1{,}247\) tokens. Original box: \(x \cdot 3000/1204\), \(y \cdot 2000/812\): about (1744.2, 985.2)–(1893.7, 1108.4). 4. \(8 + 2 \cdot 1 = 10\) for the tag; points 1–9 cost \(6 + 2 + 4 + 4 = 16\) each (144), points 10–12 cost \(6 + 4 + 4 + 4 = 18\) each (54): total 208. 5. C beats A when \(3.3/3^\alpha \lt 2.2\), so \(3^\alpha \gt 1.5\), \(\alpha \gt \ln 1.5/\ln 3 = 0.369\). 6. \(e = 0.5 \cdot 8/72 \cdot 200 = 11.1\) px.Key takeaways
- Every VLM task is \(p(\mathbf{y} \mid I, \mathbf{x})\) (191.1): only the prompt and the shape of the answer change. The format of the answer is part of the task.
- Captions and VQA need format signals (prefixes, “single word or phrase”, letter instructions); reasoning needs a strict final-answer line.
- Coordinates are text: bins (191.2) add at most half a bin of error, and the IoU bound (191.3) shows that small objects need many bins.
- Know the coordinate frame: [0, 1000), percent, or pixels of the resized image (191.4, 191.5). The resize trap turns correct answers into wrong boxes.
- Count by pointing first: more accurate and checkable, at about 16 output tokens per object (191.6).
- Reading needs pixels: the text height after the resize (191.7) sets what a document model can read for a given token budget.
- For closed-set answers, score instead of parse: options with a stated length normalization (191.8) or letters (191.9).
Key formulas
Next, session 192 turns these parsed answers into scores: VQA accuracy, CIDEr for captions, ANLS for documents, and benchmarks such as MMMU, DocVQA and POPE.
References for this session
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
- Deitke et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 2: Vision Language Model Applications
- Deitke et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models
- Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Peng, Wang, Dong, Hao, Huang, Ma and Wei (2023). Kosmos-2: Grounding Multimodal Large Language Models to the World
- Chen, Saxena, Li, Fleet and Hinton (2021). Pix2seq: A Language Modeling Framework for Object Detection
- Xiao et al. (2023). Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
- Qwen Team (2025). Qwen2.5-VL cookbook: spatial understanding (notebook, January 2025 version)
- Kim et al. (2021). OCR-free Document Understanding Transformer
- Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images