LLM & VLM courseBlog
180Track 7: Vision and VLMs · session 18 of 38

High resolution and native aspect ratio

A vision encoder trained at 224 × 224 pixels cannot read a page of text, the labels of a chart, or a small sign in a 4K photo: the pixels are gone before the model sees them. This session shows the two answers modern VLMs use: tiling (cut the image into encoder-sized crops plus a thumbnail, as in LLaVA-NeXT and InternVL) and native dynamic resolution (encode the image at its own size and aspect ratio, as in NaViT and Qwen2-VL). It ends with M-RoPE, the position encoding that tells the language model where each visual token sits in time, height and width.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 164 (ViT), 177 (anatomy of a VLM) and 178 (connectors); session 14 (RoPE) and session 18 (packing masks) help

What you will learn

  • Why a fixed 224 × 224 or 336 × 336 input loses documents, charts and small objects, computed in pixels per letter and pixels per patch
  • How AnyRes tiling picks a grid of crops (effective and wasted resolution), adds a thumbnail, removes padding and marks rows, and what it costs in tokens
  • How Qwen2-VL encodes an image at its native size: resize to multiples of 28 inside a pixel budget, 14-pixel patches, 2D RoPE in the ViT, and a 2 × 2 merge into one LLM token
  • How NaViT-style packing puts many images of different sizes into one sequence with a block-diagonal mask, and why that saves padding and attention compute
  • How M-RoPE splits the rotary frequencies into time, height and width parts (16, 24, 24 pairs in Qwen2-VL), and how text, images and video get their position IDs
  • How to choose a token budget for documents, screens and video, and the common mistakes that silently hurt accuracy

Why this matters

The first VLMs looked at every image through a small square window: 224 × 224 pixels, or 336 × 336 in LLaVA-1.5. That is fine for “is there a dog in this photo?”. It fails for a page of text, a chart with small labels, a phone screenshot, or a traffic sign far away in a 4K video frame. The detail is destroyed when the image is shrunk, before the model sees a single pixel.

Two families of fixes are used in almost every VLM today:

  • Tiling (often called AnyRes): keep the encoder at its training size, but cut a large image into several encoder-sized crops, encode each crop, and add one small thumbnail of the whole image for global context. LLaVA-1.5-HD, LLaVA-NeXT and InternVL 1.5 work this way.
  • Native dynamic resolution: let the encoder accept images of any size and shape, so a wide image gives a wide grid of tokens and a tall image a tall grid. NaViT showed how to train such an encoder efficiently by packing images of different sizes into one sequence. Qwen2-VL put it into a VLM, together with M-RoPE, a rotary position encoding with separate parts for time, height and width.

The price of both is more visual tokens. A page that LLaVA-1.5 saw as 576 tokens can become 2,000 to 5,000 tokens. Attention cost grows with the square of the sequence length, so resolution is also a serving question: the token budget per image is one of the most important knobs in a VLM product (session 183, session 196).

In session 177 we met the three parts of a VLM: vision encoder, connector, language model. session 178 showed how visual features are compressed (for example by pixel shuffle) and session 179 showed where they enter the language model. This session is about the input side: how many pixels go in, in what shape, and how their positions are encoded. The next session, session 181, then reads real systems (LLaVA, BLIP-2, Flamingo, IDEFICS) with these ideas in hand.

Why 224 × 224 is not enough

A ViT cuts the image into square patches of \(p \times p\) pixels and turns each patch into one token (session 164). For an image of height \(H\) and width \(W\) the number of tokens and the cost of self-attention are

\[ N = \frac{H}{p} \cdot \frac{W}{p}, \qquad \text{attention cost} \;\propto\; N^2 \tag{180.1} \]
Read the symbols
\(H, W\)
height and width of the image that the encoder actually sees, in pixels
\(p\)
patch size: 14 pixels for CLIP ViT-L/14 and the Qwen2-VL encoder, 16 for many other ViTs
\(N\)
number of visual tokens; at \(224 \times 224\) with \(p = 14\) this is \(16 \times 16 = 256\), at \(336 \times 336\) it is \(24 \times 24 = 576\)

A CLIP-style encoder was trained at one fixed size \(S \times S\), and its learned position embeddings have exactly \((S/p)^2\) entries. The simple way to feed it any image is to resize the whole image to \(S \times S\). This does two harmful things at once:

\[ s_x = \frac{S}{W}, \qquad s_y = \frac{S}{H}, \qquad h' = s_y \, h \tag{180.2} \]
Read the symbols
\(s_x, s_y\)
how much the image shrinks along the width and along the height; when \(W \ne H\) they differ, so shapes are squashed
\(h\)
the height of a detail in the original image, for example a line of text, in pixels
\(h'\)
the height of the same detail after resizing
Worked example: a letter page

A US letter page scanned at 200 dpi is \(1700 \times 2200\) pixels (width × height). A 10-point font is \(10/72\) inch tall, so at 200 dpi it is \(10/72 \times 200 = 27.8\) pixels (the full font size, from the top of “h” to the bottom of “g”).

Resize to \(224 \times 224\): \(s_y = 224/2200 = 0.102\), so the font becomes \(27.8 \times 0.102 = 2.83\) pixels tall. At 336 it is 4.24 pixels, at 672 it is 8.48 pixels. Letters need several pixels of height to stay readable; at 3 pixels, “e”, “a” and “o” are the same blob. [Certain for the arithmetic]

Seen the other way: one 14-pixel patch at 224 covers \(14 / 0.102 = 137.5\) original pixels of height, about five lines of text. One token has to describe five lines.

The same problem appears for other inputs. A chart of \(1200 \times 800\) with 12-pixel tick labels squashed to 224 has labels 3.4 pixels tall. In a 4K video frame (\(3840 \times 2160\)), a traffic sign 40 pixels wide becomes \(40 \times 224/3840 = 2.3\) pixels wide: one sixth of a patch. No amount of model quality brings back pixels that were thrown away.

And there is a second, quieter problem: aspect ratio. A receipt of \(400 \times 1600\) squashed into a square is stretched 4 times wider than its height scale. Letters become wide and short, shapes the encoder never saw in training. Padding to a square instead keeps the shape but wastes most of the square on empty pixels.

224
Original pixels (a 560 × 300 crop of the full image)
The same crop after the whole image is resized to S × S
Figure 1. What a fixed square input does to detail, formula (180.2). The left panel is a crop of the original image at full resolution. The right panel is the same crop after the whole image is resized to \(S \times S\), then blown up again so you can compare. The images are drawn in your browser; the numbers in the readout are exact.
How to read Figure 1

Start with the letter page at 224: the text is gone. Move the slider to the right. At 672 the text starts to come back, at 1344 it is readable. The readout gives the text height after resizing, the area of the original that one patch must summarize, and the token count \((S/14)^2\), which grows with the square of \(S\). Switch to the 4K street frame: the small sign needs a very large \(S\), and the squash factors \(s_x \ne s_y\) show the shape distortion of a non-square image.

Why not simply train one encoder at \(1344 \times 1344\)? Formula (180.1): \(N = 96^2 = 9{,}216\) tokens for every image, even a small icon, and the ViT’s attention cost grows with \(N^2\), 256 times the cost at 336. Most of those tokens would be wasted on small or simple images. We want the token count to follow the content. The two families below do this in different ways.

Tiling: a thumbnail plus high-resolution crops

The idea of tiling: do not change the encoder, change the input. The encoder still sees \(336 \times 336\) images (its training size), but a large image is first resized to fit a grid of \(r \times c\) such tiles. Each tile is encoded separately, and the tile features are put back together into one large feature map. LLaVA-1.5-HD introduced this, with one more piece: the features of a downsampled copy of the whole image are added too, because a single tile only sees a part of the scene. [Certain] In the LLaVA-1.5 paper, adding this global context raised MM-Vet from 31.9 to 35.1 and MME from 1425.8 to 1497.5 for the 7B model. [Certain]

LLaVA-NeXT (January 2024) made this the default and called it AnyRes. Its tile grids are \(\{2{\times}2,\ 1{\times}\{2,3,4\},\ \{2,3,4\}{\times}1\}\) with 336-pixel tiles, so the largest inputs are \(672 \times 672\), \(336 \times 1344\) and \(1344 \times 336\): four times the pixels of LLaVA-1.5. [Certain] Which grid should an image use? The rule (from the LLaVA-NeXT code, now in Hugging Face select_best_resolution) is: fit the image inside each candidate canvas without changing its aspect ratio, and count how many original pixels survive.

\[ s = \min\!\Big(\frac{W_c}{W},\ \frac{H_c}{H}\Big), \qquad e = \min\!\big(\lfloor sW \rfloor \lfloor sH \rfloor,\ W H\big), \qquad w = W_c H_c - e \tag{180.3} \]
\[ (r^*, c^*) = \arg\max_{(r,c)} \; e, \quad \text{ties broken by the smallest } w \tag{180.4} \]
Read the symbols
\(W_c, H_c\)
width and height of a candidate canvas: \(W_c = 336\,c\), \(H_c = 336\,r\) for a grid of \(r\) rows and \(c\) columns
\(s\)
the scale that makes the image fit inside the canvas, keeping its aspect ratio
\(e\)
effective resolution: pixels of real image content after fitting; it is capped at \(WH\) because upscaling a small image adds no information
\(w\)
wasted resolution: canvas pixels that hold padding (or upscaled duplicates)

So the rule prefers the canvas that keeps the most detail, and among equal ones the smallest canvas. The image is resized by \(s\), padded to the canvas, cut into tiles, and each tile becomes \(24 \times 24 = 576\) features. Then two clean-ups happen on the joined feature map: features that cover only padding are removed, and a learned newline token is added at the end of each row so the language model knows where a row ends (the LLaVA-1.5 paper describes both for its HD variant [Certain]). With the thumbnail first, the token count is

\[ N_{\text{AnyRes}} = 576 + h_f\,(w_f + 1) \tag{180.5} \]
Read the symbols
\(576\)
tokens of the thumbnail: the whole image resized to one \(336 \times 336\) tile
\(h_f, w_f\)
height and width of the joined tile feature map after removing padding rows or columns; before that removal it is \(24r \times 24c\)
\(+1\)
one newline token per row
Worked example: a 1000 × 600 photo

Width 1000, height 600. The seven candidates, with formula (180.3):

Grid r × cCanvas W × HScale sFitted sizeEffective eWasted w
2 × 2672 × 6720.672672 × 403270,816180,768
1 × 2672 × 3360.560560 × 336188,16037,632
1 × 31008 × 3360.560560 × 336188,160150,528
1 × 41344 × 3360.560560 × 336188,160263,424
2 × 1, 3 × 1, 4 × 1336 × 672 …0.336336 × 20167,536158,256 …

The \(2 \times 2\) grid keeps the most pixels (270,816 of the original 600,000), so it wins even though 40% of its canvas is padding. The joined feature map is \(48 \times 48\). Padding rows are removed: \(\operatorname{int}(600 \times 48/1000) = 28\) rows remain, so \(h_f = 28\), \(w_f = 48\), and formula (180.5) gives \(576 + 28 \times 49 = 1{,}948\) tokens. Without removing padding it would be \(576 + 48 \times 49 = 2{,}928\). [Certain: computed with the same code as the PyTorch section]

Algorithm 1: AnyRes tiling (LLaVA-NeXT style)
  1. input: image of size \(W \times H\); tile size \(T = 336\); candidate grids \(G\)
  2. \((r, c) \leftarrow\) the grid with the largest effective resolution \(e\), then the smallest waste \(w\) // (180.3), (180.4)
  3. resize the image by \(s\), pad it (centered) to \(cT \times rT\), cut it into \(r \cdot c\) tiles
  4. thumbnail \(\leftarrow\) the whole image resized to \(T \times T\)
  5. encode the thumbnail and every tile with the same frozen-size ViT; each gives \(24 \times 24\) features
  6. place tile features in a \(24r \times 24c\) map; drop rows or columns that cover only padding
  7. append a newline embedding at the end of each row; flatten row by row
  8. return [thumbnail tokens, tile tokens] // (180.5) tokens, then the connector

Figure 2 runs Algorithm 1 on the image you choose, one line at a time. First it scores the seven candidate grids with (180.3), then it cuts the tiles, adds the thumbnail, removes the padding features and adds the newline tokens. The token count on the right grows as each piece is added.

1000 600
fitted imagepadding336 × 336 tilepadding features removednewline tokens
  1. \(e^\ast \leftarrow 0,\ w^\ast \leftarrow \infty\)
  2. for each grid \((r, c)\) in \(G\): \(s \leftarrow \min(cT/W,\ rT/H)\)
  3. \(e \leftarrow \min(\lfloor sW \rfloor \lfloor sH \rfloor,\ WH)\), \(w \leftarrow cT \cdot rT - e\) // (180.3)
  4. if \(e \gt e^\ast\) or (\(e = e^\ast\) and \(w \lt w^\ast\)): best \(\leftarrow (r, c)\) // (180.4)
  5. resize by \(s\), pad to \(cT \times rT\), cut \(r \cdot c\) tiles; each tile \(\to 24 \times 24\) features
  6. thumbnail \(\leftarrow\) whole image resized to \(T \times T\) \(\to 576\) features
  7. drop feature rows or columns that cover only padding \(\to h_f \times w_f\)
  8. add one newline per row; \(N \leftarrow 576 + h_f (w_f + 1)\) // (180.5)
Figure 2. AnyRes tiling, Algorithm 1 and formulas (180.3) to (180.5), computed in your browser with the LLaVA-NeXT grids and 336-pixel tiles (\(T = 336\)). Left: the canvas of the grid under test (or the chosen grid), the fitted image and the padding. Right: the 336 × 336 thumbnail (the whole image squeezed into a square) and the joined tile feature map, where removed padding rows or columns turn red and the newline column is green. The table fills in as each candidate is scored; the current best row is highlighted.
How to read Figure 2

Press Play, or Step through it. With the 1000 × 600 photo the first candidate, \(2 \times 2\), gives \(s = \min(672/1000, 672/600) = 0.672\), a fitted image of 672 × 403 and \(e = 270{,}816\). The next six candidates all keep fewer pixels (the \(1 \times c\) grids reach only 188,160), so \(2 \times 2\) stays the best. Then the four tiles are cut (4 × 576 = 2,304 features), the thumbnail adds 576, the unpad step keeps 28 of the 48 feature rows, and the newlines add 28: \(576 + 28 \times 49 = 1{,}948\). Now try the presets. The receipt picks \(4 \times 1\) with zero waste, so nothing is removed. The small 336 × 336 image has no \(1 \times 1\) grid in this list; it ties on \(e\) everywhere, so the smallest waste decides: \(1 \times 2\) and \(2 \times 1\) tie again, and the first one in the list, \(1 \times 2\), wins. After the padding columns are removed it still costs \(576 + 24 \times 25 = 1{,}176\) tokens, because the thumbnail and the tile show the same pixels twice. Qwen2-VL (Figure 3) needs 144 tokens for it. Moving a slider keeps the current step, so you can stop on a candidate and compare images.

InternVL 1.5 uses the same idea with a different selection rule. It uses \(448 \times 448\) tiles and 35 predefined aspect ratios (all grids with at most 12 tiles during training), picks the ratio closest to the image’s ratio, and breaks ties by not enlarging the image to more than twice its area. A \(448 \times 448\) thumbnail is always added. A pixel shuffle in the connector cuts each tile to 256 tokens, and at test time the number of tiles can go up to 40, about 4K resolution. [Certain] So 12 tiles plus a thumbnail cost \(13 \times 256 = 3{,}328\) tokens.

Tiling has clear strengths: it reuses a strong pretrained encoder (CLIP, SigLIP, InternViT) without retraining it at a new size, every tile is a standard batch item, and the cost is easy to cap (maximum number of tiles). Its weaknesses: a fixed menu of grids, padding waste, tile borders that cut objects, and the thumbnail tokens paid again for every image.

Native dynamic resolution: Qwen2-VL

The second family removes the fixed size from the encoder itself. Qwen2-VL (September 2024) calls it Naive Dynamic Resolution: the ViT takes an image of (almost) any size, and the number of tokens follows the image. [Certain] Three changes make this possible.

  1. No absolute position table. A learned table of \((S/p)^2\) position embeddings only works for one grid size. Qwen2-VL removes it and uses 2D RoPE inside the ViT: part of each attention head is rotated by the patch’s row, the other part by its column (session 14). Rotations work for any row and column number. [Certain]
  2. Resize only a little. The image is resized to the nearest multiples of 28 pixels (14-pixel patches, then a 2 × 2 merge), and scaled down only if it exceeds a pixel budget.
  3. Merge 2 × 2 patches. After the ViT, a small MLP merges each 2 × 2 group of neighbouring patch features into one token for the language model, so one LLM token covers \(28 \times 28\) pixels. [Certain]

The resize rule, from smart_resize in the Hugging Face Qwen2-VL image processor:

\[ \tilde H = 28 \cdot \operatorname{round}\!\Big(\frac{H}{28}\Big), \quad \tilde W = 28 \cdot \operatorname{round}\!\Big(\frac{W}{28}\Big); \quad \text{if } \tilde H \tilde W \gt P_{\max}: \ \beta = \sqrt{\frac{HW}{P_{\max}}},\ \ \tilde H = 28 \Big\lfloor \frac{H}{28\beta} \Big\rfloor,\ \ \tilde W = 28 \Big\lfloor \frac{W}{28\beta} \Big\rfloor \tag{180.6} \]
\[ N_{\text{Qwen}} = \frac{\tilde H}{28} \cdot \frac{\tilde W}{28} \;(+\,2 \text{ for the vision start and end tokens}) \tag{180.7} \]
Read the symbols
\(\tilde H, \tilde W\)
the size actually fed to the ViT; both are multiples of 28 and the aspect ratio stays close to the original
\(P_{\max}\)
the pixel budget; the released processor config uses \(16{,}384 \times 28^2 = 12{,}845{,}056\) pixels (at most 16,384 tokens), and the model card suggests lower values such as \(1280 \times 28^2\) to save memory [Certain]
\(\beta\)
the shrink factor when the image is too big; a symmetric rule with \(P_{\min} = 4 \times 28^2 = 3{,}136\) pixels enlarges tiny images
\(N_{\text{Qwen}}\)
tokens the language model sees: one per \(28 \times 28\) pixel block

The merge step, for the \(2 \times 2\) block at merged position \((i, j)\):

\[ \mathbf{z}_{i,j} = \operatorname{MLP}\Big(\operatorname{LN}\big[\mathbf{x}_{2i,2j};\ \mathbf{x}_{2i,2j+1};\ \mathbf{x}_{2i+1,2j};\ \mathbf{x}_{2i+1,2j+1}\big]\Big), \qquad \mathbf{x} \in \R^{1280},\ \ \mathbf{z} \in \R^{3584} \tag{180.8} \]
Read the symbols
\(\mathbf{x}_{a,b}\)
output feature of the ViT patch in row \(a\), column \(b\); width 1,280 in the Qwen2-VL encoder (32 layers, about 675M parameters) [Certain]
\([\,;\,]\)
concatenation: four patch features give one vector of \(4 \times 1280 = 5120\) numbers
\(\mathbf{z}_{i,j}\)
the merged token, projected to the LLM width (3,584 for Qwen2-VL-7B)

This merge is the same trick as pixel shuffle in session 178: trade 4 tokens for 1 wider token, so the LLM sequence is 4 times shorter while the ViT still worked at 14-pixel detail.

Worked example: the same 1000 × 600 photo

\(\tilde H = 28 \cdot \operatorname{round}(600/28) = 28 \times 21 = 588\), \(\tilde W = 28 \cdot \operatorname{round}(1000/28) = 28 \times 36 = 1008\). The product \(588 \times 1008 = 592{,}704\) is under every budget, so no further scaling. The ViT sees \(42 \times 72 = 3{,}024\) patches; after the merge the LLM gets \(21 \times 36 = 756\) tokens, plus 2 special tokens: 758. AnyRes needed 1,948 tokens for this image and still shrank it to 672 × 403. Here the image keeps 99% of its pixels and its shape.

A \(224 \times 224\) image gives \(8 \times 8 = 64\) tokens, 66 with the start and end tokens: exactly the example in the Qwen2-VL paper. [Certain]

1000 600
Figure 3. Qwen2-VL dynamic resolution, formulas (180.6) and (180.7). Left: the resized image with one cell per LLM token (28 × 28 pixels); the highlighted cell shows the four 14-pixel ViT patches that the merger (180.8) turns into one token. Right: tokens for the same image with a fixed 336 input, AnyRes (Figure 2), and Qwen2-VL.
How to read Figure 3

Move the sliders. The token grid keeps the shape of the image: a wide image gives a wide grid. When the image is larger than the budget, the readout shows the shrink factor \(\beta\) and the grid stops growing. Switch the budget to 16,384 and a 1700 × 2200 page keeps almost its full size (4,819 tokens), so the 10-point text stays about 28 pixels tall. With the 1,280 budget the page gets 1,240 tokens and the text is about 14 pixels tall. In the bar chart, the fixed 336 input always costs 576 tokens whatever the image.

What does the dynamic approach buy? The Qwen2-VL paper has a clean ablation on the 7B model: fixed token counts versus dynamic resolution. [Certain]

Image tokensAverage countInfoVQARealWorldQAOCRBenchMMMU
Fixed6428.8556.4757253.33
Fixed57665.7265.8882852.78
Fixed160074.9969.5482452.89
Fixed313677.2770.5978653.44
Dynamic192475.8970.0786653.44

Two lessons. First, text-heavy tasks (InfoVQA, OCRBench) gain hugely from more tokens, while a reasoning benchmark like MMMU barely moves: resolution matters most for reading. Second, dynamic resolution gets close to the largest fixed setting with 39% fewer tokens on average (1,924 versus 3,136), and beats every fixed setting on OCRBench.

The cost moves into the ViT. With full attention over all patches, a 16,384-token image means 65,536 patches in one attention. Qwen2.5-VL therefore uses window attention in most ViT layers: windows of \(112 \times 112\) pixels (\(8 \times 8\) patches), with full attention only in layers 7, 15, 23 and 31. [Certain: from its config.json]

Inside the ViT: patch order and 2D RoPE

Two details of the ViT input make native resolution work. First, the patch order. The 14-pixel patches are not listed row by row. They are listed in \(2 \times 2\) blocks: the four patches that the merger (180.8) will join are always four neighbours in the sequence. Then the merge is only a reshape of four consecutive tokens. [Certain: from rot_pos_emb in the Hugging Face code]

Second, 2D RoPE. The Qwen2-VL ViT has width 1,280 and 16 heads, so each head has \(d = 80\) dimensions, which form 40 rotation pairs. The first 20 pairs are rotated by the patch’s row \(a\), the last 20 by its column \(b\), with the usual RoPE frequencies. [Certain: from the Hugging Face code, base 10,000]

\[ \begin{gathered} \phi_j(a, b) = \begin{cases} a\,\theta_j & 0 \le j \lt 20 \\ b\,\theta_{j-20} & 20 \le j \lt 40 \end{cases}, \qquad \theta_j = 10000^{-2j/40} \\ \mathbf{q}_{a,b}^\top\, \mathbf{k}_{a',b'} = \big(R_{\phi(a,b)}\mathbf{q}\big)^\top R_{\phi(a',b')}\mathbf{k} = \mathbf{q}^\top R_{\phi(a'-a,\ b'-b)}\, \mathbf{k} \end{gathered} \tag{180.9} \]
Read the symbols
\(a, b\)
row and column of the 14-pixel patch in the resized image, counted from 0
\(\phi_j\)
rotation angle of pair \(j\); pairs 0 to 19 see only the row, pairs 20 to 39 only the column
\(\theta_j\)
frequency of pair \(j\): \(\theta_0 = 1\), \(\theta_1 = 0.631\), down to \(\theta_{19} = 0.000158\) radians per patch
\(R_{\phi}\)
the block-diagonal rotation that turns every pair \(j\) by its angle \(\phi_j\)
\(\mathbf{q}, \mathbf{k}\)
a query and a key before rotation; \(\mathbf{q}_{a,b}\) is the query of the patch at \((a, b)\) after rotation

The last line is the reason this works at any size. The angles are linear in \((a, b)\), and two rotations by \(\phi\) and \(\phi'\) combine into one rotation by \(\phi' - \phi\). So the attention score sees only the 2D offset \((a' - a,\ b' - b)\), never the absolute grid size. A table of learned positions would have no entry for row 40 of a tall receipt; a rotation by \(40\,\theta_j\) is always defined.

Figure 4 runs this part on a small image: the resize of (180.6), the patches in block order with their \((a, b)\) labels, the rotation angles of the selected patch, a numerical check of (180.9), and the \(2 \times 2\) merge.

330 200
original size (dashed)patches already in the sequenceselected patch (query)key patch for the check
  1. \(\tilde H \leftarrow 28\,\operatorname{round}(H/28)\), \(\tilde W \leftarrow 28\,\operatorname{round}(W/28)\)
  2. if \(\tilde H \tilde W \gt P_{\max}\): \(\beta \leftarrow \sqrt{HW/P_{\max}}\), shrink both sides with \(\lfloor \cdot \rfloor\) // (180.6)
  3. for each \(2 \times 2\) block, row by row: append its 4 patches
  4. patch \((a, b)\): turn pairs 0–19 by \(a\theta_j\), pairs 20–39 by \(b\theta_j\) // (180.9)
  5. check: the score depends only on \((\Delta a, \Delta b)\)
  6. merge each 4 consecutive patches into 1 token; \(N = \tfrac{\tilde H}{28} \cdot \tfrac{\tilde W}{28}\) // (180.7), (180.8)
Figure 4. Native-resolution patching with 2D RoPE, formulas (180.6) to (180.9), computed in your browser. Left: the image at its original size (dashed) and after the resize, cut into 14-pixel patches (thin lines) and \(2 \times 2\) merge blocks (thick lines); patches fill in as they enter the sequence. Right: two rotation dials for the selected patch (pair 0 turns with the row, pair 20 with the column), and the first patches of the ViT sequence, colored by merge block. The check uses the real head size \(d = 80\) and fixed test vectors \(q_i = \sin(1.3i + 0.4)\), \(k_i = \cos(0.7i + 1.1)\). Click any patch to select it.
How to read Figure 4

Press Play. The default image is 330 × 200 with a budget of 64 tokens (\(P_{\max} = 64 \times 784 = 50{,}176\) pixels). Rounding gives 336 × 196 = 65,856 pixels, over the budget, so \(\beta = \sqrt{66{,}000 / 50{,}176} = 1.147\) and the image becomes 280 × 168: 12 × 20 = 240 patches. The scan then adds one row of merge blocks per step; look at the sequence strip: four patches of the same color always sit next to each other. The dials turn by \(a \cdot 1\) and \(b \cdot 1\) radians for pair 0 and pair 20. At the check step the score of the selected patch with the patch one row down and two columns right is \(0.6552\), the same as for the patches at (0, 0) and (1, 2), wherever you click. An offset of (2, 1) gives a different score, \(0.5518\). The last step merges the blocks: 6 × 10 = 60 tokens. Switch the budget to 256: the image is only rounded (336 × 196), and you get 84 tokens.

Packing many images: NaViT

If every image has its own number of tokens, how do we put them in a batch? The naive way pads every image to the longest one. NaViT (“Patch n’ Pack”, 2023) showed a better way, borrowed from language-model pretraining (session 18): pack the patches of several images into one long sequence, and use an attention mask so that each image only attends to itself. [Certain]

\[ M_{ij} = \begin{cases} 0 & \text{if tokens } i \text{ and } j \text{ belong to the same image} \\ -\infty & \text{otherwise} \end{cases}, \qquad \operatorname{Attn} = \softmax\!\Big(\frac{QK^\top}{\sqrt{d}} + M\Big) V \tag{180.10} \]
Read the symbols
\(M\)
the packing mask, added to the attention scores; it is block-diagonal: one square block per image
\(-\infty\)
after softmax this gives weight 0, so no information flows between images
\(Q, K, V, d\)
queries, keys, values and head size of normal attention (session 17)

The result is exactly the same as running each image alone (the PyTorch section checks this), but the hardware sees one dense sequence. In practice the mask is not stored: FlashAttention’s variable-length kernel takes the start offsets of each image (cu_seqlens) and skips the off-diagonal blocks. Qwen2-VL’s ViT does exactly this. [Certain: from the Hugging Face code] Pooling at the end is also per image (NaViT’s masked pooling).

Algorithm 2: packing variable-size images (first-fit decreasing)
  1. input: images with token counts \(n_1, \dots, n_B\); sequence length \(L\) (each \(n_k \le L\))
  2. sort the images by \(n_k\), largest first
  3. for each image: put it in the first sequence with room left; if none has room, open a new sequence
  4. for each sequence, record the start offsets of its images (cu_seqlens) and the image id of every token
  5. run the ViT with the block-diagonal mask (180.10), or a variable-length attention kernel with the offsets
  6. split the output by the offsets: each image gets back its own tokens // then the 2×2 merge (180.8)

NaViT added three more ideas that matter for training [Certain]:

  • Factorized position embeddings: separate learned tables for the row and for the column, summed: \(\mathbf{e}_{\text{pos}}(r, c) = \mathbf{e}^{y}_r + \mathbf{e}^{x}_c\). Two short tables instead of one table per grid size. Qwen2-VL goes one step further with 2D RoPE, which needs no table at all.
  • Resolution sampling: train on a mix of image sizes, so the model learns from both small and large inputs.
  • Token dropping per image: randomly drop a different fraction of patches from each image; packing makes the varying lengths free.

With packing, NaViT reports that typically less than 2% of tokens are padding, and that a NaViT matches the best plain ViT with about four times less training compute. [Certain]

2048
  1. sort the images by \(n_k\), largest first
  2. for each image \(k\) in that order:
  3. for each open row \(j\): if \(u_j + n_k \le L\): put \(k\) in row \(j\), break
  4. if no row had room: open a new row with \(k\)
  5. cu_seqlens of each row \(\leftarrow\) prefix sums of its image lengths; mask (180.10)
Figure 5. Packing versus padding for twelve images (a photo, a page, a chart, a receipt, a screenshot and others) resized by Qwen2-VL with the 1,280-token budget: 8,732 tokens in total. The animation runs Algorithm 2 (first-fit decreasing) one image per step; \(u_j\) is the number of tokens already in row \(j\). Left: each row is one sequence; colored segments are images, grey is free space or padding, and the image just placed has a thick outline. Right: the block-diagonal mask (180.10) of the row that received the last image (or of the row you click).
How to read Figure 5

Press Play with \(L = 2048\). Step 1 sorts the twelve images: chart 1,247, page 1,240, screen 1,222, scan 999, and so on. The chart opens row 1. The page does not fit there (\(1{,}247 + 1{,}240 = 2{,}487 \gt 2{,}048\)), so it opens row 2; the screen and the scan also open new rows. The receipt (798) is the first small image that fits next to the chart: \(1{,}247 + 798 = 2{,}045 \le 2{,}048\). At the end the twelve images fill 5 rows, and only 14.7% of the 10,240 slots are padding; row 1 has cu_seqlens \([0, 1247, 2045]\), and its two blocks need 52.3% of the work of a full \(2048 \times 2048\) attention. Switch to “Pad to the longest”: every image opens its own row padded to 1,247 tokens (the chart), so 41.6% of all slots are padding. Change \(L\) to see more or fewer rows.

M-RoPE: positions for time, height and width

The ViT knows where each patch is (2D RoPE). But after the merge, the visual tokens are placed into the language model’s sequence next to text tokens, and the language model has its own position encoding. Standard 1D RoPE gives each token a single number: its index in the sequence (session 14). For an image of \(h \times w\) tokens, flattened row by row, a token and the one directly below it are \(w\) positions apart, while it and the one to its right are 1 apart. The LLM must learn the grid width from data, and it changes with every image size. Video adds a third axis.

Qwen2-VL’s Multimodal RoPE (M-RoPE) gives every token three position numbers: temporal, height and width, \(\mathbf{p} = (p^t, p^h, p^w)\). [Certain]

\[ \mathbf{p} = \begin{cases} (n,\ n,\ n) & \text{text token at index } n \\ (s,\ s + r,\ s + c) & \text{image token in row } r, \text{ column } c \\ (s + \tau,\ s + r,\ s + c) & \text{video token in frame group } \tau \end{cases}, \qquad s_{\text{next}} = \max(\text{IDs so far}) + 1 \tag{180.11} \]
Read the symbols
\(n\)
for text, all three parts are the same number, so M-RoPE on text is exactly 1D RoPE
\(s\)
start ID of an image or video: one more than the largest ID before it
\(r, c\)
row and column of the merged token in its grid (\(0 \le r \lt h\), \(0 \le c \lt w\))
\(\tau\)
index of the frame group; Qwen2-VL samples video at 2 frames per second and merges 2 frames per token (3D patches of depth 2), so one group is about 1 second [Certain]
\(s_{\text{next}}\)
the next modality starts after the largest ID used so far; for an image this is \(s + \max(h, w)\), not \(s + hw\)

How do three numbers enter one rotation? RoPE rotates each pair of dimensions \(i\) of a query or key by an angle proportional to the position, with a frequency \(\theta_i\) that falls with \(i\). M-RoPE splits the frequency pairs into three groups and lets each group use one of the three position parts:

\[ \phi_i = p^{\,g(i)}\, \theta_i, \qquad \theta_i = b^{-2i/d}, \qquad g(i) = \begin{cases} t & 0 \le i \lt 16 \\ h & 16 \le i \lt 40 \\ w & 40 \le i \lt 64 \end{cases} \tag{180.12} \]
Read the symbols
\(\phi_i\)
rotation angle of dimension pair \(i\) for this token
\(d\)
head size, 128 in Qwen2-VL-7B (3,584 / 28 heads), so there are 64 pairs
\(b\)
RoPE base, \(10^6\) in Qwen2-VL
\(16, 24, 24\)
mrope_section in the config: 16 pairs for time, 24 for height, 24 for width [Certain]

Because each group is an ordinary RoPE, the key property of RoPE carries over: the attention score between a query at \(\mathbf{p}_q\) and a key at \(\mathbf{p}_k\) depends only on the difference \(\mathbf{p}_q - \mathbf{p}_k\), now a 3D offset \((\Delta t, \Delta h, \Delta w)\). Moving an image within the prompt does not change how its tokens attend to each other. (The PyTorch code checks this.)

Worked example: text, a 2 × 3 image, text

Take 4 tokens before the image (the last one is the vision-start token), an image of \(2 \times 3\) merged tokens, then 3 text tokens. Text gets \((0,0,0), (1,1,1), (2,2,2), (3,3,3)\). The image starts at \(s = 4\): its first row is \((4,4,4), (4,4,5), (4,4,6)\), its second row \((4,5,4), (4,5,5), (4,5,6)\). The largest ID used is 6, so text continues at 7: \((7,7,7), (8,8,8), (9,9,9)\).

With 1D RoPE the image would take positions 4 to 9 and the text would start at 10. For a \(36 \times 21\) grid (the 1000 × 600 photo) the difference is large: 1D RoPE uses 756 positions for the image, M-RoPE only \(\max(21, 36) = 36\). Long documents and videos use far fewer position IDs, which helps the model stay inside the position range it saw in training.

2 3 1
Figure 6. Position IDs from formula (180.11) for a prompt “text, image or video, text”. Each chip is one token with its IDs; colored chips are visual tokens (labelled f0, f1, … when there are several frame groups). Click any visual token to compare it with its right and lower neighbours.
How to read Figure 6

In M-RoPE mode, the image chips show \((t, h, w)\): the first number stays fixed for one frame group, the second follows the row, the third the column. The text after the image starts at the largest ID plus one. Switch to 1D RoPE: the same tokens get one number each, the lower neighbour is now \(w\) positions away, and the text after the image starts later. Add frame groups to see video: only the first number changes from group to group.

Choosing a strategy and a token budget

Here are the token counts for four real-shaped inputs, all computed with the code below (Qwen2-VL count without the 2 special tokens):

Input (W × H)Fixed 336AnyRes (LLaVA-NeXT)Qwen2-VL, budget 1,280Qwen2-VL, budget 16,384
Photo 1000 × 6005761,948 (2 × 2)756756
Letter page 1700 × 22005762,448 (2 × 2)1,2404,819
Receipt 400 × 16005762,976 (4 × 1)798798
4K frame 3840 × 21605761,948 (2 × 2)1,22210,549

Text height on the letter page (27.8 pixels at full size): 4.2 pixels at fixed 336, 8.5 with AnyRes (the page is shrunk to 519 × 672), 14.1 with the 1,280 budget, and 27.9 with the 16,384 budget. These numbers explain most of the difference between VLMs on document benchmarks such as DocVQA. Practical rules:

  • Documents, charts, screenshots: raise the budget until the smallest text you care about is at least about 10 to 15 pixels tall after resizing. [Likely: a rule of thumb, not a paper result] Measure accuracy against tokens on your own data.
  • Natural photos and classification: low budgets are enough; MMMU in the table above did not change between 64 and 3,136 tokens.
  • Video: tokens multiply by the number of frame groups. Qwen2-VL caps a video at 16,384 tokens [Certain], so more frames means fewer pixels per frame. Choose between temporal and spatial detail per task (session 205, session 206).
  • Latency: prefill cost grows with tokens, attention with their square. The 16,384-budget page has \(4819/576 = 8.4\) times the tokens of a fixed 336 input and about 70 times the attention-score work. Cap the budget per request.

The PyTorch version

The code is small, deterministic and runs on a CPU in about a second. It implements AnyRes grid selection and token counting (180.3 to 180.5), Qwen2-VL’s smart_resize (180.6, 180.7), patchify and the 2 × 2 merger (180.8), packed attention with a block-diagonal mask (180.10) checked against separate attention, and M-RoPE position IDs and rotation (180.11, 180.12) with two checks: scores depend only on 3D offsets, and on text M-RoPE equals 1D RoPE. The printed values are the ones used in the text.

import math, torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)

# 1) AnyRes: pick the best grid of 336-px tiles (formulas 180.3 to 180.5)
TILE = 336
GRIDS = [(2, 2), (1, 2), (1, 3), (1, 4), (2, 1), (3, 1), (4, 1)]      # (rows, cols)

def select_best_resolution(H, W, grids=GRIDS, tile=TILE):
    best, best_eff, best_waste = None, 0, float("inf")
    for r, c in grids:
        Hc, Wc = r * tile, c * tile
        s = min(Wc / W, Hc / H)                                       # fit inside, keep aspect
        eff = min(int(W * s) * int(H * s), W * H)                     # no credit for upscaling
        waste = Hc * Wc - eff
        if eff > best_eff or (eff == best_eff and waste < best_waste):
            best, best_eff, best_waste = (r, c), eff, waste
    return best, best_eff, best_waste

def anyres_tokens(H, W, per_side=24):
    (r, c), _, _ = select_best_resolution(H, W)
    fh, fw = r * per_side, c * per_side                               # feature map of all tiles
    if W / H > fw / fh:                                               # remove rows of padding
        fh = fh - 2 * ((fh - int(round(H * fw / W, 7))) // 2)
    else:                                                             # remove columns of padding
        fw = fw - 2 * ((fw - int(round(W * fh / H, 7))) // 2)
    return (r, c), per_side ** 2 + fh * (fw + 1)                      # thumbnail + rows with newline

print(select_best_resolution(600, 1000), anyres_tokens(600, 1000))
# ((2, 2), 270816, 180768) ((2, 2), 1948)

# 2) Qwen2-VL: resize to multiples of 28 inside a pixel budget (formula 180.6)
def smart_resize(H, W, factor=28, min_pixels=4 * 28 * 28, max_pixels=1280 * 28 * 28):
    h, w = round(H / factor) * factor, round(W / factor) * factor
    if h * w > max_pixels:
        beta = math.sqrt(H * W / max_pixels)
        h = max(factor, math.floor(H / beta / factor) * factor)
        w = max(factor, math.floor(W / beta / factor) * factor)
    elif h * w < min_pixels:
        beta = math.sqrt(min_pixels / (H * W))
        h, w = math.ceil(H * beta / factor) * factor, math.ceil(W * beta / factor) * factor
    return h, w

for H, W in [(600, 1000), (2200, 1700), (224, 224)]:
    h, w = smart_resize(H, W)
    print((H, W), "->", (h, w), "tokens", (h // 28) * (w // 28))
# (600, 1000) -> (588, 1008) tokens 756
# (2200, 1700) -> (1120, 868) tokens 1240
# (224, 224) -> (224, 224) tokens 64

# 3) Patchify at 14 px, then merge 2x2 neighbours into one LLM token (formula 180.8)
def patchify(img, p=14):                       # img (3, H, W), H and W multiples of 2p
    C, H, W = img.shape
    x = img.reshape(C, H // p, p, W // p, p).permute(1, 3, 0, 2, 4)   # (gh, gw, C, p, p)
    return x.reshape(H // p, W // p, C * p * p)

class Merger(nn.Module):                       # Qwen2-VL "PatchMerger", tiny sizes
    def __init__(self, d_vit=32, d_llm=48, m=2):
        super().__init__(); self.m = m
        self.ln = nn.LayerNorm(d_vit)
        self.mlp = nn.Sequential(nn.Linear(m * m * d_vit, m * m * d_vit), nn.GELU(), nn.Linear(m * m * d_vit, d_llm))
    def forward(self, x):                      # x (gh, gw, d_vit)
        gh, gw, d = x.shape; m = self.m
        x = self.ln(x).reshape(gh // m, m, gw // m, m, d).permute(0, 2, 1, 3, 4)
        return self.mlp(x.reshape(gh // m, gw // m, m * m * d))       # (gh/2, gw/2, d_llm)

img = torch.rand(3, 588, 1008)
patches = patchify(img)                        # (42, 72, 588)
embed = nn.Linear(3 * 14 * 14, 32)
tokens = Merger()(embed(patches))
print(tuple(patches.shape), tuple(tokens.shape), tokens.shape[0] * tokens.shape[1])
# (42, 72, 588) (21, 36, 48) 756

# 4) Patch n' Pack: several images in one sequence, block-diagonal mask (formula 180.10)
lens = [6, 4, 9]                               # tokens of three images
x = torch.randn(1, 2, sum(lens), 8)            # (batch, heads, packed length, head dim)
img_id = torch.repeat_interleave(torch.arange(len(lens)), torch.tensor(lens))
mask = img_id[:, None] == img_id[None, :]      # True = may attend
packed = F.scaled_dot_product_attention(x, x, x, attn_mask=mask)
alone = torch.cat([F.scaled_dot_product_attention(s, s, s) for s in x.split(lens, dim=2)], dim=2)
print("packed == separate:", torch.allclose(packed, alone, atol=1e-6), "| mask density", round(mask.float().mean().item(), 3))
# packed == separate: True | mask density 0.369

# 5) M-RoPE position ids for text, image, text (formula 180.11)
def mrope_ids(n_text_before, grid_h, grid_w, n_text_after):
    ids = [(p, p, p) for p in range(n_text_before)]
    s = n_text_before
    ids += [(s, s + r, s + c) for r in range(grid_h) for c in range(grid_w)]
    nxt = s + max(grid_h, grid_w)
    ids += [(p, p, p) for p in range(nxt, nxt + n_text_after)]
    return torch.tensor(ids).T                 # (3, L)

pos = mrope_ids(4, 2, 3, 3)
print(pos.tolist())
# [[0, 1, 2, 3, 4, 4, 4, 4, 4, 4, 7, 8, 9], [0, 1, 2, 3, 4, 4, 4, 5, 5, 5, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 4, 5, 6, 7, 8, 9]]

# 6) Apply M-RoPE: frequency pairs split 16 / 24 / 24 between t, h, w (formula 180.12)
def mrope_angles(pos, head_dim=128, base=1e6, sections=(16, 24, 24)):
    inv = 1.0 / base ** (torch.arange(0, head_dim, 2).float() / head_dim)   # (64,)
    which = torch.repeat_interleave(torch.arange(3), torch.tensor(sections))  # 0=t,1=h,2=w per pair
    return pos[which].T.float() * inv                                      # (L, 64)

def rotate(x, ang):                            # x (L, head_dim), half-split layout as in Qwen2
    c, s = ang.cos().repeat(1, 2), ang.sin().repeat(1, 2)
    x1, x2 = x.chunk(2, dim=-1)
    return x * c + torch.cat([-x2, x1], dim=-1) * s

q, k = torch.randn(1, 128), torch.randn(1, 128)
def score(pq, pk):
    aq = mrope_angles(torch.tensor(pq).view(3, 1)); ak = mrope_angles(torch.tensor(pk).view(3, 1))
    return (rotate(q, aq) * rotate(k, ak)).sum().item()
print(round(score((4, 5, 6), (4, 4, 4)), 4), round(score((9, 10, 11), (9, 9, 9)), 4))   # same 3D offset
# -10.1389 -10.1389
text_1d = torch.arange(5).float()[:, None] * (1.0 / 1e6 ** (torch.arange(0, 128, 2).float() / 128))
print("text M-RoPE == 1D RoPE:", torch.allclose(mrope_angles(torch.arange(5).repeat(3, 1)), text_1d))
# text M-RoPE == 1D RoPE: True

Things to try: call anyres_tokens(224, 224) and smart_resize(224, 224) and compare 1,176 tokens with 64; pass max_pixels=16384 * 28 * 28 and print the page again (4,819 tokens); change sections to (64, 0, 0) and check that score((4, 5, 6), (4, 4, 4)) now equals score((4, 4, 4), (4, 4, 4)): with only the temporal part, the model can no longer tell rows and columns apart.

Where this is used

ApplicationWhat resolution handling doesTypical setup
Document QA and OCR-free parsing (invoices, forms, scientific papers)Keeps small text readable and the page shape intactNative resolution with a high budget, or many tiles; see session 218
Chart and infographic QASmall labels and legends survive resizingDynamic resolution; InfoVQA gains most from tokens
GUI and computer-use agents (screenshots)Wide screens keep their aspect; click coordinates map back to pixelsNative resolution, coordinates in the resized frame (session 220)
Small-object questions in photos, drones, retail shelvesObjects of a few dozen pixels still cover a full tokenTiling or native resolution; or crop-then-ask
Video QA, key-moment detection and Video-RAGTrade frames against pixels under one token cap; M-RoPE time IDs order framesLow per-frame budget, more frames (session 210, session 207)
Video retrieval and fingerprintingUsually low resolution is enough; native aspect avoids letterbox artifactsSmall fixed or dynamic budget per frame (session 208, session 209)
Multi-image prompts (compare two receipts, before/after photos)Each image gets its own grid and its own M-RoPE blockPacking in the ViT; per-image token cap
Training vision encoders at mixed sizesPacking removes padding; resolution sampling and token droppingNaViT-style training; SigLIP 2 NaFlex variants follow this line [Likely]

Common mistakes

  • Squashing every image to a square. It distorts text and shapes and destroys small detail (Figure 1). Keep the aspect ratio: tile, pad, or use native resolution.
  • Using a different budget at inference than in training. A model fine-tuned at 1,280 tokens per image but served at 256 sees much smaller text than it learned on; one served far above its training sizes meets position ranges it never saw. Log min_pixels and max_pixels with every checkpoint.
  • Forgetting that tokens explode. A batch of document pages at a high budget can use more KV-cache memory than the text. Set a per-image and a per-request cap, and test the worst case (a 4K screenshot, a long receipt).
  • Mapping coordinates in the wrong frame. Qwen2-VL resizes to multiples of 28; boxes and click points predicted by the model are in the resized frame. Scale them back by \(W/\tilde W\) and \(H/\tilde H\) before drawing or clicking.
  • Computing M-RoPE positions by hand and getting them wrong. If your own batching code (packing, left padding, multi-image) gives visual tokens plain 1D positions or forgets the \(\max + 1\) rule, the model still runs but answers get worse. Use the model’s own get_rope_index, and pass the per-image grid sizes (image_grid_thw).
  • Packing without the mask. Concatenating images in the ViT without a block-diagonal mask (or cu_seqlens) lets images attend to each other. Nothing crashes; features quietly mix.
  • Ignoring tile borders. With tiling, a key word cut by a border may be misread. For critical fields, crop around the region and ask again, or use overlapping crops.
  • Paying for resolution the task does not need. MMMU did not improve from 64 to 3,136 tokens in Qwen2-VL’s ablation. For photo tagging or retrieval, a low budget is faster at the same quality.

Interview questions

Why do early VLMs fail on documents and small text?basic
Their encoder takes one fixed square input (224 or 336), so every image is shrunk to that size first. A letter page at 200 dpi (1700 × 2200) resized to 224 has 10-point text about 2.8 pixels tall, and one 14-pixel patch covers about five lines of text. The information is destroyed before the model runs. Squashing to a square also distorts the aspect ratio of non-square inputs. A bigger LLM cannot fix missing pixels.
Explain AnyRes in LLaVA-NeXT.core
Keep the 336-pixel CLIP encoder, but choose a grid of 336-pixel tiles from \(\{2 \times 2,\ 1 \times \{2,3,4\},\ \{2,3,4\} \times 1\}\). For each grid, fit the image inside the canvas keeping its aspect ratio, and count the effective pixels (capped at the original size); take the grid with the most effective pixels, ties broken by least waste. Resize, pad, cut into tiles, encode each tile (576 features each), join them, drop features that cover only padding, add a newline token per row, and prepend the features of a 336 × 336 thumbnail of the whole image. A 1000 × 600 photo gets a \(2 \times 2\) grid and \(576 + 28 \times 49 = 1{,}948\) tokens.
Why add a thumbnail when you already have high-resolution tiles?basic
Each tile is encoded alone and sees only a part of the image, so no encoder pass sees the whole layout, and objects cut by tile borders are split. The thumbnail gives one global view: where things are relative to each other, the overall scene. In LLaVA-1.5-HD the global context raised MM-Vet from 31.9 to 35.1 and MME from 1425.8 to 1497.5 (7B). The cost is 576 extra tokens per image.
What is Naive Dynamic Resolution in Qwen2-VL, and what has to change in the ViT to allow it?core
The image is resized only to the nearest multiples of 28 (and scaled down if it exceeds a pixel budget), then encoded at that size: the token grid has the image’s own shape. The ViT must not have a fixed absolute position table, so Qwen2-VL replaces it with 2D RoPE (rotations by row and column, valid for any grid). Images of different sizes in one batch are packed with a block-diagonal mask (variable-length attention). After the ViT, an MLP merges each \(2 \times 2\) block of patches into one LLM token, so one token covers 28 × 28 pixels: a 224 × 224 image becomes 64 tokens (66 with start/end tokens).
How do you batch images of different sizes efficiently?core
Pack them (NaViT, Patch n’ Pack): put the patches of several images into one sequence and use a block-diagonal attention mask so each image attends only to itself; pool per image. Use first-fit-decreasing or similar to fill sequences. With FlashAttention’s varlen kernel you pass cumulative sequence offsets (cu_seqlens) and the off-diagonal blocks are never computed, so the result equals separate processing but with dense hardware use. Padding to the longest image instead wastes compute: in Figure 5, 41.6% of slots versus about 15% for packed rows of 2,048. NaViT reports under 2% padding in its setup.
What is M-RoPE? Why not just use 1D RoPE on the flattened image?core
M-RoPE gives each token three position numbers (temporal, height, width) and splits the rotary frequency pairs into three groups (16, 24, 24 of 64 in Qwen2-VL), each rotated by one of the numbers. Text tokens get three equal numbers, so for text it is exactly 1D RoPE. Image tokens get \((s,\ s + \text{row},\ s + \text{col})\); video tokens also advance the temporal number per frame group. With 1D RoPE on a flattened grid, a token and its lower neighbour are \(w\) positions apart, and \(w\) changes with every image, so 2D structure must be learned from data; M-RoPE makes both neighbours one step away in their own axis. It also uses fewer position IDs: an image costs \(\max(h, w)\) IDs instead of \(h \cdot w\). Qwen2-VL’s ablation shows gains mostly on video and math (e.g. MathVista 39.2 → 43.4, NextQA 43.9 → 46.0).
After an image of 21 × 36 merged tokens that starts at position \(s\), where does the next text token start under M-RoPE?core
At \(s + \max(21, 36) = s + 36\). The largest ID used by the image is \(s + 35\) (the width part of the last column), and the rule is “largest previous ID plus one”. With 1D RoPE it would be \(s + 756\).
Compare tiling and native dynamic resolution. When would you choose each?advanced
Tiling: reuses any strong fixed-size encoder unchanged; every tile is a standard input, easy to batch and cap; but a fixed grid menu, padding waste, tile borders that cut text, and a thumbnail cost for each image (a 336 × 336 image still costs 1,176 tokens with the LLaVA-NeXT grids). Native resolution: tokens proportional to pixels, exact aspect ratio, no borders; but needs an encoder trained at mixed sizes (2D RoPE, packing), a variable-length attention path, a token cap, and ViT cost grows with image size (Qwen2.5-VL adds window attention). Choose tiling when you must keep a frozen CLIP/SigLIP tower or a fixed-shape deployment (for example a compiled engine with static shapes); choose native resolution for documents, screens and mixed inputs when you control the encoder.
Your document VLM misses small numbers in invoices. How do you debug it?advanced
First measure what the model sees: the resized size and the pixel height of the smallest text after preprocessing (log min_pixels, max_pixels, the resized \(\tilde H \times \tilde W\)). If the text is a few pixels tall, raise the budget or crop the region of interest and ask again. Check that training and serving use the same budget. Check aspect handling (no square squash). With tiling, check whether the numbers fall on tile borders. Then plot accuracy against tokens per page on a labeled set and pick the knee of the curve, with a hard cap for latency. Finally check coordinate mapping if the model also returns boxes.
Why does the Qwen2-VL ablation show big gains on InfoVQA but none on MMMU when tokens increase?advanced
InfoVQA and OCRBench require reading small text, so they are limited by pixels: from 64 to 3,136 fixed tokens, InfoVQA goes from 28.85 to 77.27. MMMU questions are mostly about reasoning over diagrams and subject knowledge at normal sizes; once the image is legible, more pixels add little (53.33 at 64 tokens, 53.44 at 3,136). This is why a fixed budget is wasteful: different images and tasks need very different token counts, which is the case for dynamic resolution (1,924 average tokens, close to the best scores).
How does a single image fit Qwen2-VL’s 3D patch embedding, and what does that mean for video tokens?advanced
The patch embedding works on \(2 \times 14 \times 14\) blocks (temporal_patch_size 2). A single image is repeated to two frames, so images and videos share one embedding. For video, sampled at 2 fps in the paper, every two frames become one temporal group, and each group of a 28 × 28 pixel area becomes one LLM token after the 2 × 2 merge. Video tokens are therefore \((\text{frames}/2) \times (\tilde H/28) \times (\tilde W/28)\), and the paper caps them at 16,384 per video, so longer videos must use fewer pixels per frame.

Exercises

  1. An A4 page scanned at 150 dpi is 1240 × 1754 pixels. A 10-point font at 150 dpi is how many pixels tall? How tall after squashing the page to 336 × 336? What are \(s_x\) and \(s_y\)?
  2. Use formulas (180.3) to (180.5) for a receipt of 400 × 1600 (width × height) with the LLaVA-NeXT grids. Which grid wins, how much is wasted, and how many tokens does it cost?
  3. Apply smart_resize (180.6) to a 1920 × 1080 screenshot with the 1,280-token budget and with the 16,384-token budget. How many LLM tokens in each case?
  4. A prompt has 2 text tokens, then a video of 3 frame groups with a 2 × 2 merged grid, then text. Give the M-RoPE IDs of the last video token and of the first text token after the video.
  5. Pack four images of 756, 1,240, 1,247 and 798 tokens into sequences of 2,048 with Algorithm 2. How many sequences, and what fraction is padding? Compare with padding each image to the longest.
  6. In Qwen2-VL-7B, what fraction of the rotary frequency pairs encode height? Which of the three parts gets the highest frequencies?
  7. How many times more attention-score work does one 4,819-token page need than a 576-token image, inside a single attention layer?
Answerscheck
1. \(10/72 \times 150 = 20.8\) pixels. \(s_x = 336/1240 = 0.271\), \(s_y = 336/1754 = 0.192\); the font becomes \(20.8 \times 0.192 = 4.0\) pixels tall, and letters are squashed (width kept 1.4 times more than height). 2. Grid \(4 \times 1\) (canvas 336 × 1344): \(s = \min(336/400, 1344/1600) = 0.84\), fitted size exactly 336 × 1344, effective 451,584, waste 0. Feature map \(96 \times 24\), no padding to remove: \(576 + 96 \times 25 = 2{,}976\) tokens. 3. Budget 1,280: \(\tilde H \times \tilde W = 728 \times 1316\), \(26 \times 47 = 1{,}222\) tokens. Budget 16,384: \(1092 \times 1932\), \(39 \times 69 = 2{,}691\) tokens (only rounding, no shrinking). 4. Video starts at \(s = 2\). Last video token: frame group 2, row 1, column 1: \((4, 3, 3)\). Largest ID used is 4, so the next text token is \((5, 5, 5)\). 5. Sorted: 1247, 1240, 798, 756. Sequence 1: 1247 + 798 = 2045; sequence 2: 1240 + 756 = 1996. Two sequences, padding \(1 - 4041/4096 = 1.3\%\). Padding to the longest: \(4 \times 1247 = 4988\) slots, \(1 - 4041/4988 = 19.0\%\) padding. 6. \(24/64 = 37.5\%\). Temporal: the first 16 pairs, with the largest \(\theta_i\). 7. \((4819/576)^2 = 70.0\) times.

Key takeaways

  • A fixed square input throws away detail before the model runs: a letter page at 224 has 2.8-pixel text, and squashing distorts aspect ratio (180.2). Tokens grow as \((S/p)^2\) and attention as their square (180.1), so “just use a bigger square” is wasteful.
  • Tiling (AnyRes) keeps the encoder and changes the input: pick the grid with the most effective pixels (180.3, 180.4), encode tiles plus a thumbnail, remove padding, add row newlines (180.5). LLaVA-NeXT: 336-pixel tiles up to 672 × 672 or 336 × 1344. InternVL 1.5: 448-pixel tiles, up to 12 in training and 40 at test, 256 tokens each.
  • Native dynamic resolution (Qwen2-VL): resize to multiples of 28 inside a pixel budget (180.6), 2D RoPE in the ViT, 2 × 2 merge so one LLM token is 28 × 28 pixels (180.7, 180.8). A 224 × 224 image is 64 tokens; dynamic resolution matched large fixed budgets with 1,924 tokens on average.
  • NaViT packing puts many images in one sequence with a block-diagonal mask (180.10), equal to separate processing but with almost no padding; varlen attention with cu_seqlens skips the masked blocks.
  • M-RoPE gives each token \((t, h, w)\) IDs (180.11) and splits the 64 frequency pairs 16 / 24 / 24 (180.12). Text reduces to 1D RoPE; an image uses \(\max(h, w)\) IDs, not \(hw\).
  • Resolution is a budget: reading tasks need it, reasoning and photo tasks much less. Set and log the budget, keep training and serving consistent, and map coordinates back to the original frame.

Key formulas

\[ \begin{gathered} N = \tfrac{H}{p}\cdot\tfrac{W}{p}, \qquad s = \min\!\big(\tfrac{W_c}{W}, \tfrac{H_c}{H}\big), \quad e = \min(\lfloor sW \rfloor \lfloor sH \rfloor, WH) \\ N_{\text{AnyRes}} = 576 + h_f (w_f + 1), \qquad \tilde H = 28\,\operatorname{round}(H/28), \quad N_{\text{Qwen}} = \tfrac{\tilde H}{28}\cdot\tfrac{\tilde W}{28} \\ \phi_j(a,b) = a\theta_j \ (j \lt 20), \ \ b\theta_{j-20} \ (j \ge 20), \qquad M_{ij} = 0 \text{ (same image)}, \ -\infty \text{ (otherwise)} \\ \mathbf{p}_{\text{img}} = (s,\ s+r,\ s+c), \quad s_{\text{next}} = \max + 1, \qquad \phi_i = p^{\,g(i)}\theta_i \end{gathered} \]

References for this session

  1. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  2. Dehghani et al. (2023). Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
  3. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
  5. Liu et al. (2024). LLaVA-NeXT: Improved reasoning, OCR, and world knowledge (blog)
  6. Chen et al. (2024). How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites (InternVL 1.5)
  7. Hugging Face (2024). Qwen2-VL in transformers: image processor (smart_resize) and get_rope_index
  8. Liu et al. (2023). Visual Instruction Tuning
  9. Chen et al. (2023). InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  10. Bai et al. (2025). Qwen2.5-VL Technical Report
  11. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  12. Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
  13. Laurençon et al. (2024). What matters when building vision-language models?
  14. Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images