High resolution and native aspect ratio
A vision encoder trained at 224 × 224 pixels cannot read a page of text, the labels of a chart, or a small sign in a 4K photo: the pixels are gone before the model sees them. This session shows the two answers modern VLMs use: tiling (cut the image into encoder-sized crops plus a thumbnail, as in LLaVA-NeXT and InternVL) and native dynamic resolution (encode the image at its own size and aspect ratio, as in NaViT and Qwen2-VL). It ends with M-RoPE, the position encoding that tells the language model where each visual token sits in time, height and width.
What you will learn
- Why a fixed 224 × 224 or 336 × 336 input loses documents, charts and small objects, computed in pixels per letter and pixels per patch
- How AnyRes tiling picks a grid of crops (effective and wasted resolution), adds a thumbnail, removes padding and marks rows, and what it costs in tokens
- How Qwen2-VL encodes an image at its native size: resize to multiples of 28 inside a pixel budget, 14-pixel patches, 2D RoPE in the ViT, and a 2 × 2 merge into one LLM token
- How NaViT-style packing puts many images of different sizes into one sequence with a block-diagonal mask, and why that saves padding and attention compute
- How M-RoPE splits the rotary frequencies into time, height and width parts (16, 24, 24 pairs in Qwen2-VL), and how text, images and video get their position IDs
- How to choose a token budget for documents, screens and video, and the common mistakes that silently hurt accuracy
Why this matters
The first VLMs looked at every image through a small square window: 224 × 224 pixels, or 336 × 336 in LLaVA-1.5. That is fine for “is there a dog in this photo?”. It fails for a page of text, a chart with small labels, a phone screenshot, or a traffic sign far away in a 4K video frame. The detail is destroyed when the image is shrunk, before the model sees a single pixel.
Two families of fixes are used in almost every VLM today:
- Tiling (often called AnyRes): keep the encoder at its training size, but cut a large image into several encoder-sized crops, encode each crop, and add one small thumbnail of the whole image for global context. LLaVA-1.5-HD, LLaVA-NeXT and InternVL 1.5 work this way.
- Native dynamic resolution: let the encoder accept images of any size and shape, so a wide image gives a wide grid of tokens and a tall image a tall grid. NaViT showed how to train such an encoder efficiently by packing images of different sizes into one sequence. Qwen2-VL put it into a VLM, together with M-RoPE, a rotary position encoding with separate parts for time, height and width.
The price of both is more visual tokens. A page that LLaVA-1.5 saw as 576 tokens can become 2,000 to 5,000 tokens. Attention cost grows with the square of the sequence length, so resolution is also a serving question: the token budget per image is one of the most important knobs in a VLM product (session 183, session 196).
In session 177 we met the three parts of a VLM: vision encoder, connector, language model. session 178 showed how visual features are compressed (for example by pixel shuffle) and session 179 showed where they enter the language model. This session is about the input side: how many pixels go in, in what shape, and how their positions are encoded. The next session, session 181, then reads real systems (LLaVA, BLIP-2, Flamingo, IDEFICS) with these ideas in hand.
Why 224 × 224 is not enough
A ViT cuts the image into square patches of \(p \times p\) pixels and turns each patch into one token (session 164). For an image of height \(H\) and width \(W\) the number of tokens and the cost of self-attention are
- Read the symbols
- \(H, W\)
- height and width of the image that the encoder actually sees, in pixels
- \(p\)
- patch size: 14 pixels for CLIP ViT-L/14 and the Qwen2-VL encoder, 16 for many other ViTs
- \(N\)
- number of visual tokens; at \(224 \times 224\) with \(p = 14\) this is \(16 \times 16 = 256\), at \(336 \times 336\) it is \(24 \times 24 = 576\)
A CLIP-style encoder was trained at one fixed size \(S \times S\), and its learned position embeddings have exactly \((S/p)^2\) entries. The simple way to feed it any image is to resize the whole image to \(S \times S\). This does two harmful things at once:
- Read the symbols
- \(s_x, s_y\)
- how much the image shrinks along the width and along the height; when \(W \ne H\) they differ, so shapes are squashed
- \(h\)
- the height of a detail in the original image, for example a line of text, in pixels
- \(h'\)
- the height of the same detail after resizing
A US letter page scanned at 200 dpi is \(1700 \times 2200\) pixels (width × height). A 10-point font is \(10/72\) inch tall, so at 200 dpi it is \(10/72 \times 200 = 27.8\) pixels (the full font size, from the top of “h” to the bottom of “g”).
Resize to \(224 \times 224\): \(s_y = 224/2200 = 0.102\), so the font becomes \(27.8 \times 0.102 = 2.83\) pixels tall. At 336 it is 4.24 pixels, at 672 it is 8.48 pixels. Letters need several pixels of height to stay readable; at 3 pixels, “e”, “a” and “o” are the same blob. [Certain for the arithmetic]
Seen the other way: one 14-pixel patch at 224 covers \(14 / 0.102 = 137.5\) original pixels of height, about five lines of text. One token has to describe five lines.
The same problem appears for other inputs. A chart of \(1200 \times 800\) with 12-pixel tick labels squashed to 224 has labels 3.4 pixels tall. In a 4K video frame (\(3840 \times 2160\)), a traffic sign 40 pixels wide becomes \(40 \times 224/3840 = 2.3\) pixels wide: one sixth of a patch. No amount of model quality brings back pixels that were thrown away.
And there is a second, quieter problem: aspect ratio. A receipt of \(400 \times 1600\) squashed into a square is stretched 4 times wider than its height scale. Letters become wide and short, shapes the encoder never saw in training. Padding to a square instead keeps the shape but wastes most of the square on empty pixels.
How to read Figure 1
Start with the letter page at 224: the text is gone. Move the slider to the right. At 672 the text starts to come back, at 1344 it is readable. The readout gives the text height after resizing, the area of the original that one patch must summarize, and the token count \((S/14)^2\), which grows with the square of \(S\). Switch to the 4K street frame: the small sign needs a very large \(S\), and the squash factors \(s_x \ne s_y\) show the shape distortion of a non-square image.
Why not simply train one encoder at \(1344 \times 1344\)? Formula (180.1): \(N = 96^2 = 9{,}216\) tokens for every image, even a small icon, and the ViT’s attention cost grows with \(N^2\), 256 times the cost at 336. Most of those tokens would be wasted on small or simple images. We want the token count to follow the content. The two families below do this in different ways.
Tiling: a thumbnail plus high-resolution crops
The idea of tiling: do not change the encoder, change the input. The encoder still sees \(336 \times 336\) images (its training size), but a large image is first resized to fit a grid of \(r \times c\) such tiles. Each tile is encoded separately, and the tile features are put back together into one large feature map. LLaVA-1.5-HD introduced this, with one more piece: the features of a downsampled copy of the whole image are added too, because a single tile only sees a part of the scene. [Certain] In the LLaVA-1.5 paper, adding this global context raised MM-Vet from 31.9 to 35.1 and MME from 1425.8 to 1497.5 for the 7B model. [Certain]
LLaVA-NeXT (January 2024) made this the default and called it AnyRes. Its tile grids are \(\{2{\times}2,\ 1{\times}\{2,3,4\},\ \{2,3,4\}{\times}1\}\) with 336-pixel tiles, so the largest inputs are \(672 \times 672\), \(336 \times 1344\) and \(1344 \times 336\): four times the pixels of LLaVA-1.5. [Certain] Which grid should an image use? The rule (from the LLaVA-NeXT code, now in Hugging Face select_best_resolution) is: fit the image inside each candidate canvas without changing its aspect ratio, and count how many original pixels survive.
- Read the symbols
- \(W_c, H_c\)
- width and height of a candidate canvas: \(W_c = 336\,c\), \(H_c = 336\,r\) for a grid of \(r\) rows and \(c\) columns
- \(s\)
- the scale that makes the image fit inside the canvas, keeping its aspect ratio
- \(e\)
- effective resolution: pixels of real image content after fitting; it is capped at \(WH\) because upscaling a small image adds no information
- \(w\)
- wasted resolution: canvas pixels that hold padding (or upscaled duplicates)
So the rule prefers the canvas that keeps the most detail, and among equal ones the smallest canvas. The image is resized by \(s\), padded to the canvas, cut into tiles, and each tile becomes \(24 \times 24 = 576\) features. Then two clean-ups happen on the joined feature map: features that cover only padding are removed, and a learned newline token is added at the end of each row so the language model knows where a row ends (the LLaVA-1.5 paper describes both for its HD variant [Certain]). With the thumbnail first, the token count is
- Read the symbols
- \(576\)
- tokens of the thumbnail: the whole image resized to one \(336 \times 336\) tile
- \(h_f, w_f\)
- height and width of the joined tile feature map after removing padding rows or columns; before that removal it is \(24r \times 24c\)
- \(+1\)
- one newline token per row
Width 1000, height 600. The seven candidates, with formula (180.3):
| Grid r × c | Canvas W × H | Scale s | Fitted size | Effective e | Wasted w |
|---|---|---|---|---|---|
| 2 × 2 | 672 × 672 | 0.672 | 672 × 403 | 270,816 | 180,768 |
| 1 × 2 | 672 × 336 | 0.560 | 560 × 336 | 188,160 | 37,632 |
| 1 × 3 | 1008 × 336 | 0.560 | 560 × 336 | 188,160 | 150,528 |
| 1 × 4 | 1344 × 336 | 0.560 | 560 × 336 | 188,160 | 263,424 |
| 2 × 1, 3 × 1, 4 × 1 | 336 × 672 … | 0.336 | 336 × 201 | 67,536 | 158,256 … |
The \(2 \times 2\) grid keeps the most pixels (270,816 of the original 600,000), so it wins even though 40% of its canvas is padding. The joined feature map is \(48 \times 48\). Padding rows are removed: \(\operatorname{int}(600 \times 48/1000) = 28\) rows remain, so \(h_f = 28\), \(w_f = 48\), and formula (180.5) gives \(576 + 28 \times 49 = 1{,}948\) tokens. Without removing padding it would be \(576 + 48 \times 49 = 2{,}928\). [Certain: computed with the same code as the PyTorch section]
- input: image of size \(W \times H\); tile size \(T = 336\); candidate grids \(G\)
- \((r, c) \leftarrow\) the grid with the largest effective resolution \(e\), then the smallest waste \(w\) // (180.3), (180.4)
- resize the image by \(s\), pad it (centered) to \(cT \times rT\), cut it into \(r \cdot c\) tiles
- thumbnail \(\leftarrow\) the whole image resized to \(T \times T\)
- encode the thumbnail and every tile with the same frozen-size ViT; each gives \(24 \times 24\) features
- place tile features in a \(24r \times 24c\) map; drop rows or columns that cover only padding
- append a newline embedding at the end of each row; flatten row by row
- return [thumbnail tokens, tile tokens] // (180.5) tokens, then the connector
Figure 2 runs Algorithm 1 on the image you choose, one line at a time. First it scores the seven candidate grids with (180.3), then it cuts the tiles, adds the thumbnail, removes the padding features and adds the newline tokens. The token count on the right grows as each piece is added.
- \(e^\ast \leftarrow 0,\ w^\ast \leftarrow \infty\)
- for each grid \((r, c)\) in \(G\): \(s \leftarrow \min(cT/W,\ rT/H)\)
- \(e \leftarrow \min(\lfloor sW \rfloor \lfloor sH \rfloor,\ WH)\), \(w \leftarrow cT \cdot rT - e\) // (180.3)
- if \(e \gt e^\ast\) or (\(e = e^\ast\) and \(w \lt w^\ast\)): best \(\leftarrow (r, c)\) // (180.4)
- resize by \(s\), pad to \(cT \times rT\), cut \(r \cdot c\) tiles; each tile \(\to 24 \times 24\) features
- thumbnail \(\leftarrow\) whole image resized to \(T \times T\) \(\to 576\) features
- drop feature rows or columns that cover only padding \(\to h_f \times w_f\)
- add one newline per row; \(N \leftarrow 576 + h_f (w_f + 1)\) // (180.5)
How to read Figure 2
Press Play, or Step through it. With the 1000 × 600 photo the first candidate, \(2 \times 2\), gives \(s = \min(672/1000, 672/600) = 0.672\), a fitted image of 672 × 403 and \(e = 270{,}816\). The next six candidates all keep fewer pixels (the \(1 \times c\) grids reach only 188,160), so \(2 \times 2\) stays the best. Then the four tiles are cut (4 × 576 = 2,304 features), the thumbnail adds 576, the unpad step keeps 28 of the 48 feature rows, and the newlines add 28: \(576 + 28 \times 49 = 1{,}948\). Now try the presets. The receipt picks \(4 \times 1\) with zero waste, so nothing is removed. The small 336 × 336 image has no \(1 \times 1\) grid in this list; it ties on \(e\) everywhere, so the smallest waste decides: \(1 \times 2\) and \(2 \times 1\) tie again, and the first one in the list, \(1 \times 2\), wins. After the padding columns are removed it still costs \(576 + 24 \times 25 = 1{,}176\) tokens, because the thumbnail and the tile show the same pixels twice. Qwen2-VL (Figure 3) needs 144 tokens for it. Moving a slider keeps the current step, so you can stop on a candidate and compare images.
InternVL 1.5 uses the same idea with a different selection rule. It uses \(448 \times 448\) tiles and 35 predefined aspect ratios (all grids with at most 12 tiles during training), picks the ratio closest to the image’s ratio, and breaks ties by not enlarging the image to more than twice its area. A \(448 \times 448\) thumbnail is always added. A pixel shuffle in the connector cuts each tile to 256 tokens, and at test time the number of tiles can go up to 40, about 4K resolution. [Certain] So 12 tiles plus a thumbnail cost \(13 \times 256 = 3{,}328\) tokens.
Tiling has clear strengths: it reuses a strong pretrained encoder (CLIP, SigLIP, InternViT) without retraining it at a new size, every tile is a standard batch item, and the cost is easy to cap (maximum number of tiles). Its weaknesses: a fixed menu of grids, padding waste, tile borders that cut objects, and the thumbnail tokens paid again for every image.
Native dynamic resolution: Qwen2-VL
The second family removes the fixed size from the encoder itself. Qwen2-VL (September 2024) calls it Naive Dynamic Resolution: the ViT takes an image of (almost) any size, and the number of tokens follows the image. [Certain] Three changes make this possible.
- No absolute position table. A learned table of \((S/p)^2\) position embeddings only works for one grid size. Qwen2-VL removes it and uses 2D RoPE inside the ViT: part of each attention head is rotated by the patch’s row, the other part by its column (session 14). Rotations work for any row and column number. [Certain]
- Resize only a little. The image is resized to the nearest multiples of 28 pixels (14-pixel patches, then a 2 × 2 merge), and scaled down only if it exceeds a pixel budget.
- Merge 2 × 2 patches. After the ViT, a small MLP merges each 2 × 2 group of neighbouring patch features into one token for the language model, so one LLM token covers \(28 \times 28\) pixels. [Certain]
The resize rule, from smart_resize in the Hugging Face Qwen2-VL image processor:
- Read the symbols
- \(\tilde H, \tilde W\)
- the size actually fed to the ViT; both are multiples of 28 and the aspect ratio stays close to the original
- \(P_{\max}\)
- the pixel budget; the released processor config uses \(16{,}384 \times 28^2 = 12{,}845{,}056\) pixels (at most 16,384 tokens), and the model card suggests lower values such as \(1280 \times 28^2\) to save memory [Certain]
- \(\beta\)
- the shrink factor when the image is too big; a symmetric rule with \(P_{\min} = 4 \times 28^2 = 3{,}136\) pixels enlarges tiny images
- \(N_{\text{Qwen}}\)
- tokens the language model sees: one per \(28 \times 28\) pixel block
The merge step, for the \(2 \times 2\) block at merged position \((i, j)\):
- Read the symbols
- \(\mathbf{x}_{a,b}\)
- output feature of the ViT patch in row \(a\), column \(b\); width 1,280 in the Qwen2-VL encoder (32 layers, about 675M parameters) [Certain]
- \([\,;\,]\)
- concatenation: four patch features give one vector of \(4 \times 1280 = 5120\) numbers
- \(\mathbf{z}_{i,j}\)
- the merged token, projected to the LLM width (3,584 for Qwen2-VL-7B)
This merge is the same trick as pixel shuffle in session 178: trade 4 tokens for 1 wider token, so the LLM sequence is 4 times shorter while the ViT still worked at 14-pixel detail.
\(\tilde H = 28 \cdot \operatorname{round}(600/28) = 28 \times 21 = 588\), \(\tilde W = 28 \cdot \operatorname{round}(1000/28) = 28 \times 36 = 1008\). The product \(588 \times 1008 = 592{,}704\) is under every budget, so no further scaling. The ViT sees \(42 \times 72 = 3{,}024\) patches; after the merge the LLM gets \(21 \times 36 = 756\) tokens, plus 2 special tokens: 758. AnyRes needed 1,948 tokens for this image and still shrank it to 672 × 403. Here the image keeps 99% of its pixels and its shape.
A \(224 \times 224\) image gives \(8 \times 8 = 64\) tokens, 66 with the start and end tokens: exactly the example in the Qwen2-VL paper. [Certain]
How to read Figure 3
Move the sliders. The token grid keeps the shape of the image: a wide image gives a wide grid. When the image is larger than the budget, the readout shows the shrink factor \(\beta\) and the grid stops growing. Switch the budget to 16,384 and a 1700 × 2200 page keeps almost its full size (4,819 tokens), so the 10-point text stays about 28 pixels tall. With the 1,280 budget the page gets 1,240 tokens and the text is about 14 pixels tall. In the bar chart, the fixed 336 input always costs 576 tokens whatever the image.
What does the dynamic approach buy? The Qwen2-VL paper has a clean ablation on the 7B model: fixed token counts versus dynamic resolution. [Certain]
| Image tokens | Average count | InfoVQA | RealWorldQA | OCRBench | MMMU |
|---|---|---|---|---|---|
| Fixed | 64 | 28.85 | 56.47 | 572 | 53.33 |
| Fixed | 576 | 65.72 | 65.88 | 828 | 52.78 |
| Fixed | 1600 | 74.99 | 69.54 | 824 | 52.89 |
| Fixed | 3136 | 77.27 | 70.59 | 786 | 53.44 |
| Dynamic | 1924 | 75.89 | 70.07 | 866 | 53.44 |
Two lessons. First, text-heavy tasks (InfoVQA, OCRBench) gain hugely from more tokens, while a reasoning benchmark like MMMU barely moves: resolution matters most for reading. Second, dynamic resolution gets close to the largest fixed setting with 39% fewer tokens on average (1,924 versus 3,136), and beats every fixed setting on OCRBench.
The cost moves into the ViT. With full attention over all patches, a 16,384-token image means 65,536 patches in one attention. Qwen2.5-VL therefore uses window attention in most ViT layers: windows of \(112 \times 112\) pixels (\(8 \times 8\) patches), with full attention only in layers 7, 15, 23 and 31. [Certain: from its config.json]
Inside the ViT: patch order and 2D RoPE
Two details of the ViT input make native resolution work. First, the patch order. The 14-pixel patches are not listed row by row. They are listed in \(2 \times 2\) blocks: the four patches that the merger (180.8) will join are always four neighbours in the sequence. Then the merge is only a reshape of four consecutive tokens. [Certain: from rot_pos_emb in the Hugging Face code]
Second, 2D RoPE. The Qwen2-VL ViT has width 1,280 and 16 heads, so each head has \(d = 80\) dimensions, which form 40 rotation pairs. The first 20 pairs are rotated by the patch’s row \(a\), the last 20 by its column \(b\), with the usual RoPE frequencies. [Certain: from the Hugging Face code, base 10,000]
- Read the symbols
- \(a, b\)
- row and column of the 14-pixel patch in the resized image, counted from 0
- \(\phi_j\)
- rotation angle of pair \(j\); pairs 0 to 19 see only the row, pairs 20 to 39 only the column
- \(\theta_j\)
- frequency of pair \(j\): \(\theta_0 = 1\), \(\theta_1 = 0.631\), down to \(\theta_{19} = 0.000158\) radians per patch
- \(R_{\phi}\)
- the block-diagonal rotation that turns every pair \(j\) by its angle \(\phi_j\)
- \(\mathbf{q}, \mathbf{k}\)
- a query and a key before rotation; \(\mathbf{q}_{a,b}\) is the query of the patch at \((a, b)\) after rotation
The last line is the reason this works at any size. The angles are linear in \((a, b)\), and two rotations by \(\phi\) and \(\phi'\) combine into one rotation by \(\phi' - \phi\). So the attention score sees only the 2D offset \((a' - a,\ b' - b)\), never the absolute grid size. A table of learned positions would have no entry for row 40 of a tall receipt; a rotation by \(40\,\theta_j\) is always defined.
Figure 4 runs this part on a small image: the resize of (180.6), the patches in block order with their \((a, b)\) labels, the rotation angles of the selected patch, a numerical check of (180.9), and the \(2 \times 2\) merge.
- \(\tilde H \leftarrow 28\,\operatorname{round}(H/28)\), \(\tilde W \leftarrow 28\,\operatorname{round}(W/28)\)
- if \(\tilde H \tilde W \gt P_{\max}\): \(\beta \leftarrow \sqrt{HW/P_{\max}}\), shrink both sides with \(\lfloor \cdot \rfloor\) // (180.6)
- for each \(2 \times 2\) block, row by row: append its 4 patches
- patch \((a, b)\): turn pairs 0–19 by \(a\theta_j\), pairs 20–39 by \(b\theta_j\) // (180.9)
- check: the score depends only on \((\Delta a, \Delta b)\)
- merge each 4 consecutive patches into 1 token; \(N = \tfrac{\tilde H}{28} \cdot \tfrac{\tilde W}{28}\) // (180.7), (180.8)
How to read Figure 4
Press Play. The default image is 330 × 200 with a budget of 64 tokens (\(P_{\max} = 64 \times 784 = 50{,}176\) pixels). Rounding gives 336 × 196 = 65,856 pixels, over the budget, so \(\beta = \sqrt{66{,}000 / 50{,}176} = 1.147\) and the image becomes 280 × 168: 12 × 20 = 240 patches. The scan then adds one row of merge blocks per step; look at the sequence strip: four patches of the same color always sit next to each other. The dials turn by \(a \cdot 1\) and \(b \cdot 1\) radians for pair 0 and pair 20. At the check step the score of the selected patch with the patch one row down and two columns right is \(0.6552\), the same as for the patches at (0, 0) and (1, 2), wherever you click. An offset of (2, 1) gives a different score, \(0.5518\). The last step merges the blocks: 6 × 10 = 60 tokens. Switch the budget to 256: the image is only rounded (336 × 196), and you get 84 tokens.
Packing many images: NaViT
If every image has its own number of tokens, how do we put them in a batch? The naive way pads every image to the longest one. NaViT (“Patch n’ Pack”, 2023) showed a better way, borrowed from language-model pretraining (session 18): pack the patches of several images into one long sequence, and use an attention mask so that each image only attends to itself. [Certain]
- Read the symbols
- \(M\)
- the packing mask, added to the attention scores; it is block-diagonal: one square block per image
- \(-\infty\)
- after softmax this gives weight 0, so no information flows between images
- \(Q, K, V, d\)
- queries, keys, values and head size of normal attention (session 17)
The result is exactly the same as running each image alone (the PyTorch section checks this), but the hardware sees one dense sequence. In practice the mask is not stored: FlashAttention’s variable-length kernel takes the start offsets of each image (cu_seqlens) and skips the off-diagonal blocks. Qwen2-VL’s ViT does exactly this. [Certain: from the Hugging Face code] Pooling at the end is also per image (NaViT’s masked pooling).
- input: images with token counts \(n_1, \dots, n_B\); sequence length \(L\) (each \(n_k \le L\))
- sort the images by \(n_k\), largest first
- for each image: put it in the first sequence with room left; if none has room, open a new sequence
- for each sequence, record the start offsets of its images (
cu_seqlens) and the image id of every token - run the ViT with the block-diagonal mask (180.10), or a variable-length attention kernel with the offsets
- split the output by the offsets: each image gets back its own tokens // then the 2×2 merge (180.8)
NaViT added three more ideas that matter for training [Certain]:
- Factorized position embeddings: separate learned tables for the row and for the column, summed: \(\mathbf{e}_{\text{pos}}(r, c) = \mathbf{e}^{y}_r + \mathbf{e}^{x}_c\). Two short tables instead of one table per grid size. Qwen2-VL goes one step further with 2D RoPE, which needs no table at all.
- Resolution sampling: train on a mix of image sizes, so the model learns from both small and large inputs.
- Token dropping per image: randomly drop a different fraction of patches from each image; packing makes the varying lengths free.
With packing, NaViT reports that typically less than 2% of tokens are padding, and that a NaViT matches the best plain ViT with about four times less training compute. [Certain]
- sort the images by \(n_k\), largest first
- for each image \(k\) in that order:
- for each open row \(j\): if \(u_j + n_k \le L\): put \(k\) in row \(j\), break
- if no row had room: open a new row with \(k\)
cu_seqlensof each row \(\leftarrow\) prefix sums of its image lengths; mask (180.10)
How to read Figure 5
Press Play with \(L = 2048\). Step 1 sorts the twelve images: chart 1,247, page 1,240, screen 1,222, scan 999, and so on. The chart opens row 1. The page does not fit there (\(1{,}247 + 1{,}240 = 2{,}487 \gt 2{,}048\)), so it opens row 2; the screen and the scan also open new rows. The receipt (798) is the first small image that fits next to the chart: \(1{,}247 + 798 = 2{,}045 \le 2{,}048\). At the end the twelve images fill 5 rows, and only 14.7% of the 10,240 slots are padding; row 1 has cu_seqlens \([0, 1247, 2045]\), and its two blocks need 52.3% of the work of a full \(2048 \times 2048\) attention. Switch to “Pad to the longest”: every image opens its own row padded to 1,247 tokens (the chart), so 41.6% of all slots are padding. Change \(L\) to see more or fewer rows.
M-RoPE: positions for time, height and width
The ViT knows where each patch is (2D RoPE). But after the merge, the visual tokens are placed into the language model’s sequence next to text tokens, and the language model has its own position encoding. Standard 1D RoPE gives each token a single number: its index in the sequence (session 14). For an image of \(h \times w\) tokens, flattened row by row, a token and the one directly below it are \(w\) positions apart, while it and the one to its right are 1 apart. The LLM must learn the grid width from data, and it changes with every image size. Video adds a third axis.
Qwen2-VL’s Multimodal RoPE (M-RoPE) gives every token three position numbers: temporal, height and width, \(\mathbf{p} = (p^t, p^h, p^w)\). [Certain]
- Read the symbols
- \(n\)
- for text, all three parts are the same number, so M-RoPE on text is exactly 1D RoPE
- \(s\)
- start ID of an image or video: one more than the largest ID before it
- \(r, c\)
- row and column of the merged token in its grid (\(0 \le r \lt h\), \(0 \le c \lt w\))
- \(\tau\)
- index of the frame group; Qwen2-VL samples video at 2 frames per second and merges 2 frames per token (3D patches of depth 2), so one group is about 1 second [Certain]
- \(s_{\text{next}}\)
- the next modality starts after the largest ID used so far; for an image this is \(s + \max(h, w)\), not \(s + hw\)
How do three numbers enter one rotation? RoPE rotates each pair of dimensions \(i\) of a query or key by an angle proportional to the position, with a frequency \(\theta_i\) that falls with \(i\). M-RoPE splits the frequency pairs into three groups and lets each group use one of the three position parts:
- Read the symbols
- \(\phi_i\)
- rotation angle of dimension pair \(i\) for this token
- \(d\)
- head size, 128 in Qwen2-VL-7B (3,584 / 28 heads), so there are 64 pairs
- \(b\)
- RoPE base, \(10^6\) in Qwen2-VL
- \(16, 24, 24\)
mrope_sectionin the config: 16 pairs for time, 24 for height, 24 for width [Certain]
Because each group is an ordinary RoPE, the key property of RoPE carries over: the attention score between a query at \(\mathbf{p}_q\) and a key at \(\mathbf{p}_k\) depends only on the difference \(\mathbf{p}_q - \mathbf{p}_k\), now a 3D offset \((\Delta t, \Delta h, \Delta w)\). Moving an image within the prompt does not change how its tokens attend to each other. (The PyTorch code checks this.)
Take 4 tokens before the image (the last one is the vision-start token), an image of \(2 \times 3\) merged tokens, then 3 text tokens. Text gets \((0,0,0), (1,1,1), (2,2,2), (3,3,3)\). The image starts at \(s = 4\): its first row is \((4,4,4), (4,4,5), (4,4,6)\), its second row \((4,5,4), (4,5,5), (4,5,6)\). The largest ID used is 6, so text continues at 7: \((7,7,7), (8,8,8), (9,9,9)\).
With 1D RoPE the image would take positions 4 to 9 and the text would start at 10. For a \(36 \times 21\) grid (the 1000 × 600 photo) the difference is large: 1D RoPE uses 756 positions for the image, M-RoPE only \(\max(21, 36) = 36\). Long documents and videos use far fewer position IDs, which helps the model stay inside the position range it saw in training.
How to read Figure 6
In M-RoPE mode, the image chips show \((t, h, w)\): the first number stays fixed for one frame group, the second follows the row, the third the column. The text after the image starts at the largest ID plus one. Switch to 1D RoPE: the same tokens get one number each, the lower neighbour is now \(w\) positions away, and the text after the image starts later. Add frame groups to see video: only the first number changes from group to group.
Choosing a strategy and a token budget
Here are the token counts for four real-shaped inputs, all computed with the code below (Qwen2-VL count without the 2 special tokens):
| Input (W × H) | Fixed 336 | AnyRes (LLaVA-NeXT) | Qwen2-VL, budget 1,280 | Qwen2-VL, budget 16,384 |
|---|---|---|---|---|
| Photo 1000 × 600 | 576 | 1,948 (2 × 2) | 756 | 756 |
| Letter page 1700 × 2200 | 576 | 2,448 (2 × 2) | 1,240 | 4,819 |
| Receipt 400 × 1600 | 576 | 2,976 (4 × 1) | 798 | 798 |
| 4K frame 3840 × 2160 | 576 | 1,948 (2 × 2) | 1,222 | 10,549 |
Text height on the letter page (27.8 pixels at full size): 4.2 pixels at fixed 336, 8.5 with AnyRes (the page is shrunk to 519 × 672), 14.1 with the 1,280 budget, and 27.9 with the 16,384 budget. These numbers explain most of the difference between VLMs on document benchmarks such as DocVQA. Practical rules:
- Documents, charts, screenshots: raise the budget until the smallest text you care about is at least about 10 to 15 pixels tall after resizing. [Likely: a rule of thumb, not a paper result] Measure accuracy against tokens on your own data.
- Natural photos and classification: low budgets are enough; MMMU in the table above did not change between 64 and 3,136 tokens.
- Video: tokens multiply by the number of frame groups. Qwen2-VL caps a video at 16,384 tokens [Certain], so more frames means fewer pixels per frame. Choose between temporal and spatial detail per task (session 205, session 206).
- Latency: prefill cost grows with tokens, attention with their square. The 16,384-budget page has \(4819/576 = 8.4\) times the tokens of a fixed 336 input and about 70 times the attention-score work. Cap the budget per request.
The PyTorch version
The code is small, deterministic and runs on a CPU in about a second. It implements AnyRes grid selection and token counting (180.3 to 180.5), Qwen2-VL’s smart_resize (180.6, 180.7), patchify and the 2 × 2 merger (180.8), packed attention with a block-diagonal mask (180.10) checked against separate attention, and M-RoPE position IDs and rotation (180.11, 180.12) with two checks: scores depend only on 3D offsets, and on text M-RoPE equals 1D RoPE. The printed values are the ones used in the text.
import math, torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # 1) AnyRes: pick the best grid of 336-px tiles (formulas 180.3 to 180.5) TILE = 336 GRIDS = [(2, 2), (1, 2), (1, 3), (1, 4), (2, 1), (3, 1), (4, 1)] # (rows, cols) def select_best_resolution(H, W, grids=GRIDS, tile=TILE): best, best_eff, best_waste = None, 0, float("inf") for r, c in grids: Hc, Wc = r * tile, c * tile s = min(Wc / W, Hc / H) # fit inside, keep aspect eff = min(int(W * s) * int(H * s), W * H) # no credit for upscaling waste = Hc * Wc - eff if eff > best_eff or (eff == best_eff and waste < best_waste): best, best_eff, best_waste = (r, c), eff, waste return best, best_eff, best_waste def anyres_tokens(H, W, per_side=24): (r, c), _, _ = select_best_resolution(H, W) fh, fw = r * per_side, c * per_side # feature map of all tiles if W / H > fw / fh: # remove rows of padding fh = fh - 2 * ((fh - int(round(H * fw / W, 7))) // 2) else: # remove columns of padding fw = fw - 2 * ((fw - int(round(W * fh / H, 7))) // 2) return (r, c), per_side ** 2 + fh * (fw + 1) # thumbnail + rows with newline print(select_best_resolution(600, 1000), anyres_tokens(600, 1000)) # ((2, 2), 270816, 180768) ((2, 2), 1948) # 2) Qwen2-VL: resize to multiples of 28 inside a pixel budget (formula 180.6) def smart_resize(H, W, factor=28, min_pixels=4 * 28 * 28, max_pixels=1280 * 28 * 28): h, w = round(H / factor) * factor, round(W / factor) * factor if h * w > max_pixels: beta = math.sqrt(H * W / max_pixels) h = max(factor, math.floor(H / beta / factor) * factor) w = max(factor, math.floor(W / beta / factor) * factor) elif h * w < min_pixels: beta = math.sqrt(min_pixels / (H * W)) h, w = math.ceil(H * beta / factor) * factor, math.ceil(W * beta / factor) * factor return h, w for H, W in [(600, 1000), (2200, 1700), (224, 224)]: h, w = smart_resize(H, W) print((H, W), "->", (h, w), "tokens", (h // 28) * (w // 28)) # (600, 1000) -> (588, 1008) tokens 756 # (2200, 1700) -> (1120, 868) tokens 1240 # (224, 224) -> (224, 224) tokens 64 # 3) Patchify at 14 px, then merge 2x2 neighbours into one LLM token (formula 180.8) def patchify(img, p=14): # img (3, H, W), H and W multiples of 2p C, H, W = img.shape x = img.reshape(C, H // p, p, W // p, p).permute(1, 3, 0, 2, 4) # (gh, gw, C, p, p) return x.reshape(H // p, W // p, C * p * p) class Merger(nn.Module): # Qwen2-VL "PatchMerger", tiny sizes def __init__(self, d_vit=32, d_llm=48, m=2): super().__init__(); self.m = m self.ln = nn.LayerNorm(d_vit) self.mlp = nn.Sequential(nn.Linear(m * m * d_vit, m * m * d_vit), nn.GELU(), nn.Linear(m * m * d_vit, d_llm)) def forward(self, x): # x (gh, gw, d_vit) gh, gw, d = x.shape; m = self.m x = self.ln(x).reshape(gh // m, m, gw // m, m, d).permute(0, 2, 1, 3, 4) return self.mlp(x.reshape(gh // m, gw // m, m * m * d)) # (gh/2, gw/2, d_llm) img = torch.rand(3, 588, 1008) patches = patchify(img) # (42, 72, 588) embed = nn.Linear(3 * 14 * 14, 32) tokens = Merger()(embed(patches)) print(tuple(patches.shape), tuple(tokens.shape), tokens.shape[0] * tokens.shape[1]) # (42, 72, 588) (21, 36, 48) 756 # 4) Patch n' Pack: several images in one sequence, block-diagonal mask (formula 180.10) lens = [6, 4, 9] # tokens of three images x = torch.randn(1, 2, sum(lens), 8) # (batch, heads, packed length, head dim) img_id = torch.repeat_interleave(torch.arange(len(lens)), torch.tensor(lens)) mask = img_id[:, None] == img_id[None, :] # True = may attend packed = F.scaled_dot_product_attention(x, x, x, attn_mask=mask) alone = torch.cat([F.scaled_dot_product_attention(s, s, s) for s in x.split(lens, dim=2)], dim=2) print("packed == separate:", torch.allclose(packed, alone, atol=1e-6), "| mask density", round(mask.float().mean().item(), 3)) # packed == separate: True | mask density 0.369 # 5) M-RoPE position ids for text, image, text (formula 180.11) def mrope_ids(n_text_before, grid_h, grid_w, n_text_after): ids = [(p, p, p) for p in range(n_text_before)] s = n_text_before ids += [(s, s + r, s + c) for r in range(grid_h) for c in range(grid_w)] nxt = s + max(grid_h, grid_w) ids += [(p, p, p) for p in range(nxt, nxt + n_text_after)] return torch.tensor(ids).T # (3, L) pos = mrope_ids(4, 2, 3, 3) print(pos.tolist()) # [[0, 1, 2, 3, 4, 4, 4, 4, 4, 4, 7, 8, 9], [0, 1, 2, 3, 4, 4, 4, 5, 5, 5, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 4, 5, 6, 7, 8, 9]] # 6) Apply M-RoPE: frequency pairs split 16 / 24 / 24 between t, h, w (formula 180.12) def mrope_angles(pos, head_dim=128, base=1e6, sections=(16, 24, 24)): inv = 1.0 / base ** (torch.arange(0, head_dim, 2).float() / head_dim) # (64,) which = torch.repeat_interleave(torch.arange(3), torch.tensor(sections)) # 0=t,1=h,2=w per pair return pos[which].T.float() * inv # (L, 64) def rotate(x, ang): # x (L, head_dim), half-split layout as in Qwen2 c, s = ang.cos().repeat(1, 2), ang.sin().repeat(1, 2) x1, x2 = x.chunk(2, dim=-1) return x * c + torch.cat([-x2, x1], dim=-1) * s q, k = torch.randn(1, 128), torch.randn(1, 128) def score(pq, pk): aq = mrope_angles(torch.tensor(pq).view(3, 1)); ak = mrope_angles(torch.tensor(pk).view(3, 1)) return (rotate(q, aq) * rotate(k, ak)).sum().item() print(round(score((4, 5, 6), (4, 4, 4)), 4), round(score((9, 10, 11), (9, 9, 9)), 4)) # same 3D offset # -10.1389 -10.1389 text_1d = torch.arange(5).float()[:, None] * (1.0 / 1e6 ** (torch.arange(0, 128, 2).float() / 128)) print("text M-RoPE == 1D RoPE:", torch.allclose(mrope_angles(torch.arange(5).repeat(3, 1)), text_1d)) # text M-RoPE == 1D RoPE: True
Things to try: call anyres_tokens(224, 224) and smart_resize(224, 224) and compare 1,176 tokens with 64; pass max_pixels=16384 * 28 * 28 and print the page again (4,819 tokens); change sections to (64, 0, 0) and check that score((4, 5, 6), (4, 4, 4)) now equals score((4, 4, 4), (4, 4, 4)): with only the temporal part, the model can no longer tell rows and columns apart.
Where this is used
| Application | What resolution handling does | Typical setup |
|---|---|---|
| Document QA and OCR-free parsing (invoices, forms, scientific papers) | Keeps small text readable and the page shape intact | Native resolution with a high budget, or many tiles; see session 218 |
| Chart and infographic QA | Small labels and legends survive resizing | Dynamic resolution; InfoVQA gains most from tokens |
| GUI and computer-use agents (screenshots) | Wide screens keep their aspect; click coordinates map back to pixels | Native resolution, coordinates in the resized frame (session 220) |
| Small-object questions in photos, drones, retail shelves | Objects of a few dozen pixels still cover a full token | Tiling or native resolution; or crop-then-ask |
| Video QA, key-moment detection and Video-RAG | Trade frames against pixels under one token cap; M-RoPE time IDs order frames | Low per-frame budget, more frames (session 210, session 207) |
| Video retrieval and fingerprinting | Usually low resolution is enough; native aspect avoids letterbox artifacts | Small fixed or dynamic budget per frame (session 208, session 209) |
| Multi-image prompts (compare two receipts, before/after photos) | Each image gets its own grid and its own M-RoPE block | Packing in the ViT; per-image token cap |
| Training vision encoders at mixed sizes | Packing removes padding; resolution sampling and token dropping | NaViT-style training; SigLIP 2 NaFlex variants follow this line [Likely] |
Common mistakes
- Squashing every image to a square. It distorts text and shapes and destroys small detail (Figure 1). Keep the aspect ratio: tile, pad, or use native resolution.
- Using a different budget at inference than in training. A model fine-tuned at 1,280 tokens per image but served at 256 sees much smaller text than it learned on; one served far above its training sizes meets position ranges it never saw. Log
min_pixelsandmax_pixelswith every checkpoint. - Forgetting that tokens explode. A batch of document pages at a high budget can use more KV-cache memory than the text. Set a per-image and a per-request cap, and test the worst case (a 4K screenshot, a long receipt).
- Mapping coordinates in the wrong frame. Qwen2-VL resizes to multiples of 28; boxes and click points predicted by the model are in the resized frame. Scale them back by \(W/\tilde W\) and \(H/\tilde H\) before drawing or clicking.
- Computing M-RoPE positions by hand and getting them wrong. If your own batching code (packing, left padding, multi-image) gives visual tokens plain 1D positions or forgets the \(\max + 1\) rule, the model still runs but answers get worse. Use the model’s own
get_rope_index, and pass the per-image grid sizes (image_grid_thw). - Packing without the mask. Concatenating images in the ViT without a block-diagonal mask (or
cu_seqlens) lets images attend to each other. Nothing crashes; features quietly mix. - Ignoring tile borders. With tiling, a key word cut by a border may be misread. For critical fields, crop around the region and ask again, or use overlapping crops.
- Paying for resolution the task does not need. MMMU did not improve from 64 to 3,136 tokens in Qwen2-VL’s ablation. For photo tagging or retrieval, a low budget is faster at the same quality.
Interview questions
Why do early VLMs fail on documents and small text?basic
Explain AnyRes in LLaVA-NeXT.core
Why add a thumbnail when you already have high-resolution tiles?basic
What is Naive Dynamic Resolution in Qwen2-VL, and what has to change in the ViT to allow it?core
How do you batch images of different sizes efficiently?core
What is M-RoPE? Why not just use 1D RoPE on the flattened image?core
After an image of 21 × 36 merged tokens that starts at position \(s\), where does the next text token start under M-RoPE?core
Compare tiling and native dynamic resolution. When would you choose each?advanced
Your document VLM misses small numbers in invoices. How do you debug it?advanced
Why does the Qwen2-VL ablation show big gains on InfoVQA but none on MMMU when tokens increase?advanced
How does a single image fit Qwen2-VL’s 3D patch embedding, and what does that mean for video tokens?advanced
Exercises
- An A4 page scanned at 150 dpi is 1240 × 1754 pixels. A 10-point font at 150 dpi is how many pixels tall? How tall after squashing the page to 336 × 336? What are \(s_x\) and \(s_y\)?
- Use formulas (180.3) to (180.5) for a receipt of 400 × 1600 (width × height) with the LLaVA-NeXT grids. Which grid wins, how much is wasted, and how many tokens does it cost?
- Apply
smart_resize(180.6) to a 1920 × 1080 screenshot with the 1,280-token budget and with the 16,384-token budget. How many LLM tokens in each case? - A prompt has 2 text tokens, then a video of 3 frame groups with a 2 × 2 merged grid, then text. Give the M-RoPE IDs of the last video token and of the first text token after the video.
- Pack four images of 756, 1,240, 1,247 and 798 tokens into sequences of 2,048 with Algorithm 2. How many sequences, and what fraction is padding? Compare with padding each image to the longest.
- In Qwen2-VL-7B, what fraction of the rotary frequency pairs encode height? Which of the three parts gets the highest frequencies?
- How many times more attention-score work does one 4,819-token page need than a 576-token image, inside a single attention layer?
Answerscheck
Key takeaways
- A fixed square input throws away detail before the model runs: a letter page at 224 has 2.8-pixel text, and squashing distorts aspect ratio (180.2). Tokens grow as \((S/p)^2\) and attention as their square (180.1), so “just use a bigger square” is wasteful.
- Tiling (AnyRes) keeps the encoder and changes the input: pick the grid with the most effective pixels (180.3, 180.4), encode tiles plus a thumbnail, remove padding, add row newlines (180.5). LLaVA-NeXT: 336-pixel tiles up to 672 × 672 or 336 × 1344. InternVL 1.5: 448-pixel tiles, up to 12 in training and 40 at test, 256 tokens each.
- Native dynamic resolution (Qwen2-VL): resize to multiples of 28 inside a pixel budget (180.6), 2D RoPE in the ViT, 2 × 2 merge so one LLM token is 28 × 28 pixels (180.7, 180.8). A 224 × 224 image is 64 tokens; dynamic resolution matched large fixed budgets with 1,924 tokens on average.
- NaViT packing puts many images in one sequence with a block-diagonal mask (180.10), equal to separate processing but with almost no padding; varlen attention with
cu_seqlensskips the masked blocks. - M-RoPE gives each token \((t, h, w)\) IDs (180.11) and splits the 64 frequency pairs 16 / 24 / 24 (180.12). Text reduces to 1D RoPE; an image uses \(\max(h, w)\) IDs, not \(hw\).
- Resolution is a budget: reading tasks need it, reasoning and photo tasks much less. Set and log the budget, keep training and serving consistent, and map coordinates back to the original frame.
Key formulas
References for this session
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Dehghani et al. (2023). Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
- Liu et al. (2024). LLaVA-NeXT: Improved reasoning, OCR, and world knowledge (blog)
- Chen et al. (2024). How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites (InternVL 1.5)
- Hugging Face (2024). Qwen2-VL in transformers: image processor (smart_resize) and get_rope_index
- Liu et al. (2023). Visual Instruction Tuning
- Chen et al. (2023). InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding
- Laurençon et al. (2024). What matters when building vision-language models?
- Mathew et al. (2020). DocVQA: A Dataset for VQA on Document Images