LLM & VLM courseBlog
182Track 7: Vision and VLMs · session 20 of 38

Case studies 2: Qwen2.5-VL, InternVL, PaliGemma and Gemma, SmolVLM

Four model families that most teams download and fine-tune today, taken apart with real numbers. Qwen2.5-VL keeps images at native resolution, uses window attention in its vision encoder, and gives video tokens time positions in real seconds. InternVL uses a very large vision encoder and cuts images into tiles; PaliGemma and Gemma 3 join SigLIP to Gemma with a prefix-LM or image-bidirectional mask; SmolVLM shows what changes when the whole model must run on a phone.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on the Vision Transformer (164), SigLIP (167), anatomy of a VLM (177), connectors (178), high resolution (180), RoPE (14) and masks (18)

What you will learn

  • The four design questions every VLM answers: image to grid, token reduction, positions, attention mask
  • Qwen2.5-VL: native-resolution resizing to multiples of 28, the token count, and why window attention with 4 full layers keeps the ViT linear in patches
  • M-RoPE with absolute time: why a time id of 2 × seconds lets a model localize events in video
  • InternVL: the 6B (and 300M) InternViT, aspect-ratio tiling with a thumbnail, and 2×2 pixel shuffle to 256 tokens per tile
  • PaliGemma and Gemma 3: SigLIP + Gemma, fixed squares, the prefix-LM mask, 4×4 pooling to 256 tokens, and Pan & Scan
  • SmolVLM: the design findings for small models (balanced encoder, r = 4 pixel shuffle, longer context, learned tile tokens)
  • How to compute and compare the visual token cost of one image or video across all five models

Why this matters

In session 181 we met the first generation of open vision-language models: LLaVA, BLIP-2, Flamingo and Idefics 2. This session covers the models that most teams actually download and fine-tune today: Qwen2.5-VL, InternVL, PaliGemma and Gemma 3, and SmolVLM. All of them use the same basic plan from session 177: a vision encoder, a connector, and a language model that reads image tokens in one shared sequence. But each family made a different bet on the hard questions.

Those questions are always the same four. (1) How does an image of any size become a fixed grid for the encoder? (2) How many visual tokens reach the language model, and how are they reduced? (3) How does the model know where (and, for video, when) each token is? (4) Which tokens may attend to which (the attention mask)? Qwen2.5-VL answers with native resolution, window attention and time-aligned positions. InternVL answers with a very large encoder and tiles. PaliGemma answers with a fixed square and a prefix-LM mask. SmolVLM answers with aggressive token compression so the model fits on a phone.

For video work this matters directly. A 60-second clip at 2 frames per second with 640×360 frames costs 17,940 visual tokens in Qwen2.5-VL (we compute this below). The choice of model family decides whether a video retrieval or key-moment system fits on one GPU, and whether it can say at what second something happened.

Four families on one page

Figure 1 puts the five models side by side. Read each row from left to right: the image goes in, the encoder turns it into patch vectors, the connector reduces and projects them, and the language model reads them as tokens. All numbers are from the papers and the released config.json files [Certain].

input handlingvision encoderconnectorlanguage model
Figure 1. Five open VLM designs. The visual token count on the right is for one image (before special tokens). Qwen2.5-VL’s count depends on the image size; the others use fixed squares or fixed tiles.

Two patterns are visible already. First, everyone reduces tokens before the language model, but in different ways: Qwen and InternVL merge 2×2 neighbors (4× fewer), Gemma 3 averages 4×4 blocks (16× fewer), SmolVLM-256M merges 4×4 neighbors into one wide vector (16× fewer). Second, only Qwen2.5-VL avoids cutting the image into fixed squares; everyone else resizes or tiles.

Qwen2.5-VL, part 1: native resolution

Most early VLMs resize every image to one square, such as 336×336. A receipt, a 16:9 video frame and a tall phone screenshot all become the same square, so small text is blurred and shapes are stretched. Qwen2-VL and Qwen2.5-VL do the opposite: they keep the image close to its native resolution and aspect ratio, and let the number of tokens follow the image.

The ViT uses patches of 14×14 pixels, and the connector later merges each 2×2 block of patches into one token. So each side only has to be a multiple of \(14 \times 2 = 28\) pixels. The preprocessor rounds each side to the nearest multiple of 28, and rescales the whole image only if the pixel count leaves a range \([p_{\min}, p_{\max}]\):

\[ \begin{aligned} \hat H &= 28 \cdot \operatorname{round}\!\left(\frac{H}{28}\right), \qquad \hat W = 28 \cdot \operatorname{round}\!\left(\frac{W}{28}\right) \\ \text{if } \hat H \hat W \gt p_{\max}: \ \hat H &= 28 \left\lfloor \frac{H}{28\beta} \right\rfloor, \qquad \beta = \sqrt{\frac{HW}{p_{\max}}} \end{aligned} \tag{182.1} \]

(The width is shrunk the same way, and a similar rule with \(\lceil \cdot \rceil\) grows images below \(p_{\min}\).) The number of patches and of tokens for the language model is then:

\[ N_{\text{patch}} = \frac{\hat H}{14} \cdot \frac{\hat W}{14}, \qquad V = \frac{N_{\text{patch}}}{2 \times 2} = \frac{\hat H}{28} \cdot \frac{\hat W}{28} \tag{182.2} \]
Read the symbols
\(H, W\)
height and width of the input image in pixels
\(\hat H, \hat W\)
the size after resizing; both are multiples of 28
\(p_{\min}, p_{\max}\)
the allowed pixel range. The released preprocessor uses \(p_{\min} = 3{,}136\) (that is 56 × 56) and \(p_{\max} = 12{,}845{,}056\) (that is \(16{,}384 \times 28^2\)), so at most 16,384 tokens per image [Certain]. The model card suggests a range of 256 to 1,280 tokens “to balance performance and cost” [Certain]
\(N_{\text{patch}}\)
number of 14×14 patches the ViT sees
\(V\)
number of visual tokens the language model sees (plus one vision-start and one vision-end token)
Worked example: one 1080p frame in Qwen2.5-VL

A 1920×1080 frame: \(1080/28 = 38.6 \to 39\), so \(\hat H = 1{,}092\); \(1920/28 = 68.6 \to 69\), so \(\hat W = 1{,}932\). That is 2,109,744 pixels, far below \(p_{\max}\), so no rescaling.

Patches: \(78 \times 138 = 10{,}764\). Tokens for the LLM: \(39 \times 69 = 2{,}691\). With the model card’s cap of 1,280 tokens, the same frame is shrunk to 1316×728 and costs \(47 \times 26 = 1{,}222\) tokens. A 224×224 image costs only \(8 \times 8 = 64\) tokens.

The vision encoder itself is new in Qwen2.5-VL: a 32-layer ViT (width 1,280, 16 heads) trained from scratch, with 2D RoPE for positions (so it accepts any grid size), RMSNorm and SwiGLU like the language model [Certain]. From the config, it has about 632M parameters, plus 37M to 68M for the merger depending on the LLM width (our count) [Certain]. The merger concatenates each 2×2 group of patch vectors (\(4 \times 1{,}280 = 5{,}120\) numbers) and maps them with a two-layer MLP to the LLM width: 2,048 for the 3B model, 3,584 for 7B and 8,192 for 72B [Certain].

Qwen2.5-VL, part 2: window attention in the ViT

Native resolution has a price. Full self-attention over \(N\) patches costs \(N^2\) scores per layer. For our 1080p frame, \(N = 10{,}764\), so one layer computes about 116 million scores, and a 4K frame (42,196 patches) needs 1.78 billion. The ViT becomes the bottleneck.

Qwen2.5-VL fixes this with window attention: in most layers, each patch attends only to patches in its own window of 112×112 pixels (\(8 \times 8 = 64\) patches). Only four of the 32 layers (layers 7, 15, 23 and 31, counting from 0) use full attention, so information can still travel across the whole image [Certain]. If the grid splits into windows of sizes \(n_1, n_2, \dots\), the scores per layer are:

\[ \text{scores}_{\text{full}} = N^2, \qquad \text{scores}_{\text{window}} = \sum_{w} n_w^2 \;\le\; 64\,N \tag{182.3} \]

So window layers grow linearly with the number of patches, not quadratically. The paper says exactly this [Certain]. For 1080p: \(10{,}764^2 = 115{,}863{,}696\) scores in a full layer, and 668,304 in a window layer, 173 times fewer. Over the 32 layers (4 full + 28 window), attention scores drop to 13% of an all-full ViT. Figure 2 shows the windows on top of the image.

one LLM token (\(2 \times 2\) patches)one attention window (\(8 \times 8\) patches)
Figure 2. Qwen2.5-VL’s resizing (182.1), token count (182.2) and window attention (182.3) for an image of your choice. Each small cell is one token for the language model; each orange box is one attention window. Windows at the right and bottom edges can be smaller.
How to read Figure 2

Start at 224²: there are only 4 windows and window attention saves 4×. Move to 1080p: 180 windows and 173× fewer scores per window layer. The bigger the image, the bigger the saving, because full attention grows as \(N^2\) and window attention as \(N\). Now tick the 1,280-token cap: the image is shrunk until \(V \le 1{,}280\). Try the document page: without the cap it costs 4,819 tokens. This is why serving settings for max_pixels matter so much for cost (session 183).

Qwen2.5-VL, part 3: absolute time for video

A video is a stack of frames. Qwen2.5-VL groups two consecutive frames into one temporal patch (a 3D patch of 2×14×14 pixels), so each pair of frames costs as much as one image of the same size [Certain]. With 640×360 frames (resized to 644×364), each pair costs \(23 \times 13 = 299\) tokens.

The model must also know where each token is. Qwen2-VL introduced M-RoPE (multimodal RoPE): the rotary angle is split into three parts, one driven by a time index \(t\), one by the row \(h\), and one by the column \(w\) (see session 14). For text tokens all three indices are equal, so it reduces to normal 1D RoPE. In Qwen2-VL the time index was simply the frame-pair number \(k = 0, 1, 2, \dots\). That has a problem: at 1 frame per second, pair 10 is at 20 seconds; at 4 frames per second, pair 10 is at 5 seconds. The model cannot tell speed or real time from the index.

Qwen2.5-VL aligns the time index with absolute time [Certain]. In the released code, the temporal index of pair \(k\) is:

\[ t_k = \left\lfloor k \cdot \frac{\tau}{f} \cdot s \right\rfloor, \qquad \tau = 2 \text{ frames per patch}, \quad s = 2 \text{ ids per second} \tag{182.4} \]
Read the symbols
\(k\)
index of the frame pair (temporal patch): 0, 1, 2, …
\(f\)
sampling rate in frames per second
\(\tau / f\)
seconds covered by one frame pair (second_per_grid_t in the code)
\(s\)
tokens_per_second = 2 in the config: two time ids per second of video [Certain]
\(t_k\)
the time id given to all tokens of pair \(k\); it equals twice the start time of the pair in seconds

Now the gap between ids tells the model the sampling speed: at \(f = 2\) the ids are 0, 2, 4, …; at \(f = 1\) they are 0, 4, 8, …. And a moment at 10 seconds always gets id 20, whatever the frame rate. During training the frame rate is sampled dynamically, so the model sees many different gaps [Certain]. This is what lets the model answer “at what second does the goal happen?” and is the basis for key-moment detection. Explore it in Figure 3.

Qwen2-VL: id \(= k\) (pair index)Qwen2.5-VL: id \(t_k = 2 \times\) seconds (182.4)the event
Figure 3. Temporal position ids for a video sampled at different frame rates, with 640×360 frames. Each dot is one frame pair (one temporal patch). The upper row uses the Qwen2-VL rule, the lower row the Qwen2.5-VL rule.
How to read Figure 3

Keep the event at 10 s and switch between frame rates. In the upper row the event’s id changes (2, 5, 10, 20): the index has no fixed meaning. In the lower row it stays at 20: the id is the time. Also watch the token count: 2 fps doubles the tokens of 1 fps for the same clip. Qwen2.5-VL’s evaluation caps a video at 768 frames and 24,576 tokens [Certain], so long videos must use a lower frame rate or smaller frames.

Algorithm 1: Qwen2.5-VL vision input, image or video
  1. input: image (\(H, W\)) or video frames sampled at \(f\) fps; limits \(p_{\min}, p_{\max}\)
  2. \(\hat H, \hat W \leftarrow\) smart resize to multiples of 28 inside \([p_{\min}, p_{\max}]\) // (182.1)
  3. cut into \(14 \times 14\) patches; for video, stack 2 frames per patch (an image is repeated to 2 frames)
  4. reorder patches so that each \(8 \times 8\) window is contiguous
  5. for layer \(\ell = 0 \dots 31\): attention over the whole image if \(\ell \in \{7, 15, 23, 31\}\), else inside each window // (182.3)
  6. merge each \(2 \times 2\) patch group (concatenate, 2-layer MLP) \(\to V\) tokens of LLM width // (182.2)
  7. restore the original token order
  8. position ids: \((t, h, w)\) per token, with \(t_k\) from (182.4) for video and \(t = 0\) for an image; text after the image continues from the largest id \(+\,1\)
  9. return visual tokens and their 3D position ids

One more Qwen2.5-VL choice is worth knowing for detection and grounding: boxes and points are written in absolute pixel coordinates of the resized image, not normalized to 0–1000 as in Qwen2-VL [Likely], because relative numbers “fail to effectively represent the original size and position of objects” [Certain]. When you fine-tune it on your own boxes, keep this convention.

Training: 4.1 trillion tokens in three stages. Stage 1 trains only the ViT on 1.5T tokens; stage 2 trains the ViT and LLM on 2T tokens (sequence length 8,192); stage 3 trains both on 0.6T tokens of long-context data (32,768) [Certain]. Sizes are 3B, 7B and 72B, all with the same ViT [Certain].

InternVL: a large vision encoder and dynamic tiles

The bet on a big encoder

The first InternVL paper (2023) asked a simple question: language models had grown to tens of billions of parameters, while vision encoders were still around 0.3 to 1B. Why not scale the vision side too? It trained InternViT-6B, a 6-billion-parameter ViT, and aligned it with a language model in stages, first with a contrastive loss (like CLIP) and then generatively [Certain]. InternVL 1.5 (2024) found that features from the fourth-to-last layer work best, so it dropped the last three layers (48 → 45 layers, width 3,200) [Certain].

Later versions keep the 6B encoder for the big models but use a 300M encoder for small ones. In InternVL 2.5, the 1B to 8B models use InternViT-300M (24 layers, width 1,024), and the 26B, 38B and 78B models use InternViT-6B [Certain]. The design is “ViT-MLP-LLM”: encoder, a randomly initialized MLP projector, and an off-the-shelf LLM (InternLM 2.5 or Qwen2.5) [Certain].

Dynamic high-resolution tiling

InternViT is trained at a fixed 448×448 input. To read large or long images without changing the encoder, InternVL 1.5 cuts the image into tiles of 448×448. It first picks a grid of \(i\) columns and \(j\) rows whose shape is closest to the image:

\[ (i^*, j^*) = \argmin_{(i, j):\ 1 \le ij \le n_{\max}} \left| \frac{W}{H} - \frac{i}{j} \right| \tag{182.5} \]

Ties (for example 2×1 and 4×2 have the same shape) go to the grid with more tiles if the image has more than half of its pixels, so big images get more detail. The image is resized to \(448 i \times 448 j\), cut into \(ij\) tiles, and, if \(ij \gt 1\), one extra thumbnail of the whole image at 448×448 is added for global context [Certain]. With \(n_{\max} = 12\) there are 35 candidate grids; during training \(n_{\max}\) is 12, and at test time it can go up to 40 tiles, about 4K resolution [Certain].

Pixel shuffle: 4× fewer tokens

Each 448 tile gives \(32 \times 32 = 1{,}024\) patch vectors. That is too many. InternVL applies a pixel shuffle (also called space-to-depth or pixel unshuffle): each 2×2 group of neighbors is stacked into one vector with 4 times the channels. Then an MLP maps it to the LLM width:

\[ (h, w, C) \ \xrightarrow{\ \text{shuffle}, r\ } \ \left(\frac{h}{r}, \frac{w}{r}, r^2 C\right), \qquad V = \big(i^* j^* + [\,i^* j^* \gt 1\,]\big) \cdot \frac{(448/14)^2}{r^2} \tag{182.6} \]
Read the symbols
\(r\)
shuffle ratio: 2 in InternVL (so \(1{,}024 \to 256\) tokens per tile), 4 in the small SmolVLM models
\(C\)
encoder width: 1,024 for InternViT-300M, so each shuffled vector has 4,096 numbers
\([\,i^* j^* \gt 1\,]\)
1 if there is more than one tile (then a thumbnail is added), else 0

Pixel shuffle loses no numbers: it only moves them from the spatial grid into the channel dimension. The MLP then decides what to keep. Compare this with Qwen’s merger: both turn 2×2 patches into one token. The difference is where the grid comes from: Qwen uses one native-resolution grid, InternVL uses many fixed tiles.

original image shape448×448 tilesthumbnail
Figure 4. InternVL’s dynamic tiling (182.5) and token count (182.6). The dashed outline is the true image shape, scaled to the same area as the tile grid. The gap between them is the stretch that the resize introduces.
How to read Figure 4

At 1080p with \(n_{\max} = 12\), the best shape is 2:1 (16:9 is not available), and the tie between 2×1 and 4×2 goes to 4×2: 8 tiles plus a thumbnail, 2,304 tokens. Raise \(n_{\max}\) to 40 and 7×4 (ratio 1.75) wins: 29 tiles, 7,424 tokens. Now set \(n_{\max} = 1\): every image becomes one stretched 448 square with 256 tokens, which is the old fixed-square behavior. The readout shows how much the aspect ratio is distorted.

Algorithm 2: InternVL dynamic preprocessing
  1. input: image \(W \times H\), tile size \(S = 448\), max tiles \(n_{\max}\)
  2. list all grids \((i, j)\) with \(1 \le ij \le n_{\max}\), sorted by \(ij\)
  3. pick the grid with the smallest \(|W/H - i/j|\); on a tie, take the larger grid if \(WH \gt \tfrac12 S^2 ij\) // (182.5)
  4. resize the image to \(Si \times Sj\) (aspect ratio may change slightly) and cut it into \(ij\) tiles
  5. if \(ij \gt 1\): append a thumbnail, the whole image resized to \(S \times S\)
  6. encode each tile with InternViT \(\to 32 \times 32 \times C\); pixel shuffle \(r = 2\): \(\to 16 \times 16 \times 4C\) // (182.6)
  7. MLP to LLM width; wrap each image’s tokens in <img> … </img>
  8. return \(256 \cdot (ij + [ij \gt 1])\) visual tokens

Tiling has a cost that native resolution does not: the ViT never sees across a tile border, so an object cut in half by a border is seen as two pieces (the thumbnail helps, at low detail). For multi-image or video inputs, InternVL 2.x shares the tile budget across images, and uses a single tile per video frame [Certain]. InternVL3 (2025) keeps the ViT-MLP-LLM design but trains vision and language together in one “native multimodal pretraining” stage instead of adapting a finished text LLM, and adds a variable visual position encoding (V2PE) for long multimodal contexts [Certain].

PaliGemma and Gemma 3: SigLIP plus Gemma

PaliGemma: a base model for transfer

PaliGemma (2024, Google) is deliberately simple: a SigLIP So400m/14 encoder (about 400M parameters, see session 167), a single linear layer initialized to zero, and Gemma 2B, under 3B parameters in total [Certain]. The image is always a square: 224, 448 or 896 pixels, giving \((224/14)^2 = 256\), 1,024 or 4,096 tokens [Certain]. It is meant as a base model to fine-tune (“transfer”) on one task, not as a chat assistant [Certain]. PaliGemma 2 keeps the encoder and swaps in Gemma 2 from 2B up to 27B, at the same three resolutions [Certain].

Its training is unusual: in the first multimodal stage (about 1 billion examples at 224 px), nothing is frozen. To protect the encoder from the first, misaligned gradients, its learning rate has a slow linear warmup [Certain]. Then two short stages raise the resolution to 448 (50M examples) and 896 (10M examples) [Certain].

The prefix-LM mask

The input sequence is [image tokens] [BOS] [prefix] [\n] [suffix] [EOS], where the prefix is the task or question (“caption en”, “detect car”) and the suffix is the answer [Certain]. A normal decoder uses a causal mask: token \(i\) sees only tokens \(j \le i\). PaliGemma uses a prefix-LM mask (see session 18): the image and the prefix attend to each other in both directions, and only the suffix is causal:

\[ M_{ij} = \begin{cases} 1 & \text{if } j \lt P \quad (\text{everyone sees the whole prefix}) \\ 1 & \text{if } j \le i \quad (\text{causal for the rest}) \\ 0 & \text{otherwise} \end{cases} \qquad P = V + 1 + |\text{prefix}| + 1 \tag{182.7} \]

Why? With a causal mask, the first image token can see nothing but itself, and image tokens can never look at the question. With the prefix-LM mask, image tokens can already “read” the question and focus on what is asked. The paper’s ablation found that causal masking of the prefix, or of the image and prefix, both work “but perform clearly worse” [Certain]. The loss is computed only on the suffix; adding it on the prefix also clearly reduced average quality [Certain].

Gemma 3: the same idea, in a general model

Gemma 3 (2025) adds vision to a general-purpose LLM family. The 4B, 12B and 27B models share one frozen SigLIP encoder of about 400M parameters (listed as 417M); the 1B model is text-only [Certain]. Every image is resized to 896×896, giving \(64 \times 64 = 4{,}096\) patches, and a 4×4 average pool reduces them to a fixed 256 “soft tokens” [Certain]:

\[ V_{\text{Gemma 3}} = \left(\frac{896}{14 \cdot 4}\right)^2 = 16^2 = 256 \text{ tokens per crop} \tag{182.8} \]

A fixed square hurts on documents and non-square images. Gemma 3’s answer is Pan & Scan, an inference-time option that cuts the image into equal, non-overlapping crops, resizes each to 896, and encodes each one [Certain]. It can be turned off for speed. On the pretrained 4B model it raised DocVQA from 72.8 to 81.0 and InfoVQA from 44.1 to 57.0 [Certain]. In the attention mask, Gemma 3 lets image tokens attend to each other in both directions while text stays causal (this is how the Hugging Face implementation builds the mask [Certain]). Figure 5 compares the three masks.

allowed (row attends to column)blockedselected row
Figure 5. Three attention masks for a toy sequence of 4 image tokens, 3 prefix (question) tokens and 3 suffix (answer) tokens. Row = the token that asks (query), column = the token it may look at (key). Click any row to see what that token can see.
How to read Figure 5

Click row “img1” in each mask. Causal: it sees only itself. Gemma 3: it sees all four image tokens but not the question. Prefix-LM: it sees the image and the question. Then click an answer row (“ans2”): it is the same in all three masks, because the answer must stay causal so that the model can generate it token by token. That is why the prefix-LM mask is free at inference: the prefix is processed once in prefill, and decoding is unchanged.

In counts: with 10 tokens, the causal mask allows 55 of the 100 pairs, Gemma 3’s mask 61, and the prefix-LM mask 76. The extra pairs are all inside the prefix, which is computed in one parallel prefill pass, so they cost almost nothing at serving time.

SmolVLM: design choices for small and edge models

SmolVLM (2025, Hugging Face) asks: what changes when the whole VLM must have 256M to 2.2B parameters and run on a laptop or phone? The three sizes are [Certain]:

ModelVision encoderLanguage modelTile / tokens per tile
SmolVLM-256MSigLIP-B/16 (93M), 512 pxSmolLM2-135M512 px \(\to\) 1,024 patches \(\to\) 64 tokens (\(r = 4\))
SmolVLM-500MSigLIP-B/16 (93M), 512 pxSmolLM2-360M512 px \(\to\) 64 tokens (\(r = 4\))
SmolVLM-2.2BSigLIP So400m, 384 pxSmolLM2-1.7B384 px \(\to\) 729 patches \(\to\) 81 tokens (\(r = 3\))

The token count per tile follows the pixel-shuffle formula (182.6) with a larger ratio:

\[ V_{\text{tile}} = \frac{(S/p)^2}{r^2}: \qquad \frac{(512/16)^2}{4^2} = \frac{1{,}024}{16} = 64, \qquad \frac{(384/14)^2}{3^2} = \frac{729}{9} = 81 \tag{182.9} \]

Images are split into tiles like Idefics3: the released preprocessor resizes the longest side to 2,048 pixels, rounds both sides up to multiples of 512, cuts 512 tiles and adds one downscaled global image [Certain]. A 1080p frame becomes 4×3 tiles plus the global image: \(13 \times 64 = 832\) image tokens (plus small row and column marker tokens). The 256M model runs inference in about 0.8 GB of GPU memory at batch size 1 [Certain].

The paper lists its findings explicitly [Certain]. The ones worth remembering:

  • Balance encoder and LM. With a tiny LM, a 400M encoder is too big a share of the parameters; at the 135M and 360M LM sizes, the 93M encoder is the better trade.
  • Compress harder when small. Small models do better with pixel shuffle \(r = 4\) than \(r = 2\) (the ratio used by InternVL and Idefics3). Fewer, richer tokens leave more of the short context for text.
  • Longer context helps. Raising the RoPE base from 10k to 273k and the context to 16k (2.2B) or 8k (smaller models) helped a lot.
  • Learned position tokens, not strings. Marking tiles with text strings like <row_1_col_2> caused a training plateau (“OCR loss plague”); learned positional tokens fixed it.
  • Data details matter. Do not average video frames; mask the user prompt in SFT; reusing the LLM’s own text SFT data hurt (up to 3.7% on video and 6.5% on image tasks); a tiny share (0.02–0.05%) of chain-of-thought data helps, more hurts.

The result: the paper reports that SmolVLM-256M beats the 300 times larger Idefics-80B from 18 months earlier on most benchmarks (not on MMMU and AI2D) [Certain]. The lesson for edge work: the number of visual tokens and the context budget matter more than model size alone.

Same image, five models

Token count is the single number that best predicts the prefill cost and KV-cache size of a VLM request (session 183). Figure 6 computes it for one image with each model’s released preprocessing defaults.

Figure 6. Visual tokens for the same input under each model’s default preprocessing (no special or marker tokens). Qwen2.5-VL: native resolution up to 16,384 tokens, and with the 1,280-token cap. InternVL 2.5: up to 12 tiles plus thumbnail. PaliGemma at 224 and 896 px: fixed. Gemma 3: one 896 crop (Pan & Scan off). SmolVLM-256M: 512 tiles with \(r = 4\). The scale is logarithmic.
How to read Figure 6

At 224² Qwen is the cheapest (64 tokens) because it does not upscale small images much, while PaliGemma-896 pays 4,096 for everything. At 4K, Qwen pays 10,549, while InternVL is capped at 2,304 and Gemma 3 at 256. Neither is “better”: fixed budgets are cheap but lose small text; native resolution keeps detail and pays for it. Raise the number of images to 32 (think of 32 video frames) and everything multiplies by 32; for video, Qwen2.5-VL would merge frame pairs and pay half. This is why video systems sample few frames and use small frame sizes. Note that each model’s prefill cost also depends on its LLM size.

Worked example: one 1080p frame, all models

Qwen2.5-VL: 1932×1092, \(39 \times 69 = 2{,}691\) tokens. InternVL 2.5: grid \(4 \times 2\) + thumbnail = 9 tiles, \(9 \times 256 = 2{,}304\). PaliGemma-896: 4,096. Gemma 3: 256 (or \(256\) per crop with Pan & Scan). SmolVLM-256M: 13 tiles, \(13 \times 64 = 832\). A 60 s clip at 2 fps with 640×360 frames in Qwen2.5-VL: 120 frames \(\to\) 60 pairs, \(60 \times 299 = 17{,}940\) tokens.

Code: the key mechanisms in one file

This script re-implements the parts that decide token counts and attention patterns: Qwen’s resize, windows and time ids, InternVL’s grid choice and pixel shuffle, the three masks, and Gemma 3’s pooling. It runs on CPU in a second and prints the numbers used on this page.

import math, torch
torch.manual_seed(0)

# 1) Qwen2.5-VL: native resolution, formula (182.1); sides become multiples of 28 (patch 14 x merge 2)
def smart_resize(h, w, f=28, min_px=56*56, max_px=16384*28*28):
    hb, wb = round(h / f) * f, round(w / f) * f
    if hb * wb > max_px:                       # too big: shrink, keep the aspect ratio
        b = math.sqrt(h * w / max_px); hb, wb = math.floor(h / b / f) * f, math.floor(w / b / f) * f
    elif hb * wb < min_px:                     # too small: grow
        b = math.sqrt(min_px / (h * w)); hb, wb = math.ceil(h * b / f) * f, math.ceil(w * b / f) * f
    return hb, wb

hb, wb = smart_resize(1080, 1920)
patches = (hb // 14) * (wb // 14)
print("resized", wb, "x", hb, "patches", patches, "LLM tokens", patches // 4)   # (182.2)

# 2) Window attention, formula (182.3): 8x8-patch windows (112 px) vs full attention
gh, gw = hb // 28, wb // 28                    # merged grid; one window = 4x4 merged = 8x8 patches
sizes = [min(4, gh - a) * min(4, gw - b) * 4 for a in range(0, gh, 4) for b in range(0, gw, 4)]
print("windows", len(sizes), "full/window scores", round(patches**2 / sum(s * s for s in sizes), 1))

# 3) M-RoPE time ids aligned to absolute time, formula (182.4)
def time_ids(n_grids, fps, tokens_per_second=2, temporal_patch=2):
    sec_per_grid = temporal_patch / fps
    return [int(k * sec_per_grid * tokens_per_second) for k in range(n_grids)]
print("fps 1:", time_ids(5, 1.0), " fps 2:", time_ids(5, 2.0))

# 4) InternVL dynamic tiling, formula (182.5): the grid (i, j) closest in shape to the image
def pick_grid(w, h, max_num=12, size=448):
    ratios = sorted({(i, j) for n in range(1, max_num + 1) for i in range(1, n + 1)
                     for j in range(1, n + 1) if i * j <= max_num}, key=lambda r: r[0] * r[1])
    best, grid = float("inf"), (1, 1)
    for i, j in ratios:
        d = abs(w / h - i / j)
        if d < best: best, grid = d, (i, j)
        elif d == best and w * h > 0.5 * size * size * i * j: grid = (i, j)   # tie: more tiles
    return len(ratios), grid
n_ratios, (i, j) = pick_grid(1920, 1080)
tiles = i * j + (1 if i * j > 1 else 0)       # + one thumbnail of the whole image
print("candidate grids", n_ratios, "grid", (i, j), "tiles", tiles, "tokens", tiles * 256)

# 5) Pixel shuffle (space-to-depth), formula (182.6): (B, 32, 32, C) -> (B, 16, 16, 4C)
def pixel_shuffle(x, r=2):
    B, H, W, C = x.shape
    x = x.view(B, H // r, r, W // r, r, C).permute(0, 1, 3, 2, 4, 5)
    return x.reshape(B, H // r, W // r, r * r * C)
feat = torch.randn(1, 32, 32, 1024)               # one 448 px tile from InternViT-300M
print("pixel shuffle", tuple(pixel_shuffle(feat).shape),
      "SmolVLM r=4:", tuple(pixel_shuffle(torch.randn(1, 32, 32, 768), 4).shape))

# 6) Masks, formula (182.7): causal, PaliGemma prefix-LM, Gemma 3 (image tokens bidirectional)
I, P, S = 4, 3, 3; n = I + P + S
causal = torch.tril(torch.ones(n, n, dtype=torch.bool))
prefix = causal.clone(); prefix[:, : I + P] = True       # every token sees the whole prefix
gemma3 = causal.clone(); gemma3[:I, :I] = True            # the image block is full
print("allowed pairs: causal", causal.sum().item(), "prefix-LM", prefix.sum().item(), "gemma3", gemma3.sum().item())

# 7) Gemma 3, formula (182.8): 64x64 SigLIP patches -> 4x4 average pool -> 256 soft tokens
v = torch.randn(1, 1152, 64, 64)
print("gemma3 tokens", torch.nn.functional.avg_pool2d(v, 4).flatten(2).shape[-1])

# Output:
# resized 1932 x 1092 patches 10764 LLM tokens 2691
# windows 180 full/window scores 173.4
# fps 1: [0, 4, 8, 12, 16]  fps 2: [0, 2, 4, 6, 8]
# candidate grids 35 grid (4, 2) tiles 9 tokens 2304
# pixel shuffle (1, 16, 16, 4096) SmolVLM r=4: (1, 8, 8, 12288)
# allowed pairs: causal 55 prefix-LM 76 gemma3 61
# gemma3 tokens 256

Our pixel_shuffle orders the stacked channels differently from InternVL’s own code, but the shapes and the information are the same; with a learned MLP after it, the order does not matter. To run the real models, use the Hugging Face classes Qwen2_5_VLForConditionalGeneration, PaliGemmaForConditionalGeneration, Gemma3ForConditionalGeneration and AutoModelForVision2Seq (SmolVLM); InternVL 2.5 loads with trust_remote_code=True [Certain].

Where this is used

TaskGood fitWhy
Key-moment detection, temporal grounding (“when does the goal happen?”)Qwen2.5-VLTime ids aligned to seconds (182.4) and dynamic-fps training; it can answer with timestamps
Video question answering and captioning on long clipsQwen2.5-VL with a token cap; InternVL with one tile per framePer-frame cost is the knob: 299 tokens per 360p frame pair in Qwen, 256 per frame in InternVL
Documents, charts, screenshots, OCRQwen2.5-VL (native), InternVL (tiles), Gemma 3 with Pan & ScanSmall text needs many tokens; fixed 896 squares lose it (Pan & Scan: +8.2 DocVQA on 4B)
Detection and grounding with boxesQwen2.5-VL (absolute pixel boxes), PaliGemma (“detect” prefix, then fine-tune)Both output boxes as text tokens; PaliGemma is built to be fine-tuned per task
Fine-tuning a specialist (one task, one domain)PaliGemma / PaliGemma 2Base model without chat tuning, prefix-LM mask, simple linear connector
Video retrieval and Video-RAGEncoder-level embeddings (SigLIP) for search, then a VLM such as Qwen2.5-VL to answer over the retrieved clipsEmbedding every frame with a full VLM is too expensive; generate only for the top hits
On-device, edge, or high-volume cheap captioningSmolVLM-256M / 500MAbout 0.8 GB GPU memory; 64 tokens per 512 tile

Common mistakes

  • Leaving Qwen’s max_pixels at the default when serving. A 4K screenshot then costs 10,549 visual tokens. Set a cap that matches your task (for example the 1,280-token setting) and measure accuracy.
  • Mixing up merged tokens and patches. In Qwen2.5-VL, window attention and the 4 full layers are counted in ViT patches; the LLM sees 4 times fewer tokens. A 1080p frame is 10,764 patches but 2,691 tokens.
  • Using a fixed time index for video. If you sample frames yourself, pass the real frame rate (fps / second_per_grid_ts) to Qwen2.5-VL; otherwise the time ids assume 1 s per frame pair and timestamps in the answer will be wrong.
  • Normalizing boxes for Qwen2.5-VL. It was trained with absolute pixel coordinates of the resized image; 0–1000 normalized boxes (the Qwen2-VL habit) give wrong outputs.
  • Fine-tuning PaliGemma with a causal mask or loss on the prompt. Pass token_type_ids (or use the processor with a suffix) so the prefix is bidirectional and only the suffix gets loss.
  • Using too many InternVL tiles for multi-image or video input. The tile budget is per sample; 13 tiles (12 + thumbnail) for each of 16 images is \(13 \times 256 \times 16 = 53{,}248\) tokens. Lower max_num to 1 per frame for video.
  • Expecting Gemma 3 to read small text without Pan & Scan. Every image becomes 256 tokens of an 896 square; turn on Pan & Scan for documents.
  • Comparing models by parameter count only. Prefill compute is roughly \(2 \times\) (LM parameters) \(\times\) (tokens). A model with half the parameters but four times the visual tokens per image does about twice the prefill work. Compare tokens \(\times\) LM size.

Interview questions

How many visual tokens does Qwen2.5-VL produce for a 1920×1080 image, and how do you compute it?basic
Round each side to a multiple of 28 (patch 14, merge 2: \(14 \times 2 = 28\)): \(1092 \times 1932\). The ViT sees \(78 \times 138 = 10{,}764\) patches; the \(2 \times 2\) merger gives \(39 \times 69 = 2{,}691\) tokens, plus vision-start and vision-end. If max_pixels is lower, the image is first scaled down keeping the aspect ratio; with a 1,280-token cap it becomes 1316×728 and 1,222 tokens.
Why does Qwen2.5-VL use window attention in its ViT, and why keep four full-attention layers?core
With native resolution, a big image has tens of thousands of patches and full attention costs \(N^2\) per layer (116M scores at 1080p, 1.78B at 4K). Window attention over \(8 \times 8\)-patch windows costs at most \(64N\), linear in \(N\): 173× fewer scores at 1080p. But windows alone cannot pass information across the image, so layers 7, 15, 23 and 31 use full attention to mix globally. Overall the ViT’s attention scores fall to about 13% of an all-full ViT at 1080p.
What is M-RoPE, and what did Qwen2.5-VL change about it for video?core
M-RoPE splits the rotary dimensions into three groups driven by time, height and width indices; text tokens use the same value for all three, so it is ordinary RoPE for text. In Qwen2-VL the time index was the frame-pair number, which has no fixed meaning when the sampling rate changes. Qwen2.5-VL sets the time index to \(\lfloor k \cdot (2/f) \cdot 2 \rfloor\), i.e. twice the start time in seconds. A moment at 10 s always gets id 20, and the gap between ids encodes the sampling rate. Combined with dynamic-fps training, the model can localize events in seconds.
Explain InternVL’s dynamic tiling. What does the thumbnail do?core
The encoder expects 448×448. InternVL chooses a grid \((i, j)\) with \(ij \le n_{\max}\) whose ratio \(i/j\) is closest to \(W/H\) (ties go to more tiles for large images), resizes the image to \(448i \times 448j\), and cuts it into tiles. Each tile becomes 256 tokens after a \(2 \times 2\) pixel shuffle. If there is more than one tile, a 448 thumbnail of the whole image is added so the model has global context across tile borders. A 1080p image with \(n_{\max} = 12\) gives a \(4 \times 2\) grid + thumbnail: \((8 + 1) \times 256 = 2{,}304\) tokens.
Native resolution (Qwen) vs tiling (InternVL): trade-offs?advanced
Native resolution keeps the true aspect ratio, has no tile borders, and lets the encoder see the whole image, but needs an encoder that accepts any grid (2D RoPE), variable-length batching, and something like window attention to stay affordable; cost grows with image area. Tiling reuses a fixed-size encoder unchanged, gives a hard cap on tokens (\(256 \cdot (n_{\max}+1)\)), and batches easily, but stretches the image to the nearest grid shape, cuts objects at tile borders, and needs a thumbnail for global context. For documents both work; for video, a fixed per-frame cost is often easier to plan.
What is a prefix-LM mask and why does PaliGemma use it?core
Tokens in the prefix (image tokens, BOS, the task text, separator) attend to each other in both directions; suffix (answer) tokens are causal and also see the full prefix. It lets image tokens see the question, so the image representation can depend on the task. The PaliGemma ablation found causal masking of the prefix works but clearly worse. It costs nothing extra at inference because the prefix is processed once in prefill and decoding stays causal. The loss is applied only on the suffix.
Gemma 3 makes every image 256 tokens. How does it handle documents, and what does that cost?core
The 896×896 SigLIP output (\(64 \times 64\) patches) is average-pooled \(4 \times 4\) to 256 tokens, so a document page loses small text. Pan & Scan, an optional inference step, cuts the image into equal non-overlapping crops, resizes each to 896, and encodes each as 256 more tokens. The paper reports e.g. DocVQA 72.8 → 81.0 and InfoVQA 44.1 → 57.0 for the 4B pretrained model. The cost is linear in the number of crops, and it can be turned off for speed.
Why did InternVL scale the vision encoder to 6B, and why do its small models use 300M?advanced
The original argument: LLMs had grown much faster than vision encoders, so the vision side was the weak link in perception; a 6B ViT aligned with a strong LLM gave better visual features. But encoder cost is paid on every tile of every image, and for a 1B to 8B LLM a 6B encoder would dominate compute and memory. So InternVL 2.x uses InternViT-300M for 1B–8B models and the 6B encoder only for 26B–78B models. SmolVLM made the same point at small scale: balance encoder and LM parameters.
You must run a VLM on a phone to caption photos. What design choices do you make?advanced
Follow SmolVLM: a small encoder (SigLIP-B/16, 93M) matched to a small LM (135M–360M), aggressive pixel shuffle (\(r = 4\), 64 tokens per 512 tile), few tiles (lower the longest-edge setting), learned tile-position tokens, and a context long enough for the image tokens (8k). Then quantize weights (4-bit or 8-bit), cap image tokens, and cache the encoder output if the same image is asked about twice. Measure latency in tokens per image \(\times\) LM size, not parameters alone.
How does each model know where a visual token is in the image?core
Qwen2.5-VL: 2D RoPE inside the ViT, and M-RoPE with (t, h, w) ids in the LLM. InternVL: learned absolute positions inside each 448 tile [Likely]; the order of tiles and the thumbnail is implicit in the sequence (plus tags around each image). PaliGemma and Gemma 3: SigLIP’s learned position embeddings for a fixed grid; the LLM sees image tokens in raster order with ordinary 1D positions. SmolVLM: learned tile-position tokens between tiles, which the paper found better than string tags.
A 60 s video at 2 fps, 640×360 frames. How many tokens in Qwen2.5-VL, and how would you cut it in half?basic
640×360 is resized to 644×364, so one frame pair is \(23 \times 13 = 299\) tokens. 120 frames \(\to\) 60 pairs \(\to 60 \times 299 = 17{,}940\) tokens. To halve it: sample at 1 fps (8,970 tokens; time ids still correct because they follow seconds), or lower max_pixels so each pair has about 150 tokens. Which one is better depends on whether the task needs fine time detail (fast actions) or fine spatial detail (text on screen).

Exercises

  1. Compute Qwen2.5-VL’s token count for an 800×600 image with default limits. How many ViT patches is that?
  2. With \(8 \times 8\)-patch windows, how many windows does a 448×448 image have in Qwen2.5-VL, and how many times fewer scores does a window layer compute than a full layer?
  3. A video is sampled at 0.5 fps. What are the time ids of the first four frame pairs? Which pair contains second 10, and what is its id?
  4. Which grid does InternVL pick for an 800×600 image with \(n_{\max} = 12\)? How many tokens does it produce?
  5. In Figure 5’s toy sequence, how many allowed pairs does the prefix-LM mask have if the prefix has 2 text tokens instead of 3 (still 4 image and 3 suffix tokens)?
  6. SmolVLM-2.2B uses a 384 px tile, patch 14 and \(r = 3\). What is the vector size per token after the shuffle if the encoder width is 1,152?
Answerscheck
1. \(600/28 = 21.4 \to 21\), so \(\hat H = 588\); \(800/28 = 28.6 \to 29\), so \(\hat W = 812\). Tokens \(21 \times 29 = 609\); patches \(42 \times 58 = 2{,}436\). 2. Merged grid \(16 \times 16\), windows of \(4 \times 4\) merged tokens: 16 windows of 64 patches. Full: \(1{,}024^2\); window: \(16 \times 64^2 = 65{,}536\); \(16\times\) fewer. 3. Each pair covers 4 s, so ids step by 8: 0, 8, 16, 24. Second 10 is in pair 2 (seconds 8–12), id 16. 4. \(800/600 = 1.333\) matches 4:3 exactly, and \(4 \times 3 = 12\) tiles is allowed: 12 tiles + thumbnail, \(13 \times 256 = 3{,}328\) tokens. 5. \(n = 9\), prefix \(P = 6\): the 6 prefix rows see 6 each (36), the 3 suffix rows see 7, 8, 9 (24): 60 pairs. 6. \(r^2 C = 9 \times 1{,}152 = 10{,}368\) numbers, which a linear layer maps to the LLM width 2,048.

Key takeaways

  • All four families use encoder → connector → LLM in one sequence; they differ in how an image becomes a grid, how tokens are reduced, how positions are encoded, and which mask is used.
  • Qwen2.5-VL: native resolution with sides rounded to multiples of 28 (182.1, 182.2), window attention with 4 full layers to keep the ViT linear in patches (182.3), and time ids equal to twice the seconds (182.4). 1080p = 2,691 tokens.
  • InternVL: a large encoder (6B, or 300M for small models), 448 tiles chosen by aspect ratio plus a thumbnail (182.5), and \(2 \times 2\) pixel shuffle to 256 tokens per tile (182.6). 1080p = 2,304 tokens.
  • PaliGemma: SigLIP + linear + Gemma, a fixed square (256 / 1,024 / 4,096 tokens) and a prefix-LM mask (182.7); a base model for fine-tuning. Gemma 3: frozen SigLIP, 256 pooled tokens per 896 crop (182.8), Pan & Scan for documents.
  • SmolVLM: balance encoder and LM, compress harder (\(r = 4\), 64 tokens per tile, 182.9), longer context, learned tile tokens; the 256M model runs in about 0.8 GB.

Key formulas

\[ V_{\text{Qwen}} = \frac{\hat H}{28}\cdot\frac{\hat W}{28}, \quad \text{scores}_{\text{win}} \le 64N, \quad t_k = \left\lfloor k\,\frac{2}{f}\cdot 2 \right\rfloor, \quad V_{\text{InternVL}} = 256\,(ij + [ij \gt 1]), \quad V_{\text{tile}} = \frac{(S/p)^2}{r^2} \]

References for this session

  1. Bai et al. (2025). Qwen2.5-VL Technical Report
  2. Chen et al. (2023). InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  3. Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
  4. Gemma Team (2025). Gemma 3 Technical Report
  5. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
  6. Bai et al. (2025). Qwen2.5-VL Technical Report
  7. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  8. Chen et al. (2023). InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  9. Chen et al. (2024). How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites (InternVL 1.5)
  10. Chen et al. (2024). Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (InternVL 2.5)
  11. Zhu et al. (2025). InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
  12. Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
  13. Steiner et al. (2024). PaliGemma 2: A Family of Versatile VLMs for Transfer
  14. Gemma Team (2025). Gemma 3 Technical Report
  15. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
  16. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  17. Dehghani et al. (2023). Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
  18. Laurençon et al. (2024). What matters when building vision-language models?
  19. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
  20. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models