Case studies 2: Qwen2.5-VL, InternVL, PaliGemma and Gemma, SmolVLM
Four model families that most teams download and fine-tune today, taken apart with real numbers. Qwen2.5-VL keeps images at native resolution, uses window attention in its vision encoder, and gives video tokens time positions in real seconds. InternVL uses a very large vision encoder and cuts images into tiles; PaliGemma and Gemma 3 join SigLIP to Gemma with a prefix-LM or image-bidirectional mask; SmolVLM shows what changes when the whole model must run on a phone.
What you will learn
- The four design questions every VLM answers: image to grid, token reduction, positions, attention mask
- Qwen2.5-VL: native-resolution resizing to multiples of 28, the token count, and why window attention with 4 full layers keeps the ViT linear in patches
- M-RoPE with absolute time: why a time id of 2 × seconds lets a model localize events in video
- InternVL: the 6B (and 300M) InternViT, aspect-ratio tiling with a thumbnail, and 2×2 pixel shuffle to 256 tokens per tile
- PaliGemma and Gemma 3: SigLIP + Gemma, fixed squares, the prefix-LM mask, 4×4 pooling to 256 tokens, and Pan & Scan
- SmolVLM: the design findings for small models (balanced encoder, r = 4 pixel shuffle, longer context, learned tile tokens)
- How to compute and compare the visual token cost of one image or video across all five models
Why this matters
In session 181 we met the first generation of open vision-language models: LLaVA, BLIP-2, Flamingo and Idefics 2. This session covers the models that most teams actually download and fine-tune today: Qwen2.5-VL, InternVL, PaliGemma and Gemma 3, and SmolVLM. All of them use the same basic plan from session 177: a vision encoder, a connector, and a language model that reads image tokens in one shared sequence. But each family made a different bet on the hard questions.
Those questions are always the same four. (1) How does an image of any size become a fixed grid for the encoder? (2) How many visual tokens reach the language model, and how are they reduced? (3) How does the model know where (and, for video, when) each token is? (4) Which tokens may attend to which (the attention mask)? Qwen2.5-VL answers with native resolution, window attention and time-aligned positions. InternVL answers with a very large encoder and tiles. PaliGemma answers with a fixed square and a prefix-LM mask. SmolVLM answers with aggressive token compression so the model fits on a phone.
For video work this matters directly. A 60-second clip at 2 frames per second with 640×360 frames costs 17,940 visual tokens in Qwen2.5-VL (we compute this below). The choice of model family decides whether a video retrieval or key-moment system fits on one GPU, and whether it can say at what second something happened.
Four families on one page
Figure 1 puts the five models side by side. Read each row from left to right: the image goes in, the encoder turns it into patch vectors, the connector reduces and projects them, and the language model reads them as tokens. All numbers are from the papers and the released config.json files [Certain].
Two patterns are visible already. First, everyone reduces tokens before the language model, but in different ways: Qwen and InternVL merge 2×2 neighbors (4× fewer), Gemma 3 averages 4×4 blocks (16× fewer), SmolVLM-256M merges 4×4 neighbors into one wide vector (16× fewer). Second, only Qwen2.5-VL avoids cutting the image into fixed squares; everyone else resizes or tiles.
Qwen2.5-VL, part 1: native resolution
Most early VLMs resize every image to one square, such as 336×336. A receipt, a 16:9 video frame and a tall phone screenshot all become the same square, so small text is blurred and shapes are stretched. Qwen2-VL and Qwen2.5-VL do the opposite: they keep the image close to its native resolution and aspect ratio, and let the number of tokens follow the image.
The ViT uses patches of 14×14 pixels, and the connector later merges each 2×2 block of patches into one token. So each side only has to be a multiple of \(14 \times 2 = 28\) pixels. The preprocessor rounds each side to the nearest multiple of 28, and rescales the whole image only if the pixel count leaves a range \([p_{\min}, p_{\max}]\):
(The width is shrunk the same way, and a similar rule with \(\lceil \cdot \rceil\) grows images below \(p_{\min}\).) The number of patches and of tokens for the language model is then:
- Read the symbols
- \(H, W\)
- height and width of the input image in pixels
- \(\hat H, \hat W\)
- the size after resizing; both are multiples of 28
- \(p_{\min}, p_{\max}\)
- the allowed pixel range. The released preprocessor uses \(p_{\min} = 3{,}136\) (that is 56 × 56) and \(p_{\max} = 12{,}845{,}056\) (that is \(16{,}384 \times 28^2\)), so at most 16,384 tokens per image [Certain]. The model card suggests a range of 256 to 1,280 tokens “to balance performance and cost” [Certain]
- \(N_{\text{patch}}\)
- number of 14×14 patches the ViT sees
- \(V\)
- number of visual tokens the language model sees (plus one vision-start and one vision-end token)
A 1920×1080 frame: \(1080/28 = 38.6 \to 39\), so \(\hat H = 1{,}092\); \(1920/28 = 68.6 \to 69\), so \(\hat W = 1{,}932\). That is 2,109,744 pixels, far below \(p_{\max}\), so no rescaling.
Patches: \(78 \times 138 = 10{,}764\). Tokens for the LLM: \(39 \times 69 = 2{,}691\). With the model card’s cap of 1,280 tokens, the same frame is shrunk to 1316×728 and costs \(47 \times 26 = 1{,}222\) tokens. A 224×224 image costs only \(8 \times 8 = 64\) tokens.
The vision encoder itself is new in Qwen2.5-VL: a 32-layer ViT (width 1,280, 16 heads) trained from scratch, with 2D RoPE for positions (so it accepts any grid size), RMSNorm and SwiGLU like the language model [Certain]. From the config, it has about 632M parameters, plus 37M to 68M for the merger depending on the LLM width (our count) [Certain]. The merger concatenates each 2×2 group of patch vectors (\(4 \times 1{,}280 = 5{,}120\) numbers) and maps them with a two-layer MLP to the LLM width: 2,048 for the 3B model, 3,584 for 7B and 8,192 for 72B [Certain].
Qwen2.5-VL, part 2: window attention in the ViT
Native resolution has a price. Full self-attention over \(N\) patches costs \(N^2\) scores per layer. For our 1080p frame, \(N = 10{,}764\), so one layer computes about 116 million scores, and a 4K frame (42,196 patches) needs 1.78 billion. The ViT becomes the bottleneck.
Qwen2.5-VL fixes this with window attention: in most layers, each patch attends only to patches in its own window of 112×112 pixels (\(8 \times 8 = 64\) patches). Only four of the 32 layers (layers 7, 15, 23 and 31, counting from 0) use full attention, so information can still travel across the whole image [Certain]. If the grid splits into windows of sizes \(n_1, n_2, \dots\), the scores per layer are:
So window layers grow linearly with the number of patches, not quadratically. The paper says exactly this [Certain]. For 1080p: \(10{,}764^2 = 115{,}863{,}696\) scores in a full layer, and 668,304 in a window layer, 173 times fewer. Over the 32 layers (4 full + 28 window), attention scores drop to 13% of an all-full ViT. Figure 2 shows the windows on top of the image.
How to read Figure 2
Start at 224²: there are only 4 windows and window attention saves 4×. Move to 1080p: 180 windows and 173× fewer scores per window layer. The bigger the image, the bigger the saving, because full attention grows as \(N^2\) and window attention as \(N\). Now tick the 1,280-token cap: the image is shrunk until \(V \le 1{,}280\). Try the document page: without the cap it costs 4,819 tokens. This is why serving settings for max_pixels matter so much for cost (session 183).
Qwen2.5-VL, part 3: absolute time for video
A video is a stack of frames. Qwen2.5-VL groups two consecutive frames into one temporal patch (a 3D patch of 2×14×14 pixels), so each pair of frames costs as much as one image of the same size [Certain]. With 640×360 frames (resized to 644×364), each pair costs \(23 \times 13 = 299\) tokens.
The model must also know where each token is. Qwen2-VL introduced M-RoPE (multimodal RoPE): the rotary angle is split into three parts, one driven by a time index \(t\), one by the row \(h\), and one by the column \(w\) (see session 14). For text tokens all three indices are equal, so it reduces to normal 1D RoPE. In Qwen2-VL the time index was simply the frame-pair number \(k = 0, 1, 2, \dots\). That has a problem: at 1 frame per second, pair 10 is at 20 seconds; at 4 frames per second, pair 10 is at 5 seconds. The model cannot tell speed or real time from the index.
Qwen2.5-VL aligns the time index with absolute time [Certain]. In the released code, the temporal index of pair \(k\) is:
- Read the symbols
- \(k\)
- index of the frame pair (temporal patch): 0, 1, 2, …
- \(f\)
- sampling rate in frames per second
- \(\tau / f\)
- seconds covered by one frame pair (
second_per_grid_tin the code) - \(s\)
tokens_per_second = 2in the config: two time ids per second of video [Certain]- \(t_k\)
- the time id given to all tokens of pair \(k\); it equals twice the start time of the pair in seconds
Now the gap between ids tells the model the sampling speed: at \(f = 2\) the ids are 0, 2, 4, …; at \(f = 1\) they are 0, 4, 8, …. And a moment at 10 seconds always gets id 20, whatever the frame rate. During training the frame rate is sampled dynamically, so the model sees many different gaps [Certain]. This is what lets the model answer “at what second does the goal happen?” and is the basis for key-moment detection. Explore it in Figure 3.
How to read Figure 3
Keep the event at 10 s and switch between frame rates. In the upper row the event’s id changes (2, 5, 10, 20): the index has no fixed meaning. In the lower row it stays at 20: the id is the time. Also watch the token count: 2 fps doubles the tokens of 1 fps for the same clip. Qwen2.5-VL’s evaluation caps a video at 768 frames and 24,576 tokens [Certain], so long videos must use a lower frame rate or smaller frames.
- input: image (\(H, W\)) or video frames sampled at \(f\) fps; limits \(p_{\min}, p_{\max}\)
- \(\hat H, \hat W \leftarrow\) smart resize to multiples of 28 inside \([p_{\min}, p_{\max}]\) // (182.1)
- cut into \(14 \times 14\) patches; for video, stack 2 frames per patch (an image is repeated to 2 frames)
- reorder patches so that each \(8 \times 8\) window is contiguous
- for layer \(\ell = 0 \dots 31\): attention over the whole image if \(\ell \in \{7, 15, 23, 31\}\), else inside each window // (182.3)
- merge each \(2 \times 2\) patch group (concatenate, 2-layer MLP) \(\to V\) tokens of LLM width // (182.2)
- restore the original token order
- position ids: \((t, h, w)\) per token, with \(t_k\) from (182.4) for video and \(t = 0\) for an image; text after the image continues from the largest id \(+\,1\)
- return visual tokens and their 3D position ids
One more Qwen2.5-VL choice is worth knowing for detection and grounding: boxes and points are written in absolute pixel coordinates of the resized image, not normalized to 0–1000 as in Qwen2-VL [Likely], because relative numbers “fail to effectively represent the original size and position of objects” [Certain]. When you fine-tune it on your own boxes, keep this convention.
Training: 4.1 trillion tokens in three stages. Stage 1 trains only the ViT on 1.5T tokens; stage 2 trains the ViT and LLM on 2T tokens (sequence length 8,192); stage 3 trains both on 0.6T tokens of long-context data (32,768) [Certain]. Sizes are 3B, 7B and 72B, all with the same ViT [Certain].
InternVL: a large vision encoder and dynamic tiles
The bet on a big encoder
The first InternVL paper (2023) asked a simple question: language models had grown to tens of billions of parameters, while vision encoders were still around 0.3 to 1B. Why not scale the vision side too? It trained InternViT-6B, a 6-billion-parameter ViT, and aligned it with a language model in stages, first with a contrastive loss (like CLIP) and then generatively [Certain]. InternVL 1.5 (2024) found that features from the fourth-to-last layer work best, so it dropped the last three layers (48 → 45 layers, width 3,200) [Certain].
Later versions keep the 6B encoder for the big models but use a 300M encoder for small ones. In InternVL 2.5, the 1B to 8B models use InternViT-300M (24 layers, width 1,024), and the 26B, 38B and 78B models use InternViT-6B [Certain]. The design is “ViT-MLP-LLM”: encoder, a randomly initialized MLP projector, and an off-the-shelf LLM (InternLM 2.5 or Qwen2.5) [Certain].
Dynamic high-resolution tiling
InternViT is trained at a fixed 448×448 input. To read large or long images without changing the encoder, InternVL 1.5 cuts the image into tiles of 448×448. It first picks a grid of \(i\) columns and \(j\) rows whose shape is closest to the image:
Ties (for example 2×1 and 4×2 have the same shape) go to the grid with more tiles if the image has more than half of its pixels, so big images get more detail. The image is resized to \(448 i \times 448 j\), cut into \(ij\) tiles, and, if \(ij \gt 1\), one extra thumbnail of the whole image at 448×448 is added for global context [Certain]. With \(n_{\max} = 12\) there are 35 candidate grids; during training \(n_{\max}\) is 12, and at test time it can go up to 40 tiles, about 4K resolution [Certain].
Pixel shuffle: 4× fewer tokens
Each 448 tile gives \(32 \times 32 = 1{,}024\) patch vectors. That is too many. InternVL applies a pixel shuffle (also called space-to-depth or pixel unshuffle): each 2×2 group of neighbors is stacked into one vector with 4 times the channels. Then an MLP maps it to the LLM width:
- Read the symbols
- \(r\)
- shuffle ratio: 2 in InternVL (so \(1{,}024 \to 256\) tokens per tile), 4 in the small SmolVLM models
- \(C\)
- encoder width: 1,024 for InternViT-300M, so each shuffled vector has 4,096 numbers
- \([\,i^* j^* \gt 1\,]\)
- 1 if there is more than one tile (then a thumbnail is added), else 0
Pixel shuffle loses no numbers: it only moves them from the spatial grid into the channel dimension. The MLP then decides what to keep. Compare this with Qwen’s merger: both turn 2×2 patches into one token. The difference is where the grid comes from: Qwen uses one native-resolution grid, InternVL uses many fixed tiles.
How to read Figure 4
At 1080p with \(n_{\max} = 12\), the best shape is 2:1 (16:9 is not available), and the tie between 2×1 and 4×2 goes to 4×2: 8 tiles plus a thumbnail, 2,304 tokens. Raise \(n_{\max}\) to 40 and 7×4 (ratio 1.75) wins: 29 tiles, 7,424 tokens. Now set \(n_{\max} = 1\): every image becomes one stretched 448 square with 256 tokens, which is the old fixed-square behavior. The readout shows how much the aspect ratio is distorted.
- input: image \(W \times H\), tile size \(S = 448\), max tiles \(n_{\max}\)
- list all grids \((i, j)\) with \(1 \le ij \le n_{\max}\), sorted by \(ij\)
- pick the grid with the smallest \(|W/H - i/j|\); on a tie, take the larger grid if \(WH \gt \tfrac12 S^2 ij\) // (182.5)
- resize the image to \(Si \times Sj\) (aspect ratio may change slightly) and cut it into \(ij\) tiles
- if \(ij \gt 1\): append a thumbnail, the whole image resized to \(S \times S\)
- encode each tile with InternViT \(\to 32 \times 32 \times C\); pixel shuffle \(r = 2\): \(\to 16 \times 16 \times 4C\) // (182.6)
- MLP to LLM width; wrap each image’s tokens in <img> … </img>
- return \(256 \cdot (ij + [ij \gt 1])\) visual tokens
Tiling has a cost that native resolution does not: the ViT never sees across a tile border, so an object cut in half by a border is seen as two pieces (the thumbnail helps, at low detail). For multi-image or video inputs, InternVL 2.x shares the tile budget across images, and uses a single tile per video frame [Certain]. InternVL3 (2025) keeps the ViT-MLP-LLM design but trains vision and language together in one “native multimodal pretraining” stage instead of adapting a finished text LLM, and adds a variable visual position encoding (V2PE) for long multimodal contexts [Certain].
PaliGemma and Gemma 3: SigLIP plus Gemma
PaliGemma: a base model for transfer
PaliGemma (2024, Google) is deliberately simple: a SigLIP So400m/14 encoder (about 400M parameters, see session 167), a single linear layer initialized to zero, and Gemma 2B, under 3B parameters in total [Certain]. The image is always a square: 224, 448 or 896 pixels, giving \((224/14)^2 = 256\), 1,024 or 4,096 tokens [Certain]. It is meant as a base model to fine-tune (“transfer”) on one task, not as a chat assistant [Certain]. PaliGemma 2 keeps the encoder and swaps in Gemma 2 from 2B up to 27B, at the same three resolutions [Certain].
Its training is unusual: in the first multimodal stage (about 1 billion examples at 224 px), nothing is frozen. To protect the encoder from the first, misaligned gradients, its learning rate has a slow linear warmup [Certain]. Then two short stages raise the resolution to 448 (50M examples) and 896 (10M examples) [Certain].
The prefix-LM mask
The input sequence is [image tokens] [BOS] [prefix] [\n] [suffix] [EOS], where the prefix is the task or question (“caption en”, “detect car”) and the suffix is the answer [Certain]. A normal decoder uses a causal mask: token \(i\) sees only tokens \(j \le i\). PaliGemma uses a prefix-LM mask (see session 18): the image and the prefix attend to each other in both directions, and only the suffix is causal:
Why? With a causal mask, the first image token can see nothing but itself, and image tokens can never look at the question. With the prefix-LM mask, image tokens can already “read” the question and focus on what is asked. The paper’s ablation found that causal masking of the prefix, or of the image and prefix, both work “but perform clearly worse” [Certain]. The loss is computed only on the suffix; adding it on the prefix also clearly reduced average quality [Certain].
Gemma 3: the same idea, in a general model
Gemma 3 (2025) adds vision to a general-purpose LLM family. The 4B, 12B and 27B models share one frozen SigLIP encoder of about 400M parameters (listed as 417M); the 1B model is text-only [Certain]. Every image is resized to 896×896, giving \(64 \times 64 = 4{,}096\) patches, and a 4×4 average pool reduces them to a fixed 256 “soft tokens” [Certain]:
A fixed square hurts on documents and non-square images. Gemma 3’s answer is Pan & Scan, an inference-time option that cuts the image into equal, non-overlapping crops, resizes each to 896, and encodes each one [Certain]. It can be turned off for speed. On the pretrained 4B model it raised DocVQA from 72.8 to 81.0 and InfoVQA from 44.1 to 57.0 [Certain]. In the attention mask, Gemma 3 lets image tokens attend to each other in both directions while text stays causal (this is how the Hugging Face implementation builds the mask [Certain]). Figure 5 compares the three masks.
How to read Figure 5
Click row “img1” in each mask. Causal: it sees only itself. Gemma 3: it sees all four image tokens but not the question. Prefix-LM: it sees the image and the question. Then click an answer row (“ans2”): it is the same in all three masks, because the answer must stay causal so that the model can generate it token by token. That is why the prefix-LM mask is free at inference: the prefix is processed once in prefill, and decoding is unchanged.
In counts: with 10 tokens, the causal mask allows 55 of the 100 pairs, Gemma 3’s mask 61, and the prefix-LM mask 76. The extra pairs are all inside the prefix, which is computed in one parallel prefill pass, so they cost almost nothing at serving time.
SmolVLM: design choices for small and edge models
SmolVLM (2025, Hugging Face) asks: what changes when the whole VLM must have 256M to 2.2B parameters and run on a laptop or phone? The three sizes are [Certain]:
| Model | Vision encoder | Language model | Tile / tokens per tile |
|---|---|---|---|
| SmolVLM-256M | SigLIP-B/16 (93M), 512 px | SmolLM2-135M | 512 px \(\to\) 1,024 patches \(\to\) 64 tokens (\(r = 4\)) |
| SmolVLM-500M | SigLIP-B/16 (93M), 512 px | SmolLM2-360M | 512 px \(\to\) 64 tokens (\(r = 4\)) |
| SmolVLM-2.2B | SigLIP So400m, 384 px | SmolLM2-1.7B | 384 px \(\to\) 729 patches \(\to\) 81 tokens (\(r = 3\)) |
The token count per tile follows the pixel-shuffle formula (182.6) with a larger ratio:
Images are split into tiles like Idefics3: the released preprocessor resizes the longest side to 2,048 pixels, rounds both sides up to multiples of 512, cuts 512 tiles and adds one downscaled global image [Certain]. A 1080p frame becomes 4×3 tiles plus the global image: \(13 \times 64 = 832\) image tokens (plus small row and column marker tokens). The 256M model runs inference in about 0.8 GB of GPU memory at batch size 1 [Certain].
The paper lists its findings explicitly [Certain]. The ones worth remembering:
- Balance encoder and LM. With a tiny LM, a 400M encoder is too big a share of the parameters; at the 135M and 360M LM sizes, the 93M encoder is the better trade.
- Compress harder when small. Small models do better with pixel shuffle \(r = 4\) than \(r = 2\) (the ratio used by InternVL and Idefics3). Fewer, richer tokens leave more of the short context for text.
- Longer context helps. Raising the RoPE base from 10k to 273k and the context to 16k (2.2B) or 8k (smaller models) helped a lot.
- Learned position tokens, not strings. Marking tiles with text strings like
<row_1_col_2>caused a training plateau (“OCR loss plague”); learned positional tokens fixed it. - Data details matter. Do not average video frames; mask the user prompt in SFT; reusing the LLM’s own text SFT data hurt (up to 3.7% on video and 6.5% on image tasks); a tiny share (0.02–0.05%) of chain-of-thought data helps, more hurts.
The result: the paper reports that SmolVLM-256M beats the 300 times larger Idefics-80B from 18 months earlier on most benchmarks (not on MMMU and AI2D) [Certain]. The lesson for edge work: the number of visual tokens and the context budget matter more than model size alone.
Same image, five models
Token count is the single number that best predicts the prefill cost and KV-cache size of a VLM request (session 183). Figure 6 computes it for one image with each model’s released preprocessing defaults.
How to read Figure 6
At 224² Qwen is the cheapest (64 tokens) because it does not upscale small images much, while PaliGemma-896 pays 4,096 for everything. At 4K, Qwen pays 10,549, while InternVL is capped at 2,304 and Gemma 3 at 256. Neither is “better”: fixed budgets are cheap but lose small text; native resolution keeps detail and pays for it. Raise the number of images to 32 (think of 32 video frames) and everything multiplies by 32; for video, Qwen2.5-VL would merge frame pairs and pay half. This is why video systems sample few frames and use small frame sizes. Note that each model’s prefill cost also depends on its LLM size.
Qwen2.5-VL: 1932×1092, \(39 \times 69 = 2{,}691\) tokens. InternVL 2.5: grid \(4 \times 2\) + thumbnail = 9 tiles, \(9 \times 256 = 2{,}304\). PaliGemma-896: 4,096. Gemma 3: 256 (or \(256\) per crop with Pan & Scan). SmolVLM-256M: 13 tiles, \(13 \times 64 = 832\). A 60 s clip at 2 fps with 640×360 frames in Qwen2.5-VL: 120 frames \(\to\) 60 pairs, \(60 \times 299 = 17{,}940\) tokens.
Code: the key mechanisms in one file
This script re-implements the parts that decide token counts and attention patterns: Qwen’s resize, windows and time ids, InternVL’s grid choice and pixel shuffle, the three masks, and Gemma 3’s pooling. It runs on CPU in a second and prints the numbers used on this page.
import math, torch torch.manual_seed(0) # 1) Qwen2.5-VL: native resolution, formula (182.1); sides become multiples of 28 (patch 14 x merge 2) def smart_resize(h, w, f=28, min_px=56*56, max_px=16384*28*28): hb, wb = round(h / f) * f, round(w / f) * f if hb * wb > max_px: # too big: shrink, keep the aspect ratio b = math.sqrt(h * w / max_px); hb, wb = math.floor(h / b / f) * f, math.floor(w / b / f) * f elif hb * wb < min_px: # too small: grow b = math.sqrt(min_px / (h * w)); hb, wb = math.ceil(h * b / f) * f, math.ceil(w * b / f) * f return hb, wb hb, wb = smart_resize(1080, 1920) patches = (hb // 14) * (wb // 14) print("resized", wb, "x", hb, "patches", patches, "LLM tokens", patches // 4) # (182.2) # 2) Window attention, formula (182.3): 8x8-patch windows (112 px) vs full attention gh, gw = hb // 28, wb // 28 # merged grid; one window = 4x4 merged = 8x8 patches sizes = [min(4, gh - a) * min(4, gw - b) * 4 for a in range(0, gh, 4) for b in range(0, gw, 4)] print("windows", len(sizes), "full/window scores", round(patches**2 / sum(s * s for s in sizes), 1)) # 3) M-RoPE time ids aligned to absolute time, formula (182.4) def time_ids(n_grids, fps, tokens_per_second=2, temporal_patch=2): sec_per_grid = temporal_patch / fps return [int(k * sec_per_grid * tokens_per_second) for k in range(n_grids)] print("fps 1:", time_ids(5, 1.0), " fps 2:", time_ids(5, 2.0)) # 4) InternVL dynamic tiling, formula (182.5): the grid (i, j) closest in shape to the image def pick_grid(w, h, max_num=12, size=448): ratios = sorted({(i, j) for n in range(1, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if i * j <= max_num}, key=lambda r: r[0] * r[1]) best, grid = float("inf"), (1, 1) for i, j in ratios: d = abs(w / h - i / j) if d < best: best, grid = d, (i, j) elif d == best and w * h > 0.5 * size * size * i * j: grid = (i, j) # tie: more tiles return len(ratios), grid n_ratios, (i, j) = pick_grid(1920, 1080) tiles = i * j + (1 if i * j > 1 else 0) # + one thumbnail of the whole image print("candidate grids", n_ratios, "grid", (i, j), "tiles", tiles, "tokens", tiles * 256) # 5) Pixel shuffle (space-to-depth), formula (182.6): (B, 32, 32, C) -> (B, 16, 16, 4C) def pixel_shuffle(x, r=2): B, H, W, C = x.shape x = x.view(B, H // r, r, W // r, r, C).permute(0, 1, 3, 2, 4, 5) return x.reshape(B, H // r, W // r, r * r * C) feat = torch.randn(1, 32, 32, 1024) # one 448 px tile from InternViT-300M print("pixel shuffle", tuple(pixel_shuffle(feat).shape), "SmolVLM r=4:", tuple(pixel_shuffle(torch.randn(1, 32, 32, 768), 4).shape)) # 6) Masks, formula (182.7): causal, PaliGemma prefix-LM, Gemma 3 (image tokens bidirectional) I, P, S = 4, 3, 3; n = I + P + S causal = torch.tril(torch.ones(n, n, dtype=torch.bool)) prefix = causal.clone(); prefix[:, : I + P] = True # every token sees the whole prefix gemma3 = causal.clone(); gemma3[:I, :I] = True # the image block is full print("allowed pairs: causal", causal.sum().item(), "prefix-LM", prefix.sum().item(), "gemma3", gemma3.sum().item()) # 7) Gemma 3, formula (182.8): 64x64 SigLIP patches -> 4x4 average pool -> 256 soft tokens v = torch.randn(1, 1152, 64, 64) print("gemma3 tokens", torch.nn.functional.avg_pool2d(v, 4).flatten(2).shape[-1]) # Output: # resized 1932 x 1092 patches 10764 LLM tokens 2691 # windows 180 full/window scores 173.4 # fps 1: [0, 4, 8, 12, 16] fps 2: [0, 2, 4, 6, 8] # candidate grids 35 grid (4, 2) tiles 9 tokens 2304 # pixel shuffle (1, 16, 16, 4096) SmolVLM r=4: (1, 8, 8, 12288) # allowed pairs: causal 55 prefix-LM 76 gemma3 61 # gemma3 tokens 256
Our pixel_shuffle orders the stacked channels differently from InternVL’s own code, but the shapes and the information are the same; with a learned MLP after it, the order does not matter. To run the real models, use the Hugging Face classes Qwen2_5_VLForConditionalGeneration, PaliGemmaForConditionalGeneration, Gemma3ForConditionalGeneration and AutoModelForVision2Seq (SmolVLM); InternVL 2.5 loads with trust_remote_code=True [Certain].
Where this is used
| Task | Good fit | Why |
|---|---|---|
| Key-moment detection, temporal grounding (“when does the goal happen?”) | Qwen2.5-VL | Time ids aligned to seconds (182.4) and dynamic-fps training; it can answer with timestamps |
| Video question answering and captioning on long clips | Qwen2.5-VL with a token cap; InternVL with one tile per frame | Per-frame cost is the knob: 299 tokens per 360p frame pair in Qwen, 256 per frame in InternVL |
| Documents, charts, screenshots, OCR | Qwen2.5-VL (native), InternVL (tiles), Gemma 3 with Pan & Scan | Small text needs many tokens; fixed 896 squares lose it (Pan & Scan: +8.2 DocVQA on 4B) |
| Detection and grounding with boxes | Qwen2.5-VL (absolute pixel boxes), PaliGemma (“detect” prefix, then fine-tune) | Both output boxes as text tokens; PaliGemma is built to be fine-tuned per task |
| Fine-tuning a specialist (one task, one domain) | PaliGemma / PaliGemma 2 | Base model without chat tuning, prefix-LM mask, simple linear connector |
| Video retrieval and Video-RAG | Encoder-level embeddings (SigLIP) for search, then a VLM such as Qwen2.5-VL to answer over the retrieved clips | Embedding every frame with a full VLM is too expensive; generate only for the top hits |
| On-device, edge, or high-volume cheap captioning | SmolVLM-256M / 500M | About 0.8 GB GPU memory; 64 tokens per 512 tile |
Common mistakes
- Leaving Qwen’s
max_pixelsat the default when serving. A 4K screenshot then costs 10,549 visual tokens. Set a cap that matches your task (for example the 1,280-token setting) and measure accuracy. - Mixing up merged tokens and patches. In Qwen2.5-VL, window attention and the 4 full layers are counted in ViT patches; the LLM sees 4 times fewer tokens. A 1080p frame is 10,764 patches but 2,691 tokens.
- Using a fixed time index for video. If you sample frames yourself, pass the real frame rate (
fps/second_per_grid_ts) to Qwen2.5-VL; otherwise the time ids assume 1 s per frame pair and timestamps in the answer will be wrong. - Normalizing boxes for Qwen2.5-VL. It was trained with absolute pixel coordinates of the resized image; 0–1000 normalized boxes (the Qwen2-VL habit) give wrong outputs.
- Fine-tuning PaliGemma with a causal mask or loss on the prompt. Pass
token_type_ids(or use the processor with a suffix) so the prefix is bidirectional and only the suffix gets loss. - Using too many InternVL tiles for multi-image or video input. The tile budget is per sample; 13 tiles (12 + thumbnail) for each of 16 images is \(13 \times 256 \times 16 = 53{,}248\) tokens. Lower
max_numto 1 per frame for video. - Expecting Gemma 3 to read small text without Pan & Scan. Every image becomes 256 tokens of an 896 square; turn on Pan & Scan for documents.
- Comparing models by parameter count only. Prefill compute is roughly \(2 \times\) (LM parameters) \(\times\) (tokens). A model with half the parameters but four times the visual tokens per image does about twice the prefill work. Compare tokens \(\times\) LM size.
Interview questions
How many visual tokens does Qwen2.5-VL produce for a 1920×1080 image, and how do you compute it?basic
max_pixels is lower, the image is first scaled down keeping the aspect ratio; with a 1,280-token cap it becomes 1316×728 and 1,222 tokens.Why does Qwen2.5-VL use window attention in its ViT, and why keep four full-attention layers?core
What is M-RoPE, and what did Qwen2.5-VL change about it for video?core
Explain InternVL’s dynamic tiling. What does the thumbnail do?core
Native resolution (Qwen) vs tiling (InternVL): trade-offs?advanced
What is a prefix-LM mask and why does PaliGemma use it?core
Gemma 3 makes every image 256 tokens. How does it handle documents, and what does that cost?core
Why did InternVL scale the vision encoder to 6B, and why do its small models use 300M?advanced
You must run a VLM on a phone to caption photos. What design choices do you make?advanced
How does each model know where a visual token is in the image?core
A 60 s video at 2 fps, 640×360 frames. How many tokens in Qwen2.5-VL, and how would you cut it in half?basic
max_pixels so each pair has about 150 tokens. Which one is better depends on whether the task needs fine time detail (fast actions) or fine spatial detail (text on screen).Exercises
- Compute Qwen2.5-VL’s token count for an 800×600 image with default limits. How many ViT patches is that?
- With \(8 \times 8\)-patch windows, how many windows does a 448×448 image have in Qwen2.5-VL, and how many times fewer scores does a window layer compute than a full layer?
- A video is sampled at 0.5 fps. What are the time ids of the first four frame pairs? Which pair contains second 10, and what is its id?
- Which grid does InternVL pick for an 800×600 image with \(n_{\max} = 12\)? How many tokens does it produce?
- In Figure 5’s toy sequence, how many allowed pairs does the prefix-LM mask have if the prefix has 2 text tokens instead of 3 (still 4 image and 3 suffix tokens)?
- SmolVLM-2.2B uses a 384 px tile, patch 14 and \(r = 3\). What is the vector size per token after the shuffle if the encoder width is 1,152?
Answerscheck
Key takeaways
- All four families use encoder → connector → LLM in one sequence; they differ in how an image becomes a grid, how tokens are reduced, how positions are encoded, and which mask is used.
- Qwen2.5-VL: native resolution with sides rounded to multiples of 28 (182.1, 182.2), window attention with 4 full layers to keep the ViT linear in patches (182.3), and time ids equal to twice the seconds (182.4). 1080p = 2,691 tokens.
- InternVL: a large encoder (6B, or 300M for small models), 448 tiles chosen by aspect ratio plus a thumbnail (182.5), and \(2 \times 2\) pixel shuffle to 256 tokens per tile (182.6). 1080p = 2,304 tokens.
- PaliGemma: SigLIP + linear + Gemma, a fixed square (256 / 1,024 / 4,096 tokens) and a prefix-LM mask (182.7); a base model for fine-tuning. Gemma 3: frozen SigLIP, 256 pooled tokens per 896 crop (182.8), Pan & Scan for documents.
- SmolVLM: balance encoder and LM, compress harder (\(r = 4\), 64 tokens per tile, 182.9), longer context, learned tile tokens; the 256M model runs in about 0.8 GB.
Key formulas
References for this session
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Chen et al. (2023). InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
- Gemma Team (2025). Gemma 3 Technical Report
- Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
- Bai et al. (2025). Qwen2.5-VL Technical Report
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Chen et al. (2023). InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- Chen et al. (2024). How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites (InternVL 1.5)
- Chen et al. (2024). Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (InternVL 2.5)
- Zhu et al. (2025). InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
- Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
- Steiner et al. (2024). PaliGemma 2: A Family of Versatile VLMs for Transfer
- Gemma Team (2025). Gemma 3 Technical Report
- Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
- Dehghani et al. (2023). Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
- Laurençon et al. (2024). What matters when building vision-language models?
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models