Visual token efficiency
One image can become hundreds or thousands of tokens, and a short video can become tens of thousands. These visual tokens usually far outnumber the text tokens, so they decide most of the prefill time and the KV-cache memory of a VLM. This session counts that cost, then builds the main ways to cut it: token merging (ToMe), pruning by attention inside the language model (FastV), fixed-budget resamplers and dynamic token counts. It ends with how to measure the accuracy versus cost trade-off without fooling yourself.
What you will learn
- Why visual tokens dominate the cost of a VLM: token counts, prefill FLOPs and KV-cache memory for real models (LLaVA-1.5, LLaVA-NeXT, Qwen2-VL), and what video does to them
- The three places to remove tokens: inside the vision encoder, in the connector, and inside the language model
- Token merging (ToMe): bipartite soft matching on keys, size-weighted averages and proportional attention, with the token schedule and its FLOPs
- Pruning by attention (FastV): rank image tokens by the attention they receive at layer K, drop R% of them, and the theoretical FLOPs ratio
- Fixed-budget resamplers (Q-Former, Perceiver) versus dynamic token counts (AnyRes tiling, Qwen2-VL native resolution, Matryoshka tokens)
- How to measure the trade-off honestly: wall-clock latency with FlashAttention, end-to-end time, random and pooling baselines at the same budget, and vision-centric benchmarks
Why this matters
Ask LLaVA-1.5 “What is in this picture?”. The question is about 10 text tokens. The picture is 576 tokens. The language model spends more than 90% of its prefill work on the image, and the image also fills more than 90% of the KV cache. For a video it is worse: 32 frames at 576 tokens each are 18,432 tokens, before the user has typed a word.
A visual token is one vector that the language model reads in place of a piece of the image (one patch, or a merged group of patches). In session 177 we counted them with \(V = (H/P)(W/P)/r^2\). In session 180 we saw that reading documents and charts needs more of them: tiling and native resolution push one image to thousands of tokens. This session is the other side of the same coin. Visual tokens are expensive, many of them carry little information (sky, wall, white paper), and there are good ways to remove them.
The ideas here are used in real systems. Fixed-budget resamplers (BLIP-2’s 32 queries, Flamingo’s and Idefics2’s 64 latents) were designed to cap the token count. Pixel shuffle (a 2 × 2 merge) is used in Qwen2-VL, InternVL, Idefics3 and SmolVLM (session 178). Training-free methods such as ToMe and FastV can be added to a model that is already trained. Long-video systems (session 206) depend on token compression to fit hours of video into a context window.
The plan: first we count the cost (Figure 1); then we see the three places where tokens can be removed; then token merging (Figures 2 and 3) and pruning by attention (Figures 4 and 5), each first animated step by step on a tiny example and then explored at full size; then fixed versus dynamic budgets (Figure 6); and finally how to measure the gain without fooling yourself (Figure 7). The main survey of this area is Jin et al. (2024), and the book chapter on video-language models (Noyan et al., chapter 9) shows the same problem for video.
Where the cost goes
Let a request have \(V\) visual tokens and \(T\) text tokens, so the language model reads \(n = V + T\) tokens in prefill. The visual share of the input is
With 576 visual tokens and 40 text tokens, \(f_{\text{vis}} = 93.5\%\). Two costs grow with \(n\).
Prefill compute
In prefill (the first forward pass over the whole prompt, session 78), every token passes through every layer. For one Llama-style layer with hidden width \(d\), MLP width \(m\) and a SwiGLU MLP (three weight matrices), the FLOPs are about
- Read the symbols
- \(n\)
- the number of tokens in the prompt (visual plus text)
- \(d\)
- the hidden width of the language model (4,096 for a 7B Llama)
- \(m\)
- the inner width of the MLP (11,008 for Llama 2 7B and Vicuna-7B)
- \(4d^2\)
- the four attention projections \(W_Q, W_K, W_V, W_O\), each \(d \times d\), one multiply-add each (2 FLOPs)
- \(3dm\)
- the three MLP matrices (gate, up, down)
- \(4n^2 d\)
- the score matrix \(QK^\top\) and the weighted sum with \(V\), each \(n^2 d\) multiply-adds (we ignore the causal mask, which halves this)
The first term is linear in \(n\): every token costs the same. The second term is quadratic and only matters when \(n\) is large. Multiply by the number of layers \(L\) for the whole model. Each visual token costs exactly as much as a text token, so removing visual tokens is the most direct way to cut prefill.
KV-cache memory
After prefill, the model keeps the keys and values of every token in every layer (session 80). The memory is
- Read the symbols
- \(2\)
- one key and one value per token per layer
- \(L\)
- the number of layers (32 for a 7B Llama)
- \(n_{\text{kv}}, d_h\)
- the number of key-value heads and their width; Vicuna-7B uses full multi-head attention, so \(n_{\text{kv}} d_h = 32 \times 128 = 4{,}096\)
- \(b\)
- bytes per number (2 for FP16 or BF16)
Every decode step reads this whole cache from memory, so a larger cache also makes each generated token slower and leaves less room for other users in the batch.
Tokens. 576 visual + 40 text = 616 tokens. Visual share: \(576/616 = 93.5\%\).
Prefill. With \(d = 4096\), \(m = 11008\), \(L = 32\): \(32 \cdot C(616) = 8.18 \times 10^{12}\) FLOPs. The 40 text tokens alone would cost \(32 \cdot C(40) = 0.52 \times 10^{12}\). So the image adds \(93.7\%\) of the prefill work. At this length the quadratic attention term is only about 2.4% of \(C(n)\); the cost is almost all in the weight matrices, and so almost exactly proportional to the number of tokens.
KV cache. \(2 \cdot 32 \cdot 4096 \cdot 2 = 524{,}288\) bytes = 0.5 MiB per token. The image costs \(576 \times 0.5 = 288\) MiB per request.
Video. 32 frames at 576 tokens: 18,432 visual tokens, \(4.18 \times 10^{14}\) prefill FLOPs, and 9 GiB of KV cache for one request. Now the attention term is 43% of \(C(n)\): long inputs make the quadratic part matter.
Use Figure 1 to try other models. All of them run through the same 7B language model shape here, so that only the token count changes.
How to read Figure 1
The first bar is the share of tokens, the second is the share of prefill FLOPs that the images add (formula 183.2 with and without them), and the third is the KV-cache memory. Start with LLaVA-1.5 and one image, then move the frames slider to 32: the visual part fills almost everything and the KV cache passes 9 GiB. Then pick Q-Former: 32 tokens per image makes even 64 frames cheap. That is why fixed-budget resamplers were popular for video. Raise the text tokens to 2,000 (a long document question) and see that a single LLaVA-1.5 image becomes a minority.
Three places to remove tokens
A visual token passes through three parts of the model: the vision encoder, the connector, and the language model (session 177). You can remove tokens in any of them, and the choice decides what you save and what you risk.
- Inside the vision encoder. Merge or drop patch tokens between ViT layers. This saves encoder compute and, if the reduced set is passed on, also the language model’s work. Example: ToMe. The language model never sees the dropped tokens, and the reduction does not know the user’s question.
- In the connector. Map the \(N\) patch vectors to fewer tokens: pool or merge neighbors (pixel shuffle, 2 × 2 average pooling), or pull out a fixed number \(Q\) with learned queries (Q-Former, Perceiver resampler). This is usually part of the design and trained with the model. It also does not know the question (except InstructBLIP-style Q-Formers that also read the instruction).
- Inside the language model. Run the first few layers on all tokens, then drop visual tokens that the model is not using. Example: FastV. This can use the question, because by layer \(K\) the text tokens have already looked at the image. But the first \(K\) layers still pay full price, and the encoder is not made cheaper.
There is a second axis: merge or prune. Pruning deletes a token and its information. Merging replaces several tokens by one average, so the information is blurred but not lost. Most methods are a mix: LLaVA-PruMerge, for example, keeps the tokens that the encoder’s class token attends to most, and merges the information of the pruned tokens into the kept ones, reaching about 14× fewer visual tokens on LLaVA-1.5 [Certain, from the paper’s abstract].
Token merging (ToMe)
Token merging (ToMe, Bolya et al., 2022) speeds up a trained ViT without retraining. In every block, between the attention and the MLP, it merges the \(r\) most similar pairs of tokens. After \(L\) layers, \(rL\) tokens are gone. The trick is to find similar pairs quickly: comparing all pairs, or clustering, would be too slow to do in every layer.
Bipartite soft matching
ToMe splits the current tokens into two sets by position: \(\mathcal{A}\) holds tokens 0, 2, 4, …; \(\mathcal{B}\) holds tokens 1, 3, 5, …. Then:
- Read the symbols
- \(\mathbf{k}_i\)
- the key vector of token \(i\) in this layer’s attention, averaged over heads; keys already summarize “what this token contains”, so no extra computation is needed
- \(s_{ij}\)
- cosine similarity between token \(i\) in \(\mathcal{A}\) and token \(j\) in \(\mathcal{B}\)
- \(j^*(i)\)
- the best partner in \(\mathcal{B}\) for token \(i\)
Each \(\mathcal{A}\) token keeps only one edge, to its best partner. ToMe sorts these edges by similarity and merges the top \(r\) of them; the other \(\mathcal{A}\) tokens stay. Because every edge goes from \(\mathcal{A}\) to \(\mathcal{B}\), no chains form, and the whole step is a matrix product plus a sort. At most \(|\mathcal{A}|\), about half the tokens, can be merged in one layer. The class token is protected (it is never merged).
Size-weighted merging and proportional attention
When token \(a\) merges into token \(b\), the result is the average weighted by size, where the size \(s\) counts how many original patches a token stands for:
A merged token is now one vector in place of several. In attention it would get the weight of one token, which is less than the original group had. ToMe fixes this with proportional attention: add \(\log s\) to the attention logits of each key,
Why \(\log s\)? Because \(e^{z + \log s} = s\, e^{z}\): a key that stands for \(s\) identical tokens gets exactly \(s\) times the weight, as if the copies were still there. The code below checks this: merging two identical tokens with proportional attention gives the same output as not merging, to about \(10^{-7}\).
- input: tokens \(\mathbf{x}_1..\mathbf{x}_N\), sizes \(s_1..s_N\), merge count \(r\)
- run attention with the \(\log \mathbf{s}\) bias (183.6); keep the keys \(\mathbf{k}\), averaged over heads
- \(\mathcal{A} \leftarrow\) even positions, \(\mathcal{B} \leftarrow\) odd positions; protect the class token
- for each \(i \in \mathcal{A}\): \(j^*(i) \leftarrow \argmax_j s_{ij}\), \(\;e_i \leftarrow \max_j s_{ij}\) // formula 183.4
- \(r \leftarrow \min(r, |\mathcal{A}|)\); merge the \(r\) tokens of \(\mathcal{A}\) with the largest \(e_i\) into their \(j^*(i)\) // formula 183.5
- concatenate the unmerged \(\mathcal{A}\) tokens and all \(\mathcal{B}\) tokens: \(N - r\) tokens
- run the MLP on the \(N - r\) tokens
Figure 2 runs Algorithm 1 in your browser on a tiny image of 16 patches (4 × 4): a sky with a sun at the top, grass with a small red object at the bottom. Each token’s feature, and so its key, is its color minus 0.5 (three numbers). Every similarity, every argmax and every weighted average you see is computed live with formulas (183.4) and (183.5).
- input: tokens \(\mathbf{x}\), sizes \(\mathbf{s}\), merge count \(r\)
- attention with the \(\log \mathbf{s}\) bias; keys \(\mathbf{k} \leftarrow\) features // here: color − 0.5
- \(\mathcal{A} \leftarrow\) even positions, \(\mathcal{B} \leftarrow\) odd positions
- for \(i \in \mathcal{A}\): \(j^*(i) \leftarrow \argmax_j s_{ij}\), \(e_i \leftarrow \max_j s_{ij}\) // (183.4)
- \(r \leftarrow \min(r, |\mathcal{A}|)\); pick the \(r\) largest \(e_i\)
- \(\mathbf{x}_b \leftarrow \frac{s_a \mathbf{x}_a + s_b \mathbf{x}_b}{s_a + s_b}\), \(s_b \leftarrow s_a + s_b\) // (183.5)
- keep \(N - r\) tokens; run the MLP on them // (183.7)
How to read Figure 2
Press Play, or use Step and Back. With \(r = 4\), layer 1 merges four sky or grass pairs, all with similarity 0.997 or more, and the token count goes \(16 \to 12\). The sun token’s best match has \(e = 0.142\) and the red object’s only \(-0.039\), so they are never chosen while better pairs exist. Layer 2 gives 8 tokens with almost no error (0.0003). In layer 3 only two good pairs are left (0.993 and 0.985), but \(r = 4\) forces two more merges: both sky tokens are pulled into the sun token (\(e = -0.112\) and \(-0.220\)) and the error jumps to 0.0448. In layer 4 only \(|\mathcal{A}| = 2\) merges are possible, so the count stops at 2: the red object and everything else (error 0.1481). The four layers did 47.9% of the work of four unmerged layers. Now set \(r = 2\): after four layers 8 tokens remain, every one of them a clean region, with error 0.0002 and 72.9% of the work. Too large an \(r\) merges things that are not alike.
The token schedule
With a constant \(r\), layer \(\ell\) ends with \(N_\ell = N_0 - \ell r\) tokens (until half of the tokens can no longer be merged). The total cost is the sum of the per-layer costs:
Here \(C_{\text{ViT}}\) is a ViT layer’s cost with attention on \(N_{\ell-1}\) tokens and the MLP on \(N_\ell\) tokens (merging happens in between).
CLIP ViT-L/14 at 336 px has \(N_0 = 577\) tokens (576 patches and a class token), 24 layers, width 1,024 and MLP width 4,096. With \(r = 16\) per layer, the last layer ends with \(577 - 24 \times 16 = 193\) tokens. Counting each layer as \(2N_{\ell-1} \cdot 4d^2 + 4N_{\ell-1}^2 d\) for attention and \(2N_\ell \cdot 2dm\) for the MLP, the encoder needs 64.9% of its original FLOPs (about 1.54× fewer). With \(r = 8\): 385 tokens and 82.1%.
The bigger saving is downstream: if the 192 merged patch tokens are passed to the language model in place of 576, its prefill work falls to about one third. But the language model was trained on a regular 24 × 24 grid, and merged tokens no longer sit on a grid, so without fine-tuning accuracy can drop more than in the encoder alone [Likely].
ToMe reported about 2× the throughput of ViT-L at 512 px and ViT-H at 518 px off the shelf with a 0.2–0.3% accuracy drop on ImageNet, and 2.2× for ViT-L on video, where it merges similar tokens across frames too [Certain, from the paper]. Video is where merging shines: a static background repeated over 16 frames is exactly the kind of redundancy it removes.
How to read Figure 3
With ToMe matching, sky merges with sky and grass with grass, and the small red object and the sun keep their own tokens for a long time: the error stays small while the count falls. Switch to random pairs with the same count: patches of different regions are averaged together and the error jumps. Push \(r\) above half the tokens: the count stops at what one bipartite step can merge (at most half per layer). In a real ViT the features change from layer to layer; here they do not, so the picture shows only the merging rule.
Pruning by attention inside the language model (FastV)
FastV (Chen et al., 2024) looked at the attention maps of LLaVA-1.5 and found something surprising. In the first two layers, image tokens receive a fair share of attention. After layer 2, an average image token receives only about 0.21% of the attention that an average system-prompt token receives, even though image tokens were about 64% of the input tokens in their tests [Certain, from the paper]. In deep layers the model mostly ignores most image tokens. So why keep computing them?
The rule
Run the language model normally up to layer \(K\). At layer \(K\), compute how much attention each image token receives, averaged over heads and over all query tokens:
- Read the symbols
- \(A^{(K)}_{h,i,j}\)
- the attention probability from query token \(i\) to key token \(j\) in head \(h\) of layer \(K\)
- \(H\), \(n\)
- the number of heads and the number of tokens
- \(a_j\)
- the importance score of image token \(j\): how much the rest of the sequence looks at it
Keep the top \((1 - R)\) fraction of image tokens by \(a_j\), drop the rest, and run layers \(K+1\) to \(L\) on the shorter sequence. Text tokens are never dropped. The dropped tokens also leave the KV cache of those layers, so decoding gets cheaper too. No training is needed.
- input: sequence of \(n = n_{\text{sys}} + V + T\) tokens, filter layer \(K\), drop ratio \(R\)
- run layers \(1..K\) on all \(n\) tokens; at layer \(K\), also return the attention probabilities
- \(a_j \leftarrow\) mean over heads and queries of the attention to image token \(j\) // formula 183.8
- keep \(\lceil (1-R)\,V \rceil\) image tokens with the largest \(a_j\), in their original order, with their original position ids
- run layers \(K+1..L\) on \(\hat n = n_{\text{sys}} + (1-R)V + T\) tokens
- decode as usual; layers after \(K\) keep only \(\hat n\) entries in the KV cache
What it saves, in theory
\(\rho\) is the fraction of prefill FLOPs that remains, with \(C\) from formula (183.2). The KV cache shrinks by the same kind of ratio, with \(n\) in place of \(C(n)\).
LLaVA-1.5-7B, 576 image tokens and 40 other tokens, so \(n = 616\) and \(\hat n = 40 + 288 = 328\). With \(L = 32\): \(\rho = (2\,C(616) + 30\,C(328)) / (32\,C(616)) = 0.556\). Prefill FLOPs fall by about 44%. The KV cache falls to \((2 \cdot 616 + 30 \cdot 328)/(32 \cdot 616) = 56.2\%\).
For LLaVA-1.5-13B (\(d = 5120\), \(m = 13824\), \(L = 40\)) the same setting gives \(\rho = 0.551\). The paper reports about 45% fewer FLOPs for this model with no loss on its benchmark average (73.6 before and after on captioning, A-OKVQA and MMMU) [Certain, from the paper]. Its own FLOPs formula is slightly different, so do not expect every digit to match.
With \(R = 75\%\): \(\hat n = 184\) and \(\rho = 0.338\). Moving the filter to \(K = 16\) with \(R = 50\%\) keeps \(\rho = 0.763\): late pruning saves little, because half the layers already paid full price.
Figure 4 runs Algorithm 2 on a tiny prompt: 2 system tokens, a 4 × 4 image (16 tokens) and a 4-token question, so \(n = 22\). Two attention heads with 4-number queries and keys are given (fixed numbers, not from a trained model). Four image tokens hold two objects the question asks about: two in the top half, two in the bottom half. The system tokens have large keys that every query matches, so they act as attention sinks (tokens that absorb much of the attention whatever the input), as FastV observed in real models. To show the position effect, the logits get a distance penalty \(-\lambda (i - j)\), a simple stand-in for the long-distance decay of RoPE. Every attention probability and score is computed in your browser.
- input: \(n = 2 + 16 + 4\) tokens, filter layer \(K\), drop ratio \(R\)
- run layers \(1..K\) on all \(n\) tokens
- layer \(K\), head \(h\), row \(i\): \(A_{h,i,j} = \softmax_j\!\big(\mathbf{q}_i \cdot \mathbf{k}_j / \sqrt{d_h} - \lambda (i - j)\big)\), \(j \le i\)
- \(a_j \leftarrow \frac{1}{Hn} \sum_h \sum_i A_{h,i,j}\) for image tokens // (183.8)
- keep the \((1-R)V\) image tokens with the largest \(a_j\), in order
- run layers \(K+1..L\) on \(\hat n\) tokens // (183.9)
- layers \(K+1..L\) keep \(\hat n\) entries in the KV cache
How to read Figure 4
Press Play. Watch the first two columns of the matrix: the two system tokens take a large share of every row, and their scores end near 0.18 and 0.20 at \(\lambda = 0.15\), against at most 0.055 for an image token. With \(\lambda = 0.15\) and \(R = 75\%\), the 4 kept tokens are exactly the 4 object tokens. Now set \(\lambda = 0\) and replay: the kept tokens are all in the top half (tokens 1, 2, 3 and 6 of the image), because an early token is seen by more query rows of a causal mask, so its sum collects more. Set \(\lambda = 0.3\): the decay now favors tokens close to the question, and 3 of the 4 kept tokens are in the bottom half. The score is partly content and partly position. On FLOPs: with \(K = 2\) and \(R = 75\%\) the toy keeps 48.8% of the prefill FLOPs, and at real size the same setting keeps 33.8%, as in the worked example. The toy saves less because its 6 non-image tokens are a larger part of its 22 tokens.
How to read Figure 5
The FLOPs ratio comes from formula (183.9) with the real layer shapes of each model. Move \(K\) to the right and the savings shrink quickly. Raise \(R\) and the savings grow, but the map on the right gets thin. Now raise the position bias: the kept tokens slide to the bottom rows. Real attention scores in a RoPE language model have this bias, because RoPE makes attention decay with distance and the bottom rows of the image sit closest to the text that follows. Then a question about the top of the image loses its evidence.
The practical catches
- It needs the attention matrix. Fast kernels such as FlashAttention never write the \(n \times n\) attention probabilities to memory (session 124). FastV needs them at layer \(K\), so that layer must run a slower path or recompute the scores. Wen et al. (2025) report that FastV, SparseVLM and MustDrop do not support FlashAttention, and that SparseVLM had only 2.8% more FLOPs than FastV but 26.8% more latency [Certain, from the paper].
- Position bias. Endo et al. (2024) found that pruning after a shallow layer keeps tokens concentrated at the bottom of the image, and traced it to the long-distance decay of RoPE [Certain, from the paper]. They also found that early-layer pruning keeps good scores on many benchmarks but fails on localization tasks.
- The first \(K\) layers and the encoder still pay in full. With small \(K\) this is a small part of prefill, but the vision encoder cost is untouched.
- It interacts with batching. Different requests keep different tokens, which is fine for a serving engine with paged KV memory, but harder for code that assumes fixed shapes.
Fixed budgets and dynamic token counts
Pruning and merging change a model after the fact. The connector decides the token count by design (session 178). There are two families.
A fixed-budget resampler uses \(Q\) learned query vectors that cross-attend to all patch features and return exactly \(Q\) tokens, whatever the image size. BLIP-2’s Q-Former uses 32 queries; Flamingo’s Perceiver resampler and Idefics2 use 64 latents [Certain]. The cost is constant and easy to plan for, and a video of 64 frames at 32 tokens each is only 2,048 tokens. The risk is the other side of the same fact: a dense page of text needs more than 32 or 64 vectors, so fine detail is lost.
A dynamic token count grows with the image. LLaVA-NeXT picks the best grid of 336 px tiles for the image’s aspect ratio and adds a low-resolution overview, so a 672 × 672 grid costs \(5 \times 576 = 2{,}880\) tokens before padding is removed [Certain, from the config and the tiling rule]. Qwen2-VL resizes the image to multiples of 28 pixels inside a pixel budget and makes one token per 28 × 28 area (14 px patches, then a 2 × 2 merge):
- Read the symbols
- \(Q\)
- the number of learned queries of a resampler (32 or 64 in the models above)
- \(H', W'\)
- the image size after Qwen2-VL rounds each side to a multiple of 28 and scales it into the pixel budget
- \([3{,}136,\ 12{,}845{,}056]\)
- the default
min_pixelsandmax_pixelsin Qwen2-VL’s preprocessor config, which give between 4 and 16,384 tokens per image [Certain]
A 1920 × 1080 frame becomes 1932 × 1092 and costs \(69 \times 39 = 2{,}691\) tokens. A 4K frame costs 10,549. The user can lower max_pixels to cap the cost: with \(1280 \times 28 \times 28\) pixels, the same 4K frame costs 1,222 tokens. A dynamic count spends tokens where there are pixels, which is right for documents. But the cost of a request is no longer known before you see the image, which matters for serving and for batching (session 84).
A middle path is nested tokens. Matryoshka Multimodal Models (M3, Cai et al., 2024) train one model to work with several pooled versions of LLaVA’s 24 × 24 grid, from coarse to fine, so the user can choose the number of tokens per request. They report that COCO-style benchmarks need only about 9 tokens to reach accuracy close to using all 576 [Certain, from the paper], which says as much about those benchmarks as about the method.
How to read Figure 6
At 1920 × 1080, Qwen2-VL spends 2,691 tokens and LLaVA-NeXT 2,880, while the Q-Former spends 32. Choose 3:1 and LLaVA-NeXT’s best grid becomes 1008 × 336 (three tiles plus the overview, 2,304 tokens). Slide the long side down to 448: Qwen2-VL now spends fewer tokens than LLaVA-1.5, because a small image has few pixels. Dynamic counts are cheap for small images and expensive for large ones; fixed budgets are the same price for both.
Measuring the trade-off honestly
A token-reduction paper usually shows one number: “we keep 25% of the tokens and lose only 1 point”. That number can be true and still misleading. The pitfalls are the same as in any efficiency work (session 82), plus some special to vision.
Measure time, not only FLOPs
FLOPs are a proxy. The real goal is latency (time to first token, time per output token) and throughput (requests per second on a fixed GPU budget). Only the prefill part shrinks with visual tokens; the vision encoder, the decode steps and any overhead of the method do not shrink, or shrink less. This is Amdahl’s law:
- Read the symbols
- \(e, p\)
- the fractions of the original end-to-end time spent in the vision encoder and in the LLM prefill; the rest, \(1 - e - p\), is decoding
- \(\rho\)
- the fraction of prefill time that remains after reduction (formula 183.9, if prefill is compute-bound)
- \(\delta\)
- the fraction of decode time that remains (a smaller KV cache helps a little)
- \(o\)
- the extra time the method adds (scoring, sorting, a non-flash attention layer), as a fraction of the original time
How to read Figure 7
With the defaults, the method removes 44.5% of the prefill FLOPs, a “1.80× faster prefill”, but the whole request is only about 1.25× faster, because the encoder and the decoding of the answer do not shrink much. Move \(p\) up to 80% (a long video with a one-word answer): now the same \(\rho\) gives a much larger speedup. Move \(o\) up to 15% (a method that breaks FlashAttention in a layer, or sorts on the CPU): the speedup can fall below 1.
Compare at equal budgets, against simple baselines
The most important control is boring: at the same number of kept tokens, compare against random dropping and plain average pooling. Wen et al. (2025) did this on LLaVA-1.5-7B across several benchmarks. As a percentage of the full 576-token model’s average score [Certain, Table 1 of the paper]:
| Method | Keep 144 tokens (−75%) | Keep 64 tokens (−88.9%) |
|---|---|---|
| Random dropping | 95.0% | 89.1% |
| 2D average pooling | 96.4% | 87.6% |
| FastV | 89.8% | 78.2% |
| SparseVLM | 93.5% | 87.3% |
At these aggressive budgets, the attention-based methods were not better than the two simple baselines. This does not mean attention scores are useless; FastV was designed for \(R = 50\%\), where it loses little. It means a method must beat random and pooling at the same budget before its scoring rule gets credit. Random dropping and pooling also have a hidden advantage: they cover the whole image evenly, which a position-biased score does not.
Choose benchmarks that need the pixels
Many benchmark questions can be answered from a blurry thumbnail, or even without the image. Token reduction looks free on those. Test where detail matters: text reading (TextVQA, DocVQA, ChartQA), counting, and localization (referring expressions, grounding boxes), and for video, questions about short events and exact timestamps (session 192). Endo et al. found exactly this split: early pruning kept good scores on many benchmarks but failed on localization [Certain, from the paper].
A checklist
- fix the model, the prompt template, the decoding settings and the hardware; use the fastest attention kernel for the baseline too
- for each budget \(V' \in \{V, V/2, V/4, V/9\}\): run your method, random dropping and average pooling // equal tokens
- score on a detail-heavy set (text reading, counting, grounding, short video events) and a general set
- measure time to first token, time per output token, peak memory and throughput at your real batch size
- plot accuracy against measured time (a Pareto curve), not against kept tokens alone
- report the encoder time and the method’s overhead separately // formula 183.11
Code: ToMe, proportional attention and FastV in PyTorch
This file implements bipartite soft matching with sizes, checks proportional attention, runs the FastV selection rule on a causal attention map, computes the FLOPs ratio of formula (183.9), and builds the random and pooling baselines at equal budget. It runs on a CPU in under a second and is deterministic.
import math, torch import torch.nn.functional as F torch.manual_seed(0) # ---------- 1) ToMe: bipartite soft matching with sizes (formulas 183.4, 183.5) ---------- def tome_merge(x, k, size, r): """x: (N, D) tokens, k: (N, Dk) keys (mean over heads), size: (N, 1). Remove r tokens.""" N = x.shape[0] r = min(r, N // 2) # at most half can be merged kn = k / k.norm(dim=-1, keepdim=True) a, b = kn[0::2], kn[1::2] # alternate split into sets A and B scores = a @ b.T # cosine similarity, (|A|, |B|) best, dst = scores.max(dim=-1) # best partner in B for each A token order = best.argsort(descending=True) src, keep_a = order[:r], order[r:] # the r most similar edges are merged xa, xb = x[0::2] * size[0::2], x[1::2] * size[1::2] # weight by size before summing sa, sb = size[0::2], size[1::2] xb = xb.index_add(0, dst[src], xa[src]); sb = sb.index_add(0, dst[src], sa[src]) kb = k[1::2] # (keys are recomputed by the next layer) x_new = torch.cat([xa[keep_a], xb]) / torch.cat([sa[keep_a], sb]) return x_new, torch.cat([k[0::2][keep_a], kb]), torch.cat([sa[keep_a], sb]) N, D = 576, 64 # a 24 x 24 grid of visual tokens x = torch.randn(N, D); size = torch.ones(N, 1) for layer in range(4): # 4 layers, r = 64 each x, k, size = tome_merge(x, x, size, r=64) print(layer, x.shape[0], int(size.sum().item())) # 512, 448, 384, 320 tokens; sizes always sum to 576 # ---------- 2) proportional attention (formula 183.6): a merged pair counts twice ---------- dh = 16 q = torch.randn(1, dh); keys = torch.randn(3, dh); vals = torch.randn(3, dh) keys_dup = torch.cat([keys, keys[2:3]]); vals_dup = torch.cat([vals, vals[2:3]]) # token 2 appears twice full = F.softmax(q @ keys_dup.T / math.sqrt(dh), -1) @ vals_dup s = torch.tensor([1.0, 1.0, 2.0]) # after merging: size 2 prop = F.softmax(q @ keys.T / math.sqrt(dh) + s.log(), -1) @ vals print((full - prop).abs().max().item()) # about 1e-7: identical output # ---------- 3) FastV: rank image tokens by attention received at layer K (formula 183.8) ---------- def fastv_keep(attn, img_slice, R): """attn: (H, n, n) attention probabilities of layer K. Returns indices of image tokens to keep.""" score = attn.mean(dim=0).mean(dim=0)[img_slice] # average over heads and over all queries n_keep = round((1 - R) * score.numel()) idx = score.topk(n_keep).indices.sort().values # keep the original order return idx + img_slice.start H, n_sys, V, T = 8, 35, 576, 40 n = n_sys + V + T logits = torch.randn(H, n, n) mask = torch.ones(n, n).tril().bool() attn = logits.masked_fill(~mask, float("-inf")).softmax(-1) # a causal attention map keep = fastv_keep(attn, slice(n_sys, n_sys + V), R=0.5) print(keep.numel()) # 288 image tokens survive after layer K # ---------- 4) the theoretical FLOPs ratio (formula 183.9) for LLaVA-1.5-7B ---------- def layer_flops(n, d=4096, m=11008): # Llama layer with a SwiGLU FFN, formula 183.2 return 2 * n * (4 * d * d + 3 * d * m) + 4 * n * n * d def fastv_ratio(V, T, K, R, L=32): n, nh = V + T, T + round((1 - R) * V) return (K * layer_flops(n) + (L - K) * layer_flops(nh)) / (L * layer_flops(n)) print(round(fastv_ratio(576, 40, K=2, R=0.5), 3)) # 0.556: about 44% fewer prefill FLOPs # ---------- 5) honest baselines at the same budget: random and 2x2 average pooling ---------- grid = torch.randn(24, 24, D) g = torch.Generator().manual_seed(0) rand_keep = torch.randperm(V, generator=g)[:144].sort().values pooled = F.avg_pool2d(grid.permute(2, 0, 1), 2).permute(1, 2, 0).reshape(-1, D) print(rand_keep.numel(), pooled.shape[0]) # 144 and 144: compare methods at equal tokens
In a real model, tome_merge runs inside every ViT block between attention and MLP, with the attention keys as the metric and the sizes passed to the next block’s attention as the \(\log \mathbf{s}\) bias. fastv_keep runs once, at layer \(K\) of the language model, and the kept indices are used to slice the hidden states, the position ids and the KV cache of the following layers. Keep the original position ids of the kept tokens: renumbering them would tell the model the image is smaller than it is.
Where this is used
| Setting | Typical choice | Why |
|---|---|---|
| General image chat (LLaVA-1.5 style) | MLP projector, all 576 tokens; optional FastV with \(R = 50\%\) at serving time | Few images per request; the cost is acceptable, and training-free pruning is a cheap extra |
| Documents, charts, screenshots | Dynamic counts (AnyRes tiles, native resolution) with a pixel cap; little or no pruning | Small text needs many tokens; pruning hurts text reading first (session 180) |
| Small and edge models (SmolVLM, phones) | Aggressive pixel shuffle or resamplers, trained in | Context and memory are tiny; a fixed small budget is predictable (session 182) |
| Long video QA and Video-RAG | Few tokens per frame (pooling, resamplers), merging across frames, keyframe selection | Thousands of frames must fit in one context (session 206, session 210) |
| Action recognition and video classification | ToMe in the video ViT | Static backgrounds repeat across frames; ToMe reported 2.2× throughput for ViT-L on video |
| Key-moment detection, temporal grounding | Keep frames dense in time, reduce tokens per frame | Merging across time blurs the exact boundary of an event; spatial reduction is safer |
| Video retrieval and fingerprinting | One pooled embedding per frame or clip, not many tokens | Index size scales with tokens; retrieval needs a compact vector (session 208, session 209) |
| Serving at scale | Token caps per request, prefix caching of repeated images, measured TTFT | Visual tokens decide KV-cache memory and so the batch size (session 196) |
Common mistakes
- Reporting only FLOPs or kept tokens. A method that breaks FlashAttention or sorts on the CPU can save FLOPs and lose time. Report time to first token and throughput on real hardware.
- No random or pooling baseline. Without them you cannot tell whether your scoring rule helps or whether any reduction to that budget would do as well.
- Testing only on easy benchmarks. If the questions can be answered from a thumbnail, every method looks lossless. Include text reading, counting and grounding.
- Renumbering positions after pruning. Kept tokens must keep their original position ids (or M-RoPE coordinates); otherwise the model’s sense of where things are is wrong.
- Merging without proportional attention. A merged token that stands for 10 patches then gets the attention weight of one, and the image is under-weighted.
- Forgetting the KV cache. Pruning only the prefill hidden states but storing the full KV cache saves compute but not memory, and memory is what limits the batch size.
- Ignoring GQA when estimating memory. Models with few KV heads store several times less per token; use the real config, formula (183.3).
- Merging across time for tasks that need exact time. Temporal merging is great for classification and bad for “when did the goal happen?”.
Interview questions
Why is a VLM request often much more expensive than a text request of similar length in words?basic
Explain ToMe’s bipartite soft matching. Why not cluster the tokens?core
What is proportional attention and why is it needed?core
How does FastV choose tokens, and what does it save with \(K = 2\), \(R = 50\%\) on LLaVA-1.5-7B?core
Why do attention-based pruning scores tend to keep the bottom of the image?advanced
Q-Former with 32 queries or an MLP projector with 576 tokens: how do you choose?core
A paper shows its pruning method keeps 98% accuracy at 25% of the tokens. What do you check?advanced
Halving the prefill FLOPs gave only a 1.2× faster request. Why?core
How would you cut the token cost of a 1-hour video QA system?advanced
Why can FastV not simply run with FlashAttention, and what can you do?advanced
Merge or prune: which is safer, and why?basic
Exercises
- LLaVA-1.5-7B reads 8 video frames (576 tokens each) and a 60-token question. What is the visual share of the input, and how much KV cache do the visual tokens use (FP16, formula 183.3)?
- ToMe with \(r = 16\) on CLIP ViT-L/14-336 (577 tokens, 24 layers). How many tokens leave the encoder? Could you use \(r = 300\) in the first layer?
- FastV on LLaVA-1.5-7B with \(K = 2\), \(R = 75\%\), 576 image and 40 other tokens. How many tokens do layers 3 to 32 see, and what is \(\rho\)? Check with Figure 5.
- Qwen2-VL with the default pixel limits reads a 3840 × 2160 frame. How many tokens? What if you set
max_pixelsto \(1280 \times 28 \times 28\)? Check with Figure 6. - Prefill is 30% of your request time, with no encoder cost and no overhead, and decode does not change. A method halves prefill time. What is the end-to-end speedup?
- Show that proportional attention with \(\log s\) gives the same output as \(s\) identical copies of a token.
Answerscheck
Key takeaways
- Visual tokens usually far outnumber text tokens (183.1); each costs as much as a text token in prefill (183.2) and in the KV cache (183.3). One LLaVA-1.5 image is 576 tokens and 288 MiB of KV cache; 32 frames are 9 GiB.
- Tokens can be removed in the encoder, in the connector, or inside the LLM. Earlier is cheaper and simpler; later can use the question.
- ToMe merges the \(r\) most similar pairs per layer with bipartite matching on keys (183.4), size-weighted averages (183.5) and proportional attention (183.6).
- FastV ranks image tokens by the attention they receive at layer \(K\) (183.8) and drops \(R\%\); \(K = 2\), \(R = 50\%\) keeps about 55% of prefill FLOPs (183.9). It needs attention probabilities and has a position bias.
- Resamplers fix the budget (32 or 64 tokens); AnyRes and native resolution make it grow with the image (183.10); nested tokens let the user choose.
- Measure honestly: real latency (183.11), random and pooling baselines at equal budget, and benchmarks that need the pixels.
Key formulas
Next, session 184 adds another modality with the same problem: an audio encoder turns one second of sound into many tokens, and the same tools (pooling, fixed budgets, merging over time) decide its cost.
References for this session
- Bolya et al. (2022). Token Merging: Your ViT But Faster
- Chen et al. (2024). An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
- Jin et al. (2024). Efficient Multimodal Large Language Models: A Survey
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 9: Video-Language Models
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Liu et al. (2024). LLaVA-NeXT: Improved reasoning, OCR, and world knowledge (blog)
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
- Laurençon et al. (2024). What matters when building vision-language models?
- Cai, Yang, Gao and Lee (2024). Matryoshka Multimodal Models
- Shang et al. (2024). LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models
- Endo, Wang and Yeung-Levy (2024). Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration
- Wen et al. (2025). Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?
- Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models