Connectors: projector, Q-Former, Perceiver resampler, pixel shuffle
A vision encoder gives hundreds of vectors in its own space; a language model reads vectors of another size, and every vector costs context and memory. The connector sits between them. This session builds the four connectors used in real VLMs: the linear or MLP projector of LLaVA, the Q-Former of BLIP-2, the Perceiver resampler of Flamingo and Idefics2, and pixel shuffle (patch merging) of Idefics3, SmolVLM, InternVL and Qwen2-VL. For each one we write the math, count the tokens and parameters with real model configs, and test them side by side in a small PyTorch experiment.
What you will learn
- The two jobs of a connector: change the width of each vector, and (maybe) change the number of vectors, and why the number of tokens decides most of the cost
- The linear and MLP projector of LLaVA and LLaVA-1.5: formula, parameter count (21.0M for LLaVA-1.5-7B) and why it keeps every patch
- The Q-Former of BLIP-2: 32 learned queries, cross-attention every other block, three attention masks in stage 1, and a linear layer into a frozen LLM in stage 2
- The Perceiver resampler of Flamingo: learned latents that cross-attend to features of any number of frames, with keys and values that include the latents, and its cost R(M+R)
- Pixel shuffle and patch merging: fold each r × r block of patches into one token with r² times more channels, used with r = 2, 3 or 4 in Idefics3, SmolVLM, InternVL and Qwen2-VL
- What the ablations say (Flamingo, BLIP-2, LLaVA-1.5, MM1, Idefics2, SmolVLM): token count and resolution matter more than the connector type, and small models like aggressive compression
- How to compute tokens, connector parameters and KV-cache memory per image for a real model, and how each connector handles video
Why this matters
In session 177 we split a VLM into three parts: a vision encoder, a connector and a language model. The vision encoder (a ViT such as CLIP or SigLIP) gives one vector per image patch, for example 576 vectors of width 1,024. The language model reads vectors of its own width, for example 4,096, and every vector it reads is one more position in its context. The connector is the small piece in the middle that makes the two fit.
The connector is small (from about 0.3% of the parameters in LLaVA-1.5-7B to about 5% in BLIP-2 with OPT-2.7B), but it decides two things that matter a lot:
- The width. It must map each visual vector into the space of the language model's word embeddings, so the language model can read it like a (strange) word.
- The number of visual tokens. It may keep one token per patch, or squeeze the image into a fixed, smaller set. This number sets the cost of every image: prefill compute, KV-cache memory, and how many images or video frames fit in the context (session 80).
Four designs cover almost every model you will meet:
- A linear or MLP projector (LLaVA, LLaVA-1.5): one small network applied to each patch vector. Same number of tokens, new width.
- A Q-Former (BLIP-2): a small BERT-like transformer whose 32 learned queries read the image through cross-attention, so every image becomes exactly 32 tokens.
- A Perceiver resampler (Flamingo, Idefics2): a set of learned latents that cross-attend to the features of an image or of a whole video and return a fixed number of tokens, 64 in Flamingo.
- Pixel shuffle or patch merging (Idefics3, SmolVLM, InternVL, Qwen2-VL): fold each \(2 \times 2\) (or \(3 \times 3\), \(4 \times 4\)) block of neighboring patches into one token with more channels, then project. Four to sixteen times fewer tokens, no learned queries.
The connector's job, in shapes
Let the vision encoder see an image of \(H \times W\) pixels with patch size \(P\). It returns a grid of \(N\) patch vectors of width \(d_v\). The language model has hidden width \(d\). A connector is any function
- Read the symbols
- \(N\)
- the number of patch vectors from the vision encoder (some encoders also add one [CLS] vector)
- \(d_v\)
- the width of the vision features: 1,024 for CLIP ViT-L/14, 1,152 for SigLIP-SO400M, 1,408 for EVA ViT-g/14
- \(M\)
- the number of visual tokens the language model will see; \(M = N\) for a projector, \(M\) fixed for Q-Former and resampler, \(M = N / r^2\) for pixel shuffle
- \(d\)
- the hidden width of the language model, the same width as its word embeddings (4,096 for Vicuna-7B and Llama 3 8B)
- \(\phi\)
- the parameters of the connector; they are almost always trained from scratch
Two families of designs follow from the shape. If \(M = N\), the connector only changes the width; the cost of the image is set by the vision encoder (resolution and patch size). If \(M \lt N\), the connector also compresses; we then have to ask what is lost. There are two ways to compress: learned queries that pick what to read (Q-Former, resampler), and fixed folding of neighbors (pixel shuffle). Figure 1 shows all of them for real models.
How to read Figure 1
LLaVA-1.5-7B: 576 vectors of width 1,024 go in, 576 tokens of width 4,096 come out; nothing is compressed and one image costs 288 MiB of KV cache. Click BLIP-2: 257 vectors of width 1,408 become 32 tokens, and the cache per image drops to 10 MiB. Click Idefics3-8B: pixel shuffle with \(r = 2\) turns 676 patches into 169 tokens of width 4,608 before the linear projection. Click Flamingo: zero tokens enter the language model sequence, because Flamingo reads its 64 resampled tokens through extra cross-attention layers (session 179). Move the slider to see the cost of 8 or 32 images, tiles or frames.
Connector 1: the linear and MLP projector (LLaVA)
The simplest connector maps each patch vector on its own. LLaVA (Liu et al., 2023) takes the grid features of CLIP ViT-L/14 and applies one trainable matrix [Certain]:
LLaVA-1.5 (Liu et al., 2023b) replaced the matrix by a two-layer MLP with a GELU in the middle, and reported better scores (on MME, 1,355.2 versus 1,323.8 for the same data, their Table 2) [Certain]:
- Read the symbols
- \(\mathbf{z}_i\)
- the feature of patch \(i\) from the vision encoder, a vector in \(\R^{d_v}\)
- \(\mathbf{h}_i\)
- the visual token given to the language model, a vector in \(\R^{d}\); it is placed in the sequence where the image placeholder token was
- \(W_1, \mathbf{b}_1, W_2, \mathbf{b}_2\)
- the weights and biases of the two linear layers; the same weights are used for every patch
- GELU
- the smooth activation used inside transformer MLPs (session 21)
The parameter count is easy:
Two details from the LLaVA code and papers matter in practice. First, the features are taken from the second-to-last layer of the ViT, not the last: vision_feature_layer = -2 in the Hugging Face config [Certain]. The LLaVA paper reports that the last layer was 0.96% worse on ScienceQA and suggests that the last layer of CLIP is more global and abstract, while the one before keeps more local detail [Certain]. Second, the [CLS] vector is dropped, so the 577 outputs of ViT-L/14 at 336 pixels become 576 tokens (vision_feature_select_strategy = "default") [Certain].
CLIP ViT-L/14 at 336 × 336 pixels: \(336 / 14 = 24\) patches per side, \(N = 24^2 = 576\) patch vectors of width \(d_v = 1{,}024\). Vicuna-7B has \(d = 4{,}096\).
Parameters by (178.4): \(1{,}024 \times 4{,}096 + 4{,}096 + 4{,}096^2 + 4{,}096 = 20{,}979{,}712\), about 21.0M, roughly 0.3% of the 7B language model. (LLaVA-1's single matrix with a bias would be 4,198,400.)
Tokens: \(M = N = 576\). KV cache per token in Vicuna-7B (32 layers, 32 key-value heads of size 128, 16-bit): \(2 \times 32 \times 4{,}096 \times 2 = 524{,}288\) bytes = 0.5 MiB. One image: \(576 \times 0.5 = 288\) MiB. Eight images: 2.25 GiB, before any text.
Figure 2 runs this projector for real on a toy image of 4 × 4 patches (a cat on grass under the sky) with feature width 3 and language-model width 4. Watch the loop: every patch goes through the same small network, and every patch becomes one token. The token count out is the token count in.
How to read Figure 2
Press Step three times. Patch 1 is sky, \(\mathbf{z}_1 = (0.92, 0.06, 0.10)\). Step 1 gives \(\mathbf{a} = W_1\mathbf{z}_1 + \mathbf{b}_1 = (0.738, 0.006, -0.278, 0.124)\), step 2 gives \(\operatorname{GELU}(\mathbf{a}) = (0.568, 0.003, -0.109, 0.068)\) (GELU keeps large positive numbers and squashes negative ones toward 0), and step 3 gives the token \(\mathbf{h}_1 = (0.407, -0.183, -0.043, 0.214)\), which flies into slot 1. Press Play: patch 6 (a cat patch, \((0.32, 1.06, 0.12)\)) gives a very different token, \((-0.021, 0.517, 0.080, 0.413)\), because its content differs, not because the weights change. At the end 16 patches have made 16 tokens. Switch to linear: one step per patch, 16 parameters instead of 36.
Why is something so simple strong? Because it throws nothing away. Every patch stays a token, and the 7B language model, with its many layers of attention, decides what to use. The price is the token count: 576 tokens for a low-resolution image, and thousands once the image is cut into high-resolution tiles (session 180). That price is the reason for the other three designs.
The training recipe shows why the projector is also easy to train. LLaVA-1.5 stage 1 trains only the projector on 558K image-caption pairs, with the ViT and the language model frozen; stage 2 trains the projector and the language model on 665K instruction examples [Certain]. One epoch of stage 1 with a fresh 21M-parameter MLP is enough to “align” the two spaces. The LLaVA-1.5 paper notes in its limitations that visual resamplers “cannot achieve convergence as efficiently as LLaVA with a comparable amount of training data, probably due to more trainable parameters in the resamplers” [Certain].
Connector 2: the Q-Former (BLIP-2)
BLIP-2 (Li et al., 2023) keeps both the image encoder and the language model frozen and trains only a Querying Transformer (Q-Former) between them [Certain]. The idea: instead of passing every patch, let a small set of learned queries ask the image for what matters, and pass only their answers.
The facts from the paper [Certain]:
- The Q-Former has two transformer parts that share their self-attention layers: an image transformer that runs on the queries, and a text transformer that can act as a text encoder or a text decoder.
- There are 32 queries, each of width 768, the hidden width of the Q-Former. They are parameters, learned like weights.
- The queries talk to each other through self-attention, and read the frozen image features through cross-attention layers inserted every other transformer block.
- The Q-Former is initialized from BERT-base; the cross-attention layers start random. In total it has 188M parameters.
- The output \(Z\) is \(32 \times 768\), “much smaller than the size of frozen image features (e.g. 257 × 1024 for ViT-L/14)”. The image encoder is ViT-L/14 from CLIP or ViT-g/14 from EVA-CLIP, with its last layer removed (they use the second-to-last layer).
The cross-attention is the standard one from session 20: the queries ask, the image patches answer.
- Read the symbols
- \(X\)
- the current states of the 32 queries (at the first layer: the learned query embeddings)
- \(Z_v\)
- the frozen image features: for ViT-g/14 at 224 pixels, \(16 \times 16 = 256\) patches plus one [CLS] vector, width 1,408
- \(W_Q, W_K, W_V\)
- query, key and value projections; \(W_K\) and \(W_V\) map width 1,408 to the Q-Former width (the cross-attention input size is 1,408 in the Hugging Face config,
encoder_hidden_size) [Certain] - \(d_h\)
- the width of one attention head
The result has 32 rows whatever the size of the image grid: the number of output tokens is the number of queries. Written for one query \(j\) and one patch \(i\), (178.5) is three small steps: a score, a softmax over the patches, and a weighted sum.
- Read the symbols
- \(\mathbf{x}_j\)
- query \(j\), one row of \(X\); it is a learned vector, the same for every image
- \(s_{j,i}\)
- the score: how well patch \(i\) matches what query \(j\) looks for
- \(m_j\)
- the largest score of query \(j\); subtracting it keeps the exponentials small and does not change \(a_{j,i}\)
- \(a_{j,i}\)
- the attention weight; for each query the weights over all patches sum to 1
- \(\mathbf{o}_j\)
- the output of query \(j\): an average of the patch values, weighted by \(a_{j,i}\). One query gives one output token
Figure 3 runs (178.5a) on the toy image. Each query has been set by hand to look for one kind of content, which is what stage-1 training makes real queries do. Note that a query does not know where the cat is: it finds the cat by content, wherever it sits.
How to read Figure 3
Step once: query 1 (looking for “cat”) scores all 16 patches. The best is patch 7, \(s = 3.85 \times 1.08 / \sqrt{3} = 2.401\). Step again: the softmax divides by \(\sum_i \exp(s_{1,i} - 2.401) = 5.083\), so patch 7 gets weight 0.197 and the four cat patches together get 0.693. Step again: the output \(\mathbf{o}_1 = (0.345, 0.763, 0.309)\) is mostly “cat”. Play on: query 2 (sky) ends at \((0.806, 0.205, 0.229)\), query 3 (grass) at \((0.222, 0.273, 0.841)\). Now pick “cat at bottom left” and play again: query 1 still finds the cat (weight 0.682 on the cat patches) although the cat moved. Lower \(R\): the output has \(R\) tokens, so \(R\) sets the compression, \(16 \to R\).
Stage 1: teach the queries what to look for
A new set of queries knows nothing. BLIP-2 first trains the Q-Former on image–text pairs with the three losses of session 173: contrastive (ITC), matching (ITM) and image-grounded text generation (ITG). The same weights serve all three; what changes is the self-attention mask between the queries and the text tokens [Certain]:
- ITC (unimodal mask): queries see only queries, text sees only text. The image–text score is the best match over the 32 query outputs and the text [CLS] vector:
- ITG (multimodal causal mask): queries see only queries; each text token sees all queries and the earlier text tokens. The text cannot see the image directly, so all the information needed to write the caption must pass through the 32 queries. This is what forces the queries to extract what the text needs.
- ITM (bidirectional mask): everything sees everything; each query output goes to a 2-way classifier and the 32 logits are averaged.
BLIP-2 uses in-batch negatives for ITC instead of BLIP's momentum queue, because with frozen encoders the batch can be large (2,320 with ViT-L, 1,680 with ViT-g in stage 1) [Certain]. Figure 4 shows the three masks.
How to read Figure 4
ITC: two separate green squares, 32 of 64 entries allowed; the queries and the text are kept apart so the contrastive score cannot cheat. ITM: all 64 entries allowed. ITG: the query block is full, the text rows see all queries plus a lower triangle of text, 42 of 64 allowed; the queries cannot see the text, so they must carry the image information on their own. Stage 2: the text part is gone; the readout shows the linear layer that maps width 768 to the width of the language model.
Stage 2: plug the queries into a frozen LLM
In stage 2 the Q-Former output goes through one fully connected layer to the width of the language model, and the 32 resulting vectors are put in front of the text as “soft visual prompts” [Certain]:
For OPT-2.7B (\(d = 2{,}560\)) this layer has \(768 \times 2{,}560 + 2{,}560 = 1{,}968{,}640\) parameters. The language model stays frozen and is trained through only by the language-modeling loss. For OPT it is the next-token loss; for FlanT5 a prefix loss [Certain].
Is stage 1 needed? BLIP-2 tested it: without the representation-learning stage, zero-shot VQAv2 is much lower for both OPT and FlanT5, and with OPT “performance drastically degrades as training proceeds” (catastrophic forgetting) [Certain, their Figure 5]. The paper notes that a Q-Former trained only with the generative loss is “similar to the Perceiver Resampler in Flamingo”. So the three-loss stage is what makes the 32 queries useful when the language model is frozen.
The result that made BLIP-2 famous: with ViT-g and FlanT5-XXL it reached 65.0 on zero-shot VQAv2 test-dev, against 56.3 for Flamingo-80B, with 108M trainable parameters (in their Table 2) versus 10.2B, “54x fewer” [Certain]. The cost: everything the language model learns about the image must fit in 32 vectors of width 768. That is fine for “what is in this photo”, but tight for reading a page of text or counting many small objects. Later instruction-tuned models mostly moved back to projectors with many tokens (see “What the ablations say” below).
Connector 3: the Perceiver resampler (Flamingo)
The Perceiver (Jaegle et al., 2021) solved a general problem: self-attention over \(M\) inputs costs \(O(M^2)\), too much for raw pixels (\(M = 50{,}176\) for a 224 × 224 image). Its trick is an asymmetric attention: the keys and values come from the large input, but the queries come from a small, learned latent array of size \(N \ll M\). One cross-attention then costs \(O(MN)\), and the deep part of the network runs on the latents at \(O(N^2)\) [Certain].
Flamingo (Alayrac et al., 2022) used this idea as its connector, the Perceiver resampler [Certain]:
- The frozen vision encoder (an NFNet-F6) produces a grid of features per frame. For video, frames are sampled at 1 frame per second and encoded one by one; a learned temporal embedding is added to all features of a frame. An image is a video with one frame. No spatial position encoding is added; the authors saw no gain from it and suggest the CNN features already carry position.
- All features of all frames are flattened into one long sequence \(X_f\).
- A fixed set of learned latents, 64 in Flamingo, cross-attend to \(X_f\) through 6 layers of attention + feed-forward. The keys and values are the concatenation of the visual features and the latents themselves, which the authors found slightly better than using the features alone.
- The output is always 64 tokens, “independently from the input image resolution or the number of input video frames”. The resampler has about 200M parameters, the same in all three model sizes (6 layers, width 1,536, 16 heads).
One layer of the resampler, written like the Flamingo pseudo-code:
- Read the symbols
- \(F\)
- the visual features of \(T\) frames, each with \(S\) spatial positions, shape \(T \times S \times d_r\)
- \(\mathbf{e}^{\text{time}}\)
- a learned vector per frame index, added to every position of that frame
- \(L\)
- the latent array, \(R \times d_r\); at the first layer it is the learned latents (\(R = 64\) in Flamingo)
- \([X_f; L]\)
- the visual features and the latents stacked into one sequence of length \(TS + R\), used for the keys and values
- \(\operatorname{Attn}\)
- multi-head attention; FFN is the feed-forward block (Flamingo uses a squared ReLU there)
- input: frame features \(F \in \R^{T \times S \times d_r}\); learned time embeddings \(\mathbf{e}^{\text{time}} \in \R^{T_{\max} \times d_r}\); learned latents \(L_0 \in \R^{R \times d_r}\); \(K\) layers
- \(F_t \leftarrow F_t + \mathbf{e}^{\text{time}}_t\) for every frame \(t\) // only time is encoded, not space
- \(X_f \leftarrow\) flatten \(F\) to \(TS\) rows // any \(T\), any \(S\)
- \(L \leftarrow L_0\)
- for \(k = 1 \dots K\):
- \(L \leftarrow L + \operatorname{Attn}_k(q = L,\ kv = [X_f; L])\) // \(R\) queries, \(TS + R\) keys
- \(L \leftarrow L + \operatorname{FFN}_k(L)\)
- return \(L \in \R^{R \times d_r}\) // always \(R\) tokens
Figure 5 runs Algorithm 1 on a toy video: up to 3 frames of 4 × 4 features in which the cat moves. Two details become visible. First, the latents are also keys, so each latent spends part of its attention on itself and on the other latents. Second, the number of keys grows with the number of frames, but the number of outputs does not. The open-source OpenFlamingo code applies a LayerNorm to the features and to the latents before each attention [Likely]; the toy does the same.
How to read Figure 5
Choose \(T = 1\) and step to layer 1, latent 1. Its normalized value is \(\hat{L}_1 = (-0.707, 1.414, -0.707)\), and its best key is itself, \(s = 3.118\). After the softmax it gives 0.191 to itself and 0.740 to the cat patches, and the update is \(L_1 \leftarrow (0, 1.1, 0) + (-0.524, 1.317, -0.794) = (-0.524, 2.417, -0.794)\). Now choose \(T = 3\): there are 52 keys, the weight on itself falls to 0.076 and the cat rows of all three frames get 0.878. The counter shows \(R(TS + R) = 4 \times 52 = 208\) scores per layer, and still 4 tokens come out.
The cost of one resampler layer is set by the attention scores. Per head, the \(R\) latent queries meet \(TS + R\) keys. Compare it with running full self-attention over the same visual tokens, which is what the language model would do if it received all of them:
Take 30 frames (1 per second) with \(S = 729\) features each (a 27 × 27 SigLIP grid at 384 pixels) and \(R = 64\) latents. Then \(TS = 21{,}870\).
Resampler scores per layer and head: \(64 \times (21{,}870 + 64) = 1{,}403{,}776\). Full self-attention over the same tokens: \(21{,}870^2 = 478{,}296{,}900\), which is 341 times more. And the language model sees 64 tokens instead of 21,870.
The same arithmetic for one LLaVA-size image (\(TS = 576\)): \(64 \times 640 = 40{,}960\) versus \(331{,}776\), only 8.1 times. The resampler pays off most when the input is long.
How to read Figure 6
Start at one frame of 576 features and 64 latents: the resampler cuts 576 tokens to 64 and the score count by 8.1 times. Raise the frames to 30 and pick 729 features per frame: the language model would need 21,870 tokens with a projector but still 64 with the resampler, and the score ratio grows to about 341. Raise the latents: cost and tokens grow linearly in \(R\). The resampler's output size never depends on \(T\) or \(S\); that is why Flamingo can take any number of frames.
Two more properties matter. First, the resampler is blind to order unless the features carry position: attention does not care where a key sits in the sequence. Flamingo adds only time embeddings; spatial layout survives only as far as the CNN features encode it. Second, in Flamingo the 64 tokens do not enter the language model's sequence. The frozen language model reads them through new gated cross-attention layers (GATED XATTN-DENSE) inserted between its blocks, every 4th block for Flamingo-9B and every 7th for Flamingo-80B [Certain]. That is the topic of session 179. Idefics2 later put the resampler's 64 tokens directly into the sequence of a decoder-only model.
The Flamingo ablation supports the design at equal parameter budget [Certain, their Table 3, Flamingo-3B, short training]: the overall score was 70.7 with the Perceiver resampler, 66.6 with an MLP and 66.7 with a plain Transformer, and both alternatives were also slower. A medium-size resampler beat small and large ones, and larger resamplers made training unstable when scaled with the language model (their Table 10).
Connector 4: pixel shuffle and patch merging
The fourth idea compresses with no learned queries at all. Take the \(h \times w\) grid of patch vectors. Cut it into blocks of \(r \times r\) neighbors. Concatenate the \(r^2\) vectors of each block into one long vector. You now have \(r^2\) times fewer tokens, each \(r^2\) times wider. Then a linear layer or an MLP maps the wide vector to the width of the language model.
The name comes from super-resolution. Shi et al. (2016) introduced a “sub-pixel convolution” layer that moves channels into space to make an image larger; PyTorch calls this PixelShuffle [Certain]. VLMs use the inverse operation (space to depth, PyTorch's PixelUnshuffle), but the VLM papers still call it pixel shuffle [Certain, SmolVLM paper: “pixel shuffle (space-to-depth)”]. With grid positions \((u, v)\):
- Read the symbols
- \(\mathbf{z}_{u,v}\)
- the patch vector at row \(u\), column \(v\) of the grid
- \(r\)
- the shuffle ratio (scale factor): 2 in Idefics3 and InternVL, 3 in SmolVLM-2.2B, 4 in SmolVLM-256M (from their configs) [Certain]
- \(\mathbf{y}_{a,b}\)
- the concatenation of the \(r^2\) patch vectors of block \((a, b)\); the semicolons mean “stack”
- \(W\)
- the projection (Idefics3 and SmolVLM: one linear layer without bias; InternVL: LayerNorm + MLP; Qwen2-VL: LayerNorm + MLP)
The rearrangement itself is lossless: it only moves numbers. The PyTorch code below checks that the output holds exactly the same numbers as the input. Compression happens in \(W\): it maps \(r^2 d_v\) numbers to \(d\) numbers. For Idefics3 that is \(4 \times 1{,}152 = 4{,}608\) to \(4{,}096\), almost no squeeze. For SmolVLM-256M it is \(16 \times 768 = 12{,}288\) to \(576\), 21 times. And unlike a resampler, every output token comes from one known place in the image, so the layout is kept.
Patch merging is the same operation under another name. The Swin Transformer merges each \(2 \times 2\) group of neighbors (concatenating \(4C\) channels, then a linear layer to \(2C\)) to build a pyramid [Certain]. Qwen2-VL uses an MLP after the ViT “to compress adjacent 2 × 2 tokens into a single token”; a 224 × 224 image with patch size 14 becomes 66 tokens: \(16 \times 16 / 4 = 64\), plus the special tokens <|vision_start|> and <|vision_end|> [Certain]. InternVL 1.5 uses pixel shuffle “to reduce the number of visual tokens to one-quarter”, so a 448 × 448 tile becomes 256 tokens [Certain].
How to read Figure 7
At \(r = 2\) the 64 patches become 16 tokens, each a stack of 4 patch vectors. Click the token in row 0, column 1: it holds patches (0,2), (0,3), (1,2), (1,3), a 2 × 2 square in the image, so \(\mathbf{y}_{0,1} = [0.2, 0 \mid 0.3, 0 \mid 1.2, 0 \mid 1.3, 0]\), 8 numbers, and the toy \(W\) gives \(\mathbf{h} = (0.75, 0)\). Press Play to watch all 16 blocks fold, one after the other. At \(r = 4\) there are only 4 tokens with 16 patches each. At \(r = 1\) nothing changes. The readout also gives the real sizes when the grid is SigLIP's (width 1,152): the token width grows from 1,152 to 4,608 at \(r = 2\) and 18,432 at \(r = 4\).
The grid must be divisible by \(r\). That explains a detail of SmolVLM: at 384 pixels with patch 14, SigLIP gives a \(27 \times 27\) grid, which is not divisible by 2 or 4, and the 2.2B model uses \(r = 3\): 729 patches become 81 tokens [Certain from the config; that the divisibility drove the choice is Likely]. The 256M model uses SigLIP-B/16 at 512 pixels, a \(32 \times 32\) grid, and \(r = 4\): 1,024 patches become 64 tokens [Certain].
SigLIP-SO400M with patch 14: \(364 / 14 = 26\) per side, \(N = 676\) patches of width 1,152. Pixel shuffle with \(r = 2\): \(M = 676 / 4 = 169\) tokens of width \(4 \times 1{,}152 = 4{,}608\).
Projection to Llama 3 8B (\(d = 4{,}096\)), no bias: \(4{,}608 \times 4{,}096 = 18{,}874{,}368\) parameters.
KV cache in Llama 3 8B (32 layers, 8 key-value heads of size 128, 16-bit): \(2 \times 32 \times 1{,}024 \times 2 = 131{,}072\) bytes per token. One tile: \(169 \times 131{,}072 = 22{,}151{,}168\) bytes = 21.1 MiB. Without the shuffle it would be 676 tokens and 84.5 MiB: the shuffle saves a factor of 4 in every later step.
- input: patch features \(Z \in \R^{B \times N \times d_v}\) with \(N = h \times w\); ratio \(r\) that divides \(h\) and \(w\); projection \(W \in \R^{d \times r^2 d_v}\)
- reshape \(Z\) to \(B \times \frac{h}{r} \times r \times \frac{w}{r} \times r \times d_v\) // split rows and columns into blocks
- permute to \(B \times \frac{h}{r} \times \frac{w}{r} \times r \times r \times d_v\) // bring each block's \(r \times r\) patches together
- reshape to \(B \times \frac{N}{r^2} \times r^2 d_v\) // one wide vector per block, (178.10)
- \(H \leftarrow Y W^\top\) // (178.11); InternVL and Qwen2-VL use LayerNorm + MLP here
- return \(H \in \R^{B \times \frac{N}{r^2} \times d}\) // placed where the image tokens go in the text
Small models want more compression, not less. SmolVLM compared \(r = 2\) and \(r = 4\) and found that “smaller VLMs benefit from more aggressive compression (\(r = 4\)) as the reduced token count eases attention overhead and improves long-context modeling”, while “higher ratios collapse larger spatial regions into single tokens, impairing tasks requiring precise localization, such as OCR” [Certain]. The best ratio depends on model size and on the task.
What the ablations say
Which connector is best? Many teams ran controlled tests. Their results agree on one point: the number of visual tokens and the image resolution matter more than the connector type.
| Study | What they compared | What they found [Certain, from the papers] |
|---|---|---|
| Flamingo (2022), Table 3 | Perceiver resampler vs MLP vs Transformer, same parameter budget | Resampler best: overall 70.7 vs 66.6 (MLP) and 66.7 (Transformer) |
| BLIP-2 (2023), Figure 5 | Q-Former with and without stage-1 representation learning | Without stage 1, zero-shot VQA is much lower; with OPT it collapses during training |
| LLaVA-1.5 (2023), Table 2 | Linear vs two-layer MLP projector | MLP better (MME 1,355.2 vs 1,323.8); resamplers converge less efficiently with the same data |
| MM1 (2024), connector ablation | Average pooling, attention pooling, C-Abstractor (convolution); 64 or 144 tokens; 224 or 336 pixels | “Number of visual tokens and image resolution matters most, while the type of VL connector has little effect” |
| Idefics2 (2024), Tables 3, 4, 11 | Perceiver pooling vs no pooling; 64 vs 128 latents; other poolings | Perceiver pooling: +8.5 points on average and 729 to 64 tokens per image; 64 latents as good as 128 (71.7 vs 71.2) |
| SmolVLM (2025), Figure 3 | Pixel shuffle \(r = 2\) vs \(r = 4\) | The best ratio depends on model size; small models prefer \(r = 4\) |
These findings look like they disagree (Idefics2 says 64 tokens are enough, MM1 says more tokens help), but the setups differ: training length, data, task mix and resolution. Idefics2 itself writes that with much longer training more tokens might help [Certain]. The practical summary that most recent open models follow [Likely]: use a simple projector (MLP, often after a pixel shuffle) so that nothing is learned from scratch except a small mapping, and spend the token budget on resolution and tiling. Use learned queries (resampler, Q-Former) when you must fix the token count, for example for long video or many images, or when the language model is frozen. Honeybee (Cha et al., 2023) argued that resamplers lose locality and proposed convolutional projectors (C-Abstractor) that compress while keeping neighbors together; MM1 used the C-Abstractor with 144 tokens [Certain].
| Connector | Tokens per image | Keeps layout? | New parameters | Best when |
|---|---|---|---|---|
| Linear / MLP (LLaVA) | \(N\) (576 at 336 px) | yes, one token per patch | 4M to 31M | you can pay for the tokens; detail and OCR matter |
| Q-Former (BLIP-2) | 32 (number of queries) | no, queries choose | 188M + one linear layer | frozen LLM, short answers, cheap inference |
| Perceiver resampler (Flamingo, Idefics2) | \(R\) (64), for any number of frames | only via position in the features | about 200M in Flamingo | many images or long video, interleaved inputs |
| Pixel shuffle + linear/MLP | \(N / r^2\) | yes, one token per \(r \times r\) block | 7M to 45M | high resolution, small models, documents with \(r = 2\) |
The PyTorch version
All four connectors in a few lines each, a check that pixel shuffle loses nothing, parameter counts at real sizes, and a small experiment. In the experiment, an “image” is an \(8 \times 8\) grid of patch features (with position information, as a ViT output has). Four of the 64 cells hold one of 8 objects. A question names a cell; the answer is the object in that cell, or “empty”. A small 2-layer transformer stands in for the language model: it reads a question token plus the visual tokens from the connector. Every connector gets the same reader, the same data and 1,500 training steps. Half the questions point at an object, so “always say empty” already scores about 0.47. The question is how much fine spatial detail each connector lets through. It runs on a laptop CPU in about 3 minutes.
import torch, torch.nn as nn, torch.nn.functional as F torch.manual_seed(0) # ---- 1) the four connectors ------------------------------------------------- class MLPProjector(nn.Module): # LLaVA-1.5: Linear -> GELU -> Linear def __init__(s, d_v, d_lm): super().__init__(); s.fc1 = nn.Linear(d_v, d_lm); s.fc2 = nn.Linear(d_lm, d_lm) def forward(s, z): # z: [B, N, d_v] return s.fc2(F.gelu(s.fc1(z))) # [B, N, d_lm] (same N) def pixel_shuffle(z, r): # space-to-depth on the patch grid B, N, d = z.shape; h = int(N ** 0.5) z = z.view(B, h // r, r, h // r, r, d) # split rows and columns into r x r blocks z = z.permute(0, 1, 3, 2, 4, 5) # [B, h/r, w/r, r, r, d] return z.reshape(B, (h // r) ** 2, r * r * d) # [B, N/r^2, r^2 d] class ShuffleProjector(nn.Module): # Idefics3 / SmolVLM: shuffle + Linear def __init__(s, d_v, d_lm, r): super().__init__(); s.r = r; s.proj = nn.Linear(r * r * d_v, d_lm, bias=False) def forward(s, z): return s.proj(pixel_shuffle(z, s.r)) class ResamplerLayer(nn.Module): # Flamingo: K,V = [visual ; latents] def __init__(s, d, heads): super().__init__(); s.ln_x = nn.LayerNorm(d); s.ln_l = nn.LayerNorm(d) s.att = nn.MultiheadAttention(d, heads, batch_first=True) s.ffn = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) def forward(s, x, lat): q = s.ln_l(lat); kv = torch.cat([s.ln_x(x), q], dim=1) lat = lat + s.att(q, kv, kv, need_weights=False)[0] return lat + s.ffn(lat) class PerceiverResampler(nn.Module): def __init__(s, d_v, d_lm, R, layers=2, heads=4): super().__init__(); s.inp = nn.Linear(d_v, d_lm) s.latents = nn.Parameter(torch.randn(R, d_lm) * 0.02) # R learned queries s.layers = nn.ModuleList(ResamplerLayer(d_lm, heads) for _ in range(layers)) def forward(s, z): x = s.inp(z); lat = s.latents.expand(z.shape[0], -1, -1) for layer in s.layers: lat = layer(x, lat) return lat # [B, R, d_lm] for any N class MeanPool(nn.Module): # the extreme: one token per image def __init__(s, d_v, d_lm): super().__init__(); s.proj = nn.Linear(d_v, d_lm) def forward(s, z): return s.proj(z.mean(1, keepdim=True)) # ---- 2) pixel shuffle is a pure re-arrangement: nothing is lost ------------- z = torch.arange(2 * 16 * 3.).view(2, 16, 3) # a 4x4 grid of 3-dim "patches" y = pixel_shuffle(z, 2) print("shuffle shape", tuple(y.shape), "same numbers:", torch.equal(y.flatten().sort()[0], z.flatten().sort()[0])) print("token 0 holds patches", (y[0, 0].view(4, 3)[:, 0] / 3).long().tolist()) # patches 0,1,4,5 # ---- 3) real sizes: count connector parameters -------------------------------- n = lambda m: sum(p.numel() for p in m.parameters()) print("LLaVA-1.5-7B MLP ", n(MLPProjector(1024, 4096))) print("Idefics3 shuffle r=2 ", n(ShuffleProjector(1152, 4096, 2))) print("SmolVLM-256M r=4 ", n(ShuffleProjector(768, 576, 4))) # ---- 4) a toy test: can the reader still find small details? ----------------- G, D, K, NOBJ = 8, 16, 8, 4 # 8x8 patch grid, feature size, classes, objects obj_emb = torch.randn(K, D); pos_emb = torch.randn(G * G, D) * 0.5 def batch(B, gen): cls = torch.full((B, G * G), K) # K = "empty" for b in range(B): cells = torch.randperm(G * G, generator=gen)[:NOBJ] cls[b, cells] = torch.randint(0, K, (NOBJ,), generator=gen) feats = torch.cat([obj_emb, torch.zeros(1, D)])[cls] + pos_emb + 0.3 * torch.randn(B, G * G, D, generator=gen) ask = torch.randint(0, G * G, (B,), generator=gen) ask[: B // 2] = (cls[: B // 2] < K).float().multinomial(1, generator=gen).squeeze(1) # half the questions hit an object return feats, ask, cls[torch.arange(B), ask] class Reader(nn.Module): # stands in for the LLM: question token + visual tokens def __init__(s, conn, d=64): super().__init__(); s.conn = conn; s.q = nn.Embedding(G * G, d) s.tf = nn.TransformerEncoder(nn.TransformerEncoderLayer(d, 4, 4 * d, 0.0, batch_first=True, norm_first=True), 2, enable_nested_tensor=False) s.head = nn.Linear(d, K + 1) def forward(s, feats, ask): h = torch.cat([s.q(ask)[:, None], s.conn(feats)], 1) return s.head(s.tf(h)[:, 0]) def run(name, conn, steps=1500): torch.manual_seed(1); gen = torch.Generator().manual_seed(2) m = Reader(conn); opt = torch.optim.AdamW(m.parameters(), 1e-3) for _ in range(steps): f, a, y = batch(128, gen); loss = F.cross_entropy(m(f, a), y) opt.zero_grad(); loss.backward(); opt.step() gen = torch.Generator().manual_seed(99); f, a, y = batch(2000, gen) with torch.no_grad(): acc = (m(f, a).argmax(1) == y).float().mean().item() print(f"{name:22s} tokens {conn(f[:1]).shape[1]:3d} accuracy {acc:.3f}") f, a, y = batch(2000, torch.Generator().manual_seed(99)) print("always say empty accuracy", round((y == K).float().mean().item(), 3)) torch.manual_seed(0) for name, c in [("MLP projector", MLPProjector(D, 64)), ("pixel shuffle r=2", ShuffleProjector(D, 64, 2)), ("pixel shuffle r=4", ShuffleProjector(D, 64, 4)), ("resampler 16 latents", PerceiverResampler(D, 64, 16)), ("resampler 4 latents", PerceiverResampler(D, 64, 4)), ("mean pool", MeanPool(D, 64))]: run(name, c)
Output with PyTorch 2.8 on CPU (two runs gave identical numbers): the shuffle check prints shape (2, 4, 12), the same numbers, and token 0 holds patches 0, 1, 4 and 5. The parameter counts are 20,979,712 (LLaVA-1.5-7B MLP), 18,874,368 (Idefics3) and 7,077,888 (SmolVLM-256M), matching the worked examples. Accuracy on 2,000 test questions (“always empty” scores 0.468):
| Connector | Visual tokens | Test accuracy |
|---|---|---|
| MLP projector (one token per patch) | 64 | 0.965 |
| Pixel shuffle \(r = 2\) + linear | 16 | 0.990 |
| Pixel shuffle \(r = 4\) + linear | 4 | 0.884 |
| Perceiver resampler, 16 latents | 16 | 0.697 |
| Perceiver resampler, 4 latents | 4 | 0.607 |
| Mean pool to one token | 1 | 0.468 |
Read these as a toy, not a benchmark. The task asks about one small cell, the worst case for compression. Pixel shuffle with \(r = 2\) did better than the per-patch MLP with 4 times fewer tokens: nothing is lost in the fold, and the reader has fewer positions to search. At \(r = 4\) each token holds 16 patches and the single linear layer must squeeze them, so accuracy drops. The resampler with 16 learned latents, given the same short training, is far behind the pixel shuffle with 16 tokens: the latents must first learn where to look. This is the same pattern the LLaVA-1.5 authors describe (resamplers converge more slowly with the same data), and why pixel shuffle became popular. Mean pooling to one token keeps almost nothing about where things are and scores at the level of always answering “empty”. With much longer training the resampler may close part of the gap [Likely]; exact numbers can differ slightly across PyTorch versions [Likely].
Where this is used
| Application | Connector choice and why |
|---|---|
| General chat VLMs (LLaVA family, many open models) | MLP projector, often after pixel shuffle; tokens spent on resolution and tiles (session 181, session 182) |
| Document AI and OCR | Keep many tokens: projector or pixel shuffle with \(r = 2\); aggressive ratios hurt precise localization (SmolVLM finding) (session 180) |
| On-device and small VLMs | Pixel shuffle with \(r = 3\) or 4 (SmolVLM: 64 tokens per 512 × 512 tile), so the small language model has a short context (session 183) |
| Video classification and action recognition | A resampler gives a fixed number of tokens for any clip length, with time embeddings for order; per-frame pixel shuffle plus frame sampling is the alternative (session 205) |
| Video question answering, Video-RAG and key-moment detection | Token budget per frame decides how many frames fit; few tokens per frame (e.g. 64) let an LLM see minutes of video and point to the moment that answers a question (session 210, session 206) |
| Video and image retrieval with a VLM re-ranker | Short visual prefixes (32 Q-Former tokens) make scoring many candidates cheap; BLIP-2's Q-Former also gives ITC embeddings for the first stage (session 208) |
| Multi-image and interleaved documents | Fixed small token count per image (Flamingo, Idefics2: 64) so dozens of images fit in one context |
| Frozen-LLM products | Q-Former or resampler with cross-attention keeps the text model unchanged and safe; only the connector is trained (session 179) |
Common mistakes
- Counting tokens from the wrong grid. CLIP ViT-L/14 at 336 pixels returns 577 vectors (576 patches + [CLS]); LLaVA drops the [CLS]. Qwen2-VL adds two special tokens. Always check how many placeholder tokens the processor inserts.
- Using the last ViT layer by habit. LLaVA and BLIP-2 both take the second-to-last layer; LLaVA measured the last layer as worse. Make the layer a setting you test.
- A wrong reshape in pixel shuffle. A plain
view(B, N/4, 4*d)concatenates 4 patches that are next to each other in the flattened row, not a 2 × 2 square. Split rows and columns first, then permute (Algorithm 2). Qwen2-VL avoids this by reordering patches in its image processor so that each 2 × 2 block is already consecutive [Likely]. - A grid not divisible by r. 27 × 27 cannot be shuffled with \(r = 2\). Change the resolution, pad, or choose \(r = 3\) as SmolVLM-2.2B does.
- Expecting a resampler to learn fast. A fresh Q-Former or resampler has many new parameters and must learn where to look. Without a representation stage (BLIP-2) or a lot of data, it can lag behind a plain MLP, as in the experiment above.
- Comparing connectors at different token counts. More tokens usually win (MM1). A fair comparison fixes the number of visual tokens and the resolution, then changes the connector.
- Forgetting that 32 or 64 tokens is a hard ceiling. A Q-Former cannot read a dense page of text, whatever the language model's size. Choose the connector for the task.
- Mismatched scales. Vision features and word embeddings can have very different norms. InternVL and Qwen2-VL put a LayerNorm before the MLP; skipping a norm or initializing the projection too large can make early training unstable [Likely].
Interview questions
What does the connector in a VLM do, and why can't we feed ViT outputs directly to the LLM?basic
LLaVA used a linear layer; LLaVA-1.5 an MLP. What changed and what does it cost?basic
Explain the Q-Former. Why 32 queries, and why does it need a first training stage?core
Draw the three Q-Former attention masks and explain each.core
How is the Perceiver resampler different from the Q-Former?core
What is pixel shuffle in a VLM? Does it lose information?core
How many visual tokens and how much KV cache does one image cost in LLaVA-1.5-7B? In Idefics3-8B?core
Recent open VLMs mostly use MLP projectors, not Q-Formers. Why?advanced
How would you feed a 2-minute video to each connector type?advanced
Where does spatial position information come from in each connector?advanced
Which layer of the ViT should feed the connector, and why not the last one?core
vision_feature_layer = -2), and BLIP-2 removes the last layer of its ViT and uses the second-to-last. In a CLIP-trained ViT the last layer is shaped by the contrastive objective, which cares about one global summary vector, so patch features become less local. The layer before keeps more detail. LLaVA measured the last layer as 0.96% worse on ScienceQA. Some models concatenate features from several layers, which a projector handles by making its input wider.Exercises
- Compute the parameters of the LLaVA-1.5-13B projector (\(d_v = 1{,}024\), \(d = 5{,}120\)).
- SmolVLM-2.2B uses SigLIP at 384 pixels with patch 14 and \(r = 3\), and a language model with \(d = 2{,}048\). Give the grid, the number of tokens, the width after the shuffle and the parameters of a bias-free projection.
- Qwen2-VL turns a 224 × 224 image into 66 tokens. Derive the number.
- A Perceiver resampler with \(R = 64\) latents reads 8 frames of 576 features. Compute the attention scores per layer and head, and compare with full self-attention over the 4,608 tokens.
- In the Q-Former ITG mask with 32 queries and 10 text tokens, how many entries of the \(42 \times 42\) mask are allowed?
- Write the inverse of pixel shuffle with \(r = 2\). Why does this prove that the shuffle loses nothing, and where can the connector lose information?
Answerscheck
<|vision_start|> and <|vision_end|> = 66. 4. \(TS = 4{,}608\): \(64 \times (4{,}608 + 64) = 299{,}008\) versus \(4{,}608^2 = 21{,}233{,}664\), 71 times more. 5. Query rows: \(32 \times 32 = 1{,}024\). Text rows: each sees 32 queries, and text token \(k\) sees \(k\) text tokens: \(10 \times 32 + (1 + 2 + \dots + 10) = 320 + 55 = 375\). Total 1,399 of 1,764. 6. Reshape each wide vector to \(2 \times 2 \times d_v\), and place block \((a, b)\) back at rows \(2a, 2a+1\) and columns \(2b, 2b+1\). Since an exact inverse exists, the map is a bijection (a permutation of numbers), so nothing is lost. Loss can only happen in the projection \(W\) when \(r^2 d_v \gt d\) (for example 12,288 to 576 in SmolVLM-256M), and in the language model's limited attention.Key takeaways
- A connector maps \(N\) vision vectors of width \(d_v\) to \(M\) tokens of width \(d\) (178.1). The width change is easy; the token count \(M\) sets the cost of every image.
- The MLP projector (178.3) keeps one token per patch, has about 21M parameters for a 7B model, trains fast and loses nothing; LLaVA-1.5-7B pays 576 tokens and 288 MiB of KV cache per image.
- The Q-Former reads the image with 32 learned queries through cross-attention (178.5); a three-loss first stage with different masks teaches the queries what text needs, then one linear layer (178.7) feeds a frozen LLM.
- The Perceiver resampler (178.8) turns features of any number of frames into 64 tokens at cost \(R(TS + R)\) (178.9); it is the natural tool for long video and many images, but learns slowly and keeps layout only through the features.
- Pixel shuffle (178.10, 178.11) folds \(r \times r\) neighbors into one token: lossless rearrangement, \(r^2\) fewer tokens, layout kept; \(r = 2\) for detail, \(r = 3\) or 4 for small models.
- Ablations agree that token count and resolution matter more than the connector type; most current models use pixel shuffle or pooling plus an MLP.
Key formulas
References for this session
- Liu et al. (2023). Visual Instruction Tuning
- Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
- Jaegle et al. (2021). Perceiver: General Perception with Iterative Attention
- Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
- Liu et al. (2023). Visual Instruction Tuning
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
- Jaegle et al. (2021). Perceiver: General Perception with Iterative Attention
- Laurençon et al. (2024). What matters when building vision-language models?
- Laurençon, Marafioti, Sanh & Tronchon (2024). Building and better understanding vision-language models: insights and future directions
- Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
- Chen et al. (2024). How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites (InternVL 1.5)
- Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
- McKinzie et al. (2024). MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
- Cha, Kang, Mun & Roh (2023). Honeybee: Locality-enhanced Projector for Multimodal LLM
- Shi et al. (2016). Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
- Liu et al. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Yin et al. (2023). A Survey on Multimodal Large Language Models
- Zhang et al. (2024). MM-LLMs: Recent Advances in MultiModal Large Language Models