LLM & VLM courseBlog
178Track 7: Vision and VLMs · session 16 of 38

Connectors: projector, Q-Former, Perceiver resampler, pixel shuffle

A vision encoder gives hundreds of vectors in its own space; a language model reads vectors of another size, and every vector costs context and memory. The connector sits between them. This session builds the four connectors used in real VLMs: the linear or MLP projector of LLaVA, the Q-Former of BLIP-2, the Perceiver resampler of Flamingo and Idefics2, and pixel shuffle (patch merging) of Idefics3, SmolVLM, InternVL and Qwen2-VL. For each one we write the math, count the tokens and parameters with real model configs, and test them side by side in a small PyTorch experiment.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 20 (self-attention vs cross-attention), 164 (ViT), 166 (CLIP), 173 (ALBEF and BLIP) and 177 (anatomy of a VLM)

What you will learn

  • The two jobs of a connector: change the width of each vector, and (maybe) change the number of vectors, and why the number of tokens decides most of the cost
  • The linear and MLP projector of LLaVA and LLaVA-1.5: formula, parameter count (21.0M for LLaVA-1.5-7B) and why it keeps every patch
  • The Q-Former of BLIP-2: 32 learned queries, cross-attention every other block, three attention masks in stage 1, and a linear layer into a frozen LLM in stage 2
  • The Perceiver resampler of Flamingo: learned latents that cross-attend to features of any number of frames, with keys and values that include the latents, and its cost R(M+R)
  • Pixel shuffle and patch merging: fold each r × r block of patches into one token with r² times more channels, used with r = 2, 3 or 4 in Idefics3, SmolVLM, InternVL and Qwen2-VL
  • What the ablations say (Flamingo, BLIP-2, LLaVA-1.5, MM1, Idefics2, SmolVLM): token count and resolution matter more than the connector type, and small models like aggressive compression
  • How to compute tokens, connector parameters and KV-cache memory per image for a real model, and how each connector handles video

Why this matters

In session 177 we split a VLM into three parts: a vision encoder, a connector and a language model. The vision encoder (a ViT such as CLIP or SigLIP) gives one vector per image patch, for example 576 vectors of width 1,024. The language model reads vectors of its own width, for example 4,096, and every vector it reads is one more position in its context. The connector is the small piece in the middle that makes the two fit.

The connector is small (from about 0.3% of the parameters in LLaVA-1.5-7B to about 5% in BLIP-2 with OPT-2.7B), but it decides two things that matter a lot:

  • The width. It must map each visual vector into the space of the language model's word embeddings, so the language model can read it like a (strange) word.
  • The number of visual tokens. It may keep one token per patch, or squeeze the image into a fixed, smaller set. This number sets the cost of every image: prefill compute, KV-cache memory, and how many images or video frames fit in the context (session 80).

Four designs cover almost every model you will meet:

  1. A linear or MLP projector (LLaVA, LLaVA-1.5): one small network applied to each patch vector. Same number of tokens, new width.
  2. A Q-Former (BLIP-2): a small BERT-like transformer whose 32 learned queries read the image through cross-attention, so every image becomes exactly 32 tokens.
  3. A Perceiver resampler (Flamingo, Idefics2): a set of learned latents that cross-attend to the features of an image or of a whole video and return a fixed number of tokens, 64 in Flamingo.
  4. Pixel shuffle or patch merging (Idefics3, SmolVLM, InternVL, Qwen2-VL): fold each \(2 \times 2\) (or \(3 \times 3\), \(4 \times 4\)) block of neighboring patches into one token with more channels, then project. Four to sixteen times fewer tokens, no learned queries.

The connector's job, in shapes

Let the vision encoder see an image of \(H \times W\) pixels with patch size \(P\). It returns a grid of \(N\) patch vectors of width \(d_v\). The language model has hidden width \(d\). A connector is any function

\[ f_\phi : \R^{N \times d_v} \to \R^{M \times d}, \qquad N = \frac{H}{P} \cdot \frac{W}{P} \tag{178.1} \]
Read the symbols
\(N\)
the number of patch vectors from the vision encoder (some encoders also add one [CLS] vector)
\(d_v\)
the width of the vision features: 1,024 for CLIP ViT-L/14, 1,152 for SigLIP-SO400M, 1,408 for EVA ViT-g/14
\(M\)
the number of visual tokens the language model will see; \(M = N\) for a projector, \(M\) fixed for Q-Former and resampler, \(M = N / r^2\) for pixel shuffle
\(d\)
the hidden width of the language model, the same width as its word embeddings (4,096 for Vicuna-7B and Llama 3 8B)
\(\phi\)
the parameters of the connector; they are almost always trained from scratch

Two families of designs follow from the shape. If \(M = N\), the connector only changes the width; the cost of the image is set by the vision encoder (resolution and patch size). If \(M \lt N\), the connector also compresses; we then have to ask what is lost. There are two ways to compress: learned queries that pick what to read (Q-Former, resampler), and fixed folding of neighbors (pixel shuffle). Figure 1 shows all of them for real models.

vision encoder outputconnectortokens the language model reads
Figure 1. Shapes in and out of the connector of seven real models, from their configs and papers. The width of each block is proportional to the number of vectors, its height to the width of the vectors (both on a square-root scale so small and large fit). The readout gives the connector type, its parameter count and the KV cache that the visual tokens need in the language model (16-bit, all layers).
How to read Figure 1

LLaVA-1.5-7B: 576 vectors of width 1,024 go in, 576 tokens of width 4,096 come out; nothing is compressed and one image costs 288 MiB of KV cache. Click BLIP-2: 257 vectors of width 1,408 become 32 tokens, and the cache per image drops to 10 MiB. Click Idefics3-8B: pixel shuffle with \(r = 2\) turns 676 patches into 169 tokens of width 4,608 before the linear projection. Click Flamingo: zero tokens enter the language model sequence, because Flamingo reads its 64 resampled tokens through extra cross-attention layers (session 179). Move the slider to see the cost of 8 or 32 images, tiles or frames.

Connector 1: the linear and MLP projector (LLaVA)

The simplest connector maps each patch vector on its own. LLaVA (Liu et al., 2023) takes the grid features of CLIP ViT-L/14 and applies one trainable matrix [Certain]:

\[ \mathbf{h}_i = W \mathbf{z}_i, \qquad i = 1, \dots, N, \qquad W \in \R^{d \times d_v} \tag{178.2} \]

LLaVA-1.5 (Liu et al., 2023b) replaced the matrix by a two-layer MLP with a GELU in the middle, and reported better scores (on MME, 1,355.2 versus 1,323.8 for the same data, their Table 2) [Certain]:

\[ \mathbf{h}_i = W_2\, \operatorname{GELU}\!\big(W_1 \mathbf{z}_i + \mathbf{b}_1\big) + \mathbf{b}_2, \qquad W_1 \in \R^{d \times d_v},\ W_2 \in \R^{d \times d} \tag{178.3} \]
Read the symbols
\(\mathbf{z}_i\)
the feature of patch \(i\) from the vision encoder, a vector in \(\R^{d_v}\)
\(\mathbf{h}_i\)
the visual token given to the language model, a vector in \(\R^{d}\); it is placed in the sequence where the image placeholder token was
\(W_1, \mathbf{b}_1, W_2, \mathbf{b}_2\)
the weights and biases of the two linear layers; the same weights are used for every patch
GELU
the smooth activation used inside transformer MLPs (session 21)

The parameter count is easy:

\[ |\phi_{\text{MLP}}| = d_v d + d + d^2 + d \tag{178.4} \]

Two details from the LLaVA code and papers matter in practice. First, the features are taken from the second-to-last layer of the ViT, not the last: vision_feature_layer = -2 in the Hugging Face config [Certain]. The LLaVA paper reports that the last layer was 0.96% worse on ScienceQA and suggests that the last layer of CLIP is more global and abstract, while the one before keeps more local detail [Certain]. Second, the [CLS] vector is dropped, so the 577 outputs of ViT-L/14 at 336 pixels become 576 tokens (vision_feature_select_strategy = "default") [Certain].

Worked example: LLaVA-1.5-7B, one image

CLIP ViT-L/14 at 336 × 336 pixels: \(336 / 14 = 24\) patches per side, \(N = 24^2 = 576\) patch vectors of width \(d_v = 1{,}024\). Vicuna-7B has \(d = 4{,}096\).

Parameters by (178.4): \(1{,}024 \times 4{,}096 + 4{,}096 + 4{,}096^2 + 4{,}096 = 20{,}979{,}712\), about 21.0M, roughly 0.3% of the 7B language model. (LLaVA-1's single matrix with a bias would be 4,198,400.)

Tokens: \(M = N = 576\). KV cache per token in Vicuna-7B (32 layers, 32 key-value heads of size 128, 16-bit): \(2 \times 32 \times 4{,}096 \times 2 = 524{,}288\) bytes = 0.5 MiB. One image: \(576 \times 0.5 = 288\) MiB. Eight images: 2.25 GiB, before any text.

Figure 2 runs this projector for real on a toy image of 4 × 4 patches (a cat on grass under the sky) with feature width 3 and language-model width 4. Watch the loop: every patch goes through the same small network, and every patch becomes one token. The token count out is the token count in.

sky patchcat patchgrass patchpositive valuenegative value
The projector loop
    Figure 2. The MLP projector (178.3) computed live, patch by patch. Each patch \(\mathbf{z}_i \in \R^3\) goes through \(W_1\) (\(4 \times 3\)), GELU and \(W_2\) (\(4 \times 4\)), with fixed toy weights, and lands in slot \(i\) of the language-model input. The highlighted line of the algorithm and the formula panel show the current step with the real numbers. The linear mode is LLaVA-1's single matrix (178.2). A real model does all patches at once in one matrix product; the loop only makes the per-patch view visible.
    How to read Figure 2

    Press Step three times. Patch 1 is sky, \(\mathbf{z}_1 = (0.92, 0.06, 0.10)\). Step 1 gives \(\mathbf{a} = W_1\mathbf{z}_1 + \mathbf{b}_1 = (0.738, 0.006, -0.278, 0.124)\), step 2 gives \(\operatorname{GELU}(\mathbf{a}) = (0.568, 0.003, -0.109, 0.068)\) (GELU keeps large positive numbers and squashes negative ones toward 0), and step 3 gives the token \(\mathbf{h}_1 = (0.407, -0.183, -0.043, 0.214)\), which flies into slot 1. Press Play: patch 6 (a cat patch, \((0.32, 1.06, 0.12)\)) gives a very different token, \((-0.021, 0.517, 0.080, 0.413)\), because its content differs, not because the weights change. At the end 16 patches have made 16 tokens. Switch to linear: one step per patch, 16 parameters instead of 36.

    Why is something so simple strong? Because it throws nothing away. Every patch stays a token, and the 7B language model, with its many layers of attention, decides what to use. The price is the token count: 576 tokens for a low-resolution image, and thousands once the image is cut into high-resolution tiles (session 180). That price is the reason for the other three designs.

    The training recipe shows why the projector is also easy to train. LLaVA-1.5 stage 1 trains only the projector on 558K image-caption pairs, with the ViT and the language model frozen; stage 2 trains the projector and the language model on 665K instruction examples [Certain]. One epoch of stage 1 with a fresh 21M-parameter MLP is enough to “align” the two spaces. The LLaVA-1.5 paper notes in its limitations that visual resamplers “cannot achieve convergence as efficiently as LLaVA with a comparable amount of training data, probably due to more trainable parameters in the resamplers” [Certain].

    Connector 2: the Q-Former (BLIP-2)

    BLIP-2 (Li et al., 2023) keeps both the image encoder and the language model frozen and trains only a Querying Transformer (Q-Former) between them [Certain]. The idea: instead of passing every patch, let a small set of learned queries ask the image for what matters, and pass only their answers.

    The facts from the paper [Certain]:

    • The Q-Former has two transformer parts that share their self-attention layers: an image transformer that runs on the queries, and a text transformer that can act as a text encoder or a text decoder.
    • There are 32 queries, each of width 768, the hidden width of the Q-Former. They are parameters, learned like weights.
    • The queries talk to each other through self-attention, and read the frozen image features through cross-attention layers inserted every other transformer block.
    • The Q-Former is initialized from BERT-base; the cross-attention layers start random. In total it has 188M parameters.
    • The output \(Z\) is \(32 \times 768\), “much smaller than the size of frozen image features (e.g. 257 × 1024 for ViT-L/14)”. The image encoder is ViT-L/14 from CLIP or ViT-g/14 from EVA-CLIP, with its last layer removed (they use the second-to-last layer).

    The cross-attention is the standard one from session 20: the queries ask, the image patches answer.

    \[ \operatorname{CA}(X, Z_v) = \softmax\!\Big(\frac{(X W_Q)(Z_v W_K)^\top}{\sqrt{d_h}}\Big)\, Z_v W_V, \qquad X \in \R^{32 \times 768},\ Z_v \in \R^{257 \times 1408} \tag{178.5} \]
    Read the symbols
    \(X\)
    the current states of the 32 queries (at the first layer: the learned query embeddings)
    \(Z_v\)
    the frozen image features: for ViT-g/14 at 224 pixels, \(16 \times 16 = 256\) patches plus one [CLS] vector, width 1,408
    \(W_Q, W_K, W_V\)
    query, key and value projections; \(W_K\) and \(W_V\) map width 1,408 to the Q-Former width (the cross-attention input size is 1,408 in the Hugging Face config, encoder_hidden_size) [Certain]
    \(d_h\)
    the width of one attention head

    The result has 32 rows whatever the size of the image grid: the number of output tokens is the number of queries. Written for one query \(j\) and one patch \(i\), (178.5) is three small steps: a score, a softmax over the patches, and a weighted sum.

    \[ \begin{aligned} s_{j,i} &= \frac{(W_Q \mathbf{x}_j)^\top (W_K \mathbf{z}_i)}{\sqrt{d_h}}, \qquad m_j = \max_i s_{j,i} \\ a_{j,i} &= \frac{\exp(s_{j,i} - m_j)}{\sum_{i'} \exp(s_{j,i'} - m_j)} \\ \mathbf{o}_j &= \sum_{i} a_{j,i}\, W_V \mathbf{z}_i \end{aligned} \tag{178.5a} \]
    Read the symbols
    \(\mathbf{x}_j\)
    query \(j\), one row of \(X\); it is a learned vector, the same for every image
    \(s_{j,i}\)
    the score: how well patch \(i\) matches what query \(j\) looks for
    \(m_j\)
    the largest score of query \(j\); subtracting it keeps the exponentials small and does not change \(a_{j,i}\)
    \(a_{j,i}\)
    the attention weight; for each query the weights over all patches sum to 1
    \(\mathbf{o}_j\)
    the output of query \(j\): an average of the patch values, weighted by \(a_{j,i}\). One query gives one output token

    Figure 3 runs (178.5a) on the toy image. Each query has been set by hand to look for one kind of content, which is what stage-1 training makes real queries do. Note that a query does not know where the cat is: it finds the cat by content, wherever it sits.

    query and its attention weightsoutput tokenskycatgrass
    Cross-attention of the queries, (178.5a)
      Figure 3. Learned queries read the image, computed live. Left: the 16 patch features (width 3), which are the keys and the values. Right: the queries \(\mathbf{x}_j\) and, once computed, their outputs \(\mathbf{o}_j\). Bottom: one bar per patch, first the scores \(s_{j,i}\), then the softmax weights \(a_{j,i}\) (also drawn as circles on the grid). Toy weights: \(W_Q = 3.5\,I\), \(W_K = W_V = I\), one head of width \(d_h = 3\). The real Q-Former has 32 queries of width 768, 12 heads, and does this in every other block.
      How to read Figure 3

      Step once: query 1 (looking for “cat”) scores all 16 patches. The best is patch 7, \(s = 3.85 \times 1.08 / \sqrt{3} = 2.401\). Step again: the softmax divides by \(\sum_i \exp(s_{1,i} - 2.401) = 5.083\), so patch 7 gets weight 0.197 and the four cat patches together get 0.693. Step again: the output \(\mathbf{o}_1 = (0.345, 0.763, 0.309)\) is mostly “cat”. Play on: query 2 (sky) ends at \((0.806, 0.205, 0.229)\), query 3 (grass) at \((0.222, 0.273, 0.841)\). Now pick “cat at bottom left” and play again: query 1 still finds the cat (weight 0.682 on the cat patches) although the cat moved. Lower \(R\): the output has \(R\) tokens, so \(R\) sets the compression, \(16 \to R\).

      Stage 1: teach the queries what to look for

      A new set of queries knows nothing. BLIP-2 first trains the Q-Former on image–text pairs with the three losses of session 173: contrastive (ITC), matching (ITM) and image-grounded text generation (ITG). The same weights serve all three; what changes is the self-attention mask between the queries and the text tokens [Certain]:

      • ITC (unimodal mask): queries see only queries, text sees only text. The image–text score is the best match over the 32 query outputs and the text [CLS] vector:
      \[ s(I, T) = \max_{j = 1..32}\ \mathbf{z}_j^\top \mathbf{t}_{\text{cls}} \tag{178.6} \]
      • ITG (multimodal causal mask): queries see only queries; each text token sees all queries and the earlier text tokens. The text cannot see the image directly, so all the information needed to write the caption must pass through the 32 queries. This is what forces the queries to extract what the text needs.
      • ITM (bidirectional mask): everything sees everything; each query output goes to a 2-way classifier and the 32 logits are averaged.

      BLIP-2 uses in-batch negatives for ITC instead of BLIP's momentum queue, because with frozen encoders the batch can be large (2,320 with ViT-L, 1,680 with ViT-g in stage 1) [Certain]. Figure 4 shows the three masks.

      queriestext tokensallowed to attendmasked
      Figure 4. The Q-Former and its masks, drawn with 4 queries and 4 text tokens instead of 32 and many. Left: one pair of blocks, with self-attention shared by queries and text, cross-attention to the frozen image features only on the query side, and separate feed-forward layers. Right: the self-attention mask (row = the token that looks, column = the token it may see). In stage 2 the text tokens leave the Q-Former and the 32 outputs go through one linear layer into the frozen LLM.
      How to read Figure 4

      ITC: two separate green squares, 32 of 64 entries allowed; the queries and the text are kept apart so the contrastive score cannot cheat. ITM: all 64 entries allowed. ITG: the query block is full, the text rows see all queries plus a lower triangle of text, 42 of 64 allowed; the queries cannot see the text, so they must carry the image information on their own. Stage 2: the text part is gone; the readout shows the linear layer that maps width 768 to the width of the language model.

      Stage 2: plug the queries into a frozen LLM

      In stage 2 the Q-Former output goes through one fully connected layer to the width of the language model, and the 32 resulting vectors are put in front of the text as “soft visual prompts” [Certain]:

      \[ \mathbf{h}_j = W_{\text{fc}}\, \mathbf{z}_j + \mathbf{b}_{\text{fc}}, \qquad j = 1, \dots, 32, \qquad W_{\text{fc}} \in \R^{d \times 768} \tag{178.7} \]

      For OPT-2.7B (\(d = 2{,}560\)) this layer has \(768 \times 2{,}560 + 2{,}560 = 1{,}968{,}640\) parameters. The language model stays frozen and is trained through only by the language-modeling loss. For OPT it is the next-token loss; for FlanT5 a prefix loss [Certain].

      Is stage 1 needed? BLIP-2 tested it: without the representation-learning stage, zero-shot VQAv2 is much lower for both OPT and FlanT5, and with OPT “performance drastically degrades as training proceeds” (catastrophic forgetting) [Certain, their Figure 5]. The paper notes that a Q-Former trained only with the generative loss is “similar to the Perceiver Resampler in Flamingo”. So the three-loss stage is what makes the 32 queries useful when the language model is frozen.

      The result that made BLIP-2 famous: with ViT-g and FlanT5-XXL it reached 65.0 on zero-shot VQAv2 test-dev, against 56.3 for Flamingo-80B, with 108M trainable parameters (in their Table 2) versus 10.2B, “54x fewer” [Certain]. The cost: everything the language model learns about the image must fit in 32 vectors of width 768. That is fine for “what is in this photo”, but tight for reading a page of text or counting many small objects. Later instruction-tuned models mostly moved back to projectors with many tokens (see “What the ablations say” below).

      Connector 3: the Perceiver resampler (Flamingo)

      The Perceiver (Jaegle et al., 2021) solved a general problem: self-attention over \(M\) inputs costs \(O(M^2)\), too much for raw pixels (\(M = 50{,}176\) for a 224 × 224 image). Its trick is an asymmetric attention: the keys and values come from the large input, but the queries come from a small, learned latent array of size \(N \ll M\). One cross-attention then costs \(O(MN)\), and the deep part of the network runs on the latents at \(O(N^2)\) [Certain].

      Flamingo (Alayrac et al., 2022) used this idea as its connector, the Perceiver resampler [Certain]:

      • The frozen vision encoder (an NFNet-F6) produces a grid of features per frame. For video, frames are sampled at 1 frame per second and encoded one by one; a learned temporal embedding is added to all features of a frame. An image is a video with one frame. No spatial position encoding is added; the authors saw no gain from it and suggest the CNN features already carry position.
      • All features of all frames are flattened into one long sequence \(X_f\).
      • A fixed set of learned latents, 64 in Flamingo, cross-attend to \(X_f\) through 6 layers of attention + feed-forward. The keys and values are the concatenation of the visual features and the latents themselves, which the authors found slightly better than using the features alone.
      • The output is always 64 tokens, “independently from the input image resolution or the number of input video frames”. The resampler has about 200M parameters, the same in all three model sizes (6 layers, width 1,536, 16 heads).

      One layer of the resampler, written like the Flamingo pseudo-code:

      \[ \begin{aligned} X_f &= \operatorname{flatten}\big(F + \mathbf{e}^{\text{time}}\big) \in \R^{TS \times d_r} \\ L &\leftarrow L + \operatorname{Attn}\big(q = L,\ kv = [X_f;\, L]\big) \\ L &\leftarrow L + \operatorname{FFN}(L) \end{aligned} \tag{178.8} \]
      Read the symbols
      \(F\)
      the visual features of \(T\) frames, each with \(S\) spatial positions, shape \(T \times S \times d_r\)
      \(\mathbf{e}^{\text{time}}\)
      a learned vector per frame index, added to every position of that frame
      \(L\)
      the latent array, \(R \times d_r\); at the first layer it is the learned latents (\(R = 64\) in Flamingo)
      \([X_f; L]\)
      the visual features and the latents stacked into one sequence of length \(TS + R\), used for the keys and values
      \(\operatorname{Attn}\)
      multi-head attention; FFN is the feed-forward block (Flamingo uses a squared ReLU there)
      Algorithm 1: Perceiver resampler forward pass
      1. input: frame features \(F \in \R^{T \times S \times d_r}\); learned time embeddings \(\mathbf{e}^{\text{time}} \in \R^{T_{\max} \times d_r}\); learned latents \(L_0 \in \R^{R \times d_r}\); \(K\) layers
      2. \(F_t \leftarrow F_t + \mathbf{e}^{\text{time}}_t\) for every frame \(t\) // only time is encoded, not space
      3. \(X_f \leftarrow\) flatten \(F\) to \(TS\) rows // any \(T\), any \(S\)
      4. \(L \leftarrow L_0\)
      5. for \(k = 1 \dots K\):
      6.   \(L \leftarrow L + \operatorname{Attn}_k(q = L,\ kv = [X_f; L])\) // \(R\) queries, \(TS + R\) keys
      7.   \(L \leftarrow L + \operatorname{FFN}_k(L)\)
      8. return \(L \in \R^{R \times d_r}\) // always \(R\) tokens

      Figure 5 runs Algorithm 1 on a toy video: up to 3 frames of 4 × 4 features in which the cat moves. Two details become visible. First, the latents are also keys, so each latent spends part of its attention on itself and on the other latents. Second, the number of keys grows with the number of frames, but the number of outputs does not. The open-source OpenFlamingo code applies a LayerNorm to the features and to the latents before each attention [Likely]; the toy does the same.

      latents and their attention weightsupdated latentskycatgrass
      Perceiver resampler, Algorithm 1 in the toy
        Figure 5. The Perceiver resampler computed live. Left: the features of \(T\) frames (the cat moves to the right in frame 2 and to the bottom left in frame 3), each with its time embedding added. Right: the \(R\) latents; the circle on each shows its weight when it acts as a key. Bottom: one bar per key, the \(16T\) feature rows and then the \(R\) latents. Toy weights: query scale \(c = 1.8\), keys and values are the normalized rows, one head of width 3, no FFN, and the same weights in every layer (Flamingo: 6 layers with their own weights, width 1,536, 16 heads, 64 latents).
        How to read Figure 5

        Choose \(T = 1\) and step to layer 1, latent 1. Its normalized value is \(\hat{L}_1 = (-0.707, 1.414, -0.707)\), and its best key is itself, \(s = 3.118\). After the softmax it gives 0.191 to itself and 0.740 to the cat patches, and the update is \(L_1 \leftarrow (0, 1.1, 0) + (-0.524, 1.317, -0.794) = (-0.524, 2.417, -0.794)\). Now choose \(T = 3\): there are 52 keys, the weight on itself falls to 0.076 and the cat rows of all three frames get 0.878. The counter shows \(R(TS + R) = 4 \times 52 = 208\) scores per layer, and still 4 tokens come out.

        The cost of one resampler layer is set by the attention scores. Per head, the \(R\) latent queries meet \(TS + R\) keys. Compare it with running full self-attention over the same visual tokens, which is what the language model would do if it received all of them:

        \[ C_{\text{resampler}} = R\,(TS + R), \qquad C_{\text{full}} = (TS)^2, \qquad \frac{C_{\text{full}}}{C_{\text{resampler}}} \approx \frac{TS}{R} \tag{178.9} \]
        Worked example: 30 seconds of video

        Take 30 frames (1 per second) with \(S = 729\) features each (a 27 × 27 SigLIP grid at 384 pixels) and \(R = 64\) latents. Then \(TS = 21{,}870\).

        Resampler scores per layer and head: \(64 \times (21{,}870 + 64) = 1{,}403{,}776\). Full self-attention over the same tokens: \(21{,}870^2 = 478{,}296{,}900\), which is 341 times more. And the language model sees 64 tokens instead of 21,870.

        The same arithmetic for one LLaVA-size image (\(TS = 576\)): \(64 \times 640 = 40{,}960\) versus \(331{,}776\), only 8.1 times. The resampler pays off most when the input is long.

        projector: every feature is a tokenPerceiver resampler
        Figure 6. What the resampler buys, from (178.9). Top pair: the number of visual tokens the language model must hold (projector: \(TS\); resampler: \(R\)). Bottom pair: attention scores per layer and head over the visual part (projector: full self-attention \((TS)^2\) inside the language model; resampler: \(R(TS + R)\) inside the resampler). Log scale: each grid line is 10 times more.
        How to read Figure 6

        Start at one frame of 576 features and 64 latents: the resampler cuts 576 tokens to 64 and the score count by 8.1 times. Raise the frames to 30 and pick 729 features per frame: the language model would need 21,870 tokens with a projector but still 64 with the resampler, and the score ratio grows to about 341. Raise the latents: cost and tokens grow linearly in \(R\). The resampler's output size never depends on \(T\) or \(S\); that is why Flamingo can take any number of frames.

        Two more properties matter. First, the resampler is blind to order unless the features carry position: attention does not care where a key sits in the sequence. Flamingo adds only time embeddings; spatial layout survives only as far as the CNN features encode it. Second, in Flamingo the 64 tokens do not enter the language model's sequence. The frozen language model reads them through new gated cross-attention layers (GATED XATTN-DENSE) inserted between its blocks, every 4th block for Flamingo-9B and every 7th for Flamingo-80B [Certain]. That is the topic of session 179. Idefics2 later put the resampler's 64 tokens directly into the sequence of a decoder-only model.

        The Flamingo ablation supports the design at equal parameter budget [Certain, their Table 3, Flamingo-3B, short training]: the overall score was 70.7 with the Perceiver resampler, 66.6 with an MLP and 66.7 with a plain Transformer, and both alternatives were also slower. A medium-size resampler beat small and large ones, and larger resamplers made training unstable when scaled with the language model (their Table 10).

        Connector 4: pixel shuffle and patch merging

        The fourth idea compresses with no learned queries at all. Take the \(h \times w\) grid of patch vectors. Cut it into blocks of \(r \times r\) neighbors. Concatenate the \(r^2\) vectors of each block into one long vector. You now have \(r^2\) times fewer tokens, each \(r^2\) times wider. Then a linear layer or an MLP maps the wide vector to the width of the language model.

        The name comes from super-resolution. Shi et al. (2016) introduced a “sub-pixel convolution” layer that moves channels into space to make an image larger; PyTorch calls this PixelShuffle [Certain]. VLMs use the inverse operation (space to depth, PyTorch's PixelUnshuffle), but the VLM papers still call it pixel shuffle [Certain, SmolVLM paper: “pixel shuffle (space-to-depth)”]. With grid positions \((u, v)\):

        \[ \mathbf{y}_{a,b} = \Big[\, \mathbf{z}_{ra,\, rb};\ \mathbf{z}_{ra,\, rb+1};\ \dots;\ \mathbf{z}_{ra+r-1,\, rb+r-1} \,\Big] \in \R^{r^2 d_v}, \qquad 0 \le a \lt \tfrac{h}{r},\ 0 \le b \lt \tfrac{w}{r} \tag{178.10} \]
        \[ M = \frac{N}{r^2}, \qquad \mathbf{h}_{a,b} = W\, \mathbf{y}_{a,b}, \qquad W \in \R^{d \times r^2 d_v} \tag{178.11} \]
        Read the symbols
        \(\mathbf{z}_{u,v}\)
        the patch vector at row \(u\), column \(v\) of the grid
        \(r\)
        the shuffle ratio (scale factor): 2 in Idefics3 and InternVL, 3 in SmolVLM-2.2B, 4 in SmolVLM-256M (from their configs) [Certain]
        \(\mathbf{y}_{a,b}\)
        the concatenation of the \(r^2\) patch vectors of block \((a, b)\); the semicolons mean “stack”
        \(W\)
        the projection (Idefics3 and SmolVLM: one linear layer without bias; InternVL: LayerNorm + MLP; Qwen2-VL: LayerNorm + MLP)

        The rearrangement itself is lossless: it only moves numbers. The PyTorch code below checks that the output holds exactly the same numbers as the input. Compression happens in \(W\): it maps \(r^2 d_v\) numbers to \(d\) numbers. For Idefics3 that is \(4 \times 1{,}152 = 4{,}608\) to \(4{,}096\), almost no squeeze. For SmolVLM-256M it is \(16 \times 768 = 12{,}288\) to \(576\), 21 times. And unlike a resampler, every output token comes from one known place in the image, so the layout is kept.

        Patch merging is the same operation under another name. The Swin Transformer merges each \(2 \times 2\) group of neighbors (concatenating \(4C\) channels, then a linear layer to \(2C\)) to build a pyramid [Certain]. Qwen2-VL uses an MLP after the ViT “to compress adjacent 2 × 2 tokens into a single token”; a 224 × 224 image with patch size 14 becomes 66 tokens: \(16 \times 16 / 4 = 64\), plus the special tokens <|vision_start|> and <|vision_end|> [Certain]. InternVL 1.5 uses pixel shuffle “to reduce the number of visual tokens to one-quarter”, so a 448 × 448 tile becomes 256 tokens [Certain].

        patch grid (from the ViT)tokens after the shuffleselected token and its patches
        Pixel shuffle, block by block
          Figure 7. Pixel shuffle on an 8 × 8 grid of patches, animated one block at a time. Each colored square is one patch vector; its color stands for its content. On the right, each token is drawn as a stack of the \(r^2\) patch vectors it holds, in the order of (178.10); the patches of the current block fly into their token. Each patch carries \(d_v = 2\) numbers, \(\mathbf{z}_{u,v} = (u + v/10,\ \text{content code})\), so the stacked vector shows where every number came from. The toy \(W\) averages each kind of number; a trained \(W\) is a full learned matrix. Click any token (right) or patch (left) to jump to it.
          How to read Figure 7

          At \(r = 2\) the 64 patches become 16 tokens, each a stack of 4 patch vectors. Click the token in row 0, column 1: it holds patches (0,2), (0,3), (1,2), (1,3), a 2 × 2 square in the image, so \(\mathbf{y}_{0,1} = [0.2, 0 \mid 0.3, 0 \mid 1.2, 0 \mid 1.3, 0]\), 8 numbers, and the toy \(W\) gives \(\mathbf{h} = (0.75, 0)\). Press Play to watch all 16 blocks fold, one after the other. At \(r = 4\) there are only 4 tokens with 16 patches each. At \(r = 1\) nothing changes. The readout also gives the real sizes when the grid is SigLIP's (width 1,152): the token width grows from 1,152 to 4,608 at \(r = 2\) and 18,432 at \(r = 4\).

          The grid must be divisible by \(r\). That explains a detail of SmolVLM: at 384 pixels with patch 14, SigLIP gives a \(27 \times 27\) grid, which is not divisible by 2 or 4, and the 2.2B model uses \(r = 3\): 729 patches become 81 tokens [Certain from the config; that the divisibility drove the choice is Likely]. The 256M model uses SigLIP-B/16 at 512 pixels, a \(32 \times 32\) grid, and \(r = 4\): 1,024 patches become 64 tokens [Certain].

          Worked example: Idefics3-8B, one 364 × 364 tile

          SigLIP-SO400M with patch 14: \(364 / 14 = 26\) per side, \(N = 676\) patches of width 1,152. Pixel shuffle with \(r = 2\): \(M = 676 / 4 = 169\) tokens of width \(4 \times 1{,}152 = 4{,}608\).

          Projection to Llama 3 8B (\(d = 4{,}096\)), no bias: \(4{,}608 \times 4{,}096 = 18{,}874{,}368\) parameters.

          KV cache in Llama 3 8B (32 layers, 8 key-value heads of size 128, 16-bit): \(2 \times 32 \times 1{,}024 \times 2 = 131{,}072\) bytes per token. One tile: \(169 \times 131{,}072 = 22{,}151{,}168\) bytes = 21.1 MiB. Without the shuffle it would be 676 tokens and 84.5 MiB: the shuffle saves a factor of 4 in every later step.

          Algorithm 2: pixel shuffle connector (Idefics3 / SmolVLM style)
          1. input: patch features \(Z \in \R^{B \times N \times d_v}\) with \(N = h \times w\); ratio \(r\) that divides \(h\) and \(w\); projection \(W \in \R^{d \times r^2 d_v}\)
          2. reshape \(Z\) to \(B \times \frac{h}{r} \times r \times \frac{w}{r} \times r \times d_v\) // split rows and columns into blocks
          3. permute to \(B \times \frac{h}{r} \times \frac{w}{r} \times r \times r \times d_v\) // bring each block's \(r \times r\) patches together
          4. reshape to \(B \times \frac{N}{r^2} \times r^2 d_v\) // one wide vector per block, (178.10)
          5. \(H \leftarrow Y W^\top\) // (178.11); InternVL and Qwen2-VL use LayerNorm + MLP here
          6. return \(H \in \R^{B \times \frac{N}{r^2} \times d}\) // placed where the image tokens go in the text

          Small models want more compression, not less. SmolVLM compared \(r = 2\) and \(r = 4\) and found that “smaller VLMs benefit from more aggressive compression (\(r = 4\)) as the reduced token count eases attention overhead and improves long-context modeling”, while “higher ratios collapse larger spatial regions into single tokens, impairing tasks requiring precise localization, such as OCR” [Certain]. The best ratio depends on model size and on the task.

          What the ablations say

          Which connector is best? Many teams ran controlled tests. Their results agree on one point: the number of visual tokens and the image resolution matter more than the connector type.

          StudyWhat they comparedWhat they found [Certain, from the papers]
          Flamingo (2022), Table 3Perceiver resampler vs MLP vs Transformer, same parameter budgetResampler best: overall 70.7 vs 66.6 (MLP) and 66.7 (Transformer)
          BLIP-2 (2023), Figure 5Q-Former with and without stage-1 representation learningWithout stage 1, zero-shot VQA is much lower; with OPT it collapses during training
          LLaVA-1.5 (2023), Table 2Linear vs two-layer MLP projectorMLP better (MME 1,355.2 vs 1,323.8); resamplers converge less efficiently with the same data
          MM1 (2024), connector ablationAverage pooling, attention pooling, C-Abstractor (convolution); 64 or 144 tokens; 224 or 336 pixels“Number of visual tokens and image resolution matters most, while the type of VL connector has little effect”
          Idefics2 (2024), Tables 3, 4, 11Perceiver pooling vs no pooling; 64 vs 128 latents; other poolingsPerceiver pooling: +8.5 points on average and 729 to 64 tokens per image; 64 latents as good as 128 (71.7 vs 71.2)
          SmolVLM (2025), Figure 3Pixel shuffle \(r = 2\) vs \(r = 4\)The best ratio depends on model size; small models prefer \(r = 4\)

          These findings look like they disagree (Idefics2 says 64 tokens are enough, MM1 says more tokens help), but the setups differ: training length, data, task mix and resolution. Idefics2 itself writes that with much longer training more tokens might help [Certain]. The practical summary that most recent open models follow [Likely]: use a simple projector (MLP, often after a pixel shuffle) so that nothing is learned from scratch except a small mapping, and spend the token budget on resolution and tiling. Use learned queries (resampler, Q-Former) when you must fix the token count, for example for long video or many images, or when the language model is frozen. Honeybee (Cha et al., 2023) argued that resamplers lose locality and proposed convolutional projectors (C-Abstractor) that compress while keeping neighbors together; MM1 used the C-Abstractor with 144 tokens [Certain].

          ConnectorTokens per imageKeeps layout?New parametersBest when
          Linear / MLP (LLaVA)\(N\) (576 at 336 px)yes, one token per patch4M to 31Myou can pay for the tokens; detail and OCR matter
          Q-Former (BLIP-2)32 (number of queries)no, queries choose188M + one linear layerfrozen LLM, short answers, cheap inference
          Perceiver resampler (Flamingo, Idefics2)\(R\) (64), for any number of framesonly via position in the featuresabout 200M in Flamingomany images or long video, interleaved inputs
          Pixel shuffle + linear/MLP\(N / r^2\)yes, one token per \(r \times r\) block7M to 45Mhigh resolution, small models, documents with \(r = 2\)

          The PyTorch version

          All four connectors in a few lines each, a check that pixel shuffle loses nothing, parameter counts at real sizes, and a small experiment. In the experiment, an “image” is an \(8 \times 8\) grid of patch features (with position information, as a ViT output has). Four of the 64 cells hold one of 8 objects. A question names a cell; the answer is the object in that cell, or “empty”. A small 2-layer transformer stands in for the language model: it reads a question token plus the visual tokens from the connector. Every connector gets the same reader, the same data and 1,500 training steps. Half the questions point at an object, so “always say empty” already scores about 0.47. The question is how much fine spatial detail each connector lets through. It runs on a laptop CPU in about 3 minutes.

          import torch, torch.nn as nn, torch.nn.functional as F
          torch.manual_seed(0)
          
          # ---- 1) the four connectors -------------------------------------------------
          class MLPProjector(nn.Module):                      # LLaVA-1.5: Linear -> GELU -> Linear
              def __init__(s, d_v, d_lm):
                  super().__init__(); s.fc1 = nn.Linear(d_v, d_lm); s.fc2 = nn.Linear(d_lm, d_lm)
              def forward(s, z):                              # z: [B, N, d_v]
                  return s.fc2(F.gelu(s.fc1(z)))              # [B, N, d_lm]  (same N)
          
          def pixel_shuffle(z, r):                            # space-to-depth on the patch grid
              B, N, d = z.shape; h = int(N ** 0.5)
              z = z.view(B, h // r, r, h // r, r, d)          # split rows and columns into r x r blocks
              z = z.permute(0, 1, 3, 2, 4, 5)                 # [B, h/r, w/r, r, r, d]
              return z.reshape(B, (h // r) ** 2, r * r * d)   # [B, N/r^2, r^2 d]
          
          class ShuffleProjector(nn.Module):                  # Idefics3 / SmolVLM: shuffle + Linear
              def __init__(s, d_v, d_lm, r):
                  super().__init__(); s.r = r; s.proj = nn.Linear(r * r * d_v, d_lm, bias=False)
              def forward(s, z): return s.proj(pixel_shuffle(z, s.r))
          
          class ResamplerLayer(nn.Module):                    # Flamingo: K,V = [visual ; latents]
              def __init__(s, d, heads):
                  super().__init__(); s.ln_x = nn.LayerNorm(d); s.ln_l = nn.LayerNorm(d)
                  s.att = nn.MultiheadAttention(d, heads, batch_first=True)
                  s.ffn = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
              def forward(s, x, lat):
                  q = s.ln_l(lat); kv = torch.cat([s.ln_x(x), q], dim=1)
                  lat = lat + s.att(q, kv, kv, need_weights=False)[0]
                  return lat + s.ffn(lat)
          
          class PerceiverResampler(nn.Module):
              def __init__(s, d_v, d_lm, R, layers=2, heads=4):
                  super().__init__(); s.inp = nn.Linear(d_v, d_lm)
                  s.latents = nn.Parameter(torch.randn(R, d_lm) * 0.02)   # R learned queries
                  s.layers = nn.ModuleList(ResamplerLayer(d_lm, heads) for _ in range(layers))
              def forward(s, z):
                  x = s.inp(z); lat = s.latents.expand(z.shape[0], -1, -1)
                  for layer in s.layers: lat = layer(x, lat)
                  return lat                                  # [B, R, d_lm] for any N
          
          class MeanPool(nn.Module):                          # the extreme: one token per image
              def __init__(s, d_v, d_lm): super().__init__(); s.proj = nn.Linear(d_v, d_lm)
              def forward(s, z): return s.proj(z.mean(1, keepdim=True))
          
          # ---- 2) pixel shuffle is a pure re-arrangement: nothing is lost -------------
          z = torch.arange(2 * 16 * 3.).view(2, 16, 3)       # a 4x4 grid of 3-dim "patches"
          y = pixel_shuffle(z, 2)
          print("shuffle shape", tuple(y.shape), "same numbers:", torch.equal(y.flatten().sort()[0], z.flatten().sort()[0]))
          print("token 0 holds patches", (y[0, 0].view(4, 3)[:, 0] / 3).long().tolist())   # patches 0,1,4,5
          
          # ---- 3) real sizes: count connector parameters --------------------------------
          n = lambda m: sum(p.numel() for p in m.parameters())
          print("LLaVA-1.5-7B MLP     ", n(MLPProjector(1024, 4096)))
          print("Idefics3 shuffle r=2 ", n(ShuffleProjector(1152, 4096, 2)))
          print("SmolVLM-256M r=4     ", n(ShuffleProjector(768, 576, 4)))
          
          # ---- 4) a toy test: can the reader still find small details? -----------------
          G, D, K, NOBJ = 8, 16, 8, 4                         # 8x8 patch grid, feature size, classes, objects
          obj_emb = torch.randn(K, D); pos_emb = torch.randn(G * G, D) * 0.5
          def batch(B, gen):
              cls = torch.full((B, G * G), K)                 # K = "empty"
              for b in range(B):
                  cells = torch.randperm(G * G, generator=gen)[:NOBJ]
                  cls[b, cells] = torch.randint(0, K, (NOBJ,), generator=gen)
              feats = torch.cat([obj_emb, torch.zeros(1, D)])[cls] + pos_emb + 0.3 * torch.randn(B, G * G, D, generator=gen)
              ask = torch.randint(0, G * G, (B,), generator=gen)
              ask[: B // 2] = (cls[: B // 2] < K).float().multinomial(1, generator=gen).squeeze(1)  # half the questions hit an object
              return feats, ask, cls[torch.arange(B), ask]
          
          class Reader(nn.Module):                            # stands in for the LLM: question token + visual tokens
              def __init__(s, conn, d=64):
                  super().__init__(); s.conn = conn; s.q = nn.Embedding(G * G, d)
                  s.tf = nn.TransformerEncoder(nn.TransformerEncoderLayer(d, 4, 4 * d, 0.0, batch_first=True, norm_first=True), 2,
                                                  enable_nested_tensor=False)
                  s.head = nn.Linear(d, K + 1)
              def forward(s, feats, ask):
                  h = torch.cat([s.q(ask)[:, None], s.conn(feats)], 1)
                  return s.head(s.tf(h)[:, 0])
          
          def run(name, conn, steps=1500):
              torch.manual_seed(1); gen = torch.Generator().manual_seed(2)
              m = Reader(conn); opt = torch.optim.AdamW(m.parameters(), 1e-3)
              for _ in range(steps):
                  f, a, y = batch(128, gen); loss = F.cross_entropy(m(f, a), y)
                  opt.zero_grad(); loss.backward(); opt.step()
              gen = torch.Generator().manual_seed(99); f, a, y = batch(2000, gen)
              with torch.no_grad(): acc = (m(f, a).argmax(1) == y).float().mean().item()
              print(f"{name:22s} tokens {conn(f[:1]).shape[1]:3d}  accuracy {acc:.3f}")
          
          f, a, y = batch(2000, torch.Generator().manual_seed(99))
          print("always say empty      accuracy", round((y == K).float().mean().item(), 3))
          torch.manual_seed(0)
          for name, c in [("MLP projector", MLPProjector(D, 64)), ("pixel shuffle r=2", ShuffleProjector(D, 64, 2)),
                          ("pixel shuffle r=4", ShuffleProjector(D, 64, 4)), ("resampler 16 latents", PerceiverResampler(D, 64, 16)),
                          ("resampler 4 latents", PerceiverResampler(D, 64, 4)), ("mean pool", MeanPool(D, 64))]:
              run(name, c)

          Output with PyTorch 2.8 on CPU (two runs gave identical numbers): the shuffle check prints shape (2, 4, 12), the same numbers, and token 0 holds patches 0, 1, 4 and 5. The parameter counts are 20,979,712 (LLaVA-1.5-7B MLP), 18,874,368 (Idefics3) and 7,077,888 (SmolVLM-256M), matching the worked examples. Accuracy on 2,000 test questions (“always empty” scores 0.468):

          ConnectorVisual tokensTest accuracy
          MLP projector (one token per patch)640.965
          Pixel shuffle \(r = 2\) + linear160.990
          Pixel shuffle \(r = 4\) + linear40.884
          Perceiver resampler, 16 latents160.697
          Perceiver resampler, 4 latents40.607
          Mean pool to one token10.468

          Read these as a toy, not a benchmark. The task asks about one small cell, the worst case for compression. Pixel shuffle with \(r = 2\) did better than the per-patch MLP with 4 times fewer tokens: nothing is lost in the fold, and the reader has fewer positions to search. At \(r = 4\) each token holds 16 patches and the single linear layer must squeeze them, so accuracy drops. The resampler with 16 learned latents, given the same short training, is far behind the pixel shuffle with 16 tokens: the latents must first learn where to look. This is the same pattern the LLaVA-1.5 authors describe (resamplers converge more slowly with the same data), and why pixel shuffle became popular. Mean pooling to one token keeps almost nothing about where things are and scores at the level of always answering “empty”. With much longer training the resampler may close part of the gap [Likely]; exact numbers can differ slightly across PyTorch versions [Likely].

          Where this is used

          ApplicationConnector choice and why
          General chat VLMs (LLaVA family, many open models)MLP projector, often after pixel shuffle; tokens spent on resolution and tiles (session 181, session 182)
          Document AI and OCRKeep many tokens: projector or pixel shuffle with \(r = 2\); aggressive ratios hurt precise localization (SmolVLM finding) (session 180)
          On-device and small VLMsPixel shuffle with \(r = 3\) or 4 (SmolVLM: 64 tokens per 512 × 512 tile), so the small language model has a short context (session 183)
          Video classification and action recognitionA resampler gives a fixed number of tokens for any clip length, with time embeddings for order; per-frame pixel shuffle plus frame sampling is the alternative (session 205)
          Video question answering, Video-RAG and key-moment detectionToken budget per frame decides how many frames fit; few tokens per frame (e.g. 64) let an LLM see minutes of video and point to the moment that answers a question (session 210, session 206)
          Video and image retrieval with a VLM re-rankerShort visual prefixes (32 Q-Former tokens) make scoring many candidates cheap; BLIP-2's Q-Former also gives ITC embeddings for the first stage (session 208)
          Multi-image and interleaved documentsFixed small token count per image (Flamingo, Idefics2: 64) so dozens of images fit in one context
          Frozen-LLM productsQ-Former or resampler with cross-attention keeps the text model unchanged and safe; only the connector is trained (session 179)

          Common mistakes

          • Counting tokens from the wrong grid. CLIP ViT-L/14 at 336 pixels returns 577 vectors (576 patches + [CLS]); LLaVA drops the [CLS]. Qwen2-VL adds two special tokens. Always check how many placeholder tokens the processor inserts.
          • Using the last ViT layer by habit. LLaVA and BLIP-2 both take the second-to-last layer; LLaVA measured the last layer as worse. Make the layer a setting you test.
          • A wrong reshape in pixel shuffle. A plain view(B, N/4, 4*d) concatenates 4 patches that are next to each other in the flattened row, not a 2 × 2 square. Split rows and columns first, then permute (Algorithm 2). Qwen2-VL avoids this by reordering patches in its image processor so that each 2 × 2 block is already consecutive [Likely].
          • A grid not divisible by r. 27 × 27 cannot be shuffled with \(r = 2\). Change the resolution, pad, or choose \(r = 3\) as SmolVLM-2.2B does.
          • Expecting a resampler to learn fast. A fresh Q-Former or resampler has many new parameters and must learn where to look. Without a representation stage (BLIP-2) or a lot of data, it can lag behind a plain MLP, as in the experiment above.
          • Comparing connectors at different token counts. More tokens usually win (MM1). A fair comparison fixes the number of visual tokens and the resolution, then changes the connector.
          • Forgetting that 32 or 64 tokens is a hard ceiling. A Q-Former cannot read a dense page of text, whatever the language model's size. Choose the connector for the task.
          • Mismatched scales. Vision features and word embeddings can have very different norms. InternVL and Qwen2-VL put a LayerNorm before the MLP; skipping a norm or initializing the projection too large can make early training unstable [Likely].

          Interview questions

          What does the connector in a VLM do, and why can't we feed ViT outputs directly to the LLM?basic
          The ViT produces vectors of its own width (1,024 for CLIP ViT-L) in its own space; the LLM's embedding width is different (4,096 for a 7B model) and its layers expect inputs that look like its word embeddings. The connector maps vision features into that space and often also changes how many tokens there are, \(f: \R^{N \times d_v} \to \R^{M \times d}\). The number \(M\) sets the cost of each image in prefill compute and KV cache, so the connector is small in parameters but large in impact.
          LLaVA used a linear layer; LLaVA-1.5 an MLP. What changed and what does it cost?basic
          LLaVA maps each patch with one matrix \(W\); LLaVA-1.5 uses Linear, GELU, Linear (178.3). The non-linearity gives more capacity to align the spaces, and LLaVA-1.5 measured a gain (MME 1,355.2 vs 1,323.8 with the same data). The cost is tiny: for \(d_v = 1{,}024\), \(d = 4{,}096\) it is 20,979,712 parameters versus 4,198,400 for a linear layer with bias. Both keep all 576 tokens. Stage 1 trains only this projector on 558K caption pairs.
          Explain the Q-Former. Why 32 queries, and why does it need a first training stage?core
          The Q-Former is a BERT-base-initialized transformer (188M parameters) with 32 learned query vectors of width 768. Queries self-attend, cross-attend to frozen ViT features every other block, and share self-attention with a text transformer. Its output is always \(32 \times 768\), an information bottleneck: much smaller than \(257 \times 1{,}408\) image features. Stage 1 trains it with ITC, ITM and ITG using different masks; under the ITG mask the text sees the image only through the queries, so the queries learn to extract what the text needs. Stage 2 adds a linear layer into a frozen LLM. Without stage 1, BLIP-2 found much lower zero-shot VQA and, with OPT, catastrophic forgetting. 32 is a design choice that keeps the LLM input short; it is also the ceiling on how much image detail gets through.
          Draw the three Q-Former attention masks and explain each.core
          Order the sequence as [queries | text]. ITC: block diagonal, queries see queries and text sees text, so the contrastive score (max over queries of \(\mathbf{z}_j^\top \mathbf{t}_{\text{cls}}\)) does not leak. ITM: all ones, full interaction, a 2-way classifier on each query output, logits averaged. ITG: queries see only queries; each text token sees all queries and earlier text (causal), like UniLM. With 4 queries and 4 text tokens that is 32, 64 and 42 allowed entries of 64.
          How is the Perceiver resampler different from the Q-Former?core
          Both use learned queries and cross-attention to output a fixed number of tokens. The resampler (Flamingo) has no text branch and no BERT initialization; it is trained end to end only by the language-modeling loss through the frozen LM. Its keys and values are the visual features concatenated with the latents, \(kv = [X_f; L]\). It takes features from any number of frames, with learned time embeddings, and always returns 64 tokens. Flamingo feeds them to the LM through gated cross-attention, not as prefix tokens. The Q-Former has a stage of contrastive, matching and captioning losses and gives 32 tokens that are put in front of the text.
          What is pixel shuffle in a VLM? Does it lose information?core
          It is space-to-depth on the patch grid: each \(r \times r\) block of neighboring patch vectors is concatenated into one vector of width \(r^2 d_v\), so the token count drops by \(r^2\) (178.10, 178.11). The rearrangement is a permutation and loses nothing; information can be lost only in the following projection, which maps \(r^2 d_v\) to \(d\) (for Idefics3, 4,608 to 4,096; for SmolVLM-256M, 12,288 to 576). It keeps the spatial layout because each token comes from one known block. The trade-off: larger \(r\) means fewer tokens but coarser spatial detail, which hurts OCR. The name is borrowed from super-resolution, where the original pixel shuffle does the opposite (depth to space).
          How many visual tokens and how much KV cache does one image cost in LLaVA-1.5-7B? In Idefics3-8B?core
          LLaVA-1.5-7B: \(336/14 = 24\), so \(24^2 = 576\) tokens. KV per token: \(2 \times 32 \times 4{,}096 \times 2 = 524{,}288\) bytes (K and V, 32 layers, width 4,096, 2 bytes); \(576 \times 0.5\) MiB \(= 288\) MiB. Idefics3-8B per 364-pixel tile: \(26 \times 26 = 676\) patches, pixel shuffle \(r = 2\) gives \(676 / 4 = 169\) tokens; Llama 3 8B has 8 KV heads of 128, so 131,072 bytes per token and 21.1 MiB per tile. Grouped-query attention and the shuffle together make each tile about 14 times cheaper in cache than a LLaVA-1.5 image.
          Recent open VLMs mostly use MLP projectors, not Q-Formers. Why?advanced
          Several reasons. A fixed 32 or 64 tokens limits detail, which hurts documents, charts and OCR, the tasks users care about. An MLP has few new parameters, so it aligns quickly; LLaVA-1.5 notes that resamplers converge less efficiently with the same data. MM1 found the connector type matters little compared with the number of tokens and the resolution. Once the language model is unfrozen and trained on instruction data, it can select what it needs from many tokens. Token cost is then controlled by pixel shuffle, pooling and token pruning, which keep layout. Learned queries remain useful when the token count must be fixed: long video, many images, or a frozen LLM.
          How would you feed a 2-minute video to each connector type?advanced
          Sample frames (say 1 per second, 120 frames). With a projector, \(120 \times 576 = 69{,}120\) tokens: too many, so you must lower resolution, pool, or sample fewer frames. With pixel shuffle and few tokens per frame (64 as in SmolVLM-256M), \(120 \times 64 = 7{,}680\) tokens, which fits a long context. With a Perceiver resampler over all frames plus time embeddings, always \(R\) tokens (64), but all 120 frames compete for 64 slots, so fine details of a short key moment can be lost. In practice: per-frame compression plus a frame budget, and a resampler or token merging when the context is tight.
          Where does spatial position information come from in each connector?advanced
          Projector: each token is one patch, and the ViT features already include position embeddings; the order in the sequence also follows the grid. Pixel shuffle: each token is one block, so layout is kept at coarser scale, and the order of channels inside a token encodes the position within the block. Q-Former and resampler: attention is permutation invariant over keys, so position survives only if it is in the features. Flamingo adds temporal embeddings only, relying on CNN features to carry space; queries can then mix information from anywhere, which is why such models are weaker at precise localization. Some models add special tokens or 2D/3D rotary positions (Qwen2-VL's M-RoPE) in the language model.
          Which layer of the ViT should feed the connector, and why not the last one?core
          LLaVA uses the second-to-last layer (config vision_feature_layer = -2), and BLIP-2 removes the last layer of its ViT and uses the second-to-last. In a CLIP-trained ViT the last layer is shaped by the contrastive objective, which cares about one global summary vector, so patch features become less local. The layer before keeps more detail. LLaVA measured the last layer as 0.96% worse on ScienceQA. Some models concatenate features from several layers, which a projector handles by making its input wider.

          Exercises

          1. Compute the parameters of the LLaVA-1.5-13B projector (\(d_v = 1{,}024\), \(d = 5{,}120\)).
          2. SmolVLM-2.2B uses SigLIP at 384 pixels with patch 14 and \(r = 3\), and a language model with \(d = 2{,}048\). Give the grid, the number of tokens, the width after the shuffle and the parameters of a bias-free projection.
          3. Qwen2-VL turns a 224 × 224 image into 66 tokens. Derive the number.
          4. A Perceiver resampler with \(R = 64\) latents reads 8 frames of 576 features. Compute the attention scores per layer and head, and compare with full self-attention over the 4,608 tokens.
          5. In the Q-Former ITG mask with 32 queries and 10 text tokens, how many entries of the \(42 \times 42\) mask are allowed?
          6. Write the inverse of pixel shuffle with \(r = 2\). Why does this prove that the shuffle loses nothing, and where can the connector lose information?
          Answerscheck
          1. \(1{,}024 \times 5{,}120 + 5{,}120 + 5{,}120^2 + 5{,}120 = 31{,}467{,}520\), about 31.5M. 2. \(384 / 14 = 27\) (27.4 rounded down by the processor), grid \(27 \times 27 = 729\); \(729 / 9 = 81\) tokens; width \(9 \times 1{,}152 = 10{,}368\); parameters \(10{,}368 \times 2{,}048 = 21{,}233{,}664\). 3. \(224 / 14 = 16\), \(16 \times 16 = 256\) patches; \(2 \times 2\) merging gives \(256 / 4 = 64\) tokens; plus <|vision_start|> and <|vision_end|> = 66. 4. \(TS = 4{,}608\): \(64 \times (4{,}608 + 64) = 299{,}008\) versus \(4{,}608^2 = 21{,}233{,}664\), 71 times more. 5. Query rows: \(32 \times 32 = 1{,}024\). Text rows: each sees 32 queries, and text token \(k\) sees \(k\) text tokens: \(10 \times 32 + (1 + 2 + \dots + 10) = 320 + 55 = 375\). Total 1,399 of 1,764. 6. Reshape each wide vector to \(2 \times 2 \times d_v\), and place block \((a, b)\) back at rows \(2a, 2a+1\) and columns \(2b, 2b+1\). Since an exact inverse exists, the map is a bijection (a permutation of numbers), so nothing is lost. Loss can only happen in the projection \(W\) when \(r^2 d_v \gt d\) (for example 12,288 to 576 in SmolVLM-256M), and in the language model's limited attention.

          Key takeaways

          • A connector maps \(N\) vision vectors of width \(d_v\) to \(M\) tokens of width \(d\) (178.1). The width change is easy; the token count \(M\) sets the cost of every image.
          • The MLP projector (178.3) keeps one token per patch, has about 21M parameters for a 7B model, trains fast and loses nothing; LLaVA-1.5-7B pays 576 tokens and 288 MiB of KV cache per image.
          • The Q-Former reads the image with 32 learned queries through cross-attention (178.5); a three-loss first stage with different masks teaches the queries what text needs, then one linear layer (178.7) feeds a frozen LLM.
          • The Perceiver resampler (178.8) turns features of any number of frames into 64 tokens at cost \(R(TS + R)\) (178.9); it is the natural tool for long video and many images, but learns slowly and keeps layout only through the features.
          • Pixel shuffle (178.10, 178.11) folds \(r \times r\) neighbors into one token: lossless rearrangement, \(r^2\) fewer tokens, layout kept; \(r = 2\) for detail, \(r = 3\) or 4 for small models.
          • Ablations agree that token count and resolution matter more than the connector type; most current models use pixel shuffle or pooling plus an MLP.

          Key formulas

          \[ N = \tfrac{H}{P}\cdot\tfrac{W}{P}, \qquad \mathbf{h}_i = W_2\,\operatorname{GELU}(W_1 \mathbf{z}_i + \mathbf{b}_1) + \mathbf{b}_2, \qquad |\phi| = d_v d + d^2 + 2d \]
          \[ \operatorname{CA}(X, Z_v) = \softmax\!\big(X W_Q (Z_v W_K)^\top / \sqrt{d_h}\big)\, Z_v W_V, \qquad L \leftarrow L + \operatorname{Attn}(L, [X_f; L]) \]
          \[ C_{\text{resampler}} = R\,(TS + R), \qquad M_{\text{shuffle}} = N / r^2, \qquad \text{KV bytes} = M \cdot 2\, L_{\text{layers}}\, d_{kv}\, b \]

          References for this session

          1. Liu et al. (2023). Visual Instruction Tuning
          2. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
          3. Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
          4. Jaegle et al. (2021). Perceiver: General Perception with Iterative Attention
          5. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
          6. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
          7. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
          8. Liu et al. (2023). Visual Instruction Tuning
          9. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
          10. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
          11. Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
          12. Jaegle et al. (2021). Perceiver: General Perception with Iterative Attention
          13. Laurençon et al. (2024). What matters when building vision-language models?
          14. Laurençon, Marafioti, Sanh & Tronchon (2024). Building and better understanding vision-language models: insights and future directions
          15. Marafioti et al. (2025). SmolVLM: Redefining small and efficient multimodal models
          16. Chen et al. (2024). How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites (InternVL 1.5)
          17. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
          18. McKinzie et al. (2024). MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
          19. Cha, Kang, Mun & Roh (2023). Honeybee: Locality-enhanced Projector for Multimodal LLM
          20. Shi et al. (2016). Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
          21. Liu et al. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
          22. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
          23. Yin et al. (2023). A Survey on Multimodal Large Language Models
          24. Zhang et al. (2024). MM-LLMs: Recent Advances in MultiModal Large Language Models