LLM & VLM courseBlog
177Track 7: Vision and VLMs · session 15 of 38

Anatomy of a vision-language model

Almost every modern vision-language model has three parts: a vision encoder that turns pixels into vectors, a connector that turns those vectors into something the language model can read, and a language model that writes the answer. This session takes the three parts apart, shows the two ways to join them (one shared sequence, or cross-attention adapters), and explains which parts are pretrained, which are new, and which are trained in each stage. It ends with a tour of real models, from LLaVA to Llama 3.2 Vision.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions on the Vision Transformer (164), CLIP (166), SigLIP (167), self-attention vs cross-attention (20) and the KV cache (80)

What you will learn

  • The three parts of a VLM (vision encoder, connector, language model) and the tensor shapes that flow between them
  • How an image becomes tokens: the count V = (H/P)(W/P)/r², and why this number decides much of the cost
  • Family 1, the unified sequence (LLaVA, PaliGemma, Qwen2-VL): image tokens sit in the context next to text
  • Family 2, cross-attention adapters (Flamingo, Llama 3.2 Vision): text attends to the image through new gated layers, and the language model can stay frozen
  • What is pretrained, what is new, and what is trained in each stage, with the real recipes of LLaVA-1.5, BLIP-2, Flamingo, PaliGemma, Qwen2-VL and Llama 3.2 Vision
  • The cost of each family in tokens, attention scores and KV cache, computed with real numbers
  • The wider design space: encoder-free models (Fuyu), discrete image tokens (Chameleon), hybrids (NVLM), and what ablations (MM1, Idefics2, Prismatic) found matters most

Why this matters

Open the model card of almost any vision-language model released since 2023 (LLaVA, Qwen2-VL, PaliGemma, InternVL, Llama 3.2 Vision, SmolVLM) and you find the same three boxes: a vision encoder, a connector, and a language model. Once you can name the three boxes, say what goes in and out of each one, and say which of them were trained when, you can read any new VLM paper in ten minutes.

A vision-language model (VLM) here means a model that takes images (and often video frames) plus text as input, and writes text as output. It answers “What is the score on the board?”, reads an invoice, describes a video clip, or points at an object. This is different from the models of the last sessions. CLIP and SigLIP (session 166, session 167) compare an image with a text but cannot write. BLIP and BLIP-2 (session 173, session 176) can write, and BLIP-2 already has the three-box shape. This session makes that shape explicit and general.

Two things make this the most important architecture session of the vision track. First, the design choices are few, and each one has a clear cost. How many tokens does one image become? Do those tokens go into the language model’s context, or does the language model look at them from the side? Which weights do you update, and in what order? Second, these choices decide the serving cost. In a VLM the image tokens often outnumber the text tokens by ten to one. They fill the KV cache and dominate prefill time (session 196). An engineer who can estimate “this design costs 576 tokens per frame, so 32 frames need 18,432 tokens” can design a video system before writing any code.

The plan: first the three parts and their tensor shapes; then the two families for joining them; then what is trained when; then a tour of real models; then code that builds both families in about 100 lines. The next sessions zoom into each box: connectors (session 178), fusion (session 179), high resolution (session 180), and training stages (session 186).

The three parts

Start with the problem. A language model reads a sequence of vectors, one per token, each of size \(d\) (4,096 for a 7B model). An image is a grid of pixels, for example \(336 \times 336 \times 3\) numbers. We need a function that turns the pixel grid into vectors the language model can use. It is cut into three pieces because each piece has a different job and, usually, a different origin.

  1. Vision encoder \(E_\phi\). A network pretrained on images, almost always a Vision Transformer (session 164) trained with CLIP or SigLIP. It cuts the image into \(P \times P\) patches and outputs one vector per patch. Its job is seeing: it already knows edges, objects, text in images, and their link to words.
  2. Connector \(C_\psi\) (also called projector, adapter or bridge). A small new network. Its job is translating: it maps the encoder’s vectors (width \(d_v\)) into the language model’s space (width \(d\)), and it may also change the number of vectors (merge neighbors, or pull a fixed number out with learned queries).
  3. Language model \(\text{LM}_\theta\). A pretrained decoder-only LLM. Its job is reasoning and writing: it reads the visual vectors and the text prompt and predicts the answer token by token.
\[ Z = E_\phi(x) \in \R^{N \times d_v}, \qquad N = \frac{H_{\text{img}}}{P} \cdot \frac{W_{\text{img}}}{P} \tag{177.1} \]
\[ \mathbf{H} = C_\psi(Z) \in \R^{V \times d} \tag{177.2} \]
Read the symbols
\(x\)
the input image, resized to \(H_{\text{img}} \times W_{\text{img}}\) pixels with 3 color channels
\(P\)
the patch size of the vision encoder, usually 14 or 16 pixels
\(N\), \(d_v\)
the number of patch vectors and their width (1,024 for CLIP ViT-L/14)
\(V\)
the number of visual tokens after the connector: the number of vectors the language model actually sees
\(d\)
the hidden width of the language model (4,096 for a 7B Llama)
\(\phi, \psi, \theta\)
the weights of the encoder, the connector and the language model
vision encoder (pretrained)connector (new)language model (pretrained)
Figure 1. The anatomy of LLaVA-1.5-7B with real shapes. A \(336 \times 336\) image becomes \(24 \times 24 = 576\) patch vectors of width 1,024 (taken from the second-to-last layer of CLIP ViT-L/14-336). A two-layer MLP maps each one to width 4,096. The 576 visual tokens are placed in the sequence before the text tokens, and Vicuna-7B reads them all.
Worked example: LLaVA-1.5-7B in numbers

Encoder. CLIP ViT-L/14 at 336 pixels: \(336/14 = 24\), so \(N = 24^2 = 576\) patches, each of width \(d_v = 1{,}024\). The encoder has about 0.30B weights (303.5M by count from its config) [Certain]. LLaVA takes the features of the second-to-last layer, not the last one; the original LLaVA paper found the last layer about 1 point worse [Certain]. A likely reason: the last layer is shaped by CLIP’s contrastive loss toward one global summary, so it keeps less local detail [Likely].

Connector. A two-layer MLP with a GELU: \(1{,}024 \to 4{,}096 \to 4{,}096\). Parameters: \(1024 \cdot 4096 + 4096 + 4096 \cdot 4096 + 4096 = 20{,}979{,}712\), about 21M. No merging, so \(V = N = 576\).

Language model. Vicuna-7B v1.5 (a fine-tuned Llama 2 7B), about 6.74B weights. The whole model is about 7.06B weights, and the connector is only 0.30% of them.

Cost. In prefill, every visual token passes through every layer of the 6.74B model, like a text token. The linear layers alone cost about \(2 \times 6.74 \times 10^9 \times 576 \approx 7.8 \times 10^{12}\) FLOPs per image (formula 177.9 below). The text prompt “What is in the image?” is about 10 tokens. The image is 98% of the input.

How an image becomes a number of tokens

The single most useful number about a VLM is \(V\), the visual tokens per image. Most connectors either keep the patch count or divide it by a merge factor \(r^2\) (they fold an \(r \times r\) block of neighbors into one token). A few use a fixed number of learned queries instead:

\[ V = \frac{H_{\text{img}}}{P} \cdot \frac{W_{\text{img}}}{P} \cdot \frac{1}{r^2} \;(+\text{ a few special tokens}) \qquad \text{or} \qquad V = Q \;\text{(fixed)} \tag{177.3} \]

Examples, all from the papers or configs [Certain]: LLaVA-1.5, \(P = 14\), \(r = 1\), 336 px: 576. PaliGemma, \(P = 14\), \(r = 1\): 256, 1,024 or 4,096 tokens at 224, 448 or 896 px. Qwen2-VL, \(P = 14\), \(r = 2\) (an MLP merges each \(2 \times 2\) block), 224 px: \(16 \cdot 16 / 4 = 64\), plus a vision-start and a vision-end token, so 66. BLIP-2: always \(Q = 32\) learned queries, whatever the resolution. Flamingo: always 64 per image. Figure 4 at the end of the tour lets you compute these for any image size.

Family 1: one unified sequence

The simplest way to join the parts: treat the visual tokens as if they were word embeddings. Look up the text tokens in the embedding table as usual, and put the visual tokens into the same sequence, at the place where the image appears in the prompt (marked by a placeholder token such as <image>). Then run the language model unchanged.

\[ X = \big[\, \mathbf{h}_1, \dots, \mathbf{h}_V,\; \mathbf{e}(w_1), \dots, \mathbf{e}(w_T) \,\big] \in \R^{(V + T) \times d} \tag{177.4} \]

Every layer is ordinary self-attention over all \(V + T\) positions. A text token can attend to any patch, in any layer, and patches can attend to each other too (with a causal mask, to the earlier patches). Nothing in the language model changes except that some inputs did not come from the embedding table. The training loss is the usual next-token loss, but only on the answer tokens:

\[ \mathcal{L}(\phi, \psi, \theta) = -\frac{1}{|\mathcal{A}|} \sum_{t \in \mathcal{A}} \log p_\theta\big(w_t \mid \mathbf{H}, w_{\lt t}\big) \tag{177.5} \]
Read the symbols
\(\mathbf{e}(w)\)
the embedding-table vector of text token \(w\)
\(\mathcal{A}\)
the positions of the answer tokens. Image positions and (usually) the user’s question get no loss: there is nothing to predict at an image position
\(p_\theta(w_t \mid \mathbf{H}, w_{\lt t})\)
the probability the LM gives to the correct next token, given the visual tokens and the text so far

This is the design of LLaVA, PaliGemma, Qwen2-VL, InternVL, Idefics2 and 3, SmolVLM, Molmo and most open VLMs [Certain]. It is sometimes called “decoder-only” or “fully autoregressive”. One variant matters: PaliGemma uses a prefix-LM mask, so the image and the question attend to each other in both directions, and only the answer is causal [Certain].

Strengths: no new layers in the LLM; every LLM improvement (long context, FlashAttention, quantization, serving engines) works at once; image and text mix freely, so several images in one conversation are easy. Weakness: every visual token is a full token. It costs context length, prefill compute in every layer (formula 177.9), and KV-cache memory. High-resolution images and video make \(V\) explode.

Family 2: cross-attention adapters

The second way keeps the visual tokens out of the sequence. The language model keeps reading only text. New layers are inserted between its layers, and in these layers the text tokens cross-attend to the visual tokens: queries come from the text, keys and values come from the image (session 20). This is the design of Flamingo (2022) and of Llama 3.2 Vision [Certain].

The trick that makes it safe is a gate. Flamingo’s “gated cross-attention dense” block adds the new layer’s output through \(\tanh(\alpha)\), where \(\alpha\) is one learnable number per layer, initialized to 0 [Certain]:

\[ \begin{aligned} \mathbf{x}' &= \mathbf{x} + \tanh(\alpha)\cdot \text{CrossAttn}\big(\text{LN}(\mathbf{x}),\, \mathbf{H}\big) \\ \mathbf{x}'' &= \mathbf{x}' + \tanh(\beta)\cdot \text{FFN}\big(\text{LN}(\mathbf{x}')\big) \end{aligned} \tag{177.6} \]
Read the symbols
\(\mathbf{x}\)
the hidden states of the text tokens arriving from the previous LM layer
\(\text{CrossAttn}(\cdot, \mathbf{H})\)
attention with queries from \(\mathbf{x}\) and keys and values from the visual tokens \(\mathbf{H}\)
\(\alpha, \beta\)
learnable scalar gates, both 0 at the start. \(\tanh(0) = 0\), so at initialization the block is the identity
LN, FFN
layer normalization and a feed-forward network, as in a normal transformer block

At step 0, the whole model computes exactly what the text-only LLM computes (the code later checks this: the difference is 0.0). Training then opens the gates slowly. The language model can be kept frozen: its text skills cannot get worse, because its weights never change, and on text-only inputs the new layers can be skipped entirely [Likely]. Meta states this as the reason for the Llama 3 vision design: the text weights were kept frozen “to maintain text-only performance” [Certain].

Strengths: visual tokens take no context length; text attends to them only in the \(C\) cross-attention layers (Flamingo-80B: one every 7th layer; Flamingo-9B: every 4th; Llama 3.2 11B: 8 of its 40 layers) [Certain]; the LLM can stay frozen. Weaknesses: many new weights (for Llama 3 405B, the cross-attention layers have about 100B parameters [Certain]); the model is no longer a standard LLM, so serving engines need special support; and with the LLM frozen, the model may reason about images less deeply than one that was trained end to end. Figure 2 shows both families’ attention patterns and costs.

The two families in numbers

Count the attention scores and the KV cache for one forward pass with \(V\) visual tokens, \(T\) text tokens, \(L\) LM layers and (in family 2) \(C\) cross-attention layers. Causal self-attention over \(n\) tokens computes \(n(n+1)/2\) scores per layer.

\[ S_{\text{uni}} = L \cdot \frac{(V+T)(V+T+1)}{2}, \qquad S_{\times} = L \cdot \frac{T(T+1)}{2} + C \cdot T \cdot V \tag{177.7} \]
\[ \text{KV}_{\text{uni}} = L\,(V + T), \qquad \text{KV}_{\times} = L\,T + C\,V \tag{177.8} \]
\[ F_{\text{img}} \approx 2 \cdot P_{\text{LM}} \cdot V \quad \text{(unified: linear-layer FLOPs spent on the visual tokens)} \tag{177.9} \]
Read the symbols
\(S\)
attention scores (query–key dot products) per forward pass, summed over layers
KV
KV-cache entries, counted in “one token in one layer”. In family 2 the image keys and values are computed once per cross-attention layer and reused for every generated token
\(P_{\text{LM}}\)
the number of weights in the language model; each token costs about \(2P_{\text{LM}}\) FLOPs in the forward pass
Worked example: one image, one short question

Take \(V = 576\) (one LLaVA-1.5 image), \(T = 100\), \(L = 32\), and for family 2 one cross-attention layer every 4th layer, so \(C = 8\).

Unified: \(n = 676\), so \(676 \cdot 677/2 = 228{,}826\) scores per layer and \(7{,}322{,}432\) over 32 layers. KV cache: \(32 \times 676 = 21{,}632\) entries.

Cross-attention: text self-attention \(32 \times 5{,}050 = 161{,}600\), plus cross-attention \(8 \times 100 \times 576 = 460{,}800\): total \(622{,}400\), 11.8 times fewer. KV cache: \(32 \times 100 + 8 \times 576 = 7{,}808\) entries, 2.8 times fewer.

The gap grows with \(V\): the unified cost has a \(V^2\) term and the cross cost is linear in \(V\). But attention scores are not the whole bill. In family 2 the 576 visual tokens still pass through the vision encoder and through the new cross-attention weights, and those weights are extra parameters that must be trained and stored.

visual token (key)text token (key)unified sequencecross-attention
Figure 2. Left: who attends to whom, drawn for 6 visual and 6 text tokens (rows are queries, columns are keys). In the unified sequence, all 12 tokens are in one causal self-attention. With cross-attention, text attends to text in every layer, and to all visual tokens only in the cross-attention layers. Right: the totals from formulas (177.7) and (177.8) for your settings, on a log scale.
How to read Figure 2

Start at the defaults (one LLaVA-1.5 image, a 100-token prompt, a 32-layer LM, cross-attention every 4th layer): the readout shows 7,322,432 vs 622,400 scores. Now move \(V\) to 4,096 (a high-resolution image, or 7 video frames at 576 tokens each): the unified cost grows about 38 times, the cross-attention cost only about 5.5 times, and the gap becomes 82 times. Then set \(k = 1\) (cross-attention in every layer): the gap shrinks to 21 times and the two KV caches become equal, because every layer now stores the image keys and values. Switch the mode button to see which cells of the attention pattern each family computes.

What is pretrained, what is new, and what is trained when

A VLM is almost never trained from scratch. Two of the three parts arrive pretrained, each from a training run that cost far more than the VLM run will. The connector (and, in family 2, the cross-attention layers) start from random weights. This mismatch shapes the whole recipe.

Think of what happens at step 0 if you unfreeze everything at once. The connector outputs random vectors. The LLM sees garbage inputs, its loss is high, and large gradients flow back through the whole network. These gradients can damage the carefully pretrained encoder and LLM before the connector has learned anything useful. Hence the classic recipe:

Algorithm 1: the classic two-stage recipe (LLaVA style)
  1. input: pretrained encoder \(E_\phi\), pretrained LLM \(\text{LM}_\theta\), image–caption pairs, instruction data
  2. \(C_\psi \leftarrow\) random init // the only new part
  3. Stage 1, alignment: freeze \(\phi\) and \(\theta\); train \(\psi\) only on image–caption pairs with loss (177.5) // LLaVA-1.5: 558K pairs, 21M trainable weights
  4. Stage 2, instruction tuning: keep \(\phi\) frozen; train \(\psi\) and \(\theta\) on instruction data (questions and answers about images) // LLaVA-1.5: 665K samples
  5. optional: unfreeze \(\phi\) with a small learning rate; preference tuning or RL (session 186)
  6. return \(E_\phi, C_\psi, \text{LM}_\theta\)

The full LLaVA-1.5 13B run took about one day on one node with 8 A100 GPUs [Certain]. Stage 1 teaches the interpreter the two languages while both experts stay still; stage 2 teaches the writer to use what the interpreter says.

The recipe is a default, not a law. Real models disagree on every step, and the ablation papers explain why:

  • Is stage 1 needed? Prismatic VLMs tested it on LLaVA-style models and found the separate alignment stage “unnecessary”: training projector and LLM together from the start was as good or better and saved 20–25% of the compute [Certain]. Many teams still keep it as a cheap safety step.
  • Freeze the encoder? Prismatic found fine-tuning the vision encoder in a short LLaVA-style run hurt, especially on localization tasks [Certain]. But PaliGemma trains everything in its first multimodal stage (“nothing is frozen”), with a slow learning-rate warmup for the encoder to protect it from the early, misaligned gradients [Certain]. Qwen2-VL trains the ViT first and unfreezes all weights in its second stage [Certain]. The difference is scale and data: with billions of tokens and care, unfreezing helps; with a short run, it destroys features.
  • Freeze the LLM? Flamingo, BLIP-2 and Llama 3.2 Vision keep it frozen, to protect text skills. Idefics2 measured the trade-off: with frozen backbones, cross-attention scored 66.7 and the unified design 60.3 on their benchmark average; when the backbones were also trained (with LoRA), the unified design rose to 69.5 and beat cross-attention at 67.3 [Certain]. So the families and the freezing choice are linked: cross-attention is the better choice when the LLM must stay frozen; the unified sequence is better when you can train it.
trained in this stagefrozennot used
Figure 3. A tour of real designs. Pick a model, then step through its training stages. Each box is one part; green means its weights are updated in this stage, gray means frozen. The dashed tag on a box says whether it started pretrained or new. The arrow into the LLM shows the family: tokens in the sequence, or cross-attention from the side.
How to read Figure 3

Compare LLaVA-1.5 and Llama 3.2 Vision. LLaVA trains a 21M connector first and then the LLM; the encoder never changes. Llama 3.2 does the opposite: the encoder and the new cross-attention layers learn, and the LLM never changes. BLIP-2 freezes both big parts and trains only the Q-Former in the middle. PaliGemma trains all three parts together from its first stage. Chameleon has no connector at all: image codes are extra words in the vocabulary.

A tour of the design space

With the three parts and two families, you can place almost any model on a small map. The table lists the models of Figure 3 with facts from their papers, configs and model cards.

ModelEncoderConnectorLLMFamilyVisual tokens
LLaVA-1.5 (2023)CLIP ViT-L/14, 336 px2-layer MLPVicuna 7B / 13Bunified576
BLIP-2 (2023)EVA-CLIP ViT-g/14, frozenQ-Former (188M), 32 queriesOPT or FlanT5, frozenunified (soft prompt)32
Flamingo (2022)NFNet-F6, frozenPerceiver resamplerChinchilla 70B, frozen (80B model)cross-attention, gated64 per image
PaliGemma (2024)SigLIP So400mlinear, zero-initializedGemma 2Bunified, prefix-LM256 / 1,024 / 4,096
Qwen2-VL (2024)ViT, 675M, any resolutionMLP merging 2×2 tokensQwen2 1.5B / 7.6B / 72Bunified, M-RoPEdynamic (66 at 224 px)
Llama 3.2 Vision 11B (2024)ViT-H/14, about 850M [Likely], tiles of 560 px8 cross-attention layersLlama 3.1 8B, frozencross-attention1,601 per tile, up to 4 tiles [Likely]
Fuyu-8B (2023)nonelinear projection of raw patches8B decoderunified, encoder-freegrows with image size
Chameleon (2024)VQ image tokenizernone (image codes are vocabulary)7B / 34B, trained from scratchearly fusion, discrete1,024 per 512×512 image

Three more designs extend the map:

  • Encoder-free. Fuyu-8B has “no image encoder”: raw image patches are linearly projected into the first layer of the decoder, in raster order, with a special image-newline token at the end of each row [Certain]. This handles any resolution naturally, but the LLM must learn to see from scratch, which needs much more training data [Likely].
  • Discrete image tokens (early fusion). Chameleon uses a vector-quantized tokenizer that encodes a 512×512 image into 1,024 discrete codes from a codebook of 8,192 [Certain]. Images become ordinary vocabulary items, so the same model can also generate images. Emu3 follows the same idea. The cost: quantization loses fine detail, which hurts OCR-like tasks [Likely].
  • Hybrids. NVLM compared a decoder-only design (NVLM-D), a cross-attention design (NVLM-X) and a hybrid (NVLM-H) in one controlled study, and argued that each family has strengths [Certain]. In NVLM-H, a small thumbnail goes into the sequence and the high-resolution tiles are reached by cross-attention [Likely].

What matters most, according to the ablation papers? MM1 found that the image encoder, the image resolution and the number of visual tokens have a large effect, while the connector design is “of comparatively negligible importance” [Certain]. Idefics2 found that, for a fixed parameter count, the quality of the language model matters more than the quality of the vision encoder [Certain], and that pooling 729 SigLIP tokens down to 64 with a perceiver did not hurt, it helped [Certain]. Prismatic found that fusing DINOv2 and SigLIP features gives gains on localization [Certain]. The common lesson: spend your effort on the LLM, on resolution and token count, and on data. The connector is the cheap part.

Figure 4. Visual tokens for the same input under five designs (PaliGemma at two resolutions). Fixed-size designs resize every image to one square, so their count does not change. Qwen2-VL resizes the image to multiples of 28 pixels (patch 14, then 2×2 merging), keeping the aspect ratio, so its count follows the image. Bars that go into the LLM’s context are gray; Flamingo’s are reached by cross-attention and do not use context.
How to read Figure 4

At 1280×720 (a 720p video frame), Qwen2-VL rounds to 1288×728 and produces \(46 \times 26 + 2 = 1{,}198\) tokens, while LLaVA-1.5 squeezes the frame into 336×336 and produces 576. The fixed designs are cheap but throw away detail; the dynamic one keeps detail and pays for it. Now raise the number of images to 32 (think of 32 video frames): every unified design multiplies its cost by 32. For video, Qwen2-VL also merges each pair of consecutive frames, which halves its count [Certain]. This is why video models sample few frames and merge tokens (session 183).

Code: both families in one file

Before the code, the forward pass of each family as pseudo-code.

Algorithm 2: forward pass, unified sequence
  1. input: image \(x\), text token ids \(w_{1..T}\) (an <image> placeholder marks where the image goes)
  2. \(Z \leftarrow E_\phi(x)\) // \((N, d_v)\), formula (177.1)
  3. \(\mathbf{H} \leftarrow C_\psi(Z)\) // \((V, d)\), formula (177.2)
  4. \(X \leftarrow\) the text embeddings, with the placeholder replaced by the \(V\) rows of \(\mathbf{H}\) // formula (177.4)
  5. logits \(\leftarrow \text{LM}_\theta(X)\) with a causal (or prefix-LM) mask
  6. loss \(\leftarrow\) cross-entropy on answer positions only // formula (177.5)
Algorithm 3: forward pass, gated cross-attention
  1. input: image \(x\), text token ids \(w_{1..T}\), the set \(\mathcal{X}\) of layers that get a cross-attention block
  2. \(\mathbf{H} \leftarrow C_\psi(E_\phi(x))\) // computed once per image
  3. \(\mathbf{x} \leftarrow \mathbf{e}(w_{1..T})\) // only text in the sequence
  4. for \(\ell = 1 \dots L\): if \(\ell \in \mathcal{X}\): \(\mathbf{x} \leftarrow\) GatedXAttn\((\mathbf{x}, \mathbf{H})\) // formula (177.6), gates start at 0; then \(\mathbf{x} \leftarrow \text{Block}_\ell(\mathbf{x})\) // the frozen LM layer
  5. logits \(\leftarrow\) head\((\mathbf{x})\); loss as in (177.5)

The PyTorch version builds a tiny encoder, a tiny LLM, and both families on top of the same two parts. It freezes the encoder and LLM as in stage 1, counts trainable weights, and checks that the gated model equals the text-only LLM at initialization. It runs on a CPU in about a second.

import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(0)
D_V, D, VOCAB, P, IMG = 48, 64, 100, 8, 32      # vision width, LLM width, vocab, patch, image side

class Block(nn.Module):                          # pre-norm transformer block
    def __init__(self, d, causal):
        super().__init__()
        self.causal = causal
        self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.att = nn.MultiheadAttention(d, 4, batch_first=True)
        self.ff = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
    def forward(self, x):
        T = x.shape[1]
        mask = torch.triu(torch.ones(T, T, dtype=torch.bool), 1) if self.causal else None
        h = self.n1(x)
        x = x + self.att(h, h, h, attn_mask=mask, need_weights=False)[0]
        return x + self.ff(self.n2(x))

class VisionEncoder(nn.Module):                  # part 1: pixels -> N patch vectors, formula (177.1)
    def __init__(self):
        super().__init__()
        self.patch = nn.Conv2d(3, D_V, P, stride=P)
        self.pos = nn.Parameter(0.02 * torch.randn(1, (IMG // P) ** 2, D_V))
        self.blocks = nn.ModuleList([Block(D_V, causal=False) for _ in range(2)])
    def forward(self, img):                      # (B, 3, 32, 32)
        z = self.patch(img).flatten(2).transpose(1, 2) + self.pos   # (B, 16, D_V)
        for b in self.blocks: z = b(z)
        return z

class TinyLM(nn.Module):                         # part 3: a decoder-only LM
    def __init__(self, L=4):
        super().__init__()
        self.emb = nn.Embedding(VOCAB, D)
        self.blocks = nn.ModuleList([Block(D, causal=True) for _ in range(L)])
        self.norm, self.head = nn.LayerNorm(D), nn.Linear(D, VOCAB, bias=False)
    def forward(self, x, xattn=None, img=None):  # x: embeddings (B, T, D)
        for i, b in enumerate(self.blocks):
            if xattn is not None and str(i) in xattn: x = xattn[str(i)](x, img)
            x = b(x)
        return self.head(self.norm(x))

class UnifiedVLM(nn.Module):                     # family 1: image tokens join the sequence
    def __init__(self, enc, lm):
        super().__init__()
        self.enc, self.lm = enc, lm
        self.proj = nn.Sequential(nn.Linear(D_V, D), nn.GELU(), nn.Linear(D, D))  # part 2
    def forward(self, img, ids):
        H = self.proj(self.enc(img))                       # (B, V, D), formula (177.2)
        x = torch.cat([H, self.lm.emb(ids)], dim=1)        # (B, V + T, D), formula (177.4)
        logits = self.lm(x)
        V = H.shape[1]
        pred = logits[:, V - 1:-1]                         # positions that predict the T text tokens
        return F.cross_entropy(pred.reshape(-1, VOCAB), ids.reshape(-1))   # (177.5): no loss on image slots

class GatedXAttn(nn.Module):                     # family 2: Flamingo-style adapter, formula (177.6)
    def __init__(self):
        super().__init__()
        self.n1, self.n2 = nn.LayerNorm(D), nn.LayerNorm(D)
        self.att = nn.MultiheadAttention(D, 4, batch_first=True)
        self.ff = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))
        self.a_att, self.a_ff = nn.Parameter(torch.zeros(1)), nn.Parameter(torch.zeros(1))
    def forward(self, x, H):
        x = x + torch.tanh(self.a_att) * self.att(self.n1(x), H, H, need_weights=False)[0]
        return x + torch.tanh(self.a_ff) * self.ff(self.n2(x))

class CrossVLM(nn.Module):
    def __init__(self, enc, lm, every=2):
        super().__init__()
        self.enc, self.lm = enc, lm
        self.proj = nn.Linear(D_V, D)
        self.xattn = nn.ModuleDict({str(i): GatedXAttn() for i in range(0, len(lm.blocks), every)})
    def logits(self, img, ids):
        return self.lm(self.lm.emb(ids), self.xattn, self.proj(self.enc(img)))

def count(m): return sum(p.numel() for p in m.parameters() if p.requires_grad)
enc, lm = VisionEncoder(), TinyLM()
img, ids = torch.randn(2, 3, IMG, IMG), torch.randint(0, VOCAB, (2, 10))

A = UnifiedVLM(enc, lm)
for p in list(enc.parameters()) + list(lm.parameters()): p.requires_grad_(False)   # stage 1: connector only
print("visual tokens:", enc(img).shape[1], "| stage-1 trainable:", count(A), "of", sum(p.numel() for p in A.parameters()))
print("unified loss:", round(A(img, ids).item(), 4))

B = CrossVLM(enc, lm)
text_only = lm(lm.emb(ids))
print("new cross-attn params:", count(B), "| max |logit diff| at init:", (B.logits(img, ids) - text_only).abs().max().item())
# visual tokens: 16 | stage-1 trainable: 7296 of 286736
# unified loss: 4.7533
# new cross-attn params: 103108 | max |logit diff| at init: 0.0

Read the output. The 32×32 image with 8×8 patches gives \(4 \times 4 = 16\) visual tokens. In stage 1 only the projector trains: \(48 \cdot 64 + 64 + 64 \cdot 64 + 64 = 7{,}296\) of 286,736 weights (2.5%; in LLaVA-1.5 it is 0.30%). The loss is near \(\ln 100 = 4.61\), the value for random guessing over 100 tokens. The cross-attention model adds 103,108 new weights for only two blocks (the same order as the whole 4-layer LM’s blocks), and its logits equal the text-only LM’s exactly, because \(\tanh(0) = 0\). Output from PyTorch 2.8 on CPU.

Where this is used

ApplicationTypical designWhy
General chat about images (assistants)Unified sequence, SigLIP or CLIP encoder, MLP connector, LLM trained in stage 2Simple, strong, every LLM tool works; the most common open design
Adding vision to an LLM that is already deployed for textCross-attention adapters, LLM frozen (Llama 3.2 Vision)Text behavior stays exactly the same; text-only requests can skip the new layers
Documents, charts, screenshots (OCR-heavy)Unified sequence with dynamic resolution or tiling (Qwen2-VL, InternVL)Small text needs many tokens; fixed 336 px loses it (session 180)
Video question answering and captioningUnified sequence with per-frame tokens merged or pooled; time in the position code (Qwen2-VL’s M-RoPE)Frames multiply \(V\); token budget per frame is the key knob
Key-moment detection and temporal grounding in long videoA cheap encoder over many frames to find candidates, then a VLM on a few candidate clipsRunning a VLM on every frame is too costly; the encoder part alone is enough for search
Video retrieval, Video-RAG, fingerprintingThe vision encoder alone (CLIP or SigLIP embeddings), with a VLM only to describe or re-rank resultsRetrieval needs one vector per clip, not generated text (session 169)
Edge and mobileSmall LLM, small encoder, aggressive token merging (SmolVLM)\(V\) decides latency; fewer tokens per image beat a fancier connector
Image generation plus understandingDiscrete image tokens in one vocabulary (Chameleon, Emu3)The same next-token model can read and write images

Common mistakes

  • Forgetting that visual tokens are real tokens. In the unified family, 10 LLaVA-1.5 images are 5,760 tokens: they do not fit in a 4,096-token context. Count \(V\) before choosing resolution and number of frames.
  • Putting loss on image positions. There is no target at an image slot. Set those labels to the ignore index (−100 in PyTorch), and usually mask the user’s question too (session 188).
  • Unfreezing everything at step 0 with a random connector. Large, noisy gradients can damage the pretrained encoder and LLM. Warm up the connector first, or use a slow learning-rate warmup for the pretrained parts (as PaliGemma does).
  • Using the wrong encoder layer or preprocessing. Many VLMs read the second-to-last encoder layer and expect the encoder’s own resize, crop and normalization constants. Mismatched preprocessing silently lowers accuracy.
  • Cross-attention gates that are not zero-initialized. Then the frozen LLM receives noise from step 0 and its text skills drop at once. Check that the model equals the text-only LLM before training, as in the code.
  • “Frozen” weights that still change. Weights with requires_grad=False left in an optimizer with weight decay, or BatchNorm statistics updated in train() mode. Count trainable parameters and compare checksums.
  • Judging the connector first. Ablations (MM1, Idefics2) show resolution, token count, the LLM and data matter more than the connector type. Tune those first.

Interview questions

Draw the architecture of a typical VLM and give the tensor shapes.basic
Image \((3, H, W)\) → vision encoder (a ViT pretrained with CLIP or SigLIP) → \(N = (H/P)(W/P)\) patch vectors of width \(d_v\) → connector (MLP, or a module that merges or resamples tokens) → \(V\) vectors of width \(d\), the LLM’s hidden size → the LLM, which reads them with the text and generates the answer. LLaVA-1.5-7B: \((3, 336, 336)\) → \((576, 1024)\) → \((576, 4096)\) → Vicuna-7B.
Unified sequence vs cross-attention adapters: what are the trade-offs?core
Unified (LLaVA, Qwen2-VL): visual tokens enter the context; no new layers in the LLM; the LLM is usually trained; every LLM tool works; but each visual token costs context, KV cache and full prefill compute in every layer, with a \((V+T)^2\) attention term. Cross-attention (Flamingo, Llama 3.2 Vision): text attends to the image in a few inserted gated layers; no context cost; the LLM can stay frozen so text skills are preserved; but many new parameters (about 100B for Llama 3 405B), a non-standard architecture for serving, and Idefics2 found it falls behind the unified design when the backbones are trained.
Why is Flamingo’s gate initialized to 0, and why tanh?core
With \(\tanh(0) = 0\) the new block outputs exactly its input, so at the start the model is the pretrained text LLM: no damage to its skills and a stable start. Training opens the gate gradually as the cross-attention learns something useful. The tanh keeps the gate bounded in \((-1, 1)\), so the visual signal cannot be scaled up without limit.
How many tokens does one image cost in LLaVA-1.5, and what does that mean for a 7B model?core
\((336/14)^2 = 576\) tokens. In prefill each costs about \(2P = 13.5\) GFLOPs in the linear layers, so about \(7.8 \times 10^{12}\) FLOPs per image, and 576 entries in every layer’s KV cache. A 32-frame video at this rate is 18,432 tokens, more than four times the 4,096 context of Vicuna v1.5. This is why video VLMs pool or merge tokens per frame.
Why do LLaVA-style models train only the projector first? Is that stage necessary?core
The projector starts random; if everything is trained at once, its garbage outputs produce large gradients that can damage the pretrained encoder and LLM. Stage 1 aligns the projector on simple caption data while both big parts are frozen, and it is cheap (21M weights in LLaVA-1.5). Prismatic VLMs found it is not necessary in their setting: training projector and LLM together from the start was as good or better and saved 20–25% of compute. It remains a common safety step.
Why take features from the second-to-last layer of the CLIP encoder?core
The last layer of a contrastive encoder is shaped to feed one global embedding that matches the caption, so per-patch detail is less useful there. The layer before keeps more local information. The original LLaVA paper measured the last layer about 1 point worse on ScienceQA. Llama 3’s encoder goes further and concatenates features from several intermediate layers.
When would you freeze the LLM, and when would you train it?advanced
Freeze it when text behavior must not change (a product already relies on it, safety tuning is expensive to redo) or when you have little multimodal data: use cross-attention adapters with zero-init gates, as Llama 3.2 Vision does. Train it (fully or with LoRA) when you want the best multimodal reasoning: Idefics2 found the unified design reached 69.5 vs 67.3 for cross-attention once backbones were trained, while with frozen backbones cross-attention was better (66.7 vs 60.3). Mix in text-only data to limit forgetting.
Should the vision encoder be frozen?advanced
It depends on budget and data. In short LLaVA-style runs, Prismatic found fine-tuning the encoder hurt, especially localization. In large runs with careful schedules it helps: PaliGemma trains everything from the first multimodal stage with a slow warmup for the encoder, and Qwen2-VL trains its ViT in stage 1 and everything in stage 2. Unfreezing is also how models adapt to new resolutions or domains (documents, medical images). Use a smaller learning rate for the encoder than for the rest.
What do encoder-free models (Fuyu) and discrete-token models (Chameleon) gain and lose?advanced
Fuyu projects raw patches linearly into the decoder: no separate encoder, any resolution and aspect ratio, one simple architecture; but the LLM must learn low-level vision itself, which needs much more data and compute. Chameleon turns images into codes from a VQ tokenizer (1,024 codes per 512×512 image, codebook 8,192), so one model can read and generate images in any order; but quantization loses fine detail, and image tokens are many. Most understanding-focused VLMs keep a continuous, pretrained encoder.
Design a VLM for sports broadcast Q&A with a latency target. Which family and which token budget?advanced
Start with numbers: \(V\) per frame times frames sampled must fit the prefill budget. A unified design with a SigLIP encoder and 2×2 or stronger token merging (for example 64 to 256 tokens per frame), 8 to 16 frames chosen by a cheap encoder-based key-moment detector, and an LLM fine-tuned on the domain is the usual choice. Cross-attention would cut context cost and keep a deployed text model unchanged, at the price of new layers and less mature serving support. Cache encoder outputs for frames shared across questions, and measure accuracy against token count before choosing resolution.

Exercises

  1. How many visual tokens does Qwen2-VL produce for a 448×448 image? For a 1280×720 frame? (Patch 14, 2×2 merging, image resized to multiples of 28, plus 2 special tokens.)
  2. PaliGemma maps SigLIP So400m features (\(d_v = 1{,}152\)) into Gemma 2B (\(d = 2{,}048\)) with one linear layer. How many weights does it have? How many would a two-layer MLP \(1152 \to 2048 \to 2048\) have?
  3. Use formulas (177.7) and (177.8) with \(V = 1{,}024\), \(T = 50\), \(L = 40\), \(C = 8\). Compare the two families.
  4. A user sends 10 images to LLaVA-1.5 (context 4,096 tokens). What happens? How many images fit if the prompt and answer need 64 tokens?
  5. Show that a gated cross-attention block with \(\alpha = \beta = 0\) leaves its input unchanged, whatever the image. What is the gradient of the output with respect to \(\alpha\) at \(\alpha = 0\)?
  6. You have 20,000 domain image–question–answer pairs and must not change the chatbot’s text behavior. Which parts would you train, and in which family?
Answerscheck
1. 448×448: \((448/28)^2 = 16^2 = 256\), plus 2: \(256 + 2 = 258\). 1280×720: \(720/28 = 25.7 \to 26\) (728 px), \(1280/28 = 45.7 \to 46\) (1,288 px), so \(26 \times 46 = 1{,}196\), plus 2: \(1{,}196 + 2 = 1{,}198\). 2. Linear: \(1152 \cdot 2048 + 2048 = 2{,}361{,}344\). MLP: \(2{,}361{,}344 + 2048 \cdot 2048 + 2048 = 6{,}557{,}696\). Both are tiny next to the 2B LLM. 3. Unified: \(40 \cdot 1074 \cdot 1075/2 = 23{,}091{,}000\) scores, KV \(40 \cdot 1074 = 42{,}960\). Cross: \(40 \cdot 1275 + 8 \cdot 50 \cdot 1024 = 460{,}600\) scores, KV \(2{,}000 + 8{,}192 = 10{,}192\). About 50 times fewer scores and 4.2 times fewer KV entries. 4. \(10 \times 576 = 5{,}760 \gt 4{,}096\): the input is too long and the request fails or is truncated. With 64 text tokens, \(\lfloor 4032/576 \rfloor = 7\) images fit (exactly 4,032 tokens). 5. \(\tanh(0) = 0\) makes both residual branches zero, so \(\mathbf{x}'' = \mathbf{x}\). The gradient is \(\partial \mathbf{x}'/\partial \alpha = (1 - \tanh^2 0)\,\text{CrossAttn}(\cdot) = \text{CrossAttn}(\cdot)\), which is not zero, so the gate can learn to open. (The cross-attention weights themselves get zero gradient at step 0, because they are multiplied by \(\tanh(0) = 0\); they start learning once \(\alpha\) moves.) 6. Keep the LLM frozen. Either add zero-gated cross-attention layers and train them with the connector (Llama 3.2 style), or keep a unified design but train only the connector and LoRA adapters that are switched off for text-only requests. Keep the encoder frozen or use a small learning rate, since 20K pairs is little data. Evaluate text-only benchmarks before and after.

Key takeaways

  • A VLM is a pretrained vision encoder (177.1), a small new connector (177.2), and a pretrained language model. Visual tokens per image, \(V\) (177.3), is the most important number: 576 for LLaVA-1.5, 66 for Qwen2-VL at 224 px, 32 for BLIP-2.
  • Family 1, the unified sequence (177.4), puts visual tokens into the context; it is simple and the most common, but every visual token costs full LLM compute (177.9) and KV cache. The loss is only on answer tokens (177.5).
  • Family 2, gated cross-attention (177.6), keeps images out of the context and the LLM can stay frozen, at the cost of many new weights. Costs differ as in (177.7) and (177.8): 7.3M vs 0.62M attention scores for one LLaVA image and a 100-token prompt.
  • Training order: align the new parts while the old ones are frozen, then unfreeze what your data and budget allow. Recipes differ (LLaVA, PaliGemma, Qwen2-VL, Llama 3.2), and ablations show the LLM, resolution, token count and data matter more than the connector.

Key formulas

\[ V = \frac{H_{\text{img}}\,W_{\text{img}}}{P^2 r^2}, \qquad X = [\mathbf{H};\, \mathbf{e}(w)], \qquad \mathbf{x}' = \mathbf{x} + \tanh(\alpha)\,\text{CrossAttn}(\mathbf{x}, \mathbf{H}) \]
\[ S_{\text{uni}} = L\,\frac{(V+T)(V+T+1)}{2}, \qquad S_{\times} = L\,\frac{T(T+1)}{2} + C\,T\,V, \qquad F_{\text{img}} \approx 2 P_{\text{LM}} V \]

References for this session

  1. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
  2. Yin et al. (2023). A Survey on Multimodal Large Language Models
  3. Zhang et al. (2024). MM-LLMs: Recent Advances in MultiModal Large Language Models
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 3: Vision Language Model Training
  5. Liu et al. (2023). Visual Instruction Tuning
  6. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  7. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  8. Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
  9. Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer
  10. Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
  11. Laurençon et al. (2024). What matters when building vision-language models?
  12. Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models
  13. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  14. Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training
  15. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  16. Llama Team, AI @ Meta (2024). The Llama 3 Herd of Models
  17. McKinzie et al. (2024). MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  18. Karamcheti et al. (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  19. Dai et al. (2024). NVLM: Open Frontier-Class Multimodal LLMs
  20. Adept AI (2023). Fuyu-8B: A Multimodal Architecture for AI Agents (model card)
  21. Laurençon et al. (2023). OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents