Fusion: early, late, unified sequence and cross-attention
A vision-language model must decide where image and text information meet. They can meet only at the end (late fusion), inside one shared sequence of tokens (unified sequence, as in LLaVA), from the very first layer with no separate image encoder (native early fusion, as in Chameleon), or through new cross-attention layers inserted into a language model (Flamingo, Llama 3.2 Vision). This session builds each design, explains why a tanh gate that starts at zero lets the language model stay frozen and safe, and measures the real cost: tokens in the context versus extra layers.
What you will learn
- The two meanings of “early” and “late” fusion: the classic one (where features or predictions are combined) and the modern VLM one (separate pretrained encoder or not)
- The unified sequence: image tokens and text tokens in one causal self-attention stream, with the loss only on text
- Cross-attention adapters: text queries, image keys and values, and the Flamingo rule that a text token looks only at the latest image
- Why a tanh gate initialized to 0 makes the new model equal to the frozen LLM at step 0, and why it can still learn
- The cost trade-off with real numbers from Llama 3.2 11B Vision: KV cache, prefill FLOPs, sequence length and new parameters
- What real systems chose (LLaVA, Fuyu, Chameleon, Flamingo, Llama 3, NVLM) and what their ablations say
Why this matters
In session 177 we split a VLM into three parts: a vision encoder, a connector and a language model. In session 178 we saw how the connector turns image features into a small set of vectors. This session answers the next question: where do those image vectors enter the language model, and how do they mix with the text?
The choice is called the fusion design. "Fusion" simply means "the place where information from two modalities (here: image and text) is combined". There are four common answers:
- Late fusion. Each modality is processed alone, and the results meet only at the very end. CLIP is the standard example: one image vector, one text vector, one dot product.
- Unified sequence. Image features become "visual tokens" and are placed in the same sequence as the text tokens. The language model reads them all with ordinary self-attention. LLaVA, Qwen2-VL, InternVL and most open VLMs work this way.
- Native early fusion. There is no separate pretrained image encoder at all. Raw image patches (Fuyu) or discrete image codes (Chameleon) go straight into one transformer that is trained on both modalities from the start.
- Cross-attention. The text stays alone in the language model's sequence. New cross-attention layers are inserted between the language model's layers, and through them the text "looks at" the image. Flamingo and Llama 3.2 Vision work this way.
The choice decides almost everything about cost and safety. With a unified sequence, a high-resolution image can add thousands of tokens to the context, and every one of them passes through every layer and sits in the KV cache. With cross-attention, the image never enters the sequence, but you add new layers with billions of new parameters. And cross-attention with a tanh gate that starts at zero lets you keep the language model completely frozen, so its text abilities are guaranteed not to get worse.
Four places where image and text can meet
Figure 1 shows the four designs side by side. In each panel, orange squares are image information and blue squares are text tokens. Read each panel from bottom (inputs) to top (output). Later, Figure 6 runs all four designs on real numbers, layer by layer.
Early and late fusion: two meanings of the same words
Be careful: the words "early" and "late" are used in two different ways. Interviewers sometimes mean one, sometimes the other, so say which one you mean.
Meaning 1: the classic one (where features are combined)
In classic multimodal learning, the question is at which depth the two signals are combined.
- Early fusion combines the raw inputs or low-level features, then one model processes the mix. Example: in video classification, stack 10 frames along the channel axis and feed them to the first convolution, so the very first filters see motion. Karpathy et al. (2014) called this "early fusion" and compared it with "late fusion" (two single-frame networks, 15 frames apart, merged only at the first fully connected layer) and "slow fusion" (in between). [Certain]
- Late fusion processes each input alone and combines only high-level outputs: embeddings or predictions. The two-stream video network of Simonyan and Zisserman (2014) is late fusion: an RGB network and an optical-flow network each predict class scores, and the scores are averaged. [Certain]
Two common forms of late fusion are combining embeddings with a dot product (CLIP) and combining predictions with a weighted average:
- Read the symbols
- \(I, T\)
- the image and the text
- \(f, g\)
- the image encoder and the text encoder; each outputs one vector
- \(s(I, T)\)
- cosine similarity between the two vectors: high means "they match"
- \(p_{\text{img}}, p_{\text{txt}}\)
- class probabilities predicted from each modality alone
- \(\lambda\)
- a mixing weight between 0 and 1, chosen on validation data
Late fusion is cheap and simple. Its weakness is that the two modalities never interact before the end. The image encoder cannot know which part of the image the text is asking about. In (179.1), the whole image is squeezed into one vector before the question is even seen. That is why CLIP is excellent at retrieval but cannot answer "what is written on the second sign from the left?" (session 166).
Meaning 2: the modern VLM one (separate encoder or not)
In recent VLM papers, the words mean something else. "Late fusion" means: take a vision encoder that was pretrained separately (for example a CLIP or SigLIP ViT), connect it to a pretrained LLM, and continue training. "Early fusion" means: no separate encoder; image patches or image codes enter one transformer that is trained on all modalities from the start. In this language, LLaVA counts as "late fusion", even though image and text tokens mix in every LLM layer. Shukor et al. (2025) use the words this way and report that, when both are trained from scratch with the same compute, late fusion has no built-in advantage; early fusion did better at small parameter counts and was more efficient to train. [Certain: that is their claim; it is one study.]
The unified sequence
This is the design of LLaVA and of most open VLMs today. The connector from session 178 maps each image feature to a vector of the LLM's width \(d\). These vectors are placed in the sequence, usually where a special image placeholder token was in the prompt, and the LLM runs exactly as it would on text:
- Read the symbols
- \(\mathbf{v}_j\)
- the \(j\)-th image feature from the vision encoder (one per patch, or one per resampler query)
- \(P\)
- the connector (projector): a linear layer or a small MLP
- \(E(t_i)\)
- the embedding of the \(i\)-th text token
- \(N_v, N_t\)
- number of visual tokens and of text tokens
- \(H^{(0)}\)
- the input to the first LLM layer: one matrix, one row per token
Then every layer applies causal self-attention over all \(N_v + N_t\) rows. A text token can attend to every image token that comes before it and to earlier text, all in one softmax. Nothing in the LLM knows which rows "are image". Only the position in the sequence and the content of the vectors differ.
Training uses the normal next-token loss, but only on text positions. There is no target "next image token" to predict, so image positions are masked out of the loss:
(In instruction tuning, the prompt text is usually masked out too, so only the answer tokens count. session 188 covers this.)
How many tokens is one image?
A ViT with patch size 14 on a \(336 \times 336\) image makes \((336/14)^2 = 24^2 = 576\) patches. LLaVA-1.5 keeps all of them, so one image costs 576 tokens. [Certain] The cost grows with the square of the resolution, and high-resolution methods tile the image into many crops (session 180). A video with 64 frames at 576 tokens each would need \(64 \times 576 = 36{,}864\) tokens. This is the main weakness of the unified sequence: every visual token sits in the context and passes through every layer.
Its strength is simplicity and power. The LLM's own attention, in every layer, can compare any word with any image patch. No new layer types are needed, the serving stack is the normal LLM stack, and models of this type lead most OCR and document benchmarks. NVLM, which built both designs on the same data, reports that its decoder-only (unified) model was more accurate on OCR-related tasks, while its cross-attention model trained faster on high-resolution images. [Certain: their reported finding]
Native early fusion: no separate image encoder
Two well-known models skip the pretrained vision encoder entirely.
- Fuyu-8B (Adept, 2023) cuts the image into \(30 \times 30\) pixel patches, flattens each patch and maps it with one linear layer to the decoder width. The patches go directly into the decoder sequence, with a special "image newline" token at the end of each row of patches, so any image size works. [Certain: patch size 30 is in the released config]
- Chameleon (Meta, 2024) turns a \(512 \times 512\) image into 1,024 discrete tokens from a codebook of 8,192 entries, using a learned image tokenizer. Images and text are then the same kind of thing: integer IDs in one vocabulary. The model can read and write images token by token. [Certain]
The price is training difficulty. The Chameleon team reports that training was unstable above 8B parameters and 1T tokens, with loss divergences late in training. They needed QK-norm (normalizing queries and keys before the softmax) and z-loss to fix it. [Certain] Their explanation: the image and text tokens have very different statistics but share every weight, so norms keep growing until they leave the BF16 range. Their tokenizer also struggles to reconstruct images with a lot of text, which hurts OCR.
Cross-attention: the image as keys and values
Now the other big design. The LLM's sequence holds only text. Between some of the LLM layers we insert a new cross-attention layer. In self-attention, queries, keys and values all come from the same sequence. In cross-attention, the queries come from the text and the keys and values come from the image (session 16):
- Read the symbols
- \(Y \in \R^{N_t \times d}\)
- the text hidden states at this depth: one row per text token
- \(V \in \R^{N_v \times d_v}\)
- the visual tokens (the same for every cross-attention layer)
- \(W_Q\)
- makes queries from text (color query)
- \(W_K, W_V\)
- make keys and values from the image (colors key, value)
- \(W_O\)
- output projection back to width \(d\)
- \(d_h\)
- the head dimension (the formula is written for one head)
- \(M\)
- a mask: 0 where a text token may look at an image token, \(-\infty\) where it may not
The attention matrix has shape \(N_t \times N_v\): every text token gets a weighted average of image values. There is no causal mask between image tokens, because the image is not generated; it is just read.
Which image may each text token see?
Flamingo handles documents that mix many images and texts ("interleaved" data, like a web page). Its rule: a text token cross-attends only to the visual tokens of the image that came just before it. [Certain] In mask form, for text token \(i\) and visual token \(j\) of image \(m(j)\):
Earlier images are not lost. The text that followed an earlier image has already absorbed it through cross-attention, and later text can read that text through ordinary causal self-attention. The rule keeps each cross-attention small and lets the model handle any number of images. Text that comes before any image has nothing to look at; Llama 3.2's Hugging Face implementation multiplies the cross-attention output of such rows by zero (full_text_row_masked_out_mask) so they behave like pure text. [Certain]
Figure 2 lets you compare the attention patterns of the two designs on the same short interleaved sequence.
How to read Figure 2
In the unified sequence there is one 12 × 12 causal mask: image rows are queries too, and they pass through every layer. Allowed cells: 78. In the cross-attention modes, the text has its own 6 × 6 causal self-attention (21 cells), and a separate 6 × 6 text-to-image cross-attention. With the Flamingo rule, "dog" sees only the 3 tokens of image 2 (15 cross cells in total); with all earlier images it sees 6 (24 cross cells). Note the row "Look": in cross-attention mode it has no image at all.
The tanh gate: start as the frozen LLM
Here is the problem. We take a strong pretrained LLM and insert new, randomly initialized cross-attention layers. At step 0, these layers output noise. That noise is added to the residual stream, and the LLM, which never saw such inputs, produces garbage. Training must first repair the damage, and the large early gradients flow into a model we wanted to keep. If we freeze the LLM, it cannot adapt, so the noise is even more harmful.
Flamingo's fix: multiply the output of each new layer by \(\tanh(\alpha)\), where \(\alpha\) is one learnable scalar per layer, initialized to 0. [Certain] Each inserted block (Flamingo calls it GATED XATTN-DENSE) has two gated parts, a cross-attention and a feed-forward network, each with its own gate:
- Read the symbols
- \(\alpha_{\text{x}}, \alpha_{\text{f}}\)
- the two gate scalars of this block; both start at 0
- \(\tanh\)
- squashes the gate into \((-1, 1)\), so a gate can never scale the new branch by more than 1
- \(\operatorname{LN}\)
- layer normalization before each branch (pre-norm)
- \(\operatorname{FFW}\)
- a new feed-forward network, trained together with the cross-attention
At step 0, \(\tanh(0) = 0\), so both new branches add exactly zero. The whole model is then the original LLM, bit for bit. Text-only behavior is unchanged, and on an image prompt the model ignores the image. During training the gates open slowly, as far as the data asks for.
If the output is zero, how does anything learn?
Write \(\mathbf{u} = \operatorname{CA}(\cdot)\) for the branch output and \(Y' = Y + \tanh(\alpha)\,\mathbf{u}\). By the chain rule (session 4), with \(G = \partial \mathcal{L}/\partial Y'\):
At \(\alpha = 0\), the gate's gradient is at its largest (the factor \(1 - \tanh^2 0 = 1\)), while the gradient of every weight \(W\) inside the branch is multiplied by \(\tanh 0 = 0\). So the first optimizer step changes only the gates. After that, \(\tanh(\alpha) \neq 0\), and the branch weights start to learn too. The same trick appears in LoRA, where one of the two matrices starts at zero (session 57).
Take the 8-number hidden state \(\mathbf{x}\) and branch output \(\mathbf{u}\) of Figure 3, with \(\lVert \mathbf{x} \rVert = 3.105\) and \(\lVert \mathbf{u} \rVert = 2.238\), so \(\lVert \mathbf{u} \rVert / \lVert \mathbf{x} \rVert = 0.721\). The relative change of the residual stream is \(|\tanh \alpha| \times 0.721\):
- \(\alpha = 0\): \(\tanh = 0\), change 0. The LLM is untouched.
- \(\alpha = 0.1\): \(\tanh = 0.100\), change 0.072 (7%).
- \(\alpha = 0.5\): \(\tanh = 0.462\), change 0.333.
- \(\alpha = 1\): \(\tanh = 0.762\), change 0.549, and the gate's own gradient factor has dropped to \(1 - 0.762^2 = 0.420\).
- No gate at all (factor 1): change 0.721 at step 0, from random weights. That is the shock the gate prevents.
The Flamingo ablation shows it matters: removing the tanh gating dropped the overall score of Flamingo-3B from 70.7 to 66.5, and the authors report training instabilities without it. [Certain]
How to read Figure 3
Move \(\alpha\) from 0 upward. The green bars grow and the outline moves away from the gray bars: the image starts to change the LLM's computation. The readout also shows the two gradient factors of formula (179.8): at \(\alpha = 0\) the gate gets the full gradient and the branch weights get none. Near \(\alpha = 3\) the gate is almost fully open (0.995) and nearly stops learning (slope 0.010). Tick "Remove the gate" to see the change at step 0 without gating.
Formula (179.8) is easier to believe when you watch it run. Figure 4 trains one gated branch with plain gradient descent on a tiny problem. The frozen LLM gives the hidden state \(\mathbf{x}\) (the same 8 numbers as in Figure 3). The image gives a feature \(\mathbf{z}\) with 4 numbers, and the branch output is \(\mathbf{u} = W\mathbf{z}\), a stand-in for the cross-attention output. The target \(\mathbf{y}^\ast\) is the hidden state that would use the image well. The loss is \(\tfrac12 \lVert \mathbf{y} - \mathbf{y}^\ast \rVert^2\), so \(G = \mathbf{y} - \mathbf{y}^\ast\). \(W\) starts random, \(\alpha\) starts at 0, and every number is computed in your browser at each step.
- \(\mathbf{u} \leftarrow W \mathbf{z}\) // branch output
- \(\mathbf{y} \leftarrow \mathbf{x} + \tanh(\alpha)\, \mathbf{u}\) // (179.7)
- \(G \leftarrow \mathbf{y} - \mathbf{y}^\ast,\ \ \mathcal{L} \leftarrow \tfrac12 \lVert G \rVert^2\)
- \(g_\alpha \leftarrow (1 - \tanh^2 \alpha)\, \langle G, \mathbf{u} \rangle\) // (179.8), left
- \(g_W \leftarrow \tanh(\alpha)\, G\, \mathbf{z}^\top\) // (179.8), right
- \(\alpha \leftarrow \alpha - \eta\, g_\alpha,\ \ W \leftarrow W - \eta\, g_W\)
How to read Figure 4
Press Step and read the live lines. At \(t = 0\) the gate is closed: \(\mathbf{y} = \mathbf{x}\) and the loss is 2.315, the frozen LLM's loss. Line 4 gives \(g_\alpha = (1 - 0^2) \times (-1.079) = -1.079\), but line 5 gives \(\lVert g_W \rVert = 0\) exactly, because of the factor \(\tanh 0\). So the first update (\(\eta = 0.1\)) moves only the gate: \(\alpha = 0.108\), and \(W\) is unchanged. At \(t = 1\), \(W\) gets its first gradient (norm 0.349). After 40 steps \(\alpha = 0.822\), \(\tanh\alpha = 0.676\) and the loss is 0.009; with the gate, the loss never rises above the frozen line, for all three learning rates. Now choose "no gate": the random branch moves the state by 2.773 at once (\(\lVert \mathbf{x} \rVert = 3.105\)), and the loss starts at 5.081, more than twice the frozen loss. In this one-layer linear toy the ungated branch still recovers fast. In a real VLM the damage passes through every frozen layer above, and the large early gradients are what the gate avoids.
Keeping the LLM frozen
With the gate, it becomes safe to freeze the LLM and train only the new layers (and the connector). Flamingo did this, and its ablation is clear: on Flamingo-3B, fine-tuning the LLM instead of freezing it dropped the overall score from 70.7 to 62.7, which the authors call catastrophic forgetting; training the LLM from random initialization gave 57.8. [Certain] Llama 3 made the same choice for its vision models: "The language model weights are kept frozen to maintain text-only performance." [Certain] A product team gets a strong promise from this: the text model you already evaluated, red-teamed and shipped is exactly the same network on text-only prompts.
Unified-sequence models usually unfreeze the LLM during later training stages and protect text skills differently, by mixing text-only data into the training set. NVLM did this and reports that its models' text-only scores even went up slightly. [Certain: their reported finding] So freezing is not the only way to stay safe; it is the cheapest and most certain one.
Real systems
| Model | Frozen LLM | Where cross-attention goes | Visual tokens |
|---|---|---|---|
| Flamingo-3B | Chinchilla 1.4B (24 layers) | before every LM layer; 1.2B new parameters | 64 per image (Perceiver resampler) |
| Flamingo-9B | Chinchilla 7B (40 layers) | every 4th layer; 1.6B new parameters | 64 per image |
| Flamingo-80B | Chinchilla 70B (80 layers) | every 7th layer; 10B new parameters | 64 per image |
| Llama 3.2 11B Vision | 32 text layers of width 4096 | 8 new layers at positions 3, 8, …, 38 of 40 (one after every 4 text layers), tanh-gated attention and MLP | 1,601 per 560 × 560 tile, up to 4 tiles |
| Llama 3 405B (paper) | 405B | after every 4th self-attention layer, with GQA; about 100B new parameters | from a ViT-H/14 encoder with 850M parameters |
Flamingo numbers are from its parameter table; Llama 3.2 numbers are from its released config.json (cross_attention_layers = [3, 8, 13, 18, 23, 28, 33, 38], image_size 560, patch_size 14, so \(40^2 + 1 = 1{,}601\) tokens per tile with the class token). [Certain] Flamingo also measured the insertion frequency: on the 3B model, cross-attention every 4th layer instead of every layer lowered the score from 70.7 to 68.8 but cut the training step time from 1.74 s to 1.02 s. [Certain]
The trade-off: tokens in the context vs extra layers
Now we can count. Take a base LLM with \(L\) layers, KV heads \(h_{kv}\), head size \(d_h\), and \(b\) bytes per number. The KV cache stores one key and one value vector per token per layer (session 80). For one image of \(N_v\) tokens and a prompt of \(N_t\) text tokens:
Here \(k\) is the spacing of cross-attention layers (one every \(k\) LLM layers, so \(L/k\) of them). In the cross design, the image's keys and values are computed once per cross-attention layer and reused for every generated token; text tokens never enter those layers' caches. For compute in the prefill (the first pass over the prompt), count about 2 FLOPs per parameter per token for the linear layers (session 1):
- Read the symbols
- \(P_{\text{LM}}\)
- the LLM's linear-layer parameters (all layers)
- \(P_{\text{X}}\)
- parameters of all cross-attention layers that act on text: \(W_Q\), \(W_O\) and the new FFN
- \(P_{kv}\)
- parameters of \(W_K\) and \(W_V\) in one cross-attention layer: the only part that touches image tokens
Both formulas ignore the attention-score part (the \(QK^\top\) and the weighted sum), which grows with the square of the sequence and makes the unified design look even worse for long image sequences. Both designs also pay the same vision encoder cost.
The text part has \(L = 32\) layers, \(d = 4096\), 8 KV heads of size 128, FFN width 14,336 (the Llama 3.1 8B shape). One LLM layer has \(2 \cdot 4096^2 + 2 \cdot 4096 \cdot 1024 + 3 \cdot 4096 \cdot 14336 = 218.1\)M linear parameters, so \(P_{\text{LM}} = 6.98\)B. A 4-tile image gives \(N_v = 4 \times 1601 = 6{,}404\) tokens. Take a prompt of \(N_t = 512\) text tokens, BF16 (\(b = 2\)), so one token in one layer costs \(2 \cdot 8 \cdot 128 \cdot 2 = 4{,}096\) bytes of cache.
- KV cache. Unified: \(32 \times 6916 \times 4096 = 906.5\) MB. Cross (\(k = 4\), 8 layers): \(32 \times 512 \times 4096 + 8 \times 6404 \times 4096 = 277.0\) MB. The image part alone is 839.4 MB vs 209.8 MB: 4 times less, because only 8 layers hold it instead of 32.
- Prefill compute. Unified: \(2 \times 6.98\text{B} \times 6916 = 96.5\) TFLOP. Cross: 9.72 TFLOP, about 10 times less. The image part alone is 89.4 TFLOP in the unified design and 0.86 TFLOP in the cross design, where image tokens only go through \(W_K\) and \(W_V\).
- Sequence length. Unified: 6,916 positions in the context window. Cross: 512.
- The price. Each cross-attention layer has the same linear shapes as an LLM layer: 218.1M parameters, so 8 of them add 1.745B parameters that every text token must pass through and that take memory.
These are the reasons Meta gives for cross-attention: it avoids passing full-resolution images through the LLM, and it keeps text-only performance unchanged. [Certain: their stated reasons; our numbers are linear-layer estimates]
How to read Figure 5
The defaults are the worked example: 906.5 MB vs 277.0 MB of KV cache and 96.5 vs 9.72 TFLOP. Drag \(N_v\) down to 64 (a Flamingo-style resampler): the unified design becomes cheap, and the cross design's 1.745B extra parameters become the main cost. Choose "every 1": 32 cross layers, 6.98B new parameters, almost a second LLM. Raise the number of images: the unified sequence quickly fills the context window.
The table summarizes the trade-off.
| Unified sequence | Cross-attention | |
|---|---|---|
| Image tokens in the context | yes: \(N_v\) per image, in every layer and in the KV cache | no: only in \(L/k\) cross layers' caches |
| New parameters | only the connector (millions) | new layers (billions: 1.745B for 8 layers at 8B scale) |
| Interaction depth | image and text mix in every layer | only at the cross layers |
| Text-only safety | LLM usually unfrozen; needs text data in the mix | LLM can stay frozen; tanh gate guarantees the start |
| Serving | standard LLM engine; prefix caching of the image works | needs cross-attention support and a second cache |
| Strength | fine detail, OCR, simple recipe | many or large images, long videos, cheap prefill |
A third option combines both. NVLM-H puts a low-resolution thumbnail of the image into the sequence (so self-attention sees the global picture) and sends the high-resolution tiles through gated cross-attention. [Certain] Most open models in 2025 still use the unified sequence and fight its cost with fewer tokens per image instead: pixel shuffle, resamplers and token pruning (session 183). [Likely]
Putting it together: the two forward passes
- input: images, prompt with one placeholder token per image, LLM with \(L\) layers, connector \(P\)
- \(\mathbf{v} \leftarrow\) VisionEncoder(image) // e.g. 576 features for 336 px
- \(Z \leftarrow P(\mathbf{v})\) // width d, formula (179.3)
- \(H \leftarrow\) text embeddings, with each placeholder replaced by its block of rows \(Z\)
- for \(\ell = 1 \dots L\): \(H \leftarrow\) Layer\(_\ell(H)\) with a causal mask over all \(N_v + N_t\) positions
- loss: next-token cross-entropy on answer positions only // formula (179.4)
- at inference: cache K, V of all positions, including image positions, in all \(L\) layers
- input: images, text, frozen LLM layers \(1 \dots L\), new gated blocks every \(k\) layers, gates \(\alpha\) initialized to 0
- \(V \leftarrow\) Resampler(VisionEncoder(images)) // e.g. 64 tokens per image
- \(M \leftarrow\) cross-attention mask: each text token sees its latest image // formula (179.6)
- \(Y \leftarrow\) text embeddings only
- for \(\ell = 1 \dots L\):
- if a gated block sits here: \(Y \leftarrow Y + \tanh(\alpha_x)\,\mathrm{CA}(\mathrm{LN}(Y), V, M)\); \(Y \leftarrow Y + \tanh(\alpha_f)\,\mathrm{FFW}(\mathrm{LN}(Y))\) // (179.7)
- zero the cross output on rows with no image yet
- \(Y \leftarrow\) FrozenLayer\(_\ell(Y)\) with a causal mask over the \(N_t\) text positions
- loss on text; gradients only reach the gates, the new blocks and the resampler
- at inference: compute K, V of the image once per cross layer and reuse them for every new token
Figure 6 runs all four designs on the same tiny frozen LLM, one layer per step, so you can see where the image enters and what it costs. The prompt is "Look [image] what color ?". We follow the last text token "?", the one that will predict the answer. To measure how much the image has reached it, compare its hidden state with the hidden state of the same LLM run on the text alone:
- Read the symbols
- \(\mathbf{h}_?^{(\ell)}\)
- the hidden state of the token "?" after layer \(\ell\)
- \(\Delta_\ell\)
- 0 means the image has not changed this token at all at depth \(\ell\); larger means more image information has flowed into it
- \(\mathbf{f} \leftarrow P\big(\operatorname{mean}_j \operatorname{Enc}(\text{img})_j\big)\); \(Y \leftarrow\) text embeddings only
- for \(\ell = 1 \dots L\):
- \(Y \leftarrow \text{FrozenLayer}_\ell(Y)\) // the image is not here
- \(\Delta_\ell \leftarrow 0\) // (179.11)
- \(s \leftarrow \cos(\mathbf{f}, \mathbf{y}_?)\) // (179.1): they meet once
- \(H \leftarrow [W_p \mathbf{x}_1, \dots, W_p \mathbf{x}_{N_v};\ E(\text{Look}), \dots, E(?)]\) // raw patches, no encoder
- for \(\ell = 1 \dots L\):
- \(H \leftarrow H + \operatorname{Attn}(\operatorname{LN}(H))\) // causal, all rows
- \(H \leftarrow H + \operatorname{MLP}(\operatorname{LN}(H))\)
- \(\Delta_\ell \leftarrow \lVert \mathbf{h}_? - \bar{\mathbf{h}}_? \rVert\) // (179.11)
- output \(\mathbf{h}_?^{(L)}\) to the LM head
- \(H \leftarrow [E(\text{Look});\ P(\operatorname{Enc}(\text{img}));\ E(\text{what}), E(\text{color}), E(?)]\) // (179.3)
- for \(\ell = 1 \dots L\):
- \(H \leftarrow H + \operatorname{Attn}(\operatorname{LN}(H))\) // causal, all rows
- \(H \leftarrow H + \operatorname{MLP}(\operatorname{LN}(H))\)
- \(\Delta_\ell \leftarrow \lVert \mathbf{h}_? - \bar{\mathbf{h}}_? \rVert\) // (179.11)
- output \(\mathbf{h}_?^{(L)}\) to the LM head
- \(Y \leftarrow [E(\text{Look}), E(\text{what}), E(\text{color}), E(?)]\); \(V \leftarrow P(\operatorname{Enc}(\text{img}))\)
- for \(\ell = 1 \dots L\):
- if \((\ell - 1) \bmod k = 0\): \(Y \leftarrow Y + \tanh(\alpha)\, \operatorname{CA}(\operatorname{LN}(Y), V)\) // (179.5), (179.7)
- \(Y \leftarrow \text{FrozenLayer}_\ell(Y)\) // text rows only
- \(\Delta_\ell \leftarrow \lVert \mathbf{y}_? - \bar{\mathbf{h}}_? \rVert\) // (179.11)
- output \(\mathbf{y}_?^{(L)}\) to the LM head
How to read Figure 6
Press Play. In layer 1, unified already has \(\Delta_1 = 1.518\): "?" puts 0.406 of its attention on the 4 image tokens. Early fusion has 0.462 (the raw patches are not encoded, so they carry different vectors). Cross has 0.236, from the gated block before layer 1. Late stays at 0 in every layer and meets the image once, at the top: \(\cos(\mathbf{f}, \mathbf{g}) = 0.104\). After layer 6: early 2.406, unified 3.418, cross 0.705. Now set \(\alpha = 0\): the cross column becomes exactly 0 in every layer, the frozen LLM of formula (179.7). Compare the costs at the end (\(N_v = 4\), \(k = 2\)): linear FLOPs 7,296 (late), 12,416 (early, no encoder), 13,440 (unified), 8,832 (cross); cached K/V rows 24, 48, 48, 36. Set \(N_v = 8\): unified grows to 20,736 FLOPs and 72 K/V rows, cross only to 9,984 FLOPs at \(k = 3\). Choose a design under the columns to follow its code line by line.
A runnable PyTorch version of both designs on a tiny 8-layer model. It checks the three facts of this session: at initialization the gated model equals the frozen LLM exactly; the first gradients reach only the gates; and the unified sequence changes the text outputs immediately and makes the sequence longer.
import torch, torch.nn as nn torch.manual_seed(0) d, n_heads = 64, 4 class Block(nn.Module): # one decoder layer of the (frozen) LLM def __init__(s): super().__init__() s.n1, s.n2 = nn.LayerNorm(d), nn.LayerNorm(d) s.attn = nn.MultiheadAttention(d, n_heads, batch_first=True) s.ffw = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) def forward(s, x, mask): h = s.n1(x) x = x + s.attn(h, h, h, attn_mask=mask, need_weights=False)[0] return x + s.ffw(s.n2(x)) class GatedXAttn(nn.Module): # new layer, formula (179.7) def __init__(s): super().__init__() s.n1, s.n2 = nn.LayerNorm(d), nn.LayerNorm(d) s.xattn = nn.MultiheadAttention(d, n_heads, batch_first=True) s.ffw = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d)) s.a_x = nn.Parameter(torch.zeros(1)); s.a_f = nn.Parameter(torch.zeros(1)) # gates start at 0 def forward(s, y, v, xmask=None): y = y + torch.tanh(s.a_x) * s.xattn(s.n1(y), v, v, attn_mask=xmask, need_weights=False)[0] return y + torch.tanh(s.a_f) * s.ffw(s.n2(y)) def causal(n): # True = not allowed to attend return torch.triu(torch.ones(n, n, dtype=torch.bool), 1) L, k = 8, 4 lm = nn.ModuleList([Block() for _ in range(L)]) for p in lm.parameters(): p.requires_grad_(False) # the LLM stays frozen xa = nn.ModuleList([GatedXAttn() for _ in range(L // k)]) # one new layer every k layers txt = torch.randn(1, 10, d) # 10 text-token embeddings img = torch.randn(1, 64, d) # 64 visual tokens (for example from a resampler) def lm_only(x): for b in lm: x = b(x, causal(x.shape[1])) return x def cross_model(x, v): # xattn block before every k-th LM layer for i, b in enumerate(lm): if i % k == 0: x = xa[i // k](x, v) x = b(x, causal(x.shape[1])) return x def unified(x, v): # formula (179.3): one sequence [image ; text] return lm_only(torch.cat([v, x], dim=1))[:, v.shape[1]:] # 1) at initialization the cross-attention model IS the frozen LLM print((cross_model(txt, img) - lm_only(txt)).abs().max().item()) # 0.0 # 2) gradients at init: the gates get a signal, the new attention weights get none yet (179.8) loss = cross_model(txt, img).pow(2).mean(); loss.backward() print(xa[0].a_x.grad.abs().item() > 0, xa[0].xattn.in_proj_weight.grad.abs().max().item()) # True 0.0 # 3) the unified sequence changes the text outputs at once, and makes the sequence longer print((unified(txt, img) - lm_only(txt)).abs().max().item() > 0, txt.shape[1] + img.shape[1]) # True 74 # 4) after one optimizer step the gates are open a little and the image can matter opt = torch.optim.AdamW([p for p in xa.parameters()], lr=1e-2) opt.step(); opt.zero_grad() print(round(torch.tanh(xa[0].a_x).abs().item(), 4), (cross_model(txt, img) - lm_only(txt)).abs().max().item() > 0) # 0.01 True
The printed lines are 0.0, True 0.0, True 74 and 0.01 True (checked with PyTorch 2.8 on CPU). The last value is 0.01 because Adam's first step moves each parameter by about the learning rate, whatever the gradient size (session 4).
Where this is used
| Application | Typical fusion | Why |
|---|---|---|
| Text-to-image and text-to-video retrieval, near-duplicate and video fingerprinting | Late fusion (CLIP or SigLIP embeddings, per frame, pooled) | Embeddings are computed once and indexed; one dot product per pair scales to millions (session 169) |
| Action recognition and video classification | Classic early, slow or late fusion over time; two-stream RGB + flow with score averaging | Choose how early motion information is combined; late fusion is cheap and robust |
| Chat about images, OCR, document and chart QA | Unified sequence (LLaVA, Qwen2-VL, InternVL) | Fine-grained interaction in every layer; NVLM found decoder-only better on OCR |
| Long video QA and key-moment detection | Cross-attention or heavy token reduction; Llama 3 adds video cross-attention layers and a resampler that merges frames, up to 64 frames | 64 frames × 576 tokens = 36,864 tokens would fill the context in a unified design |
| Adding vision to a shipped text LLM | Gated cross-attention with a frozen LLM (Llama 3.2 Vision) | Text-only behavior is exactly the old model; text and vision teams can work in parallel |
| Image generation and mixed image-text output | Native early fusion with discrete image tokens (Chameleon) | The same next-token head writes text and image tokens |
| Video-RAG | Late fusion to retrieve clips, then a unified-sequence VLM on the retrieved frames | Cheap search over the whole archive, expensive reasoning only on a few clips |
Common mistakes
- Mixing up the two meanings of "early" and "late". LLaVA is "late fusion" in the modern sense and "deep fusion" in the classic sense. Define your terms.
- Initializing the gate to a nonzero value (or forgetting the gate). The new random layers then disturb the frozen LLM at step 0, and you lose the guarantee. Flamingo measured a drop from 70.7 to 66.5 without the gate.
- Initializing the branch weights to zero as well as the gate. Then both \(\partial\mathcal{L}/\partial\alpha\) (needs \(\mathbf{u} \neq 0\)) and \(\partial\mathcal{L}/\partial W\) (needs \(\tanh\alpha \neq 0\)) are zero, and nothing ever learns. Exactly one of the two must start at zero.
- Putting the loss on image positions in a unified sequence. There is no correct "next image token"; mask those positions out (formula 179.4).
- Letting text before the first image cross-attend to nothing. A softmax over an all-masked row gives NaN or a uniform average. Zero these rows, as Llama 3.2 does.
- Forgetting the image KV cache in the cross design. Recomputing \(VW_K\), \(VW_V\) for every generated token wastes compute; compute them once per cross layer.
- Counting only parameters, not tokens. A unified model with 0 new layers can still be far more expensive per request: 89 TFLOP of prefill for one 4-tile image in the worked example.
- Training native early fusion without stability tricks. Chameleon needed QK-norm and z-loss above 8B parameters.
Interview questions
What is the difference between early and late fusion? Give examples.basic
How does LLaVA give an image to the LLM?basic
Why is Flamingo's gate \(\tanh(\alpha)\) with \(\alpha = 0\) at the start?core
If the gate is 0, the gradient to the cross-attention weights is 0. How does training start?core
Compare the KV cache of a unified-sequence and a cross-attention VLM for one large image.core
Why did Meta choose cross-attention for Llama 3 vision?core
In Flamingo a text token sees only the latest image. How can it answer a question about an earlier image?advanced
Which design would you choose for document and OCR tasks, and why?advanced
Why is native early fusion (Chameleon) hard to train?advanced
You must answer questions about a 64-frame video. Unified sequence or cross-attention?advanced
How often should cross-attention layers be inserted?core
Exercises
- LLaVA-1.5 uses 336 px images and patch size 14. How many tokens is one image? How much BF16 KV cache do 8 such images need on the Llama 3.1 8B shape (32 layers, 4,096 bytes per token per layer) in a unified sequence, and how much with cross layers every 4th layer?
- Flamingo-9B has a 40-layer frozen LM and inserts a gated block every 4th layer. How many gated blocks does it have? How many gate scalars?
- In Figure 3, set \(\alpha = 0.5\). Compute \(\tanh\alpha\), the relative change of the hidden state, and the gate's gradient factor. Check your answer in the readout.
- In Figure 2, count the allowed attention cells in the unified mode and in the cross-attention mode with the Flamingo rule.
- Show from (179.8) what happens if you initialize the gate to 0 and the output projection \(W_O\) of the cross-attention to 0.
- The cross design needs 1.745B extra parameters (BF16) but caches each visual token in 8 layers instead of 32. How many visual tokens must be in the KV cache at the same time (over all requests in a batch) before the cross design also uses less total memory? Use (179.9).
- In Figure 6 with \(N_v = 4\) and \(k = 2\), count by hand the attention scores that the unified column and the cross column compute over all 6 layers. Then check the totals under the columns at the last step.
Answerscheck
Key takeaways
- Fusion is where image and text meet: at the end (late, 179.1 and 179.2), in one shared sequence (unified, 179.3), from raw patches with no encoder (native early fusion), or through inserted cross-attention (179.5).
- "Early" and "late" have a classic meaning (depth of combination) and a modern VLM meaning (separate pretrained encoder or not). LLaVA is deep fusion in the first and late fusion in the second.
- The unified sequence is simple and strong (best on OCR in NVLM's comparison) but every visual token sits in every layer and in the KV cache: 576 tokens for one LLaVA-1.5 image, 6,404 for a 4-tile Llama 3.2 image.
- Gated cross-attention (179.7) with \(\tanh(\alpha)\), \(\alpha = 0\) at start, makes the model equal to the frozen LLM at step 0; the gate gets the first gradient, the branch weights follow (179.8). Freezing the LLM protected text skills in Flamingo and Llama 3.
- The trade-off is tokens vs layers (179.9, 179.10): for Llama 3.2 11B with a 4-tile image, 906.5 vs 277.0 MB of cache and 96.5 vs 9.72 TFLOP of prefill, paid for with 1.745B new parameters.
Key formulas
References for this session
- Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
- Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Liu et al. (2023). Visual Instruction Tuning
- Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Dai, Lee, Wang, Yang, Liu, Barker, Rintamaki, Shoeybi, Catanzaro & Ping (2024). NVLM: Open Frontier-Class Multimodal LLMs
- Bavishi et al. (Adept) (2023). Fuyu-8B: A Multimodal Architecture for AI Agents
- Shukor, Fini, Turrisi da Costa, Cord, Susskind & El-Nouby (2025). Scaling Laws for Native Multimodal Models
- Karpathy, Toderici, Shetty, Leung, Sukthankar & Fei-Fei (2014). Large-scale Video Classification with Convolutional Neural Networks
- Simonyan & Zisserman (2014). Two-Stream Convolutional Networks for Action Recognition in Videos