LLM & VLM courseBlog
179Track 7: Vision and VLMs · session 17 of 38

Fusion: early, late, unified sequence and cross-attention

A vision-language model must decide where image and text information meet. They can meet only at the end (late fusion), inside one shared sequence of tokens (unified sequence, as in LLaVA), from the very first layer with no separate image encoder (native early fusion, as in Chameleon), or through new cross-attention layers inserted into a language model (Flamingo, Llama 3.2 Vision). This session builds each design, explains why a tanh gate that starts at zero lets the language model stay frozen and safe, and measures the real cost: tokens in the context versus extra layers.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 18 (masks), 26 (GQA and the KV cache), 166 (CLIP), 177 (anatomy of a VLM) and 178 (connectors)

What you will learn

  • The two meanings of “early” and “late” fusion: the classic one (where features or predictions are combined) and the modern VLM one (separate pretrained encoder or not)
  • The unified sequence: image tokens and text tokens in one causal self-attention stream, with the loss only on text
  • Cross-attention adapters: text queries, image keys and values, and the Flamingo rule that a text token looks only at the latest image
  • Why a tanh gate initialized to 0 makes the new model equal to the frozen LLM at step 0, and why it can still learn
  • The cost trade-off with real numbers from Llama 3.2 11B Vision: KV cache, prefill FLOPs, sequence length and new parameters
  • What real systems chose (LLaVA, Fuyu, Chameleon, Flamingo, Llama 3, NVLM) and what their ablations say

Why this matters

In session 177 we split a VLM into three parts: a vision encoder, a connector and a language model. In session 178 we saw how the connector turns image features into a small set of vectors. This session answers the next question: where do those image vectors enter the language model, and how do they mix with the text?

The choice is called the fusion design. "Fusion" simply means "the place where information from two modalities (here: image and text) is combined". There are four common answers:

  1. Late fusion. Each modality is processed alone, and the results meet only at the very end. CLIP is the standard example: one image vector, one text vector, one dot product.
  2. Unified sequence. Image features become "visual tokens" and are placed in the same sequence as the text tokens. The language model reads them all with ordinary self-attention. LLaVA, Qwen2-VL, InternVL and most open VLMs work this way.
  3. Native early fusion. There is no separate pretrained image encoder at all. Raw image patches (Fuyu) or discrete image codes (Chameleon) go straight into one transformer that is trained on both modalities from the start.
  4. Cross-attention. The text stays alone in the language model's sequence. New cross-attention layers are inserted between the language model's layers, and through them the text "looks at" the image. Flamingo and Llama 3.2 Vision work this way.

The choice decides almost everything about cost and safety. With a unified sequence, a high-resolution image can add thousands of tokens to the context, and every one of them passes through every layer and sits in the KV cache. With cross-attention, the image never enters the sequence, but you add new layers with billions of new parameters. And cross-attention with a tanh gate that starts at zero lets you keep the language model completely frozen, so its text abilities are guaranteed not to get worse.

Four places where image and text can meet

Figure 1 shows the four designs side by side. In each panel, orange squares are image information and blue squares are text tokens. Read each panel from bottom (inputs) to top (output). Later, Figure 6 runs all four designs on real numbers, layer by layer.

(a) Late fusion(b) Unified sequence(c) Native early fusion(d) Cross-attention CLIP, SigLIPLLaVA, Qwen2-VLFuyu, ChameleonFlamingo, Llama 3.2 imageencoder textencoder dot product pixelswords meet once, at the end encoder +projector text tokens LLM: L layers ofself-attention overimage + text tokens meet in every layer raw patches orimage codes linear / lookup one transformer,trained on bothfrom the start no separate encoder encoder +resampler frozen LM layer gated x-attn frozen LM layer frozen LM layer gated x-attn image as keys and values
image informationtext tokensnew cross-attention layers
Figure 1. Four fusion designs. (a) Two separate towers meet once, at the end. (b) Image tokens are put in front of the text, and the LLM reads one long sequence. (c) The same as (b), but without a separate pretrained image encoder. (d) The text sequence holds only text. New gated cross-attention layers, placed between frozen LM layers, read the image.

Early and late fusion: two meanings of the same words

Be careful: the words "early" and "late" are used in two different ways. Interviewers sometimes mean one, sometimes the other, so say which one you mean.

Meaning 1: the classic one (where features are combined)

In classic multimodal learning, the question is at which depth the two signals are combined.

  • Early fusion combines the raw inputs or low-level features, then one model processes the mix. Example: in video classification, stack 10 frames along the channel axis and feed them to the first convolution, so the very first filters see motion. Karpathy et al. (2014) called this "early fusion" and compared it with "late fusion" (two single-frame networks, 15 frames apart, merged only at the first fully connected layer) and "slow fusion" (in between). [Certain]
  • Late fusion processes each input alone and combines only high-level outputs: embeddings or predictions. The two-stream video network of Simonyan and Zisserman (2014) is late fusion: an RGB network and an optical-flow network each predict class scores, and the scores are averaged. [Certain]

Two common forms of late fusion are combining embeddings with a dot product (CLIP) and combining predictions with a weighted average:

\[ s(I, T) = \frac{f(I)^\top g(T)}{\lVert f(I) \rVert \, \lVert g(T) \rVert} \tag{179.1} \]
\[ p(y \mid I, T) = \lambda\, p_{\text{img}}(y \mid I) + (1 - \lambda)\, p_{\text{txt}}(y \mid T) \tag{179.2} \]
Read the symbols
\(I, T\)
the image and the text
\(f, g\)
the image encoder and the text encoder; each outputs one vector
\(s(I, T)\)
cosine similarity between the two vectors: high means "they match"
\(p_{\text{img}}, p_{\text{txt}}\)
class probabilities predicted from each modality alone
\(\lambda\)
a mixing weight between 0 and 1, chosen on validation data

Late fusion is cheap and simple. Its weakness is that the two modalities never interact before the end. The image encoder cannot know which part of the image the text is asking about. In (179.1), the whole image is squeezed into one vector before the question is even seen. That is why CLIP is excellent at retrieval but cannot answer "what is written on the second sign from the left?" (session 166).

Meaning 2: the modern VLM one (separate encoder or not)

In recent VLM papers, the words mean something else. "Late fusion" means: take a vision encoder that was pretrained separately (for example a CLIP or SigLIP ViT), connect it to a pretrained LLM, and continue training. "Early fusion" means: no separate encoder; image patches or image codes enter one transformer that is trained on all modalities from the start. In this language, LLaVA counts as "late fusion", even though image and text tokens mix in every LLM layer. Shukor et al. (2025) use the words this way and report that, when both are trained from scratch with the same compute, late fusion has no built-in advantage; early fusion did better at small parameter counts and was more efficient to train. [Certain: that is their claim; it is one study.]

The unified sequence

This is the design of LLaVA and of most open VLMs today. The connector from session 178 maps each image feature to a vector of the LLM's width \(d\). These vectors are placed in the sequence, usually where a special image placeholder token was in the prompt, and the LLM runs exactly as it would on text:

\[ H^{(0)} = \big[\, P(\mathbf{v}_1), \dots, P(\mathbf{v}_{N_v}),\ E(t_1), \dots, E(t_{N_t}) \,\big] \in \R^{(N_v + N_t) \times d} \tag{179.3} \]
Read the symbols
\(\mathbf{v}_j\)
the \(j\)-th image feature from the vision encoder (one per patch, or one per resampler query)
\(P\)
the connector (projector): a linear layer or a small MLP
\(E(t_i)\)
the embedding of the \(i\)-th text token
\(N_v, N_t\)
number of visual tokens and of text tokens
\(H^{(0)}\)
the input to the first LLM layer: one matrix, one row per token

Then every layer applies causal self-attention over all \(N_v + N_t\) rows. A text token can attend to every image token that comes before it and to earlier text, all in one softmax. Nothing in the LLM knows which rows "are image". Only the position in the sequence and the content of the vectors differ.

Training uses the normal next-token loss, but only on text positions. There is no target "next image token" to predict, so image positions are masked out of the loss:

\[ \mathcal{L} = -\sum_{i \in \text{answer text}} \log p_\theta\big(t_i \mid \mathbf{v}_{1:N_v},\ t_{\lt i}\big) \tag{179.4} \]

(In instruction tuning, the prompt text is usually masked out too, so only the answer tokens count. session 188 covers this.)

How many tokens is one image?

A ViT with patch size 14 on a \(336 \times 336\) image makes \((336/14)^2 = 24^2 = 576\) patches. LLaVA-1.5 keeps all of them, so one image costs 576 tokens. [Certain] The cost grows with the square of the resolution, and high-resolution methods tile the image into many crops (session 180). A video with 64 frames at 576 tokens each would need \(64 \times 576 = 36{,}864\) tokens. This is the main weakness of the unified sequence: every visual token sits in the context and passes through every layer.

Its strength is simplicity and power. The LLM's own attention, in every layer, can compare any word with any image patch. No new layer types are needed, the serving stack is the normal LLM stack, and models of this type lead most OCR and document benchmarks. NVLM, which built both designs on the same data, reports that its decoder-only (unified) model was more accurate on OCR-related tasks, while its cross-attention model trained faster on high-resolution images. [Certain: their reported finding]

Native early fusion: no separate image encoder

Two well-known models skip the pretrained vision encoder entirely.

  • Fuyu-8B (Adept, 2023) cuts the image into \(30 \times 30\) pixel patches, flattens each patch and maps it with one linear layer to the decoder width. The patches go directly into the decoder sequence, with a special "image newline" token at the end of each row of patches, so any image size works. [Certain: patch size 30 is in the released config]
  • Chameleon (Meta, 2024) turns a \(512 \times 512\) image into 1,024 discrete tokens from a codebook of 8,192 entries, using a learned image tokenizer. Images and text are then the same kind of thing: integer IDs in one vocabulary. The model can read and write images token by token. [Certain]

The price is training difficulty. The Chameleon team reports that training was unstable above 8B parameters and 1T tokens, with loss divergences late in training. They needed QK-norm (normalizing queries and keys before the softmax) and z-loss to fix it. [Certain] Their explanation: the image and text tokens have very different statistics but share every weight, so norms keep growing until they leave the BF16 range. Their tokenizer also struggles to reconstruct images with a lot of text, which hurts OCR.

Cross-attention: the image as keys and values

Now the other big design. The LLM's sequence holds only text. Between some of the LLM layers we insert a new cross-attention layer. In self-attention, queries, keys and values all come from the same sequence. In cross-attention, the queries come from the text and the keys and values come from the image (session 16):

\[ \operatorname{CA}(Y, V) = \softmax\!\left( \frac{(Y W_Q)(V W_K)^\top}{\sqrt{d_h}} + M \right) V W_V \, W_O \tag{179.5} \]
Read the symbols
\(Y \in \R^{N_t \times d}\)
the text hidden states at this depth: one row per text token
\(V \in \R^{N_v \times d_v}\)
the visual tokens (the same for every cross-attention layer)
\(W_Q\)
makes queries from text (color query)
\(W_K, W_V\)
make keys and values from the image (colors key, value)
\(W_O\)
output projection back to width \(d\)
\(d_h\)
the head dimension (the formula is written for one head)
\(M\)
a mask: 0 where a text token may look at an image token, \(-\infty\) where it may not

The attention matrix has shape \(N_t \times N_v\): every text token gets a weighted average of image values. There is no causal mask between image tokens, because the image is not generated; it is just read.

Which image may each text token see?

Flamingo handles documents that mix many images and texts ("interleaved" data, like a web page). Its rule: a text token cross-attends only to the visual tokens of the image that came just before it. [Certain] In mask form, for text token \(i\) and visual token \(j\) of image \(m(j)\):

\[ M_{ij} = \begin{cases} 0 & \text{if } m(j) = \text{last image before token } i \\ -\infty & \text{otherwise} \end{cases} \tag{179.6} \]

Earlier images are not lost. The text that followed an earlier image has already absorbed it through cross-attention, and later text can read that text through ordinary causal self-attention. The rule keeps each cross-attention small and lets the model handle any number of images. Text that comes before any image has nothing to look at; Llama 3.2's Hugging Face implementation multiplies the cross-attention output of such rows by zero (full_text_row_masked_out_mask) so they behave like pure text. [Certain]

Figure 2 lets you compare the attention patterns of the two designs on the same short interleaved sequence.

may attend to a text tokenmay attend to an image tokenblocked
Figure 2. The sequence "Look [image 1] a cat [image 2] a dog bigger?", with 3 visual tokens per image (real models use 64 to thousands). Rows are queries, columns are keys. Click a row (or its label) to choose a query token.
How to read Figure 2

In the unified sequence there is one 12 × 12 causal mask: image rows are queries too, and they pass through every layer. Allowed cells: 78. In the cross-attention modes, the text has its own 6 × 6 causal self-attention (21 cells), and a separate 6 × 6 text-to-image cross-attention. With the Flamingo rule, "dog" sees only the 3 tokens of image 2 (15 cross cells in total); with all earlier images it sees 6 (24 cross cells). Note the row "Look": in cross-attention mode it has no image at all.

The tanh gate: start as the frozen LLM

Here is the problem. We take a strong pretrained LLM and insert new, randomly initialized cross-attention layers. At step 0, these layers output noise. That noise is added to the residual stream, and the LLM, which never saw such inputs, produces garbage. Training must first repair the damage, and the large early gradients flow into a model we wanted to keep. If we freeze the LLM, it cannot adapt, so the noise is even more harmful.

Flamingo's fix: multiply the output of each new layer by \(\tanh(\alpha)\), where \(\alpha\) is one learnable scalar per layer, initialized to 0. [Certain] Each inserted block (Flamingo calls it GATED XATTN-DENSE) has two gated parts, a cross-attention and a feed-forward network, each with its own gate:

\[ \begin{aligned} Y &\leftarrow Y + \tanh(\alpha_{\text{x}}) \cdot \operatorname{CA}\big(\operatorname{LN}(Y), V\big) \\ Y &\leftarrow Y + \tanh(\alpha_{\text{f}}) \cdot \operatorname{FFW}\big(\operatorname{LN}(Y)\big) \\ Y &\leftarrow \text{FrozenLMLayer}(Y) \end{aligned} \tag{179.7} \]
Read the symbols
\(\alpha_{\text{x}}, \alpha_{\text{f}}\)
the two gate scalars of this block; both start at 0
\(\tanh\)
squashes the gate into \((-1, 1)\), so a gate can never scale the new branch by more than 1
\(\operatorname{LN}\)
layer normalization before each branch (pre-norm)
\(\operatorname{FFW}\)
a new feed-forward network, trained together with the cross-attention

At step 0, \(\tanh(0) = 0\), so both new branches add exactly zero. The whole model is then the original LLM, bit for bit. Text-only behavior is unchanged, and on an image prompt the model ignores the image. During training the gates open slowly, as far as the data asks for.

If the output is zero, how does anything learn?

Write \(\mathbf{u} = \operatorname{CA}(\cdot)\) for the branch output and \(Y' = Y + \tanh(\alpha)\,\mathbf{u}\). By the chain rule (session 4), with \(G = \partial \mathcal{L}/\partial Y'\):

\[ \frac{\partial \mathcal{L}}{\partial \alpha} = \big(1 - \tanh^2 \alpha\big)\, \langle G, \mathbf{u} \rangle, \qquad \frac{\partial \mathcal{L}}{\partial W} = \tanh(\alpha) \cdot \frac{\partial \langle G, \mathbf{u} \rangle}{\partial W} \tag{179.8} \]

At \(\alpha = 0\), the gate's gradient is at its largest (the factor \(1 - \tanh^2 0 = 1\)), while the gradient of every weight \(W\) inside the branch is multiplied by \(\tanh 0 = 0\). So the first optimizer step changes only the gates. After that, \(\tanh(\alpha) \neq 0\), and the branch weights start to learn too. The same trick appears in LoRA, where one of the two matrices starts at zero (session 57).

Worked example: how much does an open gate change the residual stream?

Take the 8-number hidden state \(\mathbf{x}\) and branch output \(\mathbf{u}\) of Figure 3, with \(\lVert \mathbf{x} \rVert = 3.105\) and \(\lVert \mathbf{u} \rVert = 2.238\), so \(\lVert \mathbf{u} \rVert / \lVert \mathbf{x} \rVert = 0.721\). The relative change of the residual stream is \(|\tanh \alpha| \times 0.721\):

  • \(\alpha = 0\): \(\tanh = 0\), change 0. The LLM is untouched.
  • \(\alpha = 0.1\): \(\tanh = 0.100\), change 0.072 (7%).
  • \(\alpha = 0.5\): \(\tanh = 0.462\), change 0.333.
  • \(\alpha = 1\): \(\tanh = 0.762\), change 0.549, and the gate's own gradient factor has dropped to \(1 - 0.762^2 = 0.420\).
  • No gate at all (factor 1): change 0.721 at step 0, from random weights. That is the shock the gate prevents.

The Flamingo ablation shows it matters: removing the tanh gating dropped the overall score of Flamingo-3B from 70.7 to 66.5, and the authors report training instabilities without it. [Certain]

frozen LLM hidden state \(\mathbf{x}\)gated branch \(\tanh(\alpha)\,\mathbf{u}\)new state \(\mathbf{y} = \mathbf{x} + \tanh(\alpha)\,\mathbf{u}\)
Figure 3. Left: the gate value \(\tanh(\alpha)\) and its slope. Right: one hidden state of the frozen LLM (gray), the gated cross-attention output (green) and their sum (dark outline). At \(\alpha = 0\) the sum equals the frozen state exactly.
How to read Figure 3

Move \(\alpha\) from 0 upward. The green bars grow and the outline moves away from the gray bars: the image starts to change the LLM's computation. The readout also shows the two gradient factors of formula (179.8): at \(\alpha = 0\) the gate gets the full gradient and the branch weights get none. Near \(\alpha = 3\) the gate is almost fully open (0.995) and nearly stops learning (slope 0.010). Tick "Remove the gate" to see the change at step 0 without gating.

Formula (179.8) is easier to believe when you watch it run. Figure 4 trains one gated branch with plain gradient descent on a tiny problem. The frozen LLM gives the hidden state \(\mathbf{x}\) (the same 8 numbers as in Figure 3). The image gives a feature \(\mathbf{z}\) with 4 numbers, and the branch output is \(\mathbf{u} = W\mathbf{z}\), a stand-in for the cross-attention output. The target \(\mathbf{y}^\ast\) is the hidden state that would use the image well. The loss is \(\tfrac12 \lVert \mathbf{y} - \mathbf{y}^\ast \rVert^2\), so \(G = \mathbf{y} - \mathbf{y}^\ast\). \(W\) starts random, \(\alpha\) starts at 0, and every number is computed in your browser at each step.

BranchLearning rate \(\eta\)Step by
frozen state \(\mathbf{x}\)\(\tanh(\alpha)\,\mathbf{u}\)\(\mathbf{y} = \mathbf{x} + \tanh(\alpha)\,\mathbf{u}\)target \(\mathbf{y}^\ast\)loss of the frozen LLM
  1. \(\mathbf{u} \leftarrow W \mathbf{z}\) // branch output
  2. \(\mathbf{y} \leftarrow \mathbf{x} + \tanh(\alpha)\, \mathbf{u}\) // (179.7)
  3. \(G \leftarrow \mathbf{y} - \mathbf{y}^\ast,\ \ \mathcal{L} \leftarrow \tfrac12 \lVert G \rVert^2\)
  4. \(g_\alpha \leftarrow (1 - \tanh^2 \alpha)\, \langle G, \mathbf{u} \rangle\) // (179.8), left
  5. \(g_W \leftarrow \tanh(\alpha)\, G\, \mathbf{z}^\top\) // (179.8), right
  6. \(\alpha \leftarrow \alpha - \eta\, g_\alpha,\ \ W \leftarrow W - \eta\, g_W\)
Figure 4. One gated branch trained by gradient descent for 40 steps, with (179.7) and (179.8) evaluated live. The left panel shows where \(\alpha\) is on the tanh curve (dots: earlier steps). The right panel shows the loss; the dashed line is the loss of the frozen LLM alone (\(\mathbf{y} = \mathbf{x}\)), and the faint curve is the other choice of branch, for comparison. Line 5 uses \(\lVert G \mathbf{z}^\top \rVert = \lVert G \rVert \lVert \mathbf{z} \rVert\). \(W\) and \(\mathbf{z}\) come from a seeded generator, so the run is always the same.
How to read Figure 4

Press Step and read the live lines. At \(t = 0\) the gate is closed: \(\mathbf{y} = \mathbf{x}\) and the loss is 2.315, the frozen LLM's loss. Line 4 gives \(g_\alpha = (1 - 0^2) \times (-1.079) = -1.079\), but line 5 gives \(\lVert g_W \rVert = 0\) exactly, because of the factor \(\tanh 0\). So the first update (\(\eta = 0.1\)) moves only the gate: \(\alpha = 0.108\), and \(W\) is unchanged. At \(t = 1\), \(W\) gets its first gradient (norm 0.349). After 40 steps \(\alpha = 0.822\), \(\tanh\alpha = 0.676\) and the loss is 0.009; with the gate, the loss never rises above the frozen line, for all three learning rates. Now choose "no gate": the random branch moves the state by 2.773 at once (\(\lVert \mathbf{x} \rVert = 3.105\)), and the loss starts at 5.081, more than twice the frozen loss. In this one-layer linear toy the ungated branch still recovers fast. In a real VLM the damage passes through every frozen layer above, and the large early gradients are what the gate avoids.

Keeping the LLM frozen

With the gate, it becomes safe to freeze the LLM and train only the new layers (and the connector). Flamingo did this, and its ablation is clear: on Flamingo-3B, fine-tuning the LLM instead of freezing it dropped the overall score from 70.7 to 62.7, which the authors call catastrophic forgetting; training the LLM from random initialization gave 57.8. [Certain] Llama 3 made the same choice for its vision models: "The language model weights are kept frozen to maintain text-only performance." [Certain] A product team gets a strong promise from this: the text model you already evaluated, red-teamed and shipped is exactly the same network on text-only prompts.

Unified-sequence models usually unfreeze the LLM during later training stages and protect text skills differently, by mixing text-only data into the training set. NVLM did this and reports that its models' text-only scores even went up slightly. [Certain: their reported finding] So freezing is not the only way to stay safe; it is the cheapest and most certain one.

Real systems

ModelFrozen LLMWhere cross-attention goesVisual tokens
Flamingo-3BChinchilla 1.4B (24 layers)before every LM layer; 1.2B new parameters64 per image (Perceiver resampler)
Flamingo-9BChinchilla 7B (40 layers)every 4th layer; 1.6B new parameters64 per image
Flamingo-80BChinchilla 70B (80 layers)every 7th layer; 10B new parameters64 per image
Llama 3.2 11B Vision32 text layers of width 40968 new layers at positions 3, 8, …, 38 of 40 (one after every 4 text layers), tanh-gated attention and MLP1,601 per 560 × 560 tile, up to 4 tiles
Llama 3 405B (paper)405Bafter every 4th self-attention layer, with GQA; about 100B new parametersfrom a ViT-H/14 encoder with 850M parameters

Flamingo numbers are from its parameter table; Llama 3.2 numbers are from its released config.json (cross_attention_layers = [3, 8, 13, 18, 23, 28, 33, 38], image_size 560, patch_size 14, so \(40^2 + 1 = 1{,}601\) tokens per tile with the class token). [Certain] Flamingo also measured the insertion frequency: on the 3B model, cross-attention every 4th layer instead of every layer lowered the score from 70.7 to 68.8 but cut the training step time from 1.74 s to 1.02 s. [Certain]

The trade-off: tokens in the context vs extra layers

Now we can count. Take a base LLM with \(L\) layers, KV heads \(h_{kv}\), head size \(d_h\), and \(b\) bytes per number. The KV cache stores one key and one value vector per token per layer (session 80). For one image of \(N_v\) tokens and a prompt of \(N_t\) text tokens:

\[ \begin{aligned} M_{\text{unified}} &= L \,(N_v + N_t) \cdot 2\, h_{kv} d_h\, b \\ M_{\text{cross}} &= \Big( L\, N_t + \tfrac{L}{k}\, N_v \Big) \cdot 2\, h_{kv} d_h\, b \end{aligned} \tag{179.9} \]

Here \(k\) is the spacing of cross-attention layers (one every \(k\) LLM layers, so \(L/k\) of them). In the cross design, the image's keys and values are computed once per cross-attention layer and reused for every generated token; text tokens never enter those layers' caches. For compute in the prefill (the first pass over the prompt), count about 2 FLOPs per parameter per token for the linear layers (session 1):

\[ \begin{aligned} C_{\text{unified}} &\approx 2\, P_{\text{LM}}\, (N_v + N_t) \\ C_{\text{cross}} &\approx 2\, P_{\text{LM}}\, N_t + 2\, P_{\text{X}}\, N_t + 2 \cdot \tfrac{L}{k}\, P_{kv}\, N_v \end{aligned} \tag{179.10} \]
Read the symbols
\(P_{\text{LM}}\)
the LLM's linear-layer parameters (all layers)
\(P_{\text{X}}\)
parameters of all cross-attention layers that act on text: \(W_Q\), \(W_O\) and the new FFN
\(P_{kv}\)
parameters of \(W_K\) and \(W_V\) in one cross-attention layer: the only part that touches image tokens

Both formulas ignore the attention-score part (the \(QK^\top\) and the weighted sum), which grows with the square of the sequence and makes the unified design look even worse for long image sequences. Both designs also pay the same vision encoder cost.

Worked example: Llama 3.2 11B Vision with a 4-tile image

The text part has \(L = 32\) layers, \(d = 4096\), 8 KV heads of size 128, FFN width 14,336 (the Llama 3.1 8B shape). One LLM layer has \(2 \cdot 4096^2 + 2 \cdot 4096 \cdot 1024 + 3 \cdot 4096 \cdot 14336 = 218.1\)M linear parameters, so \(P_{\text{LM}} = 6.98\)B. A 4-tile image gives \(N_v = 4 \times 1601 = 6{,}404\) tokens. Take a prompt of \(N_t = 512\) text tokens, BF16 (\(b = 2\)), so one token in one layer costs \(2 \cdot 8 \cdot 128 \cdot 2 = 4{,}096\) bytes of cache.

  • KV cache. Unified: \(32 \times 6916 \times 4096 = 906.5\) MB. Cross (\(k = 4\), 8 layers): \(32 \times 512 \times 4096 + 8 \times 6404 \times 4096 = 277.0\) MB. The image part alone is 839.4 MB vs 209.8 MB: 4 times less, because only 8 layers hold it instead of 32.
  • Prefill compute. Unified: \(2 \times 6.98\text{B} \times 6916 = 96.5\) TFLOP. Cross: 9.72 TFLOP, about 10 times less. The image part alone is 89.4 TFLOP in the unified design and 0.86 TFLOP in the cross design, where image tokens only go through \(W_K\) and \(W_V\).
  • Sequence length. Unified: 6,916 positions in the context window. Cross: 512.
  • The price. Each cross-attention layer has the same linear shapes as an LLM layer: 218.1M parameters, so 8 of them add 1.745B parameters that every text token must pass through and that take memory.

These are the reasons Meta gives for cross-attention: it avoids passing full-resolution images through the LLM, and it keeps text-only performance unchanged. [Certain: their stated reasons; our numbers are linear-layer estimates]

One cross-attention layer everyLLM layers
unified sequencecross-attention
Figure 5. Cost of one prompt on a Llama 3.1 8B-shaped LLM (32 layers, \(d = 4096\), 8 KV heads of 128, FFN 14,336, BF16 cache), from formulas (179.9) and (179.10). Each pair of bars is scaled to the larger of the two.
How to read Figure 5

The defaults are the worked example: 906.5 MB vs 277.0 MB of KV cache and 96.5 vs 9.72 TFLOP. Drag \(N_v\) down to 64 (a Flamingo-style resampler): the unified design becomes cheap, and the cross design's 1.745B extra parameters become the main cost. Choose "every 1": 32 cross layers, 6.98B new parameters, almost a second LLM. Raise the number of images: the unified sequence quickly fills the context window.

The table summarizes the trade-off.

Unified sequenceCross-attention
Image tokens in the contextyes: \(N_v\) per image, in every layer and in the KV cacheno: only in \(L/k\) cross layers' caches
New parametersonly the connector (millions)new layers (billions: 1.745B for 8 layers at 8B scale)
Interaction depthimage and text mix in every layeronly at the cross layers
Text-only safetyLLM usually unfrozen; needs text data in the mixLLM can stay frozen; tanh gate guarantees the start
Servingstandard LLM engine; prefix caching of the image worksneeds cross-attention support and a second cache
Strengthfine detail, OCR, simple recipemany or large images, long videos, cheap prefill

A third option combines both. NVLM-H puts a low-resolution thumbnail of the image into the sequence (so self-attention sees the global picture) and sends the high-resolution tiles through gated cross-attention. [Certain] Most open models in 2025 still use the unified sequence and fight its cost with fewer tokens per image instead: pixel shuffle, resamplers and token pruning (session 183). [Likely]

Putting it together: the two forward passes

Algorithm 1: unified-sequence forward pass (LLaVA style)
  1. input: images, prompt with one placeholder token per image, LLM with \(L\) layers, connector \(P\)
  2. \(\mathbf{v} \leftarrow\) VisionEncoder(image) // e.g. 576 features for 336 px
  3. \(Z \leftarrow P(\mathbf{v})\) // width d, formula (179.3)
  4. \(H \leftarrow\) text embeddings, with each placeholder replaced by its block of rows \(Z\)
  5. for \(\ell = 1 \dots L\): \(H \leftarrow\) Layer\(_\ell(H)\) with a causal mask over all \(N_v + N_t\) positions
  6. loss: next-token cross-entropy on answer positions only // formula (179.4)
  7. at inference: cache K, V of all positions, including image positions, in all \(L\) layers
Algorithm 2: gated cross-attention forward pass (Flamingo / Llama 3.2 style)
  1. input: images, text, frozen LLM layers \(1 \dots L\), new gated blocks every \(k\) layers, gates \(\alpha\) initialized to 0
  2. \(V \leftarrow\) Resampler(VisionEncoder(images)) // e.g. 64 tokens per image
  3. \(M \leftarrow\) cross-attention mask: each text token sees its latest image // formula (179.6)
  4. \(Y \leftarrow\) text embeddings only
  5. for \(\ell = 1 \dots L\):
  6.   if a gated block sits here: \(Y \leftarrow Y + \tanh(\alpha_x)\,\mathrm{CA}(\mathrm{LN}(Y), V, M)\); \(Y \leftarrow Y + \tanh(\alpha_f)\,\mathrm{FFW}(\mathrm{LN}(Y))\) // (179.7)
  7.   zero the cross output on rows with no image yet
  8.   \(Y \leftarrow\) FrozenLayer\(_\ell(Y)\) with a causal mask over the \(N_t\) text positions
  9. loss on text; gradients only reach the gates, the new blocks and the resampler
  10. at inference: compute K, V of the image once per cross layer and reuse them for every new token

Figure 6 runs all four designs on the same tiny frozen LLM, one layer per step, so you can see where the image enters and what it costs. The prompt is "Look [image] what color ?". We follow the last text token "?", the one that will predict the answer. To measure how much the image has reached it, compare its hidden state with the hidden state of the same LLM run on the text alone:

\[ \Delta_\ell = \big\lVert\, \mathbf{h}_?^{(\ell)}(\text{text}, \text{image}) - \mathbf{h}_?^{(\ell)}(\text{text only}) \,\big\rVert \tag{179.11} \]
Read the symbols
\(\mathbf{h}_?^{(\ell)}\)
the hidden state of the token "?" after layer \(\ell\)
\(\Delta_\ell\)
0 means the image has not changed this token at all at depth \(\ell\); larger means more image information has flowed into it
Image tokens \(N_v\)Cross block every \(k\) layers
(a) Late fusion
(b) Early fusion
(c) Unified sequence
(d) Cross-attention
image information (layer fill and bar: size of \(\Delta_\ell\))text tokensgated cross block, labeled with \(\tanh\alpha\)
Code and live numbers for
  1. \(Y \leftarrow [E(\text{Look}), E(\text{what}), E(\text{color}), E(?)]\); \(V \leftarrow P(\operatorname{Enc}(\text{img}))\)
  2. for \(\ell = 1 \dots L\):
  3. if \((\ell - 1) \bmod k = 0\): \(Y \leftarrow Y + \tanh(\alpha)\, \operatorname{CA}(\operatorname{LN}(Y), V)\) // (179.5), (179.7)
  4. \(Y \leftarrow \text{FrozenLayer}_\ell(Y)\) // text rows only
  5. \(\Delta_\ell \leftarrow \lVert \mathbf{y}_? - \bar{\mathbf{h}}_? \rVert\) // (179.11)
  6. output \(\mathbf{y}_?^{(L)}\) to the LM head
Figure 6. The four fusion designs run side by side on one tiny model: width \(d = 4\), one attention head, 6 frozen layers with an MLP of size 8 (tanh), LayerNorm without scale and shift, no position embeddings. All four columns share the same frozen layer weights; the vision encoder is one bidirectional layer plus a projector, and the cross blocks have their own weights. Weights come from a seeded generator. The numbers to the right of each layer are \(\Delta_\ell\) from (179.11); under each column are the sequence length in the LLM and running totals of linear-layer FLOPs (2 per weight per row), attention scores and cached K/V rows.
How to read Figure 6

Press Play. In layer 1, unified already has \(\Delta_1 = 1.518\): "?" puts 0.406 of its attention on the 4 image tokens. Early fusion has 0.462 (the raw patches are not encoded, so they carry different vectors). Cross has 0.236, from the gated block before layer 1. Late stays at 0 in every layer and meets the image once, at the top: \(\cos(\mathbf{f}, \mathbf{g}) = 0.104\). After layer 6: early 2.406, unified 3.418, cross 0.705. Now set \(\alpha = 0\): the cross column becomes exactly 0 in every layer, the frozen LLM of formula (179.7). Compare the costs at the end (\(N_v = 4\), \(k = 2\)): linear FLOPs 7,296 (late), 12,416 (early, no encoder), 13,440 (unified), 8,832 (cross); cached K/V rows 24, 48, 48, 36. Set \(N_v = 8\): unified grows to 20,736 FLOPs and 72 K/V rows, cross only to 9,984 FLOPs at \(k = 3\). Choose a design under the columns to follow its code line by line.

A runnable PyTorch version of both designs on a tiny 8-layer model. It checks the three facts of this session: at initialization the gated model equals the frozen LLM exactly; the first gradients reach only the gates; and the unified sequence changes the text outputs immediately and makes the sequence longer.

import torch, torch.nn as nn
torch.manual_seed(0)
d, n_heads = 64, 4

class Block(nn.Module):                      # one decoder layer of the (frozen) LLM
    def __init__(s):
        super().__init__()
        s.n1, s.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
        s.attn = nn.MultiheadAttention(d, n_heads, batch_first=True)
        s.ffw = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
    def forward(s, x, mask):
        h = s.n1(x)
        x = x + s.attn(h, h, h, attn_mask=mask, need_weights=False)[0]
        return x + s.ffw(s.n2(x))

class GatedXAttn(nn.Module):                 # new layer, formula (179.7)
    def __init__(s):
        super().__init__()
        s.n1, s.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
        s.xattn = nn.MultiheadAttention(d, n_heads, batch_first=True)
        s.ffw = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
        s.a_x = nn.Parameter(torch.zeros(1)); s.a_f = nn.Parameter(torch.zeros(1))   # gates start at 0
    def forward(s, y, v, xmask=None):
        y = y + torch.tanh(s.a_x) * s.xattn(s.n1(y), v, v, attn_mask=xmask, need_weights=False)[0]
        return y + torch.tanh(s.a_f) * s.ffw(s.n2(y))

def causal(n):                               # True = not allowed to attend
    return torch.triu(torch.ones(n, n, dtype=torch.bool), 1)

L, k = 8, 4
lm = nn.ModuleList([Block() for _ in range(L)])
for p in lm.parameters(): p.requires_grad_(False)          # the LLM stays frozen
xa = nn.ModuleList([GatedXAttn() for _ in range(L // k)])  # one new layer every k layers

txt = torch.randn(1, 10, d)       # 10 text-token embeddings
img = torch.randn(1, 64, d)       # 64 visual tokens (for example from a resampler)

def lm_only(x):
    for b in lm: x = b(x, causal(x.shape[1]))
    return x

def cross_model(x, v):            # xattn block before every k-th LM layer
    for i, b in enumerate(lm):
        if i % k == 0: x = xa[i // k](x, v)
        x = b(x, causal(x.shape[1]))
    return x

def unified(x, v):                # formula (179.3): one sequence [image ; text]
    return lm_only(torch.cat([v, x], dim=1))[:, v.shape[1]:]

# 1) at initialization the cross-attention model IS the frozen LLM
print((cross_model(txt, img) - lm_only(txt)).abs().max().item())   # 0.0

# 2) gradients at init: the gates get a signal, the new attention weights get none yet (179.8)
loss = cross_model(txt, img).pow(2).mean(); loss.backward()
print(xa[0].a_x.grad.abs().item() > 0, xa[0].xattn.in_proj_weight.grad.abs().max().item())  # True 0.0

# 3) the unified sequence changes the text outputs at once, and makes the sequence longer
print((unified(txt, img) - lm_only(txt)).abs().max().item() > 0, txt.shape[1] + img.shape[1])  # True 74

# 4) after one optimizer step the gates are open a little and the image can matter
opt = torch.optim.AdamW([p for p in xa.parameters()], lr=1e-2)
opt.step(); opt.zero_grad()
print(round(torch.tanh(xa[0].a_x).abs().item(), 4),
      (cross_model(txt, img) - lm_only(txt)).abs().max().item() > 0)   # 0.01 True

The printed lines are 0.0, True 0.0, True 74 and 0.01 True (checked with PyTorch 2.8 on CPU). The last value is 0.01 because Adam's first step moves each parameter by about the learning rate, whatever the gradient size (session 4).

Where this is used

ApplicationTypical fusionWhy
Text-to-image and text-to-video retrieval, near-duplicate and video fingerprintingLate fusion (CLIP or SigLIP embeddings, per frame, pooled)Embeddings are computed once and indexed; one dot product per pair scales to millions (session 169)
Action recognition and video classificationClassic early, slow or late fusion over time; two-stream RGB + flow with score averagingChoose how early motion information is combined; late fusion is cheap and robust
Chat about images, OCR, document and chart QAUnified sequence (LLaVA, Qwen2-VL, InternVL)Fine-grained interaction in every layer; NVLM found decoder-only better on OCR
Long video QA and key-moment detectionCross-attention or heavy token reduction; Llama 3 adds video cross-attention layers and a resampler that merges frames, up to 64 frames64 frames × 576 tokens = 36,864 tokens would fill the context in a unified design
Adding vision to a shipped text LLMGated cross-attention with a frozen LLM (Llama 3.2 Vision)Text-only behavior is exactly the old model; text and vision teams can work in parallel
Image generation and mixed image-text outputNative early fusion with discrete image tokens (Chameleon)The same next-token head writes text and image tokens
Video-RAGLate fusion to retrieve clips, then a unified-sequence VLM on the retrieved framesCheap search over the whole archive, expensive reasoning only on a few clips

Common mistakes

  • Mixing up the two meanings of "early" and "late". LLaVA is "late fusion" in the modern sense and "deep fusion" in the classic sense. Define your terms.
  • Initializing the gate to a nonzero value (or forgetting the gate). The new random layers then disturb the frozen LLM at step 0, and you lose the guarantee. Flamingo measured a drop from 70.7 to 66.5 without the gate.
  • Initializing the branch weights to zero as well as the gate. Then both \(\partial\mathcal{L}/\partial\alpha\) (needs \(\mathbf{u} \neq 0\)) and \(\partial\mathcal{L}/\partial W\) (needs \(\tanh\alpha \neq 0\)) are zero, and nothing ever learns. Exactly one of the two must start at zero.
  • Putting the loss on image positions in a unified sequence. There is no correct "next image token"; mask those positions out (formula 179.4).
  • Letting text before the first image cross-attend to nothing. A softmax over an all-masked row gives NaN or a uniform average. Zero these rows, as Llama 3.2 does.
  • Forgetting the image KV cache in the cross design. Recomputing \(VW_K\), \(VW_V\) for every generated token wastes compute; compute them once per cross layer.
  • Counting only parameters, not tokens. A unified model with 0 new layers can still be far more expensive per request: 89 TFLOP of prefill for one 4-tile image in the worked example.
  • Training native early fusion without stability tricks. Chameleon needed QK-norm and z-loss above 8B parameters.

Interview questions

What is the difference between early and late fusion? Give examples.basic
Classic meaning: early fusion combines raw inputs or low-level features before one shared model (stacking video frames as input channels, or image and text tokens mixed from the first layer); late fusion processes each modality separately and combines only final embeddings or predictions (CLIP's dot product, two-stream video networks averaging class scores). Modern VLM meaning: "late fusion" = a separately pretrained vision encoder attached to an LLM (LLaVA), "early fusion" = no separate encoder, all modalities trained together from the start (Fuyu, Chameleon). Say which meaning you use.
How does LLaVA give an image to the LLM?basic
A CLIP ViT turns the image into patch features (576 for 336 px with patch 14), an MLP projector maps each to the LLM width, and the resulting vectors replace the image placeholder in the token sequence. The LLM then runs ordinary causal self-attention over image and text tokens together. The loss is next-token prediction on the answer text only.
Why is Flamingo's gate \(\tanh(\alpha)\) with \(\alpha = 0\) at the start?core
With \(\tanh(0) = 0\) the new cross-attention and FFN branches add nothing, so at initialization the model equals the pretrained frozen LLM exactly; random new layers cannot damage it. The gate then opens as far as the data requires. The tanh keeps the factor in \((-1, 1)\). Flamingo's ablation: removing the gate lowered the 3B model's overall score from 70.7 to 66.5 and caused training instabilities.
If the gate is 0, the gradient to the cross-attention weights is 0. How does training start?core
The gradient to the gate itself is \((1 - \tanh^2\alpha)\langle G, \mathbf{u}\rangle\), which is maximal at \(\alpha = 0\) because the branch output \(\mathbf{u}\) is not zero (the branch weights are random). So the first step opens the gates a little; from then on \(\tanh\alpha \neq 0\) and the branch weights receive gradients. If you also zero-initialized the branch output, nothing would ever learn.
Compare the KV cache of a unified-sequence and a cross-attention VLM for one large image.core
Unified: every visual token is cached in every layer, \(L N_v \cdot 2 h_{kv} d_h b\). Cross: the image is cached only in the \(L/k\) cross layers. On a Llama 3.1 8B-shaped model (32 layers, 8 KV heads of 128, BF16), a 4-tile Llama 3.2 image of 6,404 tokens costs 839.4 MB unified vs 209.8 MB with 8 cross layers. With 512 text tokens added: 906.5 MB vs 277.0 MB.
Why did Meta choose cross-attention for Llama 3 vision?core
Two stated reasons: the LLM weights stay frozen, so text-only performance is unchanged (important for a model already shipped and evaluated), and full-resolution images do not have to pass through the LLM, which saves compute. It also lets vision and language development proceed in parallel. The price is new parameters: about 100B for the 405B model, and 1.745B for the 8 cross layers of the 11B model by our count.
In Flamingo a text token sees only the latest image. How can it answer a question about an earlier image?advanced
The text that followed the earlier image cross-attended to it, so its hidden states contain that image's information. Later text reads those hidden states through the LLM's causal self-attention. The rule keeps cross-attention small and lets the model generalize to more images than seen in training.
Which design would you choose for document and OCR tasks, and why?advanced
Usually the unified sequence: fine text needs fine-grained interaction between each word and many patches in every layer, and NVLM, which compared both designs on the same data, found the decoder-only model better on OCR-related tasks. The cost is many tokens per page, so combine it with tiling and token reduction (pixel shuffle). If cost dominates, a hybrid like NVLM-H (thumbnail in the sequence, tiles via cross-attention) is an option.
Why is native early fusion (Chameleon) hard to train?advanced
Image and text tokens have very different statistics but share every weight. The Chameleon team saw norm growth that eventually leaves the BF16 range and late loss divergences above 8B parameters and 1T tokens. They fixed it with QK-norm (normalizing queries and keys before the softmax), z-loss on the softmax normalizer, and for 7B also dropout. Its image tokenizer also loses detail in text-heavy images, which hurts OCR.
You must answer questions about a 64-frame video. Unified sequence or cross-attention?advanced
At 576 tokens per frame, a unified sequence needs 36,864 visual tokens in every layer and in the cache, which is very expensive and may exceed the context. Options: reduce tokens per frame (pooling, resampler, pixel shuffle), or use cross-attention so frames are only keys and values. Llama 3 used cross-attention plus a temporal aggregator (a perceiver resampler) that merges groups of frames, with up to 64 frames in fine-tuning. The unified design keeps an advantage for fine spatial detail, so for short clips with few frames it is still a good choice.
How often should cross-attention layers be inserted?core
It is a compute and quality trade-off. Every cross layer adds about one LLM layer of parameters and compute per text token. Flamingo-3B with a cross layer every 4th layer instead of every layer scored 68.8 instead of 70.7 but ran a training step in 1.02 s instead of 1.74 s; the larger Flamingo models used every 4th (9B) and every 7th (80B), and Llama 3 uses one after every 4 self-attention layers.

Exercises

  1. LLaVA-1.5 uses 336 px images and patch size 14. How many tokens is one image? How much BF16 KV cache do 8 such images need on the Llama 3.1 8B shape (32 layers, 4,096 bytes per token per layer) in a unified sequence, and how much with cross layers every 4th layer?
  2. Flamingo-9B has a 40-layer frozen LM and inserts a gated block every 4th layer. How many gated blocks does it have? How many gate scalars?
  3. In Figure 3, set \(\alpha = 0.5\). Compute \(\tanh\alpha\), the relative change of the hidden state, and the gate's gradient factor. Check your answer in the readout.
  4. In Figure 2, count the allowed attention cells in the unified mode and in the cross-attention mode with the Flamingo rule.
  5. Show from (179.8) what happens if you initialize the gate to 0 and the output projection \(W_O\) of the cross-attention to 0.
  6. The cross design needs 1.745B extra parameters (BF16) but caches each visual token in 8 layers instead of 32. How many visual tokens must be in the KV cache at the same time (over all requests in a batch) before the cross design also uses less total memory? Use (179.9).
  7. In Figure 6 with \(N_v = 4\) and \(k = 2\), count by hand the attention scores that the unified column and the cross column compute over all 6 layers. Then check the totals under the columns at the last step.
Answerscheck
1. \((336/14)^2 = 576\) tokens. Unified: \(32 \times 8 \times 576 \times 4096 = 604.0\) MB. Cross (8 layers): \(8 \times 8 \times 576 \times 4096 = 151.0\) MB, 4 times less. 2. \(40 / 4 = 10\) gated blocks, each with two gates (\(\alpha_x\), \(\alpha_f\)), so 20 scalars. 3. \(\tanh 0.5 = 0.462\); change \(0.462 \times 0.721 = 0.333\); gate factor \(1 - 0.462^2 = 0.786\). 4. Unified: \(12 \cdot 13/2 = 78\). Cross: text self-attention \(6 \cdot 7/2 = 21\), plus cross cells: "Look" 0, "a" and "cat" 3 each, the last three tokens 3 each, so 15; total 36. 5. With \(W_O = 0\) the branch output is \(\mathbf{u} = 0\), so \(\partial\mathcal{L}/\partial\alpha = (1 - \tanh^2\alpha)\langle G, \mathbf{u}\rangle = 0\). With \(\alpha = 0\), every branch weight, including \(W_O\), gets a gradient multiplied by \(\tanh\alpha = 0\). No parameter of the new block ever moves: the model stays the frozen LLM forever. Exactly one of the two must start at zero. 6. Each visual token kept in the cache costs \(32 \times 4096\) bytes in the unified design and \(8 \times 4096\) with cross layers, a saving of \(24 \times 4096 = 98{,}304\) bytes. The 1.745B extra parameters take \(3.49\) GB in BF16. Break-even: \(3.49 \times 10^9 / 98{,}304 \approx 35{,}500\) visual tokens held in the cache at the same time, summed over all requests in the batch (about 5.5 four-tile images, or 62 LLaVA-1.5 images). Above that, the cross design also wins on memory; compute favors it much earlier. 7. Unified: 8 rows in every layer, \(8 \cdot 9 / 2 = 36\) causal scores per layer, so \(6 \times 36 = 216\). Cross: the 4 text rows give \(4 \cdot 5/2 = 10\) self-attention scores per layer, \(6 \times 10 = 60\); the 3 gated blocks (before layers 1, 3 and 5) each score 3 text rows ("Look" has no image yet) against 4 image tokens, \(3 \times 12 = 36\); total 96.

Key takeaways

  • Fusion is where image and text meet: at the end (late, 179.1 and 179.2), in one shared sequence (unified, 179.3), from raw patches with no encoder (native early fusion), or through inserted cross-attention (179.5).
  • "Early" and "late" have a classic meaning (depth of combination) and a modern VLM meaning (separate pretrained encoder or not). LLaVA is deep fusion in the first and late fusion in the second.
  • The unified sequence is simple and strong (best on OCR in NVLM's comparison) but every visual token sits in every layer and in the KV cache: 576 tokens for one LLaVA-1.5 image, 6,404 for a 4-tile Llama 3.2 image.
  • Gated cross-attention (179.7) with \(\tanh(\alpha)\), \(\alpha = 0\) at start, makes the model equal to the frozen LLM at step 0; the gate gets the first gradient, the branch weights follow (179.8). Freezing the LLM protected text skills in Flamingo and Llama 3.
  • The trade-off is tokens vs layers (179.9, 179.10): for Llama 3.2 11B with a 4-tile image, 906.5 vs 277.0 MB of cache and 96.5 vs 9.72 TFLOP of prefill, paid for with 1.745B new parameters.

Key formulas

\[ H^{(0)} = [P(\mathbf{v}_{1:N_v});\, E(t_{1:N_t})], \qquad Y \leftarrow Y + \tanh(\alpha)\,\operatorname{CA}(\operatorname{LN}(Y), V), \ \ \alpha_0 = 0 \]
\[ \operatorname{CA}(Y, V) = \softmax\!\left(\frac{YW_Q (VW_K)^\top}{\sqrt{d_h}} + M\right) VW_V W_O, \qquad \frac{\partial \mathcal{L}}{\partial \alpha} = (1 - \tanh^2\alpha)\langle G, \mathbf{u}\rangle \]
\[ M_{\text{unified}} = L(N_v + N_t)\, 2h_{kv}d_h b, \qquad M_{\text{cross}} = \big(L N_t + \tfrac{L}{k} N_v\big)\, 2h_{kv}d_h b \]

References for this session

  1. Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
  2. Llama Team, Meta (2024). The Llama 3 Herd of Models
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
  4. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
  5. Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
  6. Llama Team, Meta (2024). The Llama 3 Herd of Models
  7. Liu et al. (2023). Visual Instruction Tuning
  8. Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models
  9. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  10. Dai, Lee, Wang, Yang, Liu, Barker, Rintamaki, Shoeybi, Catanzaro & Ping (2024). NVLM: Open Frontier-Class Multimodal LLMs
  11. Bavishi et al. (Adept) (2023). Fuyu-8B: A Multimodal Architecture for AI Agents
  12. Shukor, Fini, Turrisi da Costa, Cord, Susskind & El-Nouby (2025). Scaling Laws for Native Multimodal Models
  13. Karpathy, Toderici, Shetty, Leung, Sukthankar & Fei-Fei (2014). Large-scale Video Classification with Convolutional Neural Networks
  14. Simonyan & Zisserman (2014). Two-Stream Convolutional Networks for Action Recognition in Videos