Self-attention vs cross-attention
Self-attention and cross-attention use the same formula. The only difference is where the keys and values come from: the same sequence, or a different one such as an encoder output, an image or an audio clip. This session follows that one change through shapes, masks, positions, cost and caching, then shows how translation, speech recognition, Flamingo and Llama 3.2 Vision are built on it.
What you will learn
- Self-attention: Q, K and V from one sequence; cross-attention: Q from one sequence, K and V from another
- Why the cross-attention score matrix is T × M but the output always has the query's length
- Masks, positions, cost and the KV cache: what changes when the source changes
- Encoder–decoder attention in the original Transformer and in Whisper, step by step
- Learned queries (Perceiver, Q-Former, DETR) as cross-attention compressors
- Flamingo's gated cross-attention and Llama 3.2 Vision's layout, with real numbers
- Cross-attention fusion versus putting image tokens in the sequence: cost and trade-offs
Why this matters
Attention lets one vector look at a set of other vectors and pull in what it needs. Until now, the vectors that look and the vectors being looked at came from the same sentence. In this session we separate the two roles. When the questions and the answers come from the same sequence, we call it self-attention. When the questions come from one sequence and the answers from a different one, we call it cross-attention.
The formula does not change at all. Only the source of the keys and values changes. But this small change decides how a model connects two kinds of data:
- Translation. The original Transformer reads a German sentence with an encoder, then writes English with a decoder. Every decoder layer uses cross-attention to look at the German sentence. [Certain]
- Speech recognition. Whisper encodes 30 seconds of audio into 1,500 vectors. The text decoder looks at them through cross-attention while it writes the words. [Certain]
- Vision-language models. Flamingo and Llama 3.2 Vision keep a text LLM and add new cross-attention layers so that text tokens can look at image features. Other VLMs (LLaVA, Qwen2-VL) do not use cross-attention: they put image tokens directly into the text sequence and use only self-attention. Choosing between these two designs is a classic VLM interview question.
- Image generation. Stable Diffusion's denoiser uses cross-attention so that image positions can look at the text prompt. [Certain]
In Session 7 you saw Bahdanau attention: an RNN decoder that looks back at all encoder states. That was cross-attention before the word existed. In session 17 and session 19 we built the attention formula and split it into heads. Here we plug two different sequences into that formula and follow the consequences: shapes, masks, positions, cost, caching, and the VLM designs built on it.
One formula, two sources
Recall scaled dot-product attention. Given queries \(Q\), keys \(K\) and values \(V\):
Each query is compared with every key. The softmax turns the scores of one query into weights that sum to 1. The output for that query is the weighted average of the values. The formula does not say where \(Q\), \(K\) and \(V\) come from. That is the only thing we change.
Self-attention
Take one sequence of \(T\) vectors, stacked as rows of \(X \in \R^{T \times d}\). Self-attention builds all three from \(X\):
Every token asks a question, and every token (often including itself) can answer. The score matrix is square.
Cross-attention
Now take two sequences: \(X \in \R^{T \times d}\), the sequence that asks (for example the text being written), and \(Z \in \R^{M \times d_z}\), the sequence that is read (for example encoder outputs, image patches, or audio frames). People often call \(Z\) the memory or the context. Cross-attention takes queries from \(X\) and keys and values from \(Z\):
- Read the symbols
- \(X\)
- the sequence that asks: \(T\) rows, each of width \(d\) (the model width of the decoder or LLM)
- \(Z\)
- the memory: \(M\) rows, each of width \(d_z\). \(d_z\) can differ from \(d\); for example a vision encoder may output 1,280-dimensional features while the LLM uses 4,096
- \(W_Q \in \R^{d \times d_k}\)
- learned matrix that turns a row of \(X\) into a query
- \(W_K, W_V \in \R^{d_z \times d_k}\)
- learned matrices that turn a row of \(Z\) into a key and a value. They must accept width \(d_z\)
- \(T \times M\)
- the score matrix is rectangular: one row per query, one column per memory item
Look at the last line of (20.3). The output has one row per query. Each row \(i\) is
So the output has length \(T\), the length of the asking sequence, and is a mix of values from the memory. The memory can have 4 items or 6,404 items; the output shape stays \(T \times d\). This is why cross-attention fits into a residual stream: we can add its output to \(X\) row by row.
Let \(d_k = 2\). Two text tokens ask with queries \(\mathbf{q}_1 = (1, 0)\) and \(\mathbf{q}_2 = (0, 1)\). Three image regions have keys \(\mathbf{k}_1 = (2, 0)\), \(\mathbf{k}_2 = (0, 2)\), \(\mathbf{k}_3 = (1, 1)\) and values \(\mathbf{v}_1 = (1, 0)\), \(\mathbf{v}_2 = (0, 1)\), \(\mathbf{v}_3 = (0.5, 0.5)\).
Scores for query 1, divided by \(\sqrt{2}\): \((2, 0, 1)/1.414 = (1.414,\ 0,\ 0.707)\). Softmax: \((0.576,\ 0.140,\ 0.284)\). Output: \(0.576\,(1, 0) + 0.140\,(0, 1) + 0.284\,(0.5, 0.5) = (0.718,\ 0.282)\).
The score matrix is \(2 \times 3\) (2 queries, 3 keys), but the output is \(2 \times 2\): one vector per query. Query 2 gives the mirror image, weights \((0.140, 0.576, 0.284)\) and output \((0.282, 0.718)\) (Exercise 2).
Figure 1 puts both kinds side by side on a toy example. The text is “a red car”. In self mode, the three text tokens attend to each other. In cross mode, the same three queries attend to four image regions instead. The vectors are hand-made (already projected), so you can follow every number.
How to read Figure 1
Darker cells mean larger weights; each row sums to 1. In cross mode, the query of “car” puts 0.632 of its weight on the “red car” region, so its output is mostly that region’s value. In self mode, the same query spreads its weight over the three text tokens. Turn on the causal mask: the matrix becomes lower-triangular and the row for “a” can only see itself (weight 1.000). The mask box does nothing in cross mode, because the image is fully known before the text starts; there is no “future” in the memory to hide.
Figure 1 shows the finished weights. Figure 2 computes them in front of you. The text “a red car” is \(X\) (\(T_q = 3\) rows, \(d = 4\)). The image is \(Z\): up to five patches with \(d_z = 3\) features each. The weights \(W_Q\) (\(4 \times 2\)), \(W_K, W_V\) (\(3 \times 2\)) are small fixed matrices, so \(d_k = 2\). Every step runs one line of the algorithm on the real numbers and puts them into formula (20.4). Switch to self-attention to see the same loop with \(K\) and \(V\) taken from the text instead.
- \(Q \leftarrow X W_Q\) // \(T_q \times d_k\), always from the text
- \(S \leftarrow X\) (self) or \(Z\) (cross); \(K \leftarrow S W_K\); \(V \leftarrow S W_V\) // \(T_{kv} \times d_k\)
- for \(i = 1, \dots, T_q\): // one row per query
- for \(j = 1, \dots, T_{kv}\): \(s_{ij} \leftarrow \mathbf{q}_i \cdot \mathbf{k}_j / \sqrt{d_k}\) // \(-\infty\) if masked
- \(\mathbf{a}_i \leftarrow \softmax(\mathbf{s}_i)\) // (20.4), row sums to 1
- \(\mathbf{o}_i \leftarrow \sum_j a_{ij} \mathbf{v}_j\) // (20.4)
- return \(O\) // \(T_q \times d_k\): the query's length
How to read Figure 2
Press Play, or use Step and Back. Steps 1 and 2 build \(Q\), \(K\) and \(V\): in cross mode \(K\) and \(V\) come from \(Z\), which has width 3, so \(W_K\) and \(W_V\) have 3 rows while \(W_Q\) has 4. Then each cell gets its score (small number) and, after the softmax, its weight (large number, darker = more). For the query “car” with \(M = 4\), the patch “red car” gets weight 0.823 and the output is \(\mathbf{o}_3 = (0.896, 0.853)\). Change \(M\): the matrix gets wider (\(3 \times M\)) but the output column always has 3 rows. In self mode the matrix is \(3 \times 3\); turn on the causal mask and the row of “a” keeps weight 1.000 on itself. The last step checks the loop against the matrix formula \(\softmax(QK^\top/\sqrt{d_k})V\).
What changes when the source changes
The formula is the same, but four practical things are different. Each one causes real bugs and comes up in interviews.
1. Masks
In a decoder, self-attention needs a causal mask: token \(t\) may not see tokens after \(t\), or training would let the model copy the answer (session 18). Cross-attention in an encoder-decoder model needs no causal mask. The memory (the source sentence, the audio, the image) is fully known before the decoder starts. Every decoder token may look at all of it. The only mask cross-attention usually needs is a padding mask over the memory, so that queries ignore padding positions when memories of different lengths share one batch. VLMs add one more kind: a mask that decides which image each text token may see (Figure 5).
2. Positions
Attention by itself ignores order. If you shuffle the rows of \(K\) and \(V\) together with the same permutation \(P\), every query gets exactly the same output:
The weights are permuted in the same way as the values, so the weighted sum does not change. In self-attention we fix this by adding position information (learned positions, sinusoids, or RoPE inside the attention). In cross-attention, the position of a memory item must already be inside its vector: the encoder adds positions to its own tokens, a ViT adds patch positions, Whisper’s encoder adds positions to audio frames. The cross-attention layer itself usually adds none. For example, the cross-attention layers of Llama 3.2 Vision in the Hugging Face code apply no RoPE; the image features already carry their positions from the vision encoder. [Certain] T5 uses a relative position bias in self-attention but not in cross-attention. [Likely]
3. Cost
The score matrix has \(T \times T\) entries for self-attention and \(T \times M\) for cross-attention. Per layer, with model width \(d\), the two matrix products \(QK^\top\) and (weights)\(\,V\) cost about
(Each product needs about \(2 \cdot T \cdot M \cdot d\) operations, summed over all heads.) Cross-attention is linear in each length. A short text reading a long memory is cheap; it is the memory’s own self-attention (inside the encoder) that is quadratic. Whisper is a good example: the encoder’s self-attention has \(1500^2 = 2{,}250{,}000\) scores per head per layer, while the decoder’s cross-attention for its longest output (448 tokens) has \(448 \times 1500 = 672{,}000\).
4. Caching during generation
When a decoder generates text one token at a time, its self-attention keys and values grow by one row per step, so the KV cache of self-attention grows with the output (session 80). The cross-attention keys and values depend only on the memory \(Z\), which does not change. So we compute \(K = ZW_K\) and \(V = ZW_V\) once, before the first output token, and reuse them at every step. Each new token only needs its own query. The memory needed for this cache, for \(N\) memory items, is
- Read the symbols
- \(N\)
- number of memory items (encoder tokens, image tokens, audio frames)
- \(L_\times\)
- number of layers that have cross-attention
- \(2\)
- one key and one value per item
- \(n_{kv}\,, d_h\)
- number of key/value heads and the size of each head. With grouped-query attention, \(n_{kv}\) is smaller than the number of query heads (session 26)
- \(b\)
- bytes per number: 2 for FP16 or BF16
Whisper large-v3 has \(d = 1280\), 20 heads, 32 decoder layers, and one cross-attention in each decoder layer. Its config gives 1,500 encoder positions for one 30-second window. [Certain] Without grouping, \(n_{kv} d_h = 1280\). In FP16: \(1500 \times 32 \times 2 \times 1280 \times 2 = 245{,}760{,}000\) bytes, about 245.8 MB per audio window. This is computed once and read at every one of the up to 448 output steps. A batch of 16 windows needs about 3.93 GB just for this cache.
| Self-attention | Cross-attention | |
|---|---|---|
| Q from | \(X\) | \(X\) (the sequence being built) |
| K, V from | \(X\) | \(Z\) (memory: encoder, image, audio) |
| Score shape | \(T \times T\) | \(T \times M\) |
| Output length | \(T\) | \(T\) (never \(M\)) |
| Causal mask in a decoder | yes | no; padding mask on \(Z\), and per-image masks in VLMs |
| Positions | added by the layer (RoPE) or the input | must already be inside \(Z\) |
| Cost per layer | \(4T^2 d\) | \(4TMd\) |
| KV cache in generation | grows by one row per new token | fixed; computed once from \(Z\) |
Encoder–decoder attention
The Transformer of Vaswani et al. (2017) was built for translation. It has an encoder stack that reads the source sentence and a decoder stack that writes the target sentence. The base model has 6 encoder layers and 6 decoder layers with \(d = 512\) and 8 heads. [Certain] Each encoder layer has two sublayers: self-attention without a causal mask (every source word sees every source word) and a feed-forward network. Each decoder layer has three:
(The original paper puts a layer normalization after each residual sum; modern models put it before each sublayer. That detail does not change the attention.) \(Z_{\text{enc}}\) is the output of the last encoder layer, and the same \(Z_{\text{enc}}\) is used by all six decoder layers. Each decoder layer has its own \(W_K, W_V\), so each one reads the memory in its own way. The original paper calls this sublayer encoder–decoder attention.
Why this order? Self-attention first lets each target token gather what has been written so far (“I am about to write an adjective”). Cross-attention then uses that summary as the question to the source (“which source word is the adjective?”). The FFN processes the answer. Stacking six such layers lets the decoder ask more refined questions each time.
Figure 3 shows one decoding run of a toy model translating “the black cat sleeps” into French, “le chat noir dort”. Note the word order: French puts the adjective after the noun. The cross-attention weights in the figure were set by hand to look like the alignment a trained model learns; the attention mechanics are real.
- \(Z \leftarrow \mathrm{Encoder}(\text{source})\) // once, bidirectional
- \(K^\times \leftarrow Z W_K\); \(V^\times \leftarrow Z W_V\) // cross cache, fixed size \(M\)
- for \(t = 1, 2, \dots\): // one new token per step
- self cache \(\leftarrow\) append \((\mathbf{k}_t, \mathbf{v}_t)\); causal self-attention // grows by 1
- \(\mathbf{a}_t \leftarrow \softmax(\mathbf{q}_t K^{\times\top} / \sqrt{d_k})\); \(\mathbf{c}_t \leftarrow \mathbf{a}_t V^\times\) // (20.4), no new keys
- FFN, then pick the next word // (20.8)
How to read Figure 3
Step 1 is the encoder: the four source words get their keys and values once (the orange chips). After that, each step adds one blue chip to the self-attention cache and none to the cross cache. At step 4 the decoder is about to write “noir” and its cross-attention puts 0.803 of the weight on “black”, even though “black” is before “cat” in the source. Cross-attention has no causal mask and no order constraint over the source, so reordering costs nothing. Look at the grey arcs: the decoder’s self-attention is causal, so step \(t\) sees only the \(t\) tokens written so far. Also note what does not change from step to step: the encoder outputs and their keys and values. That is the cached part.
- input: asking sequence \(X\) (\(T \times d\)), optional memory \(Z\) (\(M \times d_z\)), optional mask, weights \(W_Q, W_K, W_V, W_O\)
- \(S \leftarrow X\) if no memory is given, else \(S \leftarrow Z\) // the only difference
- \(Q \leftarrow X W_Q\); \(K \leftarrow S W_K\); \(V \leftarrow S W_V\); split each into \(h\) heads
- scores \(\leftarrow QK^\top / \sqrt{d_k}\) // \(T \times T\) or \(T \times M\) per head
- apply the mask: causal for decoder self-attention; padding (and image selection) for cross-attention
- \(A \leftarrow\) softmax over each row; \(O \leftarrow A V\) // \(T\) rows in both cases
- concatenate heads; return \(O W_O\)
- \(Z \leftarrow \mathrm{Encoder}(\text{source})\) // run once: bidirectional self-attention over the source
- for each decoder layer \(\ell\): \(K^\times_\ell \leftarrow Z W_K^{(\ell)}\), \(V^\times_\ell \leftarrow Z W_V^{(\ell)}\) // cross cache, fixed size
- \(y \leftarrow [\text{start token}]\); self-attention caches empty
- repeat:
- for each layer: causal self-attention of the newest position (append its \(k, v\) to the self cache)
- then cross-attention: its query against \(K^\times_\ell, V^\times_\ell\) // no new keys, no mask except padding
- then FFN; at the top, pick the next token and append it to \(y\)
- until the end token or the length limit
Speech recognition: Whisper
Whisper is an encoder–decoder transformer of the same shape. The input is a log-Mel spectrogram (an image-like table of audio energy over time and frequency) of a 30-second window. Two convolution layers reduce it to 1,500 frames, the encoder turns those into 1,500 vectors, and the decoder writes text while cross-attending to them. Whisper large-v3 has 32 encoder layers and 32 decoder layers with \(d = 1280\) and 20 heads, and its decoder handles up to 448 tokens. [Certain] The same pattern is used by T5 and BART for text-to-text tasks such as summarization (session 24).
Learned queries: cross-attention as a compressor
So far the queries came from real tokens. A powerful trick is to make the queries learned parameters: a fixed set of \(N\) vectors that do not depend on the input. Cross-attention from \(N\) learned queries into \(M\) input features always returns \(N\) outputs, whatever \(M\) is:
This turns a long, variable-length input into a short, fixed-length summary. It is used in many places:
- Perceiver uses a small latent array that cross-attends to a very large input (pixels, audio), so the expensive quadratic self-attention runs only over the latents. [Certain]
- Flamingo’s Perceiver Resampler turns a variable number of image or video features into 64 visual tokens. [Certain] For video, frames are sampled at 1 frame per second and encoded separately, with a learned time embedding added. [Certain]
- BLIP-2’s Q-Former uses 32 learned queries to read features from a frozen image encoder. [Certain]
- DETR uses learned “object queries” in a decoder that cross-attends to image features; each query output becomes one predicted box. [Certain] Video moment detectors such as Moment-DETR use the same idea with “moment queries” over video features. [Likely]
These connectors are the topic of session 178. Here the point is only that they are cross-attention with learned queries.
Cross-attention inside a VLM
A VLM must let text tokens use image information. There are two main ways to wire it (session 179 goes deeper):
- Put image tokens into the sequence (LLaVA, Qwen2-VL, many others). The image encoder output is projected to the LLM width and placed in the token sequence next to the text. The LLM’s own self-attention then handles everything. No new attention layers are added.
- Keep image tokens outside, and read them with cross-attention (Flamingo, Llama 3.2 Vision). The LLM sequence contains only text (plus a marker where an image appears). New cross-attention layers are inserted between the LLM’s layers; their queries come from the text, and their keys and values come from the image features.
These two are more alike than they look. Put the memory \(Z\) in front of the text \(X\) and run one self-attention over the joint sequence \([Z; X]\). The score matrix splits into four blocks:
The bottom-left block \(Q_X K_Z^\top\) is exactly a cross-attention from text to image. The bottom-right block is the text’s own self-attention. So a decoder-only VLM contains cross-attention as one block of its self-attention, with three differences from a dedicated cross-attention layer: the same weights serve both blocks, one softmax runs over image and text keys together, and the image tokens also pass through every LLM layer (they are updated too). The PyTorch code below checks the block claim numerically.
Flamingo: gated cross-attention in a frozen LM
Flamingo (Alayrac et al., 2022) starts from a pretrained, frozen Chinchilla language model (1.4B, 7B or 70B) and inserts new layers called GATED XATTN-DENSE between the frozen layers. [Certain] Each new layer is a cross-attention to the visual tokens, then a feed-forward layer, each with a tanh gate:
- Read the symbols
- \(\mathbf{y}\)
- the language features (the residual stream of the text tokens)
- \(X_{\text{vis}}\)
- the visual tokens from the Perceiver Resampler (64 per image)
- \(\alpha\)
- one learnable number per layer. \(\tanh(0) = 0\), so at initialization the new layers add exactly nothing
Why the gate? The frozen LM is very good at language. A new, randomly initialized layer would add noise to its residual stream and damage it on the first training step. With \(\alpha = 0\), the model starts as exactly the original LM and opens the visual path slowly as training moves \(\alpha\) away from 0 (\(\tanh(0.5) = 0.462\), \(\tanh(2) = 0.964\)). The paper says it uses the gate “to ensure that at initialization, the conditioned model yields the same results as the original language model”. [Certain]
Figure 4 runs one gated cross-attention layer on a toy example. The last text token of “This is a photo of a” has the residual vector \(\mathbf{y} = (0.3, 0.1)\). A frozen “LM head” \(U\) turns \(\mathbf{y}\) into scores for three next words: cat, dog and car. Alone, the text model guesses “cat” with probability 0.548. Four visual tokens (already projected, \(d_k = 2\)) come from a photo. Play the figure to move \(\alpha\) from 0 to 2, as training would do, and watch the visual path open.
- if no image: return \(\mathbf{y}\) // layer skipped
- \(\mathbf{a} \leftarrow \softmax(\mathbf{q} \cdot \mathbf{k}_j / \sqrt{d_k})\); \(\mathbf{c} \leftarrow \sum_j a_j \mathbf{v}_j\) // \(\mathbf{q}\) from text, \(\mathbf{k}, \mathbf{v}\) from image
- \(\mathbf{y}' \leftarrow \mathbf{y} + \tanh(\alpha)\, \mathbf{c}\) // (20.11), \(\alpha = 0\) at the start
- \(\mathbf{p} \leftarrow \softmax(U \mathbf{y}')\) // the frozen LM continues
How to read Figure 4
Each value of \(\alpha\) takes three steps: cross-attention (line 2), the gated add (line 3) and the frozen head (line 4). At \(\alpha = 0\) the green arrow has length zero and the bars equal the grey text-only bars exactly: this is the “same results as the original language model” property. With the dog photo, \(\alpha = 1\) gives \(\tanh(1) = 0.762\) and \(P(\text{dog}) = 0.798\). Note the attention weights: the query comes from the text, which leans to “cat”, so with the dog photo the plain background token gets the largest weight (0.312); the output still points to “dog” because three of the four values do. Choose “no image”: line 1 returns at once and \(\alpha\) has no effect. The toy has no FFN branch; in the real layer it is gated the same way.
How often do you insert these layers? More layers give better quality but more compute and more trainable parameters. Flamingo-3B adds one before every LM block (1.4B new parameters), Flamingo-9B before every fourth block (1.8B), and Flamingo-80B before every seventh block (10B). [Certain] In their ablation, inserting every fourth block made training 66% faster and lowered the overall score by only 1.9%. [Certain]
Flamingo also has a careful mask for interleaved documents (text with several images). A text token cross-attends only to the visual tokens of the last image before it, not to all earlier images. [Certain] It still knows about earlier images indirectly, through self-attention over earlier text tokens that did see them. This scheme let Flamingo train on sequences with up to 5 images and still use up to 32 image–text examples at evaluation. [Certain] Figure 5 compares this mask with the decoder-only design.
How to read Figure 5
In decoder-only mode, all 14 positions are in the LLM sequence and the mask is one big causal triangle: “dog” sees both images directly. In Flamingo mode, the visual tokens are not part of the LLM sequence (their rows are empty). Text rows have two parts: a causal triangle over text only (self-attention) and a block over the visual tokens of the most recent image (cross-attention). “dog” sees 4 visual tokens of image 2 and none of image 1. It can still learn about image 1 through the text tokens “A cat .”, which did see it.
Llama 3.2 Vision
Llama 3.2 Vision (from the Llama 3 paper, Section 7) follows the Flamingo recipe at a larger scale. A ViT-H/14 image encoder produces features; cross-attention layers are “applied after every fourth self-attention layer in the core language model” and use grouped-query attention. [Certain] During adapter training, “we intentionally do not update the language-model parameters”, so the text model stays as it was. [Certain] The public configs show the layout:
- 11B: 40 decoder layers in total, of which the 8 at indices 3, 8, 13, …, 38 are cross-attention layers; the other 32 are the self-attention layers of the Llama 3.1 8B text model. Width 4,096, 32 query heads, 8 key/value heads. [Certain]
- 90B: 100 layers, 20 of them cross-attention (indices 3, 8, …, 98), width 8,192. [Certain]
- Images: tiles of \(560 \times 560\) pixels with \(14 \times 14\) patches give \(40 \times 40 = 1600\) patches plus one class token, so 1,601 vectors per tile, and up to 4 tiles per image. [Certain]
- For the 405B model (not released with vision), the paper reports about 100B parameters in the cross-attention layers. [Certain]
The Hugging Face implementation shows two more details. The cross-attention block has tanh gates initialized at zero, as in Flamingo. And if a text token has no image to look at, its cross-attention output is multiplied by zero, so the layer is a no-op for it; for text-only prompts, the cross-attention layers are skipped entirely. [Certain]
One image with 4 tiles gives \(4 \times 1601 = 6404\) visual tokens. Each token stored in the KV cache of one layer costs \(2 \times 8 \times 128 \times 2 = 4096\) bytes (K and V, 8 KV heads, head size 128, BF16).
Cross-attention design (the real model): the visual tokens are cached only in the 8 cross-attention layers: \(6404 \times 8 \times 4096 = 209.8\) MB.
Same LM, decoder-only design (hypothetical): the visual tokens would sit in the sequence and be cached in all 32 self-attention layers: \(6404 \times 32 \times 4096 = 839.4\) MB, 4 times more. Worse, each of the 6,404 visual tokens would have to run through the whole 8B model in prefill, about \(2 \times 8.03 \times 10^9\) FLOPs per token, or about 103 TFLOPs for one image. In the cross-attention design, each visual token only needs its key and value projections in 8 layers: \(8 \times 2 \times (2 \times 4096 \times 1024) = 134\) MFLOPs per token, about 120 times less. Figure 6 lets you change these numbers.
How to read Figure 6
Three costs are shown. KV cache: formula (20.7) for both designs, text tokens included. Attention per new token: formula (20.6) with one query, over all layers. Image prefill: the work done for the visual tokens before the first output token; this is where cross-attention wins most, because visual tokens skip the LLM’s FFNs and self-attention. With 64 visual tokens (Flamingo) and 1,000 text tokens, the KV cache and decode costs are almost equal: text dominates. With thousands of visual tokens, or several images, the gap grows. The figure ignores the vision encoder (the same in both designs) and the extra weights of the cross-attention layers.
So which design is better?
Neither wins everywhere. Cross-attention designs keep the text model untouched (good for text-only quality and for adding vision to an existing LLM), and they are cheap when there are many visual tokens. Decoder-only designs are simpler: no new layer type, and the LLM can use its full depth to process image tokens together with text, which many recent open VLMs have chosen. [Likely] Most open VLMs released after LLaVA use token concatenation; Llama 3.2 Vision and Flamingo-style models such as IDEFICS (first version) and OpenFlamingo use cross-attention. [Likely] The book (Noyan et al., Chapter 6) describes both families.
Code: one attention module, used both ways
The module below is self-attention when called with one argument and cross-attention when given a memory. The checks print True for each claim of this session: output length, self as a special case, order invariance (20.5), cached cross K/V, the joint-sequence block (20.10), and the zero gate (20.11). It runs on CPU in about a second.
import math, torch import torch.nn as nn torch.manual_seed(0) class Attention(nn.Module): """One module for both kinds. Self: attn(x). Cross: attn(x, memory).""" def __init__(self, d, h, d_mem=None): super().__init__() d_mem = d_mem or d self.h, self.dk = h, d // h self.wq = nn.Linear(d, d, bias=False) # queries always from x self.wk = nn.Linear(d_mem, d, bias=False) # keys from the source self.wv = nn.Linear(d_mem, d, bias=False) # values from the source self.wo = nn.Linear(d, d, bias=False) def split(self, t): # (B, N, d) -> (B, h, N, dk) B, N, _ = t.shape return t.view(B, N, self.h, self.dk).transpose(1, 2) def forward(self, x, memory=None, mask=None, kv=None): src = x if memory is None else memory # the only difference q = self.split(self.wq(x)) if kv is None: # kv = cached (k, v) for cross-attention kv = (self.split(self.wk(src)), self.split(self.wv(src))) k, v = kv s = q @ k.transpose(-2, -1) / math.sqrt(self.dk) # (B, h, T, M) if mask is not None: s = s.masked_fill(~mask, float("-inf")) o = torch.softmax(s, dim=-1) @ v # (B, h, T, dk) o = o.transpose(1, 2).reshape(x.shape[0], x.shape[1], -1) return self.wo(o), kv d, h, T, M = 64, 4, 5, 9 x = torch.randn(1, T, d) # e.g. 5 text tokens (decoder side) mem = torch.randn(1, M, d) # e.g. 9 encoder outputs or image patches att = Attention(d, h) # 1) shapes: the output has one row per QUERY, whatever M is y_self, _ = att(x) y_cross, kv = att(x, mem) print(y_self.shape, y_cross.shape) # both (1, 5, 64) # 2) self-attention is cross-attention with memory = x print(torch.allclose(att(x, x)[0], y_self)) # True # 3) cross-attention ignores the ORDER of the memory, formula (20.5) perm = torch.randperm(M) print(torch.allclose(att(x, mem[:, perm])[0], y_cross, atol=1e-6)) # True # 4) the cross K/V can be computed once and reused at every decoding step y_cached, _ = att(x[:, -1:], mem, kv=kv) # only the newest query print(torch.allclose(y_cached, y_cross[:, -1:], atol=1e-6)) # True # 5) cross-attention = one block of self-attention over [memory; x], formula (20.10) joint = torch.cat([mem, x], dim=1) # (1, M+T, d) allow = torch.zeros(M + T, M + T, dtype=torch.bool) allow[M:, :M] = True # text rows may see memory columns only allow[:M, :M] = True # (memory rows: anything, ignored here) y_joint, _ = att(joint, mask=allow) print(torch.allclose(y_joint[:, M:], y_cross, atol=1e-6)) # True # 6) Flamingo-style tanh gate, formula (20.11): at alpha = 0 the LM is unchanged alpha = nn.Parameter(torch.zeros(1)) y_gated = x + torch.tanh(alpha) * y_cross print(torch.equal(y_gated, x)) # True: identical at initialization # 7) memory with a different width (for example 1,601 vectors of a 1280-d vision encoder) att_v = Attention(d, h, d_mem=1280) print(att_v(x, torch.randn(1, 1601, 1280))[0].shape) # (1, 5, 64)
In real code you would call F.scaled_dot_product_attention (from torch.nn.functional) with arguments q, k, v, attn_mask, is_causal, which runs a fused kernel. Pass is_causal=True only for self-attention; for cross-attention pass a padding mask over the memory, or nothing.
- input: text residual stream \(\mathbf{y}\) (\(T \times d\)), visual tokens \(X_{\text{vis}}\), image-selection mask \(m\) (\(T \times M\))
- if no image is present: return \(\mathbf{y}\) // the layer is skipped
- \(\mathbf{c} \leftarrow \mathrm{CrossAttn}(\mathrm{Norm}(\mathbf{y}),\ X_{\text{vis}},\ \text{mask } m)\) // trainable
- zero the rows of \(\mathbf{c}\) whose mask row is empty // text before any image
- \(\mathbf{y} \leftarrow \mathbf{y} + \tanh(\alpha_{\text{xattn}})\,\mathbf{c}\)
- \(\mathbf{y} \leftarrow \mathbf{y} + \tanh(\alpha_{\text{dense}})\,\mathrm{FFN}(\mathrm{Norm}(\mathbf{y}))\) // trainable
- \(\mathbf{y} \leftarrow\) next frozen LM layer(s) \((\mathbf{y})\) // self-attention + FFN, weights not updated
Where this is used
| System or task | Queries from | Keys and values from | Notes |
|---|---|---|---|
| Machine translation (Transformer, 2017) | target-language decoder | source-sentence encoder | one cross-attention per decoder layer |
| Speech recognition (Whisper) | text decoder | 1,500 audio-frame vectors per 30 s | cross K/V cached once per window |
| Summarization, text-to-text (T5, BART) | decoder | encoder over the document | see session 24 |
| Image and video question answering (Flamingo) | text tokens of the frozen LM | 64 resampled visual tokens per image or video | video frames at 1 FPS; tanh gates |
| Image reasoning, document reading (Llama 3.2 Vision) | text tokens | up to 4 × 1,601 tile features | 8 of 40 layers (11B); LM frozen during adapter training |
| Video understanding (Llama 3 video adapter) | text tokens | frame features merged by a Perceiver-style aggregator (32 frames into 1) | extra video cross-attention layers, as described in the Llama 3 paper [Certain]; not part of the public Llama 3.2 release [Likely] |
| Object detection (DETR), key-moment detection (Moment-DETR) | learned object or moment queries | image or video features | each query output is one box or one time span |
| Video object segmentation (SAM 2) | features of the current frame | memory bank of earlier frames and prompts | “memory attention” uses cross-attention to the stored memories [Likely]; see session 204 |
| Text-to-image generation (Stable Diffusion) | image latent positions | text-encoder tokens of the prompt | how the prompt controls the picture |
| Video retrieval, fingerprinting, Video-RAG | learned queries (Q-Former style) or text | many frame features | compress a clip to a few vectors for indexing (session 208) |
Common mistakes
- Using a causal mask in cross-attention. With a causal mask, decoder position \(t\) would see only the first \(t\) source tokens, an arbitrary and harmful cut. Cross-attention needs a padding mask on the memory, not a causal one. In PyTorch, do not pass
is_causal=Truewhen Q and K have different lengths. - Masking the wrong axis. The padding mask for the memory must hide columns (keys) of the \(T \times M\) matrix, not rows. Hiding rows only changes which outputs you ignore, and padded keys still receive weight.
- Expecting the output to have the memory’s length. The output always has the length of the query sequence. If you need \(M\) outputs, the memory must be the query side.
- Recomputing cross K/V at every generation step. They depend only on the memory. Compute them once per request (Algorithm 2), or decoding wastes time and memory bandwidth on every token.
- Forgetting positions in the memory. Cross-attention is order-blind over the memory (20.5). If the memory vectors carry no position (for example raw frame features without a time embedding), the model cannot tell first from last.
- Calling a “cross-encoder” cross-attention. A cross-encoder reranker (used in retrieval and RAG) concatenates query and document into one sequence and uses ordinary self-attention. The word “cross” there means “the two texts are encoded together”, not cross-attention.
- Initializing new cross-attention layers without a gate when adding them to a pretrained LLM. Random outputs added to the residual stream damage the LLM in the first steps; zero-initialized gates (or zero-initialized output projections) avoid it.
- Wrong widths. \(W_K\) and \(W_V\) take inputs of width \(d_z\) (the memory), not \(d\). With a vision encoder of width 1,280 and an LLM of width 4,096, either use \(d_z = 1280\) in \(W_K, W_V\) or project the features to 4,096 first.
Interview questions
What is the difference between self-attention and cross-attention?basic
In cross-attention, which sequence decides the output length, and why does that matter?basic
List the sublayers of an original Transformer decoder layer and the mask each uses.core
Why does cross-attention in an encoder–decoder model not need a causal mask?core
How does the KV cache work for cross-attention during generation? How large is it for Whisper large-v3?core
Why does Flamingo multiply its new layers by \(\tanh(\alpha)\) with \(\alpha\) initialized to 0?core
Cross-attention fusion (Flamingo, Llama 3.2 Vision) or token concatenation (LLaVA)? Compare.advanced
Show that cross-attention is a block inside self-attention over a concatenated sequence. What is different in practice?advanced
Cross-attention is permutation invariant over the memory. How does a model know where things are in the image or audio?advanced
How many parameters does a cross-attention layer have if the memory has width \(d_z\) and the model has width \(d\) (no biases, full heads)?core
What does a text token do in Llama 3.2 Vision’s cross-attention when no image comes before it?advanced
Exercises
- A decoder has \(T = 12\) target tokens, a memory of \(M = 50\) encoder tokens, \(d = 512\) and \(h = 8\) heads. What is the shape of the cross-attention score tensor for one example, and of the output?
- Finish the worked example: compute the weights and the output for \(\mathbf{q}_2 = (0, 1)\).
- Whisper large-v3 in FP16: how much memory does the cross-attention cache need for a batch of 16 audio windows?
- Llama 3.2 90B has 100 layers, 20 of them cross-attention, 8 KV heads and head size 128. How many self-attention layers does it have? How large is the cross-attention cache for 6,404 visual tokens in BF16, and how large would it be if the visual tokens were cached in all self-attention layers?
- In Figure 5, how many keys does the final “.” see in decoder-only mode, and how many by self- and by cross-attention in Flamingo mode?
- Show that if all keys in the memory are equal, the cross-attention output is the mean of the values, whatever the query.
Answerscheck
Key takeaways
- Self- and cross-attention use the same formula (20.1). Only the source of \(K\) and \(V\) differs: the same sequence (20.2) or another one, the memory (20.3).
- The cross-attention score matrix is \(T \times M\), and the output has one row per query (20.4), so it fits in the residual stream and learned queries can compress any input to a fixed size (20.9).
- Cross-attention needs no causal mask, only padding or image-selection masks; it is order-blind over the memory (20.5), so positions must live inside the memory vectors.
- Cost is \(4TMd\) per layer (20.6), and the cross K/V cache is computed once and never grows (20.7): about 245.8 MB per 30-second window in Whisper large-v3.
- Encoder–decoder layers stack causal self-attention, cross-attention and an FFN (20.8): translation, speech recognition, T5 and BART.
- Flamingo and Llama 3.2 Vision add zero-gated cross-attention layers (20.11) to a frozen LLM; decoder-only VLMs instead contain cross-attention as one block of a joint self-attention (20.10). Cross-attention is far cheaper when there are thousands of visual tokens.
Key formulas
References for this session
- Vaswani et al. (2017). Attention Is All You Need
- Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
- Bahdanau et al. (2014). Neural Machine Translation by Jointly Learning to Align and Translate
- Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
- Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Jaegle et al. (2021). Perceiver: General Perception with Iterative Attention
- Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
- Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
- Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
- Carion, Massa, Synnaeve, Usunier, Kirillov, Zagoruyko (2020). End-to-End Object Detection with Transformers
- Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
- Hugging Face (2024). Transformers: modeling_mllama.py (Llama 3.2 Vision implementation)