LLM & VLM courseBlog
20Track 2: Attention and transformers · session 5 of 15

Self-attention vs cross-attention

Self-attention and cross-attention use the same formula. The only difference is where the keys and values come from: the same sequence, or a different one such as an encoder output, an image or an audio clip. This session follows that one change through shapes, masks, positions, cost and caching, then shows how translation, speech recognition, Flamingo and Llama 3.2 Vision are built on it.

Time: 2.5 to 3.5 hours (reading 2 hours, exercises 1 hour)Level: BeginnerBefore this: Sessions 16 to 19 (attention, masks, multi-head attention); session 7 helps

What you will learn

  • Self-attention: Q, K and V from one sequence; cross-attention: Q from one sequence, K and V from another
  • Why the cross-attention score matrix is T × M but the output always has the query's length
  • Masks, positions, cost and the KV cache: what changes when the source changes
  • Encoder–decoder attention in the original Transformer and in Whisper, step by step
  • Learned queries (Perceiver, Q-Former, DETR) as cross-attention compressors
  • Flamingo's gated cross-attention and Llama 3.2 Vision's layout, with real numbers
  • Cross-attention fusion versus putting image tokens in the sequence: cost and trade-offs

Why this matters

Attention lets one vector look at a set of other vectors and pull in what it needs. Until now, the vectors that look and the vectors being looked at came from the same sentence. In this session we separate the two roles. When the questions and the answers come from the same sequence, we call it self-attention. When the questions come from one sequence and the answers from a different one, we call it cross-attention.

The formula does not change at all. Only the source of the keys and values changes. But this small change decides how a model connects two kinds of data:

  • Translation. The original Transformer reads a German sentence with an encoder, then writes English with a decoder. Every decoder layer uses cross-attention to look at the German sentence. [Certain]
  • Speech recognition. Whisper encodes 30 seconds of audio into 1,500 vectors. The text decoder looks at them through cross-attention while it writes the words. [Certain]
  • Vision-language models. Flamingo and Llama 3.2 Vision keep a text LLM and add new cross-attention layers so that text tokens can look at image features. Other VLMs (LLaVA, Qwen2-VL) do not use cross-attention: they put image tokens directly into the text sequence and use only self-attention. Choosing between these two designs is a classic VLM interview question.
  • Image generation. Stable Diffusion's denoiser uses cross-attention so that image positions can look at the text prompt. [Certain]

In Session 7 you saw Bahdanau attention: an RNN decoder that looks back at all encoder states. That was cross-attention before the word existed. In session 17 and session 19 we built the attention formula and split it into heads. Here we plug two different sequences into that formula and follow the consequences: shapes, masks, positions, cost, caching, and the VLM designs built on it.

One formula, two sources

Recall scaled dot-product attention. Given queries \(Q\), keys \(K\) and values \(V\):

\[ \mathrm{Attn}(Q, K, V) = \softmax\!\left( \frac{Q K^\top}{\sqrt{d_k}} \right) V \tag{20.1} \]

Each query is compared with every key. The softmax turns the scores of one query into weights that sum to 1. The output for that query is the weighted average of the values. The formula does not say where \(Q\), \(K\) and \(V\) come from. That is the only thing we change.

Self-attention

Take one sequence of \(T\) vectors, stacked as rows of \(X \in \R^{T \times d}\). Self-attention builds all three from \(X\):

\[ Q = X W_Q, \qquad K = X W_K, \qquad V = X W_V, \qquad \text{scores } \frac{QK^\top}{\sqrt{d_k}} \in \R^{T \times T} \tag{20.2} \]

Every token asks a question, and every token (often including itself) can answer. The score matrix is square.

Cross-attention

Now take two sequences: \(X \in \R^{T \times d}\), the sequence that asks (for example the text being written), and \(Z \in \R^{M \times d_z}\), the sequence that is read (for example encoder outputs, image patches, or audio frames). People often call \(Z\) the memory or the context. Cross-attention takes queries from \(X\) and keys and values from \(Z\):

\[ Q = X W_Q, \qquad K = Z W_K, \qquad V = Z W_V, \qquad \text{scores } \frac{QK^\top}{\sqrt{d_k}} \in \R^{T \times M} \tag{20.3} \]
Read the symbols
\(X\)
the sequence that asks: \(T\) rows, each of width \(d\) (the model width of the decoder or LLM)
\(Z\)
the memory: \(M\) rows, each of width \(d_z\). \(d_z\) can differ from \(d\); for example a vision encoder may output 1,280-dimensional features while the LLM uses 4,096
\(W_Q \in \R^{d \times d_k}\)
learned matrix that turns a row of \(X\) into a query
\(W_K, W_V \in \R^{d_z \times d_k}\)
learned matrices that turn a row of \(Z\) into a key and a value. They must accept width \(d_z\)
\(T \times M\)
the score matrix is rectangular: one row per query, one column per memory item

Look at the last line of (20.3). The output has one row per query. Each row \(i\) is

\[ \mathbf{o}_i = \sum_{j=1}^{M} a_{ij}\, \mathbf{v}_j, \qquad a_{ij} = \frac{\exp(\mathbf{q}_i \cdot \mathbf{k}_j / \sqrt{d_k})}{\sum_{m=1}^{M} \exp(\mathbf{q}_i \cdot \mathbf{k}_m / \sqrt{d_k})} \tag{20.4} \]

So the output has length \(T\), the length of the asking sequence, and is a mix of values from the memory. The memory can have 4 items or 6,404 items; the output shape stays \(T \times d\). This is why cross-attention fits into a residual stream: we can add its output to \(X\) row by row.

Worked example: two text queries read three image regions

Let \(d_k = 2\). Two text tokens ask with queries \(\mathbf{q}_1 = (1, 0)\) and \(\mathbf{q}_2 = (0, 1)\). Three image regions have keys \(\mathbf{k}_1 = (2, 0)\), \(\mathbf{k}_2 = (0, 2)\), \(\mathbf{k}_3 = (1, 1)\) and values \(\mathbf{v}_1 = (1, 0)\), \(\mathbf{v}_2 = (0, 1)\), \(\mathbf{v}_3 = (0.5, 0.5)\).

Scores for query 1, divided by \(\sqrt{2}\): \((2, 0, 1)/1.414 = (1.414,\ 0,\ 0.707)\). Softmax: \((0.576,\ 0.140,\ 0.284)\). Output: \(0.576\,(1, 0) + 0.140\,(0, 1) + 0.284\,(0.5, 0.5) = (0.718,\ 0.282)\).

The score matrix is \(2 \times 3\) (2 queries, 3 keys), but the output is \(2 \times 2\): one vector per query. Query 2 gives the mirror image, weights \((0.140, 0.576, 0.284)\) and output \((0.282, 0.718)\) (Exercise 2).

Figure 1 puts both kinds side by side on a toy example. The text is “a red car”. In self mode, the three text tokens attend to each other. In cross mode, the same three queries attend to four image regions instead. The vectors are hand-made (already projected), so you can follow every number.

query (from the text)key and value source
Figure 1. The same attention formula with two sources of keys and values. Rows are queries (always the text “a red car”). Columns are keys: the text itself (self-attention, a \(3 \times 3\) matrix) or four image regions (cross-attention, a \(3 \times 4\) matrix). Click a row to read its weights and output.
How to read Figure 1

Darker cells mean larger weights; each row sums to 1. In cross mode, the query of “car” puts 0.632 of its weight on the “red car” region, so its output is mostly that region’s value. In self mode, the same query spreads its weight over the three text tokens. Turn on the causal mask: the matrix becomes lower-triangular and the row for “a” can only see itself (weight 1.000). The mask box does nothing in cross mode, because the image is fully known before the text starts; there is no “future” in the memory to hide.

Figure 1 shows the finished weights. Figure 2 computes them in front of you. The text “a red car” is \(X\) (\(T_q = 3\) rows, \(d = 4\)). The image is \(Z\): up to five patches with \(d_z = 3\) features each. The weights \(W_Q\) (\(4 \times 2\)), \(W_K, W_V\) (\(3 \times 2\)) are small fixed matrices, so \(d_k = 2\). Every step runs one line of the algorithm on the real numbers and puts them into formula (20.4). Switch to self-attention to see the same loop with \(K\) and \(V\) taken from the text instead.

query \(\mathbf{q}_i\) (from the text)key \(\mathbf{k}_j\)value \(\mathbf{v}_j\) and output \(\mathbf{o}_i\)cell being computed
  1. \(Q \leftarrow X W_Q\) // \(T_q \times d_k\), always from the text
  2. \(S \leftarrow X\) (self) or \(Z\) (cross); \(K \leftarrow S W_K\); \(V \leftarrow S W_V\) // \(T_{kv} \times d_k\)
  3. for \(i = 1, \dots, T_q\): // one row per query
  4. for \(j = 1, \dots, T_{kv}\): \(s_{ij} \leftarrow \mathbf{q}_i \cdot \mathbf{k}_j / \sqrt{d_k}\) // \(-\infty\) if masked
  5. \(\mathbf{a}_i \leftarrow \softmax(\mathbf{s}_i)\) // (20.4), row sums to 1
  6. \(\mathbf{o}_i \leftarrow \sum_j a_{ij} \mathbf{v}_j\) // (20.4)
  7. return \(O\) // \(T_q \times d_k\): the query's length
Figure 2. Attention computed step by step. Rows are the three text queries; columns are the keys and values: image patches (cross) or the text tokens themselves (self). The score matrix fills one cell per step, then each row goes through the softmax and mixes the values into one output row.
How to read Figure 2

Press Play, or use Step and Back. Steps 1 and 2 build \(Q\), \(K\) and \(V\): in cross mode \(K\) and \(V\) come from \(Z\), which has width 3, so \(W_K\) and \(W_V\) have 3 rows while \(W_Q\) has 4. Then each cell gets its score (small number) and, after the softmax, its weight (large number, darker = more). For the query “car” with \(M = 4\), the patch “red car” gets weight 0.823 and the output is \(\mathbf{o}_3 = (0.896, 0.853)\). Change \(M\): the matrix gets wider (\(3 \times M\)) but the output column always has 3 rows. In self mode the matrix is \(3 \times 3\); turn on the causal mask and the row of “a” keeps weight 1.000 on itself. The last step checks the loop against the matrix formula \(\softmax(QK^\top/\sqrt{d_k})V\).

What changes when the source changes

The formula is the same, but four practical things are different. Each one causes real bugs and comes up in interviews.

1. Masks

In a decoder, self-attention needs a causal mask: token \(t\) may not see tokens after \(t\), or training would let the model copy the answer (session 18). Cross-attention in an encoder-decoder model needs no causal mask. The memory (the source sentence, the audio, the image) is fully known before the decoder starts. Every decoder token may look at all of it. The only mask cross-attention usually needs is a padding mask over the memory, so that queries ignore padding positions when memories of different lengths share one batch. VLMs add one more kind: a mask that decides which image each text token may see (Figure 5).

2. Positions

Attention by itself ignores order. If you shuffle the rows of \(K\) and \(V\) together with the same permutation \(P\), every query gets exactly the same output:

\[ \mathrm{Attn}(Q,\ PK,\ PV) = \mathrm{Attn}(Q,\ K,\ V) \quad \text{for any permutation matrix } P \tag{20.5} \]

The weights are permuted in the same way as the values, so the weighted sum does not change. In self-attention we fix this by adding position information (learned positions, sinusoids, or RoPE inside the attention). In cross-attention, the position of a memory item must already be inside its vector: the encoder adds positions to its own tokens, a ViT adds patch positions, Whisper’s encoder adds positions to audio frames. The cross-attention layer itself usually adds none. For example, the cross-attention layers of Llama 3.2 Vision in the Hugging Face code apply no RoPE; the image features already carry their positions from the vision encoder. [Certain] T5 uses a relative position bias in self-attention but not in cross-attention. [Likely]

3. Cost

The score matrix has \(T \times T\) entries for self-attention and \(T \times M\) for cross-attention. Per layer, with model width \(d\), the two matrix products \(QK^\top\) and (weights)\(\,V\) cost about

\[ \text{self: } 4\,T^2 d \ \text{FLOPs}, \qquad \text{cross: } 4\,T M d \ \text{FLOPs} \tag{20.6} \]

(Each product needs about \(2 \cdot T \cdot M \cdot d\) operations, summed over all heads.) Cross-attention is linear in each length. A short text reading a long memory is cheap; it is the memory’s own self-attention (inside the encoder) that is quadratic. Whisper is a good example: the encoder’s self-attention has \(1500^2 = 2{,}250{,}000\) scores per head per layer, while the decoder’s cross-attention for its longest output (448 tokens) has \(448 \times 1500 = 672{,}000\).

4. Caching during generation

When a decoder generates text one token at a time, its self-attention keys and values grow by one row per step, so the KV cache of self-attention grows with the output (session 80). The cross-attention keys and values depend only on the memory \(Z\), which does not change. So we compute \(K = ZW_K\) and \(V = ZW_V\) once, before the first output token, and reuse them at every step. Each new token only needs its own query. The memory needed for this cache, for \(N\) memory items, is

\[ \text{bytes} = N \cdot L_\times \cdot 2 \cdot n_{kv} \cdot d_h \cdot b \tag{20.7} \]
Read the symbols
\(N\)
number of memory items (encoder tokens, image tokens, audio frames)
\(L_\times\)
number of layers that have cross-attention
\(2\)
one key and one value per item
\(n_{kv}\,, d_h\)
number of key/value heads and the size of each head. With grouped-query attention, \(n_{kv}\) is smaller than the number of query heads (session 26)
\(b\)
bytes per number: 2 for FP16 or BF16
Worked example: the cross-attention cache of Whisper large-v3

Whisper large-v3 has \(d = 1280\), 20 heads, 32 decoder layers, and one cross-attention in each decoder layer. Its config gives 1,500 encoder positions for one 30-second window. [Certain] Without grouping, \(n_{kv} d_h = 1280\). In FP16: \(1500 \times 32 \times 2 \times 1280 \times 2 = 245{,}760{,}000\) bytes, about 245.8 MB per audio window. This is computed once and read at every one of the up to 448 output steps. A batch of 16 windows needs about 3.93 GB just for this cache.

Self-attentionCross-attention
Q from\(X\)\(X\) (the sequence being built)
K, V from\(X\)\(Z\) (memory: encoder, image, audio)
Score shape\(T \times T\)\(T \times M\)
Output length\(T\)\(T\) (never \(M\))
Causal mask in a decoderyesno; padding mask on \(Z\), and per-image masks in VLMs
Positionsadded by the layer (RoPE) or the inputmust already be inside \(Z\)
Cost per layer\(4T^2 d\)\(4TMd\)
KV cache in generationgrows by one row per new tokenfixed; computed once from \(Z\)

Encoder–decoder attention

The Transformer of Vaswani et al. (2017) was built for translation. It has an encoder stack that reads the source sentence and a decoder stack that writes the target sentence. The base model has 6 encoder layers and 6 decoder layers with \(d = 512\) and 8 heads. [Certain] Each encoder layer has two sublayers: self-attention without a causal mask (every source word sees every source word) and a feed-forward network. Each decoder layer has three:

\[ \begin{aligned} \mathbf{h} &= \mathbf{x} + \mathrm{SelfAttn}_{\text{causal}}(\mathbf{x}) \\ \mathbf{h}' &= \mathbf{h} + \mathrm{CrossAttn}(\mathbf{h},\ Z_{\text{enc}}) \\ \mathbf{y} &= \mathbf{h}' + \mathrm{FFN}(\mathbf{h}') \end{aligned} \tag{20.8} \]

(The original paper puts a layer normalization after each residual sum; modern models put it before each sublayer. That detail does not change the attention.) \(Z_{\text{enc}}\) is the output of the last encoder layer, and the same \(Z_{\text{enc}}\) is used by all six decoder layers. Each decoder layer has its own \(W_K, W_V\), so each one reads the memory in its own way. The original paper calls this sublayer encoder–decoder attention.

Why this order? Self-attention first lets each target token gather what has been written so far (“I am about to write an adjective”). Cross-attention then uses that summary as the question to the source (“which source word is the adjective?”). The FFN processes the answer. Stacking six such layers lets the decoder ask more refined questions each time.

Figure 3 shows one decoding run of a toy model translating “the black cat sleeps” into French, “le chat noir dort”. Note the word order: French puts the adjective after the noun. The cross-attention weights in the figure were set by hand to look like the alignment a trained model learns; the attention mechanics are real.

encoder output = memory, and its cross cache \((K, V)\)decoder token asking (query), and its self cachecausal self-attention among decoder tokens
  1. \(Z \leftarrow \mathrm{Encoder}(\text{source})\) // once, bidirectional
  2. \(K^\times \leftarrow Z W_K\); \(V^\times \leftarrow Z W_V\) // cross cache, fixed size \(M\)
  3. for \(t = 1, 2, \dots\): // one new token per step
  4. self cache \(\leftarrow\) append \((\mathbf{k}_t, \mathbf{v}_t)\); causal self-attention // grows by 1
  5. \(\mathbf{a}_t \leftarrow \softmax(\mathbf{q}_t K^{\times\top} / \sqrt{d_k})\); \(\mathbf{c}_t \leftarrow \mathbf{a}_t V^\times\) // (20.4), no new keys
  6. FFN, then pick the next word // (20.8)
Figure 3. Encoder–decoder attention during generation. Step 1 runs the encoder once and fills the cross cache. At each step, the newest decoder position sends one query to the memory; line thickness is its cross-attention weight on each source word. On the right, the full cross-attention matrix fills one row per step.
How to read Figure 3

Step 1 is the encoder: the four source words get their keys and values once (the orange chips). After that, each step adds one blue chip to the self-attention cache and none to the cross cache. At step 4 the decoder is about to write “noir” and its cross-attention puts 0.803 of the weight on “black”, even though “black” is before “cat” in the source. Cross-attention has no causal mask and no order constraint over the source, so reordering costs nothing. Look at the grey arcs: the decoder’s self-attention is causal, so step \(t\) sees only the \(t\) tokens written so far. Also note what does not change from step to step: the encoder outputs and their keys and values. That is the cached part.

Algorithm 1: one attention call, self or cross
  1. input: asking sequence \(X\) (\(T \times d\)), optional memory \(Z\) (\(M \times d_z\)), optional mask, weights \(W_Q, W_K, W_V, W_O\)
  2. \(S \leftarrow X\) if no memory is given, else \(S \leftarrow Z\) // the only difference
  3. \(Q \leftarrow X W_Q\); \(K \leftarrow S W_K\); \(V \leftarrow S W_V\); split each into \(h\) heads
  4. scores \(\leftarrow QK^\top / \sqrt{d_k}\) // \(T \times T\) or \(T \times M\) per head
  5. apply the mask: causal for decoder self-attention; padding (and image selection) for cross-attention
  6. \(A \leftarrow\) softmax over each row; \(O \leftarrow A V\) // \(T\) rows in both cases
  7. concatenate heads; return \(O W_O\)
Algorithm 2: encoder–decoder generation with a cross-attention cache
  1. \(Z \leftarrow \mathrm{Encoder}(\text{source})\) // run once: bidirectional self-attention over the source
  2. for each decoder layer \(\ell\): \(K^\times_\ell \leftarrow Z W_K^{(\ell)}\), \(V^\times_\ell \leftarrow Z W_V^{(\ell)}\) // cross cache, fixed size
  3. \(y \leftarrow [\text{start token}]\); self-attention caches empty
  4. repeat:
  5. for each layer: causal self-attention of the newest position (append its \(k, v\) to the self cache)
  6. then cross-attention: its query against \(K^\times_\ell, V^\times_\ell\) // no new keys, no mask except padding
  7. then FFN; at the top, pick the next token and append it to \(y\)
  8. until the end token or the length limit

Speech recognition: Whisper

Whisper is an encoder–decoder transformer of the same shape. The input is a log-Mel spectrogram (an image-like table of audio energy over time and frequency) of a 30-second window. Two convolution layers reduce it to 1,500 frames, the encoder turns those into 1,500 vectors, and the decoder writes text while cross-attending to them. Whisper large-v3 has 32 encoder layers and 32 decoder layers with \(d = 1280\) and 20 heads, and its decoder handles up to 448 tokens. [Certain] The same pattern is used by T5 and BART for text-to-text tasks such as summarization (session 24).

Learned queries: cross-attention as a compressor

So far the queries came from real tokens. A powerful trick is to make the queries learned parameters: a fixed set of \(N\) vectors that do not depend on the input. Cross-attention from \(N\) learned queries into \(M\) input features always returns \(N\) outputs, whatever \(M\) is:

\[ \underbrace{L}_{N \times d} \ \xrightarrow{\ \text{cross-attend to } Z \ (M \times d_z)\ } \ \underbrace{\mathrm{Attn}(L W_Q,\ Z W_K,\ Z W_V)}_{N \times d}, \qquad \text{cost} \propto N M \ \text{instead of} \ M^2 \tag{20.9} \]

This turns a long, variable-length input into a short, fixed-length summary. It is used in many places:

  • Perceiver uses a small latent array that cross-attends to a very large input (pixels, audio), so the expensive quadratic self-attention runs only over the latents. [Certain]
  • Flamingo’s Perceiver Resampler turns a variable number of image or video features into 64 visual tokens. [Certain] For video, frames are sampled at 1 frame per second and encoded separately, with a learned time embedding added. [Certain]
  • BLIP-2’s Q-Former uses 32 learned queries to read features from a frozen image encoder. [Certain]
  • DETR uses learned “object queries” in a decoder that cross-attends to image features; each query output becomes one predicted box. [Certain] Video moment detectors such as Moment-DETR use the same idea with “moment queries” over video features. [Likely]

These connectors are the topic of session 178. Here the point is only that they are cross-attention with learned queries.

Cross-attention inside a VLM

A VLM must let text tokens use image information. There are two main ways to wire it (session 179 goes deeper):

  1. Put image tokens into the sequence (LLaVA, Qwen2-VL, many others). The image encoder output is projected to the LLM width and placed in the token sequence next to the text. The LLM’s own self-attention then handles everything. No new attention layers are added.
  2. Keep image tokens outside, and read them with cross-attention (Flamingo, Llama 3.2 Vision). The LLM sequence contains only text (plus a marker where an image appears). New cross-attention layers are inserted between the LLM’s layers; their queries come from the text, and their keys and values come from the image features.

These two are more alike than they look. Put the memory \(Z\) in front of the text \(X\) and run one self-attention over the joint sequence \([Z; X]\). The score matrix splits into four blocks:

\[ \frac{1}{\sqrt{d_k}} \begin{bmatrix} Q_Z K_Z^\top & Q_Z K_X^\top \\ Q_X K_Z^\top & Q_X K_X^\top \end{bmatrix} \qquad \begin{matrix} \leftarrow \text{image rows} \\ \leftarrow \text{text rows} \end{matrix} \tag{20.10} \]

The bottom-left block \(Q_X K_Z^\top\) is exactly a cross-attention from text to image. The bottom-right block is the text’s own self-attention. So a decoder-only VLM contains cross-attention as one block of its self-attention, with three differences from a dedicated cross-attention layer: the same weights serve both blocks, one softmax runs over image and text keys together, and the image tokens also pass through every LLM layer (they are updated too). The PyTorch code below checks the block claim numerically.

Flamingo: gated cross-attention in a frozen LM

Flamingo (Alayrac et al., 2022) starts from a pretrained, frozen Chinchilla language model (1.4B, 7B or 70B) and inserts new layers called GATED XATTN-DENSE between the frozen layers. [Certain] Each new layer is a cross-attention to the visual tokens, then a feed-forward layer, each with a tanh gate:

\[ \begin{aligned} \mathbf{y} &\leftarrow \mathbf{y} + \tanh(\alpha_{\text{xattn}}) \cdot \mathrm{CrossAttn}(\mathbf{y},\ X_{\text{vis}}) \\ \mathbf{y} &\leftarrow \mathbf{y} + \tanh(\alpha_{\text{dense}}) \cdot \mathrm{FFN}(\mathbf{y}) \end{aligned} \qquad \alpha_{\text{xattn}} = \alpha_{\text{dense}} = 0 \ \text{at the start} \tag{20.11} \]
Read the symbols
\(\mathbf{y}\)
the language features (the residual stream of the text tokens)
\(X_{\text{vis}}\)
the visual tokens from the Perceiver Resampler (64 per image)
\(\alpha\)
one learnable number per layer. \(\tanh(0) = 0\), so at initialization the new layers add exactly nothing

Why the gate? The frozen LM is very good at language. A new, randomly initialized layer would add noise to its residual stream and damage it on the first training step. With \(\alpha = 0\), the model starts as exactly the original LM and opens the visual path slowly as training moves \(\alpha\) away from 0 (\(\tanh(0.5) = 0.462\), \(\tanh(2) = 0.964\)). The paper says it uses the gate “to ensure that at initialization, the conditioned model yields the same results as the original language model”. [Certain]

Figure 4 runs one gated cross-attention layer on a toy example. The last text token of “This is a photo of a” has the residual vector \(\mathbf{y} = (0.3, 0.1)\). A frozen “LM head” \(U\) turns \(\mathbf{y}\) into scores for three next words: cat, dog and car. Alone, the text model guesses “cat” with probability 0.548. Four visual tokens (already projected, \(d_k = 2\)) come from a photo. Play the figure to move \(\alpha\) from 0 to 2, as training would do, and watch the visual path open.

0.00
text residual \(\mathbf{y}\)visual tokens (keys)gated update \(\tanh(\alpha)\,\mathbf{c}\)new residual \(\mathbf{y}'\)
  1. if no image: return \(\mathbf{y}\) // layer skipped
  2. \(\mathbf{a} \leftarrow \softmax(\mathbf{q} \cdot \mathbf{k}_j / \sqrt{d_k})\); \(\mathbf{c} \leftarrow \sum_j a_j \mathbf{v}_j\) // \(\mathbf{q}\) from text, \(\mathbf{k}, \mathbf{v}\) from image
  3. \(\mathbf{y}' \leftarrow \mathbf{y} + \tanh(\alpha)\, \mathbf{c}\) // (20.11), \(\alpha = 0\) at the start
  4. \(\mathbf{p} \leftarrow \softmax(U \mathbf{y}')\) // the frozen LM continues
Figure 4. A Flamingo-style gated cross-attention layer, computed live. Left: the four visual tokens and their attention weights. Middle: the residual vector \(\mathbf{y}\) (blue), the gated update (green) and the result \(\mathbf{y}'\) (purple), with the three LM-head directions. Right: \(\tanh(\alpha)\) and the next-word probabilities; grey bars are the text-only model.
How to read Figure 4

Each value of \(\alpha\) takes three steps: cross-attention (line 2), the gated add (line 3) and the frozen head (line 4). At \(\alpha = 0\) the green arrow has length zero and the bars equal the grey text-only bars exactly: this is the “same results as the original language model” property. With the dog photo, \(\alpha = 1\) gives \(\tanh(1) = 0.762\) and \(P(\text{dog}) = 0.798\). Note the attention weights: the query comes from the text, which leans to “cat”, so with the dog photo the plain background token gets the largest weight (0.312); the output still points to “dog” because three of the four values do. Choose “no image”: line 1 returns at once and \(\alpha\) has no effect. The toy has no FFN branch; in the real layer it is gated the same way.

How often do you insert these layers? More layers give better quality but more compute and more trainable parameters. Flamingo-3B adds one before every LM block (1.4B new parameters), Flamingo-9B before every fourth block (1.8B), and Flamingo-80B before every seventh block (10B). [Certain] In their ablation, inserting every fourth block made training 66% faster and lowered the overall score by only 1.9%. [Certain]

Flamingo also has a careful mask for interleaved documents (text with several images). A text token cross-attends only to the visual tokens of the last image before it, not to all earlier images. [Certain] It still knows about earlier images indirectly, through self-attention over earlier text tokens that did see them. This scheme let Flamingo train on sequences with up to 5 images and still use up to 32 image–text examples at evaluation. [Certain] Figure 5 compares this mask with the decoder-only design.

text attends to text (self)text attends to image (cross)image attends (decoder-only only)
Figure 5. Who can see whom in the sequence “[image 1] A cat . [image 2] A dog .”, with 4 visual tokens per image (a real model would use 64 to thousands). Rows ask, columns answer. Click any row.
How to read Figure 5

In decoder-only mode, all 14 positions are in the LLM sequence and the mask is one big causal triangle: “dog” sees both images directly. In Flamingo mode, the visual tokens are not part of the LLM sequence (their rows are empty). Text rows have two parts: a causal triangle over text only (self-attention) and a block over the visual tokens of the most recent image (cross-attention). “dog” sees 4 visual tokens of image 2 and none of image 1. It can still learn about image 1 through the text tokens “A cat .”, which did see it.

Llama 3.2 Vision

Llama 3.2 Vision (from the Llama 3 paper, Section 7) follows the Flamingo recipe at a larger scale. A ViT-H/14 image encoder produces features; cross-attention layers are “applied after every fourth self-attention layer in the core language model” and use grouped-query attention. [Certain] During adapter training, “we intentionally do not update the language-model parameters”, so the text model stays as it was. [Certain] The public configs show the layout:

  • 11B: 40 decoder layers in total, of which the 8 at indices 3, 8, 13, …, 38 are cross-attention layers; the other 32 are the self-attention layers of the Llama 3.1 8B text model. Width 4,096, 32 query heads, 8 key/value heads. [Certain]
  • 90B: 100 layers, 20 of them cross-attention (indices 3, 8, …, 98), width 8,192. [Certain]
  • Images: tiles of \(560 \times 560\) pixels with \(14 \times 14\) patches give \(40 \times 40 = 1600\) patches plus one class token, so 1,601 vectors per tile, and up to 4 tiles per image. [Certain]
  • For the 405B model (not released with vision), the paper reports about 100B parameters in the cross-attention layers. [Certain]

The Hugging Face implementation shows two more details. The cross-attention block has tanh gates initialized at zero, as in Flamingo. And if a text token has no image to look at, its cross-attention output is multiplied by zero, so the layer is a no-op for it; for text-only prompts, the cross-attention layers are skipped entirely. [Certain]

Worked example: what cross-attention saves in Llama 3.2 11B

One image with 4 tiles gives \(4 \times 1601 = 6404\) visual tokens. Each token stored in the KV cache of one layer costs \(2 \times 8 \times 128 \times 2 = 4096\) bytes (K and V, 8 KV heads, head size 128, BF16).

Cross-attention design (the real model): the visual tokens are cached only in the 8 cross-attention layers: \(6404 \times 8 \times 4096 = 209.8\) MB.

Same LM, decoder-only design (hypothetical): the visual tokens would sit in the sequence and be cached in all 32 self-attention layers: \(6404 \times 32 \times 4096 = 839.4\) MB, 4 times more. Worse, each of the 6,404 visual tokens would have to run through the whole 8B model in prefill, about \(2 \times 8.03 \times 10^9\) FLOPs per token, or about 103 TFLOPs for one image. In the cross-attention design, each visual token only needs its key and value projections in 8 layers: \(8 \times 2 \times (2 \times 4096 \times 1024) = 134\) MFLOPs per token, about 120 times less. Figure 6 lets you change these numbers.

decoder-only (image tokens in all 32 layers)cross-attention (image tokens in 8 layers)
Figure 6. The same Llama-3.1-8B-sized text model (32 self-attention layers, 8 KV heads of size 128, BF16) used in two ways: image tokens placed in the sequence, or read by 8 cross-attention layers. Bars are relative to the decoder-only value in each group.
How to read Figure 6

Three costs are shown. KV cache: formula (20.7) for both designs, text tokens included. Attention per new token: formula (20.6) with one query, over all layers. Image prefill: the work done for the visual tokens before the first output token; this is where cross-attention wins most, because visual tokens skip the LLM’s FFNs and self-attention. With 64 visual tokens (Flamingo) and 1,000 text tokens, the KV cache and decode costs are almost equal: text dominates. With thousands of visual tokens, or several images, the gap grows. The figure ignores the vision encoder (the same in both designs) and the extra weights of the cross-attention layers.

So which design is better?

Neither wins everywhere. Cross-attention designs keep the text model untouched (good for text-only quality and for adding vision to an existing LLM), and they are cheap when there are many visual tokens. Decoder-only designs are simpler: no new layer type, and the LLM can use its full depth to process image tokens together with text, which many recent open VLMs have chosen. [Likely] Most open VLMs released after LLaVA use token concatenation; Llama 3.2 Vision and Flamingo-style models such as IDEFICS (first version) and OpenFlamingo use cross-attention. [Likely] The book (Noyan et al., Chapter 6) describes both families.

Code: one attention module, used both ways

The module below is self-attention when called with one argument and cross-attention when given a memory. The checks print True for each claim of this session: output length, self as a special case, order invariance (20.5), cached cross K/V, the joint-sequence block (20.10), and the zero gate (20.11). It runs on CPU in about a second.

import math, torch
import torch.nn as nn
torch.manual_seed(0)

class Attention(nn.Module):
    """One module for both kinds. Self: attn(x). Cross: attn(x, memory)."""
    def __init__(self, d, h, d_mem=None):
        super().__init__()
        d_mem = d_mem or d
        self.h, self.dk = h, d // h
        self.wq = nn.Linear(d, d, bias=False)        # queries always from x
        self.wk = nn.Linear(d_mem, d, bias=False)    # keys from the source
        self.wv = nn.Linear(d_mem, d, bias=False)    # values from the source
        self.wo = nn.Linear(d, d, bias=False)

    def split(self, t):                              # (B, N, d) -> (B, h, N, dk)
        B, N, _ = t.shape
        return t.view(B, N, self.h, self.dk).transpose(1, 2)

    def forward(self, x, memory=None, mask=None, kv=None):
        src = x if memory is None else memory        # the only difference
        q = self.split(self.wq(x))
        if kv is None:                               # kv = cached (k, v) for cross-attention
            kv = (self.split(self.wk(src)), self.split(self.wv(src)))
        k, v = kv
        s = q @ k.transpose(-2, -1) / math.sqrt(self.dk)   # (B, h, T, M)
        if mask is not None:
            s = s.masked_fill(~mask, float("-inf"))
        o = torch.softmax(s, dim=-1) @ v                    # (B, h, T, dk)
        o = o.transpose(1, 2).reshape(x.shape[0], x.shape[1], -1)
        return self.wo(o), kv

d, h, T, M = 64, 4, 5, 9
x   = torch.randn(1, T, d)     # e.g. 5 text tokens (decoder side)
mem = torch.randn(1, M, d)     # e.g. 9 encoder outputs or image patches
att = Attention(d, h)

# 1) shapes: the output has one row per QUERY, whatever M is
y_self, _  = att(x)
y_cross, kv = att(x, mem)
print(y_self.shape, y_cross.shape)                 # both (1, 5, 64)

# 2) self-attention is cross-attention with memory = x
print(torch.allclose(att(x, x)[0], y_self))        # True

# 3) cross-attention ignores the ORDER of the memory, formula (20.5)
perm = torch.randperm(M)
print(torch.allclose(att(x, mem[:, perm])[0], y_cross, atol=1e-6))   # True

# 4) the cross K/V can be computed once and reused at every decoding step
y_cached, _ = att(x[:, -1:], mem, kv=kv)           # only the newest query
print(torch.allclose(y_cached, y_cross[:, -1:], atol=1e-6))          # True

# 5) cross-attention = one block of self-attention over [memory; x], formula (20.10)
joint = torch.cat([mem, x], dim=1)                 # (1, M+T, d)
allow = torch.zeros(M + T, M + T, dtype=torch.bool)
allow[M:, :M] = True                               # text rows may see memory columns only
allow[:M, :M] = True                               # (memory rows: anything, ignored here)
y_joint, _ = att(joint, mask=allow)
print(torch.allclose(y_joint[:, M:], y_cross, atol=1e-6))            # True

# 6) Flamingo-style tanh gate, formula (20.11): at alpha = 0 the LM is unchanged
alpha = nn.Parameter(torch.zeros(1))
y_gated = x + torch.tanh(alpha) * y_cross
print(torch.equal(y_gated, x))                     # True: identical at initialization

# 7) memory with a different width (for example 1,601 vectors of a 1280-d vision encoder)
att_v = Attention(d, h, d_mem=1280)
print(att_v(x, torch.randn(1, 1601, 1280))[0].shape)                # (1, 5, 64)

In real code you would call F.scaled_dot_product_attention (from torch.nn.functional) with arguments q, k, v, attn_mask, is_causal, which runs a fused kernel. Pass is_causal=True only for self-attention; for cross-attention pass a padding mask over the memory, or nothing.

Algorithm 3: a Flamingo-style block inserted into a frozen LM
  1. input: text residual stream \(\mathbf{y}\) (\(T \times d\)), visual tokens \(X_{\text{vis}}\), image-selection mask \(m\) (\(T \times M\))
  2. if no image is present: return \(\mathbf{y}\) // the layer is skipped
  3. \(\mathbf{c} \leftarrow \mathrm{CrossAttn}(\mathrm{Norm}(\mathbf{y}),\ X_{\text{vis}},\ \text{mask } m)\) // trainable
  4. zero the rows of \(\mathbf{c}\) whose mask row is empty // text before any image
  5. \(\mathbf{y} \leftarrow \mathbf{y} + \tanh(\alpha_{\text{xattn}})\,\mathbf{c}\)
  6. \(\mathbf{y} \leftarrow \mathbf{y} + \tanh(\alpha_{\text{dense}})\,\mathrm{FFN}(\mathrm{Norm}(\mathbf{y}))\) // trainable
  7. \(\mathbf{y} \leftarrow\) next frozen LM layer(s) \((\mathbf{y})\) // self-attention + FFN, weights not updated

Where this is used

System or taskQueries fromKeys and values fromNotes
Machine translation (Transformer, 2017)target-language decodersource-sentence encoderone cross-attention per decoder layer
Speech recognition (Whisper)text decoder1,500 audio-frame vectors per 30 scross K/V cached once per window
Summarization, text-to-text (T5, BART)decoderencoder over the documentsee session 24
Image and video question answering (Flamingo)text tokens of the frozen LM64 resampled visual tokens per image or videovideo frames at 1 FPS; tanh gates
Image reasoning, document reading (Llama 3.2 Vision)text tokensup to 4 × 1,601 tile features8 of 40 layers (11B); LM frozen during adapter training
Video understanding (Llama 3 video adapter)text tokensframe features merged by a Perceiver-style aggregator (32 frames into 1)extra video cross-attention layers, as described in the Llama 3 paper [Certain]; not part of the public Llama 3.2 release [Likely]
Object detection (DETR), key-moment detection (Moment-DETR)learned object or moment queriesimage or video featureseach query output is one box or one time span
Video object segmentation (SAM 2)features of the current framememory bank of earlier frames and prompts“memory attention” uses cross-attention to the stored memories [Likely]; see session 204
Text-to-image generation (Stable Diffusion)image latent positionstext-encoder tokens of the prompthow the prompt controls the picture
Video retrieval, fingerprinting, Video-RAGlearned queries (Q-Former style) or textmany frame featurescompress a clip to a few vectors for indexing (session 208)

Common mistakes

  • Using a causal mask in cross-attention. With a causal mask, decoder position \(t\) would see only the first \(t\) source tokens, an arbitrary and harmful cut. Cross-attention needs a padding mask on the memory, not a causal one. In PyTorch, do not pass is_causal=True when Q and K have different lengths.
  • Masking the wrong axis. The padding mask for the memory must hide columns (keys) of the \(T \times M\) matrix, not rows. Hiding rows only changes which outputs you ignore, and padded keys still receive weight.
  • Expecting the output to have the memory’s length. The output always has the length of the query sequence. If you need \(M\) outputs, the memory must be the query side.
  • Recomputing cross K/V at every generation step. They depend only on the memory. Compute them once per request (Algorithm 2), or decoding wastes time and memory bandwidth on every token.
  • Forgetting positions in the memory. Cross-attention is order-blind over the memory (20.5). If the memory vectors carry no position (for example raw frame features without a time embedding), the model cannot tell first from last.
  • Calling a “cross-encoder” cross-attention. A cross-encoder reranker (used in retrieval and RAG) concatenates query and document into one sequence and uses ordinary self-attention. The word “cross” there means “the two texts are encoded together”, not cross-attention.
  • Initializing new cross-attention layers without a gate when adding them to a pretrained LLM. Random outputs added to the residual stream damage the LLM in the first steps; zero-initialized gates (or zero-initialized output projections) avoid it.
  • Wrong widths. \(W_K\) and \(W_V\) take inputs of width \(d_z\) (the memory), not \(d\). With a vision encoder of width 1,280 and an LLM of width 4,096, either use \(d_z = 1280\) in \(W_K, W_V\) or project the features to 4,096 first.

Interview questions

What is the difference between self-attention and cross-attention?basic
Both use \(\softmax(QK^\top/\sqrt{d_k})V\). In self-attention, \(Q\), \(K\) and \(V\) are all projections of the same sequence, so the score matrix is \(T \times T\). In cross-attention, \(Q\) comes from one sequence (for example the decoder or the text) and \(K, V\) come from another (encoder output, image features, audio frames), so the score matrix is \(T \times M\). The output always has one row per query.
In cross-attention, which sequence decides the output length, and why does that matter?basic
The query sequence. Each query produces one weighted average of the memory’s values, so the output is \(T \times d\) whatever \(M\) is. That is why the output can be added to the decoder’s residual stream, and why learned queries (Perceiver, Q-Former, Flamingo’s resampler) compress a variable-length input into a fixed number of vectors.
List the sublayers of an original Transformer decoder layer and the mask each uses.core
(1) Masked (causal) self-attention over the target tokens so far, plus padding mask; (2) encoder–decoder cross-attention with queries from the decoder and keys/values from the final encoder output, with only a padding mask over the source; (3) a position-wise FFN. Each has a residual connection and layer normalization. All decoder layers read the same final encoder output, each with its own \(W_K, W_V\).
Why does cross-attention in an encoder–decoder model not need a causal mask?core
The causal mask exists to stop a position from seeing future target tokens, which would leak the answer during teacher-forced training. The memory is the input (source sentence, audio, image), fully known before decoding starts and not something the model predicts. So every decoder position may see all of it. Only padding positions in the memory are masked.
How does the KV cache work for cross-attention during generation? How large is it for Whisper large-v3?core
The cross keys and values depend only on the encoder output, so they are computed once per input, stored per decoder layer, and reused for every generated token; only the new query is computed each step. The self-attention cache, in contrast, grows by one row per token. For Whisper large-v3: \(1500\) frames \(\times\) 32 layers \(\times\, 2 \times 1280 \times 2\) bytes \(\approx 245.8\) MB per 30-second window in FP16.
Why does Flamingo multiply its new layers by \(\tanh(\alpha)\) with \(\alpha\) initialized to 0?core
The language model is pretrained and frozen. New cross-attention and FFN layers start random; adding their output directly would corrupt the residual stream and hurt the LM at the start. With \(\tanh(0) = 0\), the model at initialization is exactly the original LM, and training opens the visual path gradually. Llama 3.2 Vision’s cross-attention blocks in the Hugging Face code use the same zero-initialized tanh gates.
Cross-attention fusion (Flamingo, Llama 3.2 Vision) or token concatenation (LLaVA)? Compare.advanced
Cross-attention keeps image tokens out of the LLM sequence: visual tokens are stored only in the cross layers’ KV cache (8 of 40 layers in Llama 3.2 11B) and skip the LLM’s FFNs and self-attention, so prefill and memory are much cheaper with many visual tokens (about 120 times less work per visual token in our 11B example), and the text model can stay frozen, protecting text-only quality. The costs: a new layer type, new parameters, and image information only enters at a few depths. Concatenation is simpler, reuses the LLM fully, and lets every layer mix image and text, but visual tokens cost as much as text tokens, so long videos and high resolutions need token reduction. Many recent open VLMs choose concatenation [Likely]; the right answer depends on token budget, whether text quality must be preserved exactly, and serving cost.
Show that cross-attention is a block inside self-attention over a concatenated sequence. What is different in practice?advanced
For the joint sequence \([Z; X]\), the score matrix has blocks \(Q_Z K_Z^\top, Q_Z K_X^\top, Q_X K_Z^\top, Q_X K_X^\top\); the text rows restricted to the image columns, \(Q_X K_Z^\top\), are cross-attention scores (20.10). If the text rows are masked to see only image columns, the joint attention’s text outputs equal cross-attention exactly (the code checks this). In practice a decoder-only model differs in three ways: one set of weights for both blocks, a single softmax over image and text keys together (they compete for weight), and the image tokens are also processed and updated by every layer.
Cross-attention is permutation invariant over the memory. How does a model know where things are in the image or audio?advanced
Permuting \(K\) and \(V\) rows together permutes the weights the same way, so the weighted sum is unchanged (20.5). Position must be encoded in the memory vectors themselves: the encoder adds positional embeddings (patch positions in a ViT, frame positions in Whisper, learned time embeddings for video frames in Flamingo) before or during its own self-attention. The cross-attention layer usually adds none; Llama 3.2 Vision’s cross-attention applies no RoPE.
How many parameters does a cross-attention layer have if the memory has width \(d_z\) and the model has width \(d\) (no biases, full heads)?core
\(W_Q\) and \(W_O\) are \(d \times d\); \(W_K\) and \(W_V\) are \(d_z \times d\). Total \(2d^2 + 2 d_z d\). With \(d_z = d\) it is \(4d^2\), the same as self-attention: for the Transformer base, \(4 \times 512^2 = 1{,}048{,}576\). With grouped-query attention, \(W_K\) and \(W_V\) shrink to \(d_z \times n_{kv} d_h\).
What does a text token do in Llama 3.2 Vision’s cross-attention when no image comes before it?advanced
Its row of the cross-attention mask is empty, so softmax over no keys is undefined. The Hugging Face implementation detects such fully masked rows and multiplies the cross-attention (and gated MLP) output for those tokens by zero, so the layer acts as the identity for them. For text-only prompts the cross-attention layers are skipped entirely, and the model behaves as the text LLM.

Exercises

  1. A decoder has \(T = 12\) target tokens, a memory of \(M = 50\) encoder tokens, \(d = 512\) and \(h = 8\) heads. What is the shape of the cross-attention score tensor for one example, and of the output?
  2. Finish the worked example: compute the weights and the output for \(\mathbf{q}_2 = (0, 1)\).
  3. Whisper large-v3 in FP16: how much memory does the cross-attention cache need for a batch of 16 audio windows?
  4. Llama 3.2 90B has 100 layers, 20 of them cross-attention, 8 KV heads and head size 128. How many self-attention layers does it have? How large is the cross-attention cache for 6,404 visual tokens in BF16, and how large would it be if the visual tokens were cached in all self-attention layers?
  5. In Figure 5, how many keys does the final “.” see in decoder-only mode, and how many by self- and by cross-attention in Flamingo mode?
  6. Show that if all keys in the memory are equal, the cross-attention output is the mean of the values, whatever the query.
Answerscheck
1. Scores: \(8 \times 12 \times 50\) (heads × queries × keys). Output: \(12 \times 512\), one row per target token. 2. Scores \((0, 2, 1)/\sqrt{2} = (0, 1.414, 0.707)\); weights \((0.140, 0.576, 0.284)\); output \(0.140(1,0) + 0.576(0,1) + 0.284(0.5,0.5) = (0.282, 0.718)\). 3. \(16 \times 245.76\) MB \(= 3{,}932\) MB, about 3.93 GB. 4. \(100 - 20 = 80\) self-attention layers. Cross cache: \(6404 \times 20 \times 4096\) bytes \(= 524.6\) MB. In all 80 self layers: \(6404 \times 80 \times 4096 = 2{,}098.5\) MB, 4 times more. 5. Decoder-only: 14 keys (all positions 0 to 13, itself included). Flamingo: 6 text keys by self-attention and 4 visual keys (image 2) by cross-attention. 6. If \(\mathbf{k}_j = \mathbf{k}\) for all \(j\), every score equals \(\mathbf{q}\cdot\mathbf{k}/\sqrt{d_k}\), so the softmax gives \(a_{ij} = 1/M\) and \(\mathbf{o}_i = \frac{1}{M}\sum_j \mathbf{v}_j\). The query has no effect; attention can only select through differences between keys.

Key takeaways

  • Self- and cross-attention use the same formula (20.1). Only the source of \(K\) and \(V\) differs: the same sequence (20.2) or another one, the memory (20.3).
  • The cross-attention score matrix is \(T \times M\), and the output has one row per query (20.4), so it fits in the residual stream and learned queries can compress any input to a fixed size (20.9).
  • Cross-attention needs no causal mask, only padding or image-selection masks; it is order-blind over the memory (20.5), so positions must live inside the memory vectors.
  • Cost is \(4TMd\) per layer (20.6), and the cross K/V cache is computed once and never grows (20.7): about 245.8 MB per 30-second window in Whisper large-v3.
  • Encoder–decoder layers stack causal self-attention, cross-attention and an FFN (20.8): translation, speech recognition, T5 and BART.
  • Flamingo and Llama 3.2 Vision add zero-gated cross-attention layers (20.11) to a frozen LLM; decoder-only VLMs instead contain cross-attention as one block of a joint self-attention (20.10). Cross-attention is far cheaper when there are thousands of visual tokens.

Key formulas

\[ \text{self: } Q, K, V = XW_Q,\ XW_K,\ XW_V; \qquad \text{cross: } Q = XW_Q,\ \ K = ZW_K,\ \ V = ZW_V; \qquad \mathrm{Attn} = \softmax\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V \]
\[ \text{cost}_\times = 4TMd, \qquad \text{cache}_\times = N L_\times \cdot 2\, n_{kv} d_h\, b, \qquad \mathbf{y} \leftarrow \mathbf{y} + \tanh(\alpha)\,\mathrm{CrossAttn}(\mathbf{y}, X_{\text{vis}}),\ \alpha_0 = 0 \]

References for this session

  1. Vaswani et al. (2017). Attention Is All You Need
  2. Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning
  3. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 6: Core Architectures of Vision Language Models
  4. Bahdanau et al. (2014). Neural Machine Translation by Jointly Learning to Align and Translate
  5. Radford et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision
  6. Raffel et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  7. Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  8. Jaegle et al. (2021). Perceiver: General Perception with Iterative Attention
  9. Li et al. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  10. Llama Team, Meta (2024). The Llama 3 Herd of Models
  11. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning
  12. Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
  13. Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos
  14. Carion, Massa, Synnaeve, Usunier, Kirillov, Zagoruyko (2020). End-to-End Object Detection with Transformers
  15. Lei et al. (2021). QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
  16. Hugging Face (2024). Transformers: modeling_mllama.py (Llama 3.2 Vision implementation)