Multi-head attention
One attention map can only look at one mix of places at a time. Multi-head attention cuts the model width into h smaller pieces, runs h attention maps side by side, and joins the results with one output matrix. This session builds it step by step with real numbers, shows why it costs the same as one big head, counts its 4·dmodel² parameters, and looks at what real heads learn.
What you will learn
- Why one softmax over the keys is a limit: it must share one unit of weight between everything the token needs
- The split: dmodel = h · dk, and the reshape (B, T, d) → (B, h, T, dk) that runs all heads in one matrix multiplication
- Concatenation and the output projection WO, and the equivalent view as a sum of per-head contributions
- Counting: 4·dmodel² weights per layer, and FLOPs that do not depend on the number of heads
- What heads learn in practice: previous-token, syntax, coreference and delimiter heads, and why many heads can be pruned
- A from-scratch PyTorch module checked against nn.MultiheadAttention and scaled_dot_product_attention
- How the same block is used in vision and video transformers
Why this matters
In session 17 each token computed one set of attention weights: one softmax over all the keys. That single map is a hard limit. The weights add up to 1, so if a token needs two different things from two different places, it must split its attention between them, and the two pieces of information arrive mixed together in one vector.
Think of the word it in “The animal did not cross the street because it was tired”. To understand it, the model wants several things at once: which noun it refers to (animal), what the word just before was (because), and what verb comes with it. One attention map has to choose. Multi-head attention runs \(h\) small attention maps in parallel. Each one, called a head, has its own queries, keys and values, so each can look at a different place for a different reason. Then the results are joined.
The surprising part: this costs almost nothing extra. The model width \(d_{\text{model}}\) is cut into \(h\) slices of size \(d_k = d_{\text{model}}/h\), so the total work stays the same as one big head. Every transformer you will meet uses it: the original Transformer (\(h = 8\), \(d_k = 64\)), GPT-2 small and BERT-base (\(h = 12\), \(d_k = 64\)), Llama 2 7B (\(h = 32\), \(d_k = 128\)), and the Vision Transformer ViT-B/16 (\(h = 12\), \(d_k = 64\)). [Certain: from the paper and the published config files]
This session builds the layer from the pieces you know. You will do the full forward pass by hand with real numbers, write it in PyTorch and check it against the built-in module, count its parameters and FLOPs, and see what real heads learn. Multi-head attention is also the starting point of the KV-cache savings in session 26 (MQA, GQA and MLA change how many heads share keys and values).
The limit of a single head
Recall single-head attention from session 17. For queries \(Q\), keys \(K\), values \(V\) and an optional mask \(M\) (session 18):
Each row of the softmax is a set of weights that are positive and add up to 1. The output for a token is a weighted average of the value vectors. An average has two weaknesses when a token needs several things:
- The weight is shared. If the token needs \(m\) different positions equally, each can get at most \(1/m\) of the weight. The rest of the sequence also takes some weight, so it is less than that.
- The information is mixed. The value of position A and the value of position B are added into one vector. Later layers cannot tell which part came from where, unless A and B happened to write into different dimensions.
The authors of the Transformer said it in one sentence: “With a single attention head, averaging inhibits this.” [Certain: Vaswani et al., section 3.2.2] Here “this” is attending to information from different representation subspaces at different positions.
We can measure the first weakness. Say a query has \(n\) keys. It wants \(m\) of them (the targets), and gives each target a score \(g\) higher than the other keys. With one head, the weight on each target is
If instead we have \(m\) heads and each head looks for only one target, each head puts on its target the weight
- Read the symbols
- \(n\)
- number of keys the query can see
- \(m\)
- number of different positions the query needs at the same time
- \(g\)
- how much higher the score of a target is than the score of the other keys (the gap)
- \(p\)
- the softmax weight that lands on one target
With \(n = 8\), \(m = 2\) and \(g = 4\): one head gives each target 0.474; two heads give each target 0.886, each in its own head. Even with a huge gap, the single head never passes 0.5. Try it in Figure 1.
How to read Figure 1
Set \(m = 1\): both sides are equal, because one target needs only one head. Now set \(m = 2\) and push \(g\) to 8: the single head stays just under 0.5 per target, while each of the two heads goes to almost 1.0. With \(m = 4\) the single head is capped near 0.25. Also notice the second weakness: on the left, all targets are summed into one output vector; on the right, each head writes its own slice of the output, so the pieces stay separate.
Multi-head attention: split, attend, concatenate
The idea is simple. Instead of one set of projections \(W_Q, W_K, W_V\) of size \(d_{\text{model}} \times d_{\text{model}}\), give each head \(i = 1, \dots, h\) its own smaller set, of size \(d_{\text{model}} \times d_k\). Each head runs ordinary attention (19.1) in its own small space:
Then the \(h\) outputs are put side by side (concatenated) and multiplied by one more matrix, the output projection \(W_O\):
And the head size is chosen so that the total width does not change:
- Read the symbols
- \(X\)
- the input, of shape \(T \times d_{\text{model}}\): \(T\) token vectors, each of width \(d_{\text{model}}\) (for cross-attention, keys and values come from another sequence; see session 20)
- \(h\)
- number of heads
- \(W_Q^{i}, W_K^{i}, W_V^{i}\)
- the query, key and value projections of head \(i\), each of shape \(d_{\text{model}} \times d_k\); each head has its own, learned separately
- \(d_k, d_v\)
- width of one head’s queries and keys (\(d_k\)) and values (\(d_v\)); almost always equal
- \(\operatorname{Concat}\)
- put the \(h\) matrices of shape \(T \times d_k\) next to each other: the result is \(T \times h d_k = T \times d_{\text{model}}\)
- \(W_O\)
- the output projection: mixes the heads’ results and maps them back into the model’s space
In the original Transformer, \(d_{\text{model}} = 512\) and \(h = 8\), so \(d_k = 64\). [Certain] Notice the scale \(\sqrt{d_k}\) in (19.1) uses the head size \(d_k\), not \(d_{\text{model}}\): each dot product is a sum of \(d_k\) terms.
Shapes: all heads in one matrix multiplication
We never run \(h\) separate small multiplications in real code. Stack the \(h\) small matrices \(W_Q^{i}\) side by side and you get one big \(d_{\text{model}} \times d_{\text{model}}\) matrix \(W_Q = [W_Q^{1} \cdots W_Q^{h}]\). So the first step is exactly the single-head projection \(Q = XW_Q\). Then we only reshape the result. With a batch of \(B\) sequences:
- input: \(X\) of shape \((B, T, d)\) with \(d = d_{\text{model}}\); weights \(W_Q, W_K, W_V, W_O\) of shape \((d, d)\); number of heads \(h\); mask \(M\)
- \(Q, K, V \leftarrow XW_Q,\; XW_K,\; XW_V\) // each \((B, T, d)\): one big matmul each, or one fused \((d, 3d)\) matmul
- reshape each to \((B, T, h, d_k)\) // no data moves: dimension \(d\) is read as \(h\) blocks of \(d_k\)
- transpose to \((B, h, T, d_k)\) // the head axis becomes a batch axis
- \(S \leftarrow Q K^\top / \sqrt{d_k} + M\) // \((B, h, T, T)\): one score map per head
- \(A \leftarrow \softmax(S)\) along the last axis
- \(O \leftarrow A V\) // \((B, h, T, d_k)\)
- transpose back to \((B, T, h, d_k)\), reshape to \((B, T, d)\) // this is the concatenation
- return \(O\, W_O\) // \((B, T, d)\)
The key trick is step 4. After the transpose, the head axis sits next to the batch axis, and a batched matrix multiplication treats “4 sequences × 12 heads” the same as “48 independent sequences”. GPUs are very good at that. Use Figure 3 to see the split and the costs for real models.
How to read Figure 3
Move the heads slider for GPT-2 small. The parameters (2,359,296 per layer) and the FLOPs per token do not change at all: they depend only on \(d_{\text{model}}\) and \(T\). Only two things change: the head size \(d_k\) gets smaller, and the number of \(T \times T\) score maps grows. If the score maps are stored (plain attention, not FlashAttention), their memory grows with \(h\). Now pick Llama 2 7B and \(T = 4096\): 32 heads store about 1.07 GB of BF16 scores per layer for one sequence.
The whole forward pass with real numbers
Let us run every step on a tiny example: \(T = 3\) tokens, \(d_{\text{model}} = 4\), \(h = 2\) heads, so \(d_k = 2\). No mask, so every token can see every token. The weights are chosen by hand so that the two heads do clearly different things.
Figure 4 runs Algorithm 1 on these numbers in your browser, one operation at a time. The strip at the top follows the shape of the tensor through the layer. Below it, each matrix fills in when it is computed: first \(Q\), \(K\), \(V\); then the split into heads (the columns move apart); then, for every head, its score map row by row, its softmax and its output; then the concatenation and \(W_O\). The program on the left highlights the line being run, and the panel on the right writes that line with the live numbers. You can also change the number of heads: the weights stay the same, only the cut changes.
- input \(X\) of shape \((T, d)\); \(d_k \leftarrow d / h\)
- \(Q, K, V \leftarrow XW_Q,\ XW_K,\ XW_V\) // each \((T, d)\)
- view each as \((T, h, d_k)\) // cut the last axis
- transpose to \((h, T, d_k)\) // heads act like a batch
- for head \(i\), row \(t\): \(S_i[t] \leftarrow \mathbf{q}^{i}_t K_i^\top / \sqrt{d_k} + M_t\) // (19.1)
- \(A_i[t] \leftarrow \softmax(S_i[t])\)
- \(\text{head}_i \leftarrow A_i V_i\) // \((T, d_k)\), (19.4)
- \(C \leftarrow\) transpose back, reshape to \((T, d)\) // concat
- return \(Y \leftarrow C\,W_O = \sum_i \text{head}_i W_O^{i}\) // (19.5), (19.7)
How to read Figure 4
Press Play with \(h = 2\). At the “view” step the columns of \(Q\), \(K\) and \(V\) move apart into two blocks of \(d_k = 2\): no number changes. Then head 1 fills its score map: tokens 1 and 2 score highest against themselves (2.828), so after the softmax they keep 0.768 on themselves. Head 2 uses columns 3–4 and gets the crossed pattern: token 1 puts 0.768 on token 2. Token 3 spreads 1/3 everywhere in both heads. The last step writes \(y_1\) as the sum of the two heads’ contributions, (0.616, 0.977, 0.339, −0.745). Now switch to \(h = 1\): one head of size 4 adds the two patterns together, every score becomes \(4/\sqrt 4 = 2\), and every weight is 1/3. The same weights, cut into one head, lose both patterns. With \(h = 4\) (\(d_k = 1\)) each head compares single numbers. Turn on the causal mask: masked scores show \(-\infty\) and get weight 0, and token 1 can only read itself.
Projections. \(\mathbf{q}_1 = (2, 0, 2, 0)\). Head 1 uses \((2, 0)\), head 2 uses \((2, 0)\) as well. The keys of head 1 are \((2,0), (0,2), (1,1)\); the keys of head 2 are \((0,2), (2,0), (1,1)\).
Scores (divided by \(\sqrt{d_k} = \sqrt 2\)). Head 1: \((4, 0, 2)/\sqrt2 = (2.828, 0, 1.414)\). Head 2: \((0, 4, 2)/\sqrt2 = (0, 2.828, 1.414)\).
Softmax. Head 1: \((0.768, 0.045, 0.187)\). Head 2: \((0.045, 0.768, 0.187)\).
Head outputs. The values of head 1 are \((1,0), (0,1), (1,1)\), so \(\text{head}_1 = 0.768(1,0) + 0.045(0,1) + 0.187(1,1) = (0.955, 0.232)\). The values of head 2 are \((2,0), (0,2), (1,1)\), so \(\text{head}_2 = (0.277, 1.723)\).
Concatenate and project. \([0.955, 0.232, 0.277, 1.723]\, W_O = (0.616, 0.977, 0.339, -0.745)\). This \(W_O\) writes the average of the two heads into the first two output dimensions and half their difference into the last two. A real \(W_O\) is learned, but it does the same kind of job: it decides how the heads’ findings are combined.
What \(W_O\) really does: a sum over heads
Concatenation followed by \(W_O\) looks like one step, but we can split it. Cut \(W_O\) into \(h\) row blocks \(W_O^{i}\) of shape \(d_k \times d_{\text{model}}\), one per head. A row vector times a matrix is a sum of (piece of the vector) times (matching rows of the matrix), so
This is a very useful view. Each head computes its result and adds its own contribution, of full width \(d_{\text{model}}\), into the output. In the worked example, for token 1, head 1 adds \((0.477, 0.116, 0.477, 0.116)\) and head 2 adds \((0.139, 0.861, -0.139, -0.861)\). Their sum is the output above. In a real model this output is added to the residual stream (session 22), so every head writes into the same shared vector.
Figure 5 runs this sum view (Algorithm 2 below, written for one token) on a bigger example: \(T = 4\) tokens, \(d_{\text{model}} = 8\), a causal mask, and random weights from a fixed seed. Pick the token and the number of heads. The output \(\mathbf{y}\) starts at zero; each head computes its attention row, its small output \(\text{head}_i\), and then adds \(\text{head}_i W_O^{i}\) as one colored layer of the bars. Only the rows of \(W_O\) that belong to the current head are used in that step.
- \(\mathbf{y} \leftarrow \mathbf{0} \in \R^{d}\)
- for \(i = 1\) to \(h\) do
- \(\mathbf{a} \leftarrow \softmax(\mathbf{q}^{i}_t K_i^\top / \sqrt{d_k} + M_t)\) // row \(t\) of \(A_i\)
- \(\text{head}_i \leftarrow \mathbf{a}\, V_i\) // \(d_k\) numbers
- \(\mathbf{y} \leftarrow \mathbf{y} + \text{head}_i\, W_O^{i}\) // head \(i\)'s rows of \(W_O\)
- check: \(\mathbf{y} = \operatorname{Concat}(\text{head}_1, \dots, \text{head}_h)\, W_O\) // (19.7)
How to read Figure 5
With \(h = 2\) and token 4, head 1 puts 0.811 of its weight on token 1 and head 2 puts 0.578 on token 2: the two heads read different tokens. Head 1 then adds (−0.459, 1.301, −0.693, −0.351, 0.823, −0.739, −1.063, −0.863) and head 2 adds (−0.066, 0.177, −0.055, 0.427, −0.690, −0.198, −0.619, 0.392). Both contributions reach all 8 dimensions, and the last step checks that their sum equals \(\mathbf{c}\,W_O\) up to float rounding. Now tick “Remove \(W_O\)”: head 1 can only write into dimensions 1–4 and head 2 into 5–8, so the heads never mix. Change \(h\) with the weights fixed: the output changes (for \(h = 8\), \(d_k = 1\) and each head owns a single row of \(W_O\)), but the number of weights does not.
Putting (19.4) into (19.7) gives a second insight. Head \(i\) uses its weights only in two products:
- Read the symbols
- \(W_Q^{i} (W_K^{i})^\top\)
- a \(d_{\text{model}} \times d_{\text{model}}\) matrix that decides where head \(i\) looks; called the QK circuit
- \(W_V^{i} W_O^{i}\)
- a \(d_{\text{model}} \times d_{\text{model}}\) matrix that decides what head \(i\) copies from there into the output; called the OV circuit
This is the view of Elhage et al., A Mathematical Framework for Transformer Circuits. Both circuits are products of a \(d_{\text{model}} \times d_k\) and a \(d_k \times d_{\text{model}}\) matrix, so both have rank at most \(d_k\):
So each head is a low-rank attention: it can only compare and copy along \(d_k\) directions. That is the price of many heads. With too many heads, \(d_k\) gets so small that a head cannot express the comparison it needs; Bhojanapalli et al. call this the low-rank bottleneck. The Transformer paper’s own ablation shows the trade-off: [Certain: Table 3, rows A, English–German dev set]
| Heads \(h\) | \(d_k = d_v\) | Dev perplexity | Dev BLEU |
|---|---|---|---|
| 1 | 512 | 5.29 | 24.9 |
| 4 | 128 | 5.00 | 25.5 |
| 8 (base) | 64 | 4.92 | 25.8 |
| 16 | 32 | 4.91 | 25.8 |
| 32 | 16 | 5.01 | 25.4 |
One head is 0.9 BLEU worse than the best; 32 heads of size 16 are worse again. In practice almost all LLMs use \(d_k = 64\) or \(128\) and choose \(h = d_{\text{model}} / d_k\). [Likely]
Counting parameters and FLOPs
Parameters
The four matrices \(W_Q, W_K, W_V, W_O\) are each \(d_{\text{model}} \times d_{\text{model}}\) (stack the heads). The number of heads does not appear:
GPT-2 small (\(d_{\text{model}} = 768\), with biases): \(4 \cdot 768^2 + 4 \cdot 768 = 2{,}362{,}368\) per layer, about 28.3 million over its 12 layers.
Llama 2 7B (\(d_{\text{model}} = 4096\), no biases): \(4 \cdot 4096^2 = 67{,}108{,}864\) per layer. Over 32 layers that is 2.15 billion, about 32% of the model’s 6.74 billion parameters. (Most of the rest is the feed-forward network, session 21.)
FLOPs
A matrix multiply of \((T \times a)\) by \((a \times b)\) costs about \(2Tab\) FLOPs (one multiply and one add per term). Per token, for a sequence of length \(T\):
Where does \(4 T d_{\text{model}}\) come from? Each head computes \(T\) dot products of length \(d_k\) for the scores (\(2Td_k\)) and a weighted sum of \(T\) vectors of length \(d_k\) (\(2Td_k\)). Times \(h\) heads: \(4 T h d_k = 4 T d_{\text{model}}\). Again \(h\) cancels. So more heads are free in FLOPs; what grows is the number of softmax maps (\(h \cdot T^2\) entries) and, on real GPUs, some overhead for small \(d_k\). For Llama 2 7B at \(T = 4096\): \(8 \cdot 4096^2 = 134\) million FLOPs for the projections and \(4 \cdot 4096 \cdot 4096 = 67\) million for the attention part, per token per layer. Session 25 does the full model.
What do real heads learn?
Nobody tells a head what to do. Each head’s matrices start random, and training pushes different heads toward different jobs because that lowers the loss. Studies that look at trained models find heads with clear roles:
- Positional heads: attend to the previous token, or the next one (in BERT). Clark et al. found heads that attend to specific relative positions. [Certain]
- Syntactic heads: a verb attends to its direct object, a noun to its determiner, a preposition to its object. Clark et al. report BERT heads that match these relations well. [Certain]
- Coreference heads: a pronoun attends to the noun it refers to. [Certain: Clark et al. report a head that does coreference well]
- Delimiter and “sink” heads: many heads put most of their weight on [SEP] or [CLS] in BERT, or on the first token in GPT-style models. This acts like a “do nothing” option when the head has nothing useful to read. [Likely: Clark et al. for BERT; Xiao et al. call these attention sinks in LLMs]
- Induction heads: in a sequence like “… A B … A”, a pair of heads in two layers predicts “B” again: copy what came after the last time you saw this token. Olsson et al. connect them to in-context learning. [Likely]
Figure 6 shows a hand-built version of four such heads on one sentence. The patterns are designed by us to look like the ones reported in the papers; they are not read from a trained model.
How to read Figure 6
Click it. The coreference head puts 0.886 of its weight on animal, and the previous-token head puts 0.949 on because. Each head gets a clean answer to its own question. Turn on the average row: one head that tried to do all four jobs would put only 0.243 on animal, mixed with The and because. Click cross or tired: the verb-to-subject head finds animal and it.
Are all heads needed?
No, not after training. Michel et al. found that a large share of heads can be removed at test time without a big loss in quality, and some layers can be reduced to a single head. [Certain] Voita et al. pruned 38 of 48 encoder heads of an English–Russian translation model and lost only 0.15 BLEU; the heads that survive are the specialized ones (positional, syntactic, rare-word heads). [Certain] Michel et al. also give early evidence that many heads help training more than the final model. [Likely] This redundancy is why head pruning works (session 115), and also why sharing keys and values across heads (MQA and GQA, session 26) loses little quality.
Multi-head attention in PyTorch
- input: \(X \in \R^{T \times d}\), per-head weights \(W_Q^{i}, W_K^{i}, W_V^{i} \in \R^{d \times d_k}\), \(W_O^{i} \in \R^{d_k \times d}\), mask \(M\)
- \(Y \leftarrow 0 \in \R^{T \times d}\)
- for \(i = 1\) to \(h\) do // in real code, all heads run at once (Algorithm 1)
- \(Q_i, K_i, V_i \leftarrow XW_Q^{i},\; XW_K^{i},\; XW_V^{i}\) // each \((T, d_k)\)
- \(A_i \leftarrow \softmax(Q_i K_i^\top / \sqrt{d_k} + M)\) // \((T, T)\): where head \(i\) looks
- \(Y \leftarrow Y + A_i V_i W_O^{i}\) // formula (19.7): each head adds its contribution
- return \(Y\)
The module below follows Algorithm 1 with one fused \((d, 3d)\) projection for Q, K and V, as GPT-2 does. It then checks the result three ways: against PyTorch’s fused kernel, against nn.MultiheadAttention with the same weights, and against the sum-over-heads view of Algorithm 2.
import math, torch import torch.nn as nn import torch.nn.functional as F torch.manual_seed(0) class MultiHeadAttention(nn.Module): def __init__(self, d_model, n_heads, bias=False): super().__init__() assert d_model % n_heads == 0, "d_model must be divisible by n_heads" self.h, self.dk = n_heads, d_model // n_heads self.qkv = nn.Linear(d_model, 3 * d_model, bias=bias) # W_Q, W_K, W_V in one matmul self.out = nn.Linear(d_model, d_model, bias=bias) # W_O def forward(self, x, causal=True): B, T, D = x.shape q, k, v = self.qkv(x).split(D, dim=-1) # each (B, T, D) # split heads: (B, T, D) -> (B, T, h, dk) -> (B, h, T, dk) q, k, v = (t.view(B, T, self.h, self.dk).transpose(1, 2) for t in (q, k, v)) s = q @ k.transpose(-2, -1) / math.sqrt(self.dk) # (B, h, T, T), formula (19.1) if causal: mask = torch.triu(torch.ones(T, T, dtype=torch.bool), 1) s = s.masked_fill(mask, float("-inf")) a = s.softmax(dim=-1) # one T x T map per head o = a @ v # (B, h, T, dk) o = o.transpose(1, 2).reshape(B, T, D) # concat heads, formula (19.5) return self.out(o), a B, T, D, H = 2, 5, 16, 4 x = torch.randn(B, T, D) mha = MultiHeadAttention(D, H) y, a = mha(x) print(y.shape, a.shape) # torch.Size([2, 5, 16]) torch.Size([2, 4, 5, 5]) # 1) the same result from PyTorch's fused kernel q, k, v = mha.qkv(x).split(D, dim=-1) q, k, v = (t.view(B, T, H, D // H).transpose(1, 2) for t in (q, k, v)) o = F.scaled_dot_product_attention(q, k, v, is_causal=True) y2 = mha.out(o.transpose(1, 2).reshape(B, T, D)) print((y - y2).abs().max().item()) # about 6e-08: equal up to float rounding # 2) the same result from nn.MultiheadAttention with copied weights ref = nn.MultiheadAttention(D, H, bias=False, batch_first=True) with torch.no_grad(): ref.in_proj_weight.copy_(mha.qkv.weight) # stacked (3D, D): q, k, v rows ref.out_proj.weight.copy_(mha.out.weight) mask = torch.triu(torch.ones(T, T, dtype=torch.bool), 1) # True = not allowed y3, _ = ref(x, x, x, attn_mask=mask, need_weights=False) print((y - y3).abs().max().item()) # about 6e-08 # 3) concat-then-W_O equals a sum over heads of head_i @ W_O^i, formula (19.7) Wo = mha.out.weight.T # (D, D); nn.Linear stores W transposed heads = a @ v # (B, h, T, dk) y4 = sum(heads[:, i] @ Wo[i * (D // H):(i + 1) * (D // H)] for i in range(H)) print((y - y4).abs().max().item()) # about 6e-08 # 4) parameter count, formula (19.10): 4 * d_model^2 without biases print(sum(p.numel() for p in mha.parameters()), 4 * D * D) # 1024 1024
Two details trip people up. First, view then transpose is right, but view(B, h, T, dk) directly is wrong: it would cut the sequence into pieces instead of cutting each token’s vector. Second, after the transpose the tensor is not contiguous in memory, so going back needs reshape (or contiguous().view).
Where this is used
| Setting | Typical heads | What the heads do there |
|---|---|---|
| Decoder LLMs (GPT-2, Llama) | GPT-2 small: 12 heads of 64; Llama 2 7B: 32 heads of 128 [Certain] | Causal self-attention in every layer; newer models share K and V across heads (GQA) to shrink the KV cache (session 26) |
| Encoders for classification and retrieval (BERT, sentence embedders) | BERT-base: 12 heads of 64 [Certain] | Bidirectional attention; syntactic and coreference heads found by probing (Clark et al.) |
| Vision Transformer (images) | ViT-B/16: 12 heads of 64 [Certain] | Patches attend to patches; some heads stay local like a small convolution, others look across the whole image (Dosovitskiy et al.) (session 164) |
| Video encoders for action recognition and video classification | Same block, on space–time patch tokens | Some designs split heads or layers into spatial and temporal attention, so a head can follow motion across frames [Likely] (session 202) |
| Text-to-video retrieval and Video-RAG | Cross-attention heads in fusion layers | Text queries attend to frame tokens; different heads can pick objects, actions or on-screen text [Speculation about exact roles] (session 208) |
| Key-moment detection and temporal grounding | Heads in a temporal transformer over frame features | Heads compare a moment with nearby and far frames to find boundaries (session 207) |
| Model compression | Prune heads, or share K and V | Redundant heads (Michel et al., Voita et al.) are removed or merged |
Common mistakes
- Wrong reshape.
x.view(B, h, T, dk)instead ofx.view(B, T, h, dk).transpose(1, 2). Both have the right shape, so nothing crashes, but the first mixes tokens. Always split the last axis. - Scaling by \(\sqrt{d_{\text{model}}}\) instead of \(\sqrt{d_k}\). The scores become \(\sqrt h\) times too small, and attention is too flat at the start of training.
- Forgetting \(W_O\), or thinking it is optional. Without it, head \(i\) can only write into its own \(d_k\) slice of the output, and heads cannot be combined.
- Thinking heads add compute or parameters. With \(d_k = d_{\text{model}}/h\), both stay at \(4d^2\) parameters and \(8d^2 + 4Td\) FLOPs per token (19.10, 19.11). What grows is the number of score maps.
- Choosing \(h\) that does not divide \(d_{\text{model}}\). The reshape fails. (Some models pick \(d_k\) first and let \(h\,d_k \ne d_{\text{model}}\); then \(W_O\) is \(h d_k \times d_{\text{model}}\), as in 19.5.)
- Mask with the wrong shape. A mask of shape \((B, T, T)\) must become \((B, 1, T, T)\) to broadcast over heads.
nn.MultiheadAttentionwants \((B \cdot h, T, T)\) for 3-D masks. - Mixing the mask conventions. In
nn.MultiheadAttention, a booleanTruemeans “do not attend”; inF.scaled_dot_product_attention, a booleanTruemeans “attend”. Test with a tiny example. - Reading too much into one head’s map. Where a head looks is not what it contributes (see the OV circuit in 19.8).
Interview questions
What is multi-head attention, in two sentences?basic
Why use several heads instead of one head of size \(d_{\text{model}}\)?core
How many parameters does one multi-head attention layer have? Does it depend on h?basic
How does the number of heads change FLOPs and memory?core
What does \(W_O\) do? Could you drop it?core
Write the reshape that splits heads, and explain why the order matters.core
q.view(B, T, h, dk).transpose(1, 2) gives \((B, h, T, d_k)\). The view must cut the last axis (each token’s vector) into \(h\) blocks; view(B, h, T, dk) has the same shape but cuts the flat memory differently, mixing tokens across heads. After attention: o.transpose(1, 2).reshape(B, T, d); reshape (or contiguous()) is needed because the transpose makes the tensor non-contiguous.Why scale by \(\sqrt{d_k}\) and not \(\sqrt{d_{\text{model}}}\)?basic
What is the low-rank bottleneck of attention heads?advanced
Are all heads necessary? What does the pruning literature say?advanced
Explain the QK and OV circuits of a head.advanced
How is multi-head attention related to MQA and GQA?core
Exercises
- A model has \(d_{\text{model}} = 1024\) and \(h = 16\). What is \(d_k\)? How many weights do \(W_Q, W_K, W_V, W_O\) have together, without biases?
- With \(n = 8\) keys, \(m = 2\) targets and \(g = 6\), compute the weight per target for one head (19.2) and for one head per target (19.3). Check in Figure 1.
- In the worked example, compute the output of head 1 and head 2 for token 2. Use symmetry.
- Show that changing \(h\) from 8 to 16 with \(d_{\text{model}} = 512\) does not change the FLOPs of \(QK^\top\) for one token and \(T = 1000\).
- For Llama 2 7B at \(T = 2048\), how many bytes do the BF16 score maps of one layer need for one sequence, if they are stored?
- You want each head to write into the full output, but you remove \(W_O\) and keep the concatenation. Which output dimensions can head 3 change, with \(d_{\text{model}} = 768\) and \(h = 12\)?
- In Figure 4, set \(h = 1\) and play to the end. Why does every token get the weights (1/3, 1/3, 1/3) and the same output, although the same weights give two sharp patterns with \(h = 2\)?
Answerscheck
Key takeaways
- One softmax shares one unit of weight and averages the values; a token that needs \(m\) things gets less than \(1/m\) for each (19.2). Several heads give several independent attention maps (19.3).
- Multi-head attention: project, split into \(h\) heads of size \(d_k = d_{\text{model}}/h\), attend in each head, concatenate, multiply by \(W_O\) (19.4–19.6).
- In code it is one big projection plus a reshape \((B, T, d) \to (B, h, T, d_k)\); the head axis acts like a batch axis.
- Concatenation then \(W_O\) is a sum of per-head contributions (19.7); each head has a QK circuit (where to look) and an OV circuit (what to copy), both of rank at most \(d_k\) (19.8, 19.9).
- Parameters are \(4d_{\text{model}}^2\) and FLOPs are \(8d^2 + 4Td\) per token, whatever \(h\) is (19.10, 19.11). Too many heads make \(d_k\) too small; 64 or 128 is common.
- Trained heads specialize (previous token, syntax, coreference, sinks, induction), and many of them can be pruned after training.
Key formulas
References for this session
- Vaswani et al. (2017). Attention Is All You Need
- Elhage et al. (2021). A Mathematical Framework for Transformer Circuits
- Raschka, Build a Large Language Model (From Scratch), chapter 3: Coding attention mechanisms
- Michel, Levy, Neubig (2019). Are Sixteen Heads Really Better than One?
- Voita, Talbot, Moiseev, Sennrich, Titov (2019). Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- Clark, Khandelwal, Levy, Manning (2019). What Does BERT Look At? An Analysis of BERT's Attention
- Bhojanapalli, Yun, Rawat, Reddi, Kumar (2020). Low-Rank Bottleneck in Multi-head Attention Models
- Olsson et al. (2022). In-context Learning and Induction Heads
- Xiao et al. (2023). Efficient Streaming Language Models with Attention Sinks
- Radford et al. (2019). Language Models are Unsupervised Multitask Learners
- Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need
- Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness