Attention intuition: query, key, value
Attention lets every token pull in information from other tokens. This session builds the idea from a plain dictionary lookup: a query is what a token looks for, a key is the label each token shows, and a value is what it hands over when it is chosen. We see why a transformer learns three separate projections for these roles, and why the attention weights of one token always form a probability distribution over positions.
What you will learn
- Attention as a soft dictionary lookup: a hard lookup returns one value, a soft lookup returns a weighted average of all values
- The three roles of a token: query (what I look for), key (the label I show), value (what I give if chosen)
- How q, k and v are made from the same token vector by three learned matrices, with a worked example in real numbers
- Why three different projections: without them tokens look mostly at themselves, and one shared matrix forces symmetric scores
- Attention weights as a probability distribution over positions: sum to one, shift invariance, entropy and effective number of positions
- How sharpness, query length and the number of competing tokens change where attention goes
- Where the same query, key, value pattern appears in LLMs, VLMs and video search
Why this matters
Every LLM and every VLM is built from one operation repeated many times: attention. In a 32-layer model with 32 heads per layer, attention runs 1,024 times for every token. If you understand what one attention step does, you understand the core of the transformer.
The problem attention solves is simple to say. After the embedding layer (session 11), each token has its own vector, but that vector knows nothing about the other tokens. In “the animal crossed the street because it was tired”, the vector for “it” is the same as in any other sentence. To understand the sentence, “it” must pull in information from “animal”. Attention is the mechanism that decides, for each token, which other tokens to read from and how much.
In Session 7 we met the first attention: a decoder that looks back at all encoder states. The transformer (Vaswani et al., 2017) made this the only way tokens talk to each other, and gave its parts three names: query, key and value. These names come from databases and dictionaries, and that is the best way to understand them. This session builds the intuition; session 17 then gives the full matrix math, the \(\sqrt{d_k}\) scale and the shapes.
Interviewers ask about this all the time: “Explain attention in one minute”, “What are Q, K and V?”, “Why not use the token vector directly?”, “Why do the weights sum to one, and what problems does that cause?”. These are not trick questions, but a vague answer is easy to spot. By the end you can answer each one with a formula and a small example in numbers.
A dictionary, made soft
A hard lookup
A Python dictionary stores pairs (key, value). You give it a query, it finds the key that is exactly equal, and it returns that key’s value:
This lookup is hard: it returns exactly one value, and if no key matches exactly, it fails. It also cannot be trained by gradient descent. A small change in the query either changes nothing or jumps to a different value, so the gradient is zero almost everywhere.
A soft lookup
Now make three changes. (1) Keys and queries are vectors, and instead of “equal or not” we compute a score that says how well they match. The dot product from session 2 is the usual choice. (2) We turn the scores into weights with a softmax, so the weights are positive and sum to 1. (3) We return a weighted average of all values instead of one value:
- Read the symbols
- \(\mathbf{q}\)
- the query: a vector that describes what we are looking for
- \(\mathbf{k}_j\)
- the key of entry \(j\): a vector that describes what entry \(j\) is about. It is only used to compute scores.
- \(\mathbf{v}_j\)
- the value of entry \(j\): the content we get back from entry \(j\). It is only used in the weighted sum.
- \(s_j\)
- the score (also called the compatibility or the logit) of entry \(j\): large when \(\mathbf{q}\) and \(\mathbf{k}_j\) point the same way
- \(a_j\)
- the attention weight of entry \(j\): a number between 0 and 1; all \(n\) weights add up to 1
- \(n\)
- the number of entries (in a transformer: the number of positions the token can see)
This is the definition in the transformer paper, in words: an attention function maps a query and a set of key–value pairs to an output; the output is a weighted sum of the values, and each weight comes from a compatibility function of the query with the matching key. [Certain] The hard lookup is a special case: if one score is much larger than all the others, the softmax puts almost all weight on it, and (16.3) returns almost exactly that one value.
The soft lookup fixes all three problems of the hard one. It never fails, because there is always a best partial match. It can mix several entries when the query is between them. And every step is smooth, so gradients flow back into the query, the keys and the values. This is why attention can be trained end to end.
How to read Figure 1
The left side shows the query arrow and the five key arrows. The bars on the right are the attention weights \(a_j\); they always add up to 1. The readout gives the scores, the weights and the output \(\sum_j a_j v_j\). Start at 30° (orange, halfway between red and yellow): red and yellow get 0.475 each and the output is 632.8 nm, between their wavelengths, a sensible answer for a color that is not in the dictionary. Set the length to 0: every score is 0 and the output is the plain average of all values, 538 nm. Now move the direction to 50° and the length to 8: yellow, the closest key, gets 0.934 of the weight. Tick “Hard lookup” to compare with a normal dictionary: its output jumps when the query crosses the middle between two keys, while the soft output moves smoothly.
Figure 1 shows a fact we will use again: the length of the query acts like a sharpness knob. Since \(\mathbf{q} \cdot \mathbf{k}_j = \lVert\mathbf{q}\rVert\,\lVert\mathbf{k}_j\rVert \cos\theta_j\), doubling the query doubles every score, and a softmax of doubled scores is sharper. A model can learn to make some queries long (to pick one position) and others short (to average over many).
An old idea with a new name
Weighted averages of stored values are much older than transformers. In statistics, kernel regression (Nadaraya and Watson, 1964) predicts a value at a new point as an average of known values, weighted by how close each known point is. Formula (16.3) is exactly that, with the kernel \(\exp(\mathbf{q} \cdot \mathbf{k})\). [Certain] In neural networks, memory networks used the same split: Miller et al. (2016), in “Key-Value Memory Networks”, used one encoding of each memory slot to find it and a different encoding to read from it. [Certain] What the transformer added is that the queries, keys and values are all learned from the tokens themselves, at every layer.
Query, key, value: three roles for one token
In self-attention, the dictionary is the sentence itself. Every token is at the same time a reader and an entry:
- As a reader, token \(i\) has a query \(\mathbf{q}_i\): what am I looking for? For “it”, something like “a noun earlier in the sentence that can be tired”.
- As an entry, token \(j\) has a key \(\mathbf{k}_j\): the label I show to others. For “animal”, something like “I am a noun, I am a living thing”.
- Also as an entry, token \(j\) has a value \(\mathbf{v}_j\): what I give if chosen. For “animal”, the information that the reader should copy, for example its meaning.
All three are made from the same token vector \(\mathbf{x}_t\) (a row vector of size \(d\)) by three learned matrices:
- Read the symbols
- \(\mathbf{x}_t\)
- the vector of the token at position \(t\) as it enters the attention layer (in layer 1, its embedding plus position information; in later layers, the output of the layer before)
- \(W_Q \in \R^{d \times d_k}\)
- the query projection: turns a token vector into a query of size \(d_k\)
- \(W_K \in \R^{d \times d_k}\)
- the key projection. Queries and keys must have the same size \(d_k\), because we take their dot product.
- \(W_V \in \R^{d \times d_v}\)
- the value projection. \(d_v\) may differ from \(d_k\); in most models \(d_v = d_k\).
- \(d, d_k, d_v\)
- the model width and the sizes of queries/keys and of values. In the original transformer, \(d = 512\) and \(d_k = d_v = 64\) per head. [Certain]
The matrices are the same for every position: “it” and “animal” use the same \(W_Q\). What differs is the input vector. So the model does not learn a query for each word; it learns a rule that turns any token vector into a question, a label and a message.
Now watch one reader do the whole lookup. Figure 3 runs Algorithm 1 (in the pseudo-code section below) on the sentence “the animal crossed the street because it was tired”, one line at a time, with real numbers. Each token has 6 hand-made features: noun, living thing, verb, function word, pronoun, adjective. The key matrix \(W_K\) copies two of them, “noun” and “living thing” (times 1.5), so the keys say “I am a noun” and “I am alive”. The query matrix \(W_Q\) lets verbs ask for nouns, the adjective ask for living things, and the pronoun ask for both. The value matrix \(W_V\) sends each token to a point in a 2-dimensional “value plane”, so we can draw the output as a point that moves.
- input: \(\mathbf{q} = \mathbf{x}_{\text{reader}} W_Q\); keys \(\mathbf{k}_j\), values \(\mathbf{v}_j\); \(\beta\) // (16.4)
- for \(j = 1 \dots n\): \(s_j \leftarrow \beta\, \mathbf{q} \cdot \mathbf{k}_j\) // (16.2)
- \(m \leftarrow \max_j s_j\) // allowed by (16.8)
- for \(j = 1 \dots n\): \(e_j \leftarrow \exp(s_j - m)\)
- \(a_j \leftarrow e_j / \sum_m e_m\) // sums to 1
- \(\mathbf{o} \leftarrow \mathbf{0}\); for \(j = 1 \dots n\): \(\mathbf{o} \leftarrow \mathbf{o} + a_j \mathbf{v}_j\) // (16.3)
- return \(\mathbf{o}\)
How to read Figure 3
Press Play, or Step through it. Steps 2 to 10 compute one score each: watch the keys of “animal” and “street” light up for the reader “it”, because they show “noun”, which “it” asks for. Steps 11 to 13 turn the scores into weights: “it” gives 0.845 to “animal”, 0.089 to “street” and 0.009 to each of the other 7 tokens. Steps 14 to 22 add the values one by one: the output point jumps toward “animal” at j = 2 and ends at (1.812, 1.662), close to the value of “animal”, (2, 2), and far from the value of “it” itself, (−2, 0). Now pick “crossed”: its query asks only for “noun”, so “animal” and “street” tie at 0.365 each and the output, (1.230, 0.615), lands between them. Pick “animal”: its query is zero, every weight is 1/9 = 0.111 and the output is the plain average of all values. Raise \(\beta\) to 2 and “it” gives 0.988 to “animal”: almost a hard lookup.
Put the pieces together for one reading token \(i\). The score with token \(j\) is the query of \(i\) times the key of \(j\), and by (16.4) it can be written with the raw token vectors:
- Read the symbols
- \(s_{ij}\)
- how much reader \(i\) wants to read from token \(j\), before the softmax
- \(W_Q W_K^\top\)
- a \(d \times d\) matrix that defines the “compatibility” between two token vectors. It is learned, but only through its two thin factors.
Three tokens, “the cat it”, with \(d = 3\) features (function word, noun, pronoun) and \(d_k = d_v = 2\). The token vectors are \(\mathbf{x}_{\text{the}} = (1, 0, 0)\), \(\mathbf{x}_{\text{cat}} = (0, 1, 0)\) and \(\mathbf{x}_{\text{it}} = (0, 0.5, 1)\) (“it” is mostly a pronoun and a bit noun-like). The three matrices, written by their rows, are
\(W_Q\): rows \((0, 0), (0, 1), (2, 0)\); \(\quad W_K\): rows \((0, 0), (2, 0), (0, 1)\); \(\quad W_V\): rows \((1, 0), (0, 2), (0, 0)\).
Project. \(\mathbf{q}_{\text{it}} = 0.5\cdot(0, 1) + 1\cdot(2, 0) = (2, 0.5)\). The keys are \(\mathbf{k}_{\text{the}} = (0, 0)\), \(\mathbf{k}_{\text{cat}} = (2, 0)\), \(\mathbf{k}_{\text{it}} = (1, 1)\). The values are \(\mathbf{v}_{\text{the}} = (1, 0)\), \(\mathbf{v}_{\text{cat}} = (0, 2)\), \(\mathbf{v}_{\text{it}} = (0, 1)\).
Score. \(\mathbf{q}_{\text{it}} \cdot \mathbf{k}_j = 0,\ 4,\ 2.5\) for the, cat, it.
Softmax. \(e^0 = 1\), \(e^4 = 54.60\), \(e^{2.5} = 12.18\); the sum is 67.78, so the weights are \(0.0148,\ 0.8055,\ 0.1797\). They add up to 1.
Mix. \(\mathbf{o}_{\text{it}} = 0.0148\,(1, 0) + 0.8055\,(0, 2) + 0.1797\,(0, 1) = (0.0148,\ 1.7908)\). The new vector for “it” is mostly the value of “cat”.
Without projections (query = key = token vector): the scores of “it” would be \(\mathbf{x}_{\text{it}} \cdot \mathbf{x}_j = 0,\ 0.5,\ 1.25\), and the largest weight, 0.5685, would go to “it” itself. The projections are what turn “it” toward “cat”.
- input: \(X\) (\(3 \times 3\)), \(W_Q, W_K, W_V\) (\(3 \times 2\))
- for \(t\): \(\mathbf{q}_t \leftarrow \mathbf{x}_t W_Q\); \(\mathbf{k}_t \leftarrow \mathbf{x}_t W_K\); \(\mathbf{v}_t \leftarrow \mathbf{x}_t W_V\) // (16.4)
- for each reader \(i\):
- for each \(j\): \(s_{ij} \leftarrow \mathbf{q}_i \cdot \mathbf{k}_j\) // (16.5)
- \(\mathbf{a}_i \leftarrow \softmax(\mathbf{s}_i)\) // (16.2)
- \(\mathbf{o}_i \leftarrow \sum_j a_{ij}\, \mathbf{v}_j\) // (16.3)
- return \(O\)
- compare: \(Q = K = X\), no projections
How to read Figure 4
Press Play or Step. Blue cells are queries, orange cells keys, green cells values. The box under the tables writes out the formula of the current line with the numbers that go in. Follow reader “it” (steps 15 to 19): the scores 0, 4 and 2.5 appear one by one, the softmax turns them into 0.0148, 0.8055 and 0.1797, and the output is (0.0148, 1.7908). Check two things: every row of weights adds up to 1, and the row of “the” is flat (1/3 each) because its query is \((0, 0)\): a token with nothing to ask simply averages. The last step shows the “no projections” weights next to the real ones: there “it” reads itself most (0.5685).
Why three different projections
Why not skip the matrices and use the token vector itself as query, key and value? Or use one matrix for all three? Each simpler choice breaks something. Figure 5 lets you compare them on one sentence.
No projections: every token looks at itself
If \(\mathbf{q}_i = \mathbf{k}_i = \mathbf{x}_i\), the score of a token with itself is \(\mathbf{x}_i \cdot \mathbf{x}_i = \lVert\mathbf{x}_i\rVert^2\). By the Cauchy–Schwarz inequality, \(\mathbf{x}_i \cdot \mathbf{x}_j \le \lVert\mathbf{x}_i\rVert\,\lVert\mathbf{x}_j\rVert\), so when token vectors have similar lengths, the self-score is the largest score in the row. Each token mostly reads itself, plus tokens that are similar to it. But attention is useful exactly when a token must read something different from itself: “it” must read “animal”, a verb must find its subject, a question must find the answer in a document.
One shared matrix: scores are forced to be symmetric
With one matrix \(W\) for both queries and keys, every token is mapped to one vector \(\mathbf{a}_t = \mathbf{x}_t W\), and
So “how much ‘it’ wants to read ‘animal’” must equal “how much ‘animal’ wants to read ‘it’”. Language is full of one-way relations: a pronoun looks for its noun, but the noun has no reason to look for the pronoun; an adjective looks for the noun it describes; a closing bracket looks for the opening one. A symmetric score cannot express “A needs B, B does not need A”. The self-score problem also stays: \(s_{ii} = \lVert\mathbf{a}_i\rVert^2\) is still the largest unless another vector is longer.
Separate \(W_Q\) and \(W_K\): a learned, one-way compatibility
With two matrices, the score (16.5) uses the matrix \(M = W_Q W_K^\top\), which can be any \(d \times d\) matrix of rank at most \(d_k\), symmetric or not. The query side can ask about one set of features (“pronoun-ness”) while the key side answers with a different set (“noun-ness”, “living thing”). The self-score is no longer special.
The factored form is also cheap. A full \(d \times d\) matrix for GPT-2 small (\(d = 768\)) would have 589,824 numbers per head. The two factors with \(d_k = 64\) have \(2 \times 768 \times 64 = 98{,}304\). [Certain] The low rank is not a loss: one head only needs to match a few features, and session 19 runs many such heads side by side.
Separate \(W_V\): what you match on is not what you pass on
The key is an address; the value is the content. They are needed for different jobs. To be found by “it”, “animal” should advertise “noun, living thing”. But what “it” should copy is different: maybe the full meaning of “animal”, or only the fact that it is singular. A separate \(W_V\) lets the model choose what to send, independently of how it is found. This is the same split as in key–value memory networks, and it is why the KV cache (session 80) stores two tensors, not one.
How to read Figure 5
Row \(i\) is the attention distribution of reader \(i\) over the 9 tokens. Click a row (a word on the left) to see its weights in the readout. With no projections, the darkest cell in most rows is on the diagonal: each token reads itself, and “it” gives itself 0.254 and “animal” only 0.093. With one shared W, the matrix is mirror-symmetric, and “it” gives the same weight, 0.190, to itself and to “animal”. With separate projections, “it” puts 0.845 on “animal” and only 0.089 on “street”, “tired” finds “animal” too, while “animal” itself has nothing to ask and reads every token equally (1/9 each). The readout also reports the largest gap \(\lvert s_{ij} - s_{ji} \rvert\) between mirrored scores.
Attention weights are a probability distribution
For a fixed reader \(i\), the softmax turns its \(n\) scores into \(n\) weights. These weights have exactly the properties of a probability distribution over positions:
- Read the symbols
- \(\mathbf{a}_i\)
- row \(i\) of the attention matrix: the distribution of reader \(i\) over the positions \(j = 1, \dots, n\)
- \(\E_{j \sim \mathbf{a}_i}[\mathbf{v}_j]\)
- the expected value of \(\mathbf{v}_j\) when the position \(j\) is drawn at random with probabilities \(\mathbf{a}_i\). The output is this average.
Read it as a story: reader \(i\) picks one position at random, with the probabilities \(\mathbf{a}_i\), and takes its value. The output is the average of what it would get. Four consequences matter in practice.
1. The output stays inside the values. A weighted average with positive weights that sum to 1 lies in the convex hull of the values (the smallest shape that contains all of them). Attention can mix and select, but it cannot amplify: one head cannot make a value twice as large. Making new things is the job of the MLP that follows (session 21).
2. Only score differences matter. Adding the same number \(c\) to all scores of a row changes nothing, because the factor \(e^c\) cancels:
A direct result: a bias on the keys is useless. If \(\mathbf{k}_j = \mathbf{x}_j W_K + \mathbf{b}\), then \(\mathbf{q}_i \cdot \mathbf{k}_j = \mathbf{q}_i \cdot \mathbf{x}_j W_K + \mathbf{q}_i \cdot \mathbf{b}\), and the second term is the same for every \(j\) in row \(i\), so it cancels. [Certain] Many modern models drop the biases in these projections; the Llama 3 8B configuration sets attention_bias to false. [Certain]
3. Sharpness: from a hard pick to a plain average. Multiply all scores by a number \(\beta \ge 0\) (an inverse temperature; a longer query does exactly this, as in Figure 1):
To measure how spread out a row is, use its entropy, and turn it into an effective number of positions:
\(n_{\text{eff}} = 1\) means the reader looks at one position (a hard lookup); \(n_{\text{eff}} = n\) means it averages all positions equally. In the worked example, the row of “it” has entropy 0.545 nats, so \(n_{\text{eff}} = 1.72\): mostly “cat”, a little “it”. The row of “the” has \(H = \log 3 = 1.099\) and \(n_{\text{eff}} = 3\). Why the transformer divides scores by \(\sqrt{d_k}\) is a question about \(\beta\): without it, scores in large dimensions are so big that every row becomes a hard pick. That is the subject of session 17.
4. Weights must go somewhere. The weights must sum to 1, even when nothing in the context is useful. Two effects follow. First, dilution: every extra position takes a share. In Figure 6 the row of “it” puts 0.845 on “animal” in the 9-token sentence; add 100 unrelated tokens with score 0 and the same scores give “animal” only 0.436; with 1,000 such tokens, 0.081. This is one reason long contexts are hard (session 28). Second, attention sinks: trained LLMs often put a large weight on the first token of the sequence even when it carries little meaning, as a place to park weight they do not need. Xiao et al. (2023) found that keeping the keys and values of these first tokens is what makes sliding-window generation work on very long streams. [Certain]
How to read Figure 6
Each bar is one weight; the last bar sums all the extra tokens together. The readout shows the weight on “animal”, the entropy \(H\) and \(n_{\text{eff}}\) from (16.10), and the total, which is always 1. Move \(\beta\) to 0: all weights become \(1/9 = 0.111\) and \(n_{\text{eff}} = 9\). Move it to 3: “animal” gets 0.999. Move the constant \(c\): nothing changes at all. Add 100 extra tokens: “animal” drops to 0.436 although its score did not change; raise \(\beta\) to win it back. A sharper query is how a trained model fights dilution.
A distribution, not an explanation
Because attention weights look like probabilities over the input, it is tempting to read them as “what the model used”. Be careful. The output also depends on the values (a large weight on a small value matters little), there are many heads and layers mixing information, and Jain and Wallace (2019) showed that very different attention patterns can often give the same prediction. [Certain] Attention maps are a useful first look, not a proof. Better tools are covered in session 157.
All positions at once
A transformer does the soft lookup for every reader at the same time. Stack the token vectors as rows of \(X \in \R^{n \times d}\); then \(Q = XW_Q\), \(K = XW_K\), \(V = XW_V\), and one matrix product computes every score. The transformer also divides the scores by \(\sqrt{d_k}\):
Row \(i\) of \(O\) is formula (16.3) for reader \(i\). This is formula (2.20) from session 2, now with a meaning for each letter. Two more pieces turn it into what LLMs use: a causal mask so that a token cannot read the future (session 18), and many heads, each with its own \(W_Q, W_K, W_V\) (session 19). Real sizes:
| Model | Width \(d\) | Heads | \(d_k\) per head | Notes |
|---|---|---|---|---|
| Transformer base (2017) | 512 | 8 | 64 | \(d_k = d_v = d/h\) [Certain] |
| GPT-2 small | 768 | 12 | 64 | one fused \(768 \times 2304\) matrix computes Q, K and V together (\(3 \times 768 = 2304\)), with biases [Certain] |
| Llama 3 8B | 4,096 | 32 query heads, 8 key/value heads | 128 | no projection biases; keys and values shared across groups of 4 query heads (session 26) [Certain] |
Sizes from the original paper and the public model configurations.
Beyond one sentence: images, video and memory
Nothing in (16.2)–(16.3) says that queries, keys and values must come from the same sequence. That freedom is what makes attention the glue of multimodal models.
- Self-attention: Q, K and V all come from the same tokens. This is what we used above, and what every LLM layer does.
- Cross-attention: Q comes from one sequence and K, V from another. In a translation model, the decoder asks and the encoder answers (session 20). In a VLM with cross-attention layers, text tokens ask and image patches answer: the query of “red” is compared with the keys of all patches, and the values of the red patches flow into the text (session 179).
- Images in self-attention: In the Vision Transformer (session 164) each \(16 \times 16\) patch is a token with its own query, key and value; a patch of a dog’s ear can read the patch of its nose. [Certain]
- Video: Frames become many tokens. A text query such as “the moment the goal is scored” can be matched against keys of frame tokens, and the attention weights over time act as a soft pointer to the moment (session 207).
- Memory: At generation time, the keys and values of past tokens do not change, so they are stored and reused; only the new token computes a fresh query. This is the KV cache (session 80): a literal dictionary of keys and values that grows with every token.
Where this is used
| Application | Query | Keys and values | Link |
|---|---|---|---|
| Every LLM layer (decoder self-attention) | Each token | All earlier tokens (causal mask) | session 18 |
| Fast generation | The newest token only | Cached K, V of all earlier tokens | session 80 |
| VLM with cross-attention layers | Text tokens | Image patch features | session 179 |
| Image classification and detection (ViT) | Each patch (or a class token) | All patches | session 164 |
| Video action recognition | Each space–time patch | Patches in the same frame and in other frames | session 202 |
| Key-moment detection, temporal grounding | A text description of the moment | Frame or clip features over time | session 207 |
| Video-RAG and video question answering | Question tokens | Retrieved frames, captions and transcript chunks | session 210 |
| Text-to-video retrieval (re-ranking) | Caption tokens | Frame tokens of a candidate clip | session 208 |
Pseudo-code
- input: query \(\mathbf{q}\), keys \(\mathbf{k}_1, \dots, \mathbf{k}_n\), values \(\mathbf{v}_1, \dots, \mathbf{v}_n\), sharpness \(\beta\)
- for \(j = 1\) to \(n\) do \(s_j \leftarrow \beta\, \mathbf{q} \cdot \mathbf{k}_j\) // compatibility, formula (16.2)
- \(m \leftarrow \max_j s_j\) // subtract the max for stability; allowed by (16.8)
- for \(j = 1\) to \(n\) do \(e_j \leftarrow \exp(s_j - m)\)
- \(a_j \leftarrow e_j / \sum_m e_m\) for all \(j\) // a distribution over the entries
- return \(\sum_j a_j \mathbf{v}_j\) // weighted average of values, formula (16.3)
- input: token vectors \(\mathbf{x}_1, \dots, \mathbf{x}_n\), matrices \(W_Q, W_K, W_V\)
- for \(t = 1\) to \(n\) do \(\mathbf{q}_t \leftarrow \mathbf{x}_t W_Q;\ \mathbf{k}_t \leftarrow \mathbf{x}_t W_K;\ \mathbf{v}_t \leftarrow \mathbf{x}_t W_V\) // formula (16.4)
- for each reader \(i = 1\) to \(n\) do
- \(\mathbf{o}_i \leftarrow\) Algorithm 1 with query \(\mathbf{q}_i\), all keys and values, \(\beta = 1/\sqrt{d_k}\) // formula (16.11)
- return \(\mathbf{o}_1, \dots, \mathbf{o}_n\) // each token now carries information from the tokens it chose
The same steps in PyTorch, with the numbers of the worked example. Each block checks one claim from this session:
import torch import torch.nn as nn import torch.nn.functional as F torch.manual_seed(0) # 1) the worked example: 3 tokens "the cat it", d = 3, d_k = 2 X = torch.tensor([[1., 0, 0], [0, 1, 0], [0, 0.5, 1]]) WQ = torch.tensor([[0., 0], [0, 1], [2, 0]]) WK = torch.tensor([[0., 0], [2, 0], [0, 1]]) WV = torch.tensor([[1., 0], [0, 2], [0, 0]]) Q, K, V = X @ WQ, X @ WK, X @ WV # (16.4) S = Q @ K.T # scores, (16.5) A = torch.softmax(S, dim=-1) # each row sums to 1, (16.7) O = A @ V # weighted sum of values, (16.3) print(S[2], A[2].round(decimals=4), O[2].round(decimals=4)) # tensor([0.0000, 4.0000, 2.5000]) tensor([0.0148, 0.8055, 0.1797]) tensor([0.0148, 1.7908]) # 2) the same with PyTorch's fused function (scale=1: no 1/sqrt(d_k) here) print(torch.allclose(F.scaled_dot_product_attention(Q, K, V, scale=1.0), O)) # True # 3) adding the same number to a whole row of scores changes nothing, (16.8) print(torch.allclose(torch.softmax(S + 7.0, dim=-1), A)) # True # 4) a key bias is useless: it adds q_i . b to every score in row i b = torch.randn(2) print(torch.allclose(torch.softmax(Q @ (K + b).T, dim=-1), A, atol=1e-6)) # True # 5) one shared matrix (W_Q = W_K) always gives a symmetric score matrix, (16.6) W = torch.randn(3, 2) S_shared = (X @ W) @ (X @ W).T print(torch.allclose(S_shared, S_shared.T)) # True print(torch.allclose(S, S.T)) # False # 6) a self-attention layer with learned projections (d = 16, d_k = d_v = 8) class SoftLookup(nn.Module): def __init__(self, d, d_k, d_v): super().__init__() self.W_Q = nn.Linear(d, d_k, bias=False) self.W_K = nn.Linear(d, d_k, bias=False) self.W_V = nn.Linear(d, d_v, bias=False) def forward(self, x): # x: (T, d) q, k, v = self.W_Q(x), self.W_K(x), self.W_V(x) a = torch.softmax(q @ k.T, dim=-1) # (T, T): one distribution per query return a @ v, a # (T, d_v), (T, T) layer = SoftLookup(16, 8, 8) out, a = layer(torch.randn(5, 16)) print(out.shape, a.shape, a.sum(-1)) # (5, 8) (5, 5) all ones ent = -(a * a.log()).sum(-1) # entropy of each row, (16.10) print(bool((ent >= 0).all() and (ent <= torch.log(torch.tensor(5.))).all())) # True
Tested with PyTorch 2.8 on CPU. Block 6 leaves out the \(1/\sqrt{d_k}\) scale on purpose, to match formula (16.2); the next session adds it.
Common mistakes
- Softmax over the wrong axis. The weights of one reader must sum to 1 over the keys, so the softmax runs over the last axis of \(QK^\top\) (
dim=-1). A softmax over the queries gives columns that sum to 1, a different and wrong operation that still runs without error. - Mixing up the roles of K and V. Keys decide where to read; values decide what is read. Changing a value never changes the weights; changing a key never changes what is copied, only how much.
- Thinking queries are only for “the current word”. In self-attention every token is a reader and an entry at the same time: each one has a query, a key and a value.
- Using the token vectors directly as Q and K. The self-score \(\lVert\mathbf{x}_i\rVert^2\) then tends to win, and the pattern is forced to be symmetric. The projections are not decoration.
- Expecting attention to amplify. The output is an average inside the convex hull of the values (16.7). If you need a larger or new feature, that comes from \(W_V\), the output projection and the MLP.
- Forgetting dilution. The same scores give much smaller weights when more positions compete. A pattern that works at 512 tokens can fade at 32,000 tokens.
- Reading attention maps as explanations. High weight does not prove that a token caused the output; values, other heads and later layers all matter.
Interview questions
Explain attention in one minute, using queries, keys and values.basic
What is the difference between a key and a value? Why not use one vector for both?basic
Why do we need learned projections at all? What goes wrong with \(Q = K = V = X\)?core
If \(W_Q = W_K\), what property do the scores have, and why is it a problem?core
Only \(W_Q W_K^\top\) enters the scores. Why store two matrices instead of one \(d \times d\) matrix?advanced
Why do the attention weights of a token sum to one, and what are the consequences?core
Show that a bias in the key projection has no effect. Is the same true for a query bias or a value bias?advanced
How does the length of the query affect attention?core
What is the entropy of an attention row, and what does it tell you?core
In a VLM with cross-attention, which modality gives the queries and which gives the keys and values? Why?core
Can we read attention weights as an explanation of the model’s decision?advanced
Exercises
- A reader has scores \((2, 0, 0)\) over three keys with values \(v = (10, 20, 30)\) (scalars). Compute the weights and the output.
- For the same scores, add 100 to each score. What are the new weights? What if you multiply each score by 2 instead?
- In Figure 1, the query has length 0. What is the output, and why does it not depend on the direction?
- Show that with \(W_Q = W_K = W\), the score matrix \(S = X W W^\top X^\top\) is symmetric and has non-negative diagonal.
- What are the smallest and largest possible entropies of an attention row over 9 positions? Give \(n_{\text{eff}}\) for both.
- In the worked example, change \(\mathbf{v}_{\text{cat}}\) to \((0, 4)\). Which numbers in the computation change and which do not?
- How many parameters do \(W_Q\), \(W_K\) and \(W_V\) have together in one head of GPT-2 small (\(d = 768\), \(d_k = d_v = 64\)), without biases? How many in one layer with 12 heads?
Answerscheck
Key takeaways
- Attention is a soft dictionary lookup: score every key against the query, turn the scores into weights with a softmax, and return the weighted average of the values (16.2, 16.3).
- Each token plays three roles: query (what I look for), key (the label I show), value (what I give if chosen), made by three learned matrices from the same vector (16.4).
- Without projections, tokens mostly attend to themselves; with one shared matrix, scores are symmetric (16.6). Separate \(W_Q, W_K\) give a learned, one-way, low-rank compatibility \(W_Q W_K^\top\) (16.5); a separate \(W_V\) decides what is passed on.
- The weights of each reader are a probability distribution over positions (16.7): the output is an expected value inside the convex hull of the values, only score differences matter (16.8), and a key bias is useless.
- Sharpness (query length, temperature) moves attention between a hard pick and a plain average (16.9); entropy and \(n_{\text{eff}}\) measure it (16.10). More positions dilute the weights.
- The same pattern, with queries and keys/values from different sources, is cross-attention in VLMs, temporal grounding in video, and the KV cache in generation.
Key formulas
References for this session
- Vaswani et al. (2017). Attention Is All You Need
- Alammar (2018). The Illustrated Transformer (blog)
- Raschka, Build a Large Language Model (From Scratch), chapter 3: Coding attention mechanisms
- Bahdanau et al. (2014). Neural Machine Translation by Jointly Learning to Align and Translate
- Miller et al. (2016). Key-Value Memory Networks for Directly Reading Documents
- Xiao et al. (2023). Efficient Streaming Language Models with Attention Sinks
- Jain and Wallace (2019). Attention is not Explanation
- Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale