LLM & VLM courseBlog
16Track 2: Attention and transformers · session 1 of 15

Attention intuition: query, key, value

Attention lets every token pull in information from other tokens. This session builds the idea from a plain dictionary lookup: a query is what a token looks for, a key is the label each token shows, and a value is what it hands over when it is chosen. We see why a transformer learns three separate projections for these roles, and why the attention weights of one token always form a probability distribution over positions.

Time: 2.5 to 3 hours (reading 1.5 hours, exercises 1 hour)Level: BeginnerBefore this: Sessions 2, 7 and 11 (dot products and softmax, the first attention, embeddings)

What you will learn

  • Attention as a soft dictionary lookup: a hard lookup returns one value, a soft lookup returns a weighted average of all values
  • The three roles of a token: query (what I look for), key (the label I show), value (what I give if chosen)
  • How q, k and v are made from the same token vector by three learned matrices, with a worked example in real numbers
  • Why three different projections: without them tokens look mostly at themselves, and one shared matrix forces symmetric scores
  • Attention weights as a probability distribution over positions: sum to one, shift invariance, entropy and effective number of positions
  • How sharpness, query length and the number of competing tokens change where attention goes
  • Where the same query, key, value pattern appears in LLMs, VLMs and video search

Why this matters

Every LLM and every VLM is built from one operation repeated many times: attention. In a 32-layer model with 32 heads per layer, attention runs 1,024 times for every token. If you understand what one attention step does, you understand the core of the transformer.

The problem attention solves is simple to say. After the embedding layer (session 11), each token has its own vector, but that vector knows nothing about the other tokens. In “the animal crossed the street because it was tired”, the vector for “it” is the same as in any other sentence. To understand the sentence, “it” must pull in information from “animal”. Attention is the mechanism that decides, for each token, which other tokens to read from and how much.

In Session 7 we met the first attention: a decoder that looks back at all encoder states. The transformer (Vaswani et al., 2017) made this the only way tokens talk to each other, and gave its parts three names: query, key and value. These names come from databases and dictionaries, and that is the best way to understand them. This session builds the intuition; session 17 then gives the full matrix math, the \(\sqrt{d_k}\) scale and the shapes.

Interviewers ask about this all the time: “Explain attention in one minute”, “What are Q, K and V?”, “Why not use the token vector directly?”, “Why do the weights sum to one, and what problems does that cause?”. These are not trick questions, but a vague answer is easy to spot. By the end you can answer each one with a formula and a small example in numbers.

A dictionary, made soft

A hard lookup

A Python dictionary stores pairs (key, value). You give it a query, it finds the key that is exactly equal, and it returns that key’s value:

\[ \operatorname{lookup}(q) = v_{j^*}, \qquad j^* = \text{the index with } k_{j^*} = q \tag{16.1} \]

This lookup is hard: it returns exactly one value, and if no key matches exactly, it fails. It also cannot be trained by gradient descent. A small change in the query either changes nothing or jumps to a different value, so the gradient is zero almost everywhere.

A soft lookup

Now make three changes. (1) Keys and queries are vectors, and instead of “equal or not” we compute a score that says how well they match. The dot product from session 2 is the usual choice. (2) We turn the scores into weights with a softmax, so the weights are positive and sum to 1. (3) We return a weighted average of all values instead of one value:

\[ s_j = \mathbf{q} \cdot \mathbf{k}_j, \qquad a_j = \frac{\exp(s_j)}{\sum_{m=1}^{n} \exp(s_m)} \tag{16.2} \]
\[ \operatorname{attend}(\mathbf{q}) = \sum_{j=1}^{n} a_j\, \mathbf{v}_j \tag{16.3} \]
Read the symbols
\(\mathbf{q}\)
the query: a vector that describes what we are looking for
\(\mathbf{k}_j\)
the key of entry \(j\): a vector that describes what entry \(j\) is about. It is only used to compute scores.
\(\mathbf{v}_j\)
the value of entry \(j\): the content we get back from entry \(j\). It is only used in the weighted sum.
\(s_j\)
the score (also called the compatibility or the logit) of entry \(j\): large when \(\mathbf{q}\) and \(\mathbf{k}_j\) point the same way
\(a_j\)
the attention weight of entry \(j\): a number between 0 and 1; all \(n\) weights add up to 1
\(n\)
the number of entries (in a transformer: the number of positions the token can see)

This is the definition in the transformer paper, in words: an attention function maps a query and a set of key–value pairs to an output; the output is a weighted sum of the values, and each weight comes from a compatibility function of the query with the matching key. [Certain] The hard lookup is a special case: if one score is much larger than all the others, the softmax puts almost all weight on it, and (16.3) returns almost exactly that one value.

The soft lookup fixes all three problems of the hard one. It never fails, because there is always a best partial match. It can mix several entries when the query is between them. And every step is smooth, so gradients flow back into the query, the keys and the values. This is why attention can be trained end to end.

Figure 1. A soft dictionary lookup, formulas (16.2) and (16.3). The dictionary has five entries. Each key (orange) is a unit vector pointing to a hue on a color wheel: red 0°, yellow 60°, green 120°, blue 240°, violet 280°. Each value (green) is a number: a typical light wavelength for that color in nanometers (rounded, for illustration). The query (blue) is the color you ask for.
How to read Figure 1

The left side shows the query arrow and the five key arrows. The bars on the right are the attention weights \(a_j\); they always add up to 1. The readout gives the scores, the weights and the output \(\sum_j a_j v_j\). Start at 30° (orange, halfway between red and yellow): red and yellow get 0.475 each and the output is 632.8 nm, between their wavelengths, a sensible answer for a color that is not in the dictionary. Set the length to 0: every score is 0 and the output is the plain average of all values, 538 nm. Now move the direction to 50° and the length to 8: yellow, the closest key, gets 0.934 of the weight. Tick “Hard lookup” to compare with a normal dictionary: its output jumps when the query crosses the middle between two keys, while the soft output moves smoothly.

Figure 1 shows a fact we will use again: the length of the query acts like a sharpness knob. Since \(\mathbf{q} \cdot \mathbf{k}_j = \lVert\mathbf{q}\rVert\,\lVert\mathbf{k}_j\rVert \cos\theta_j\), doubling the query doubles every score, and a softmax of doubled scores is sharper. A model can learn to make some queries long (to pick one position) and others short (to average over many).

An old idea with a new name

Weighted averages of stored values are much older than transformers. In statistics, kernel regression (Nadaraya and Watson, 1964) predicts a value at a new point as an average of known values, weighted by how close each known point is. Formula (16.3) is exactly that, with the kernel \(\exp(\mathbf{q} \cdot \mathbf{k})\). [Certain] In neural networks, memory networks used the same split: Miller et al. (2016), in “Key-Value Memory Networks”, used one encoding of each memory slot to find it and a different encoding to read from it. [Certain] What the transformer added is that the queries, keys and values are all learned from the tokens themselves, at every layer.

Query, key, value: three roles for one token

In self-attention, the dictionary is the sentence itself. Every token is at the same time a reader and an entry:

  • As a reader, token \(i\) has a query \(\mathbf{q}_i\): what am I looking for? For “it”, something like “a noun earlier in the sentence that can be tired”.
  • As an entry, token \(j\) has a key \(\mathbf{k}_j\): the label I show to others. For “animal”, something like “I am a noun, I am a living thing”.
  • Also as an entry, token \(j\) has a value \(\mathbf{v}_j\): what I give if chosen. For “animal”, the information that the reader should copy, for example its meaning.

All three are made from the same token vector \(\mathbf{x}_t\) (a row vector of size \(d\)) by three learned matrices:

\[ \mathbf{q}_t = \mathbf{x}_t W_Q, \qquad \mathbf{k}_t = \mathbf{x}_t W_K, \qquad \mathbf{v}_t = \mathbf{x}_t W_V \tag{16.4} \]
Read the symbols
\(\mathbf{x}_t\)
the vector of the token at position \(t\) as it enters the attention layer (in layer 1, its embedding plus position information; in later layers, the output of the layer before)
\(W_Q \in \R^{d \times d_k}\)
the query projection: turns a token vector into a query of size \(d_k\)
\(W_K \in \R^{d \times d_k}\)
the key projection. Queries and keys must have the same size \(d_k\), because we take their dot product.
\(W_V \in \R^{d \times d_v}\)
the value projection. \(d_v\) may differ from \(d_k\); in most models \(d_v = d_k\).
\(d, d_k, d_v\)
the model width and the sizes of queries/keys and of values. In the original transformer, \(d = 512\) and \(d_k = d_v = 64\) per head. [Certain]

The matrices are the same for every position: “it” and “animal” use the same \(W_Q\). What differs is the input vector. So the model does not learn a query for each word; it learns a rule that turns any token vector into a question, a label and a message.

token xₜ WQ WK WV query qₜwhat I look for key kₜthe label I show value vₜwhat I give if chosen used in scores: qᵢ · kⱼ for every j (my query, their keys) weights: softmax over j (one distribution per reader) output: Σⱼ aᵢⱼ vⱼ (a mix of their values)
Figure 2. One token vector, three roles. The same \(\mathbf{x}_t\) goes through three different learned matrices, formula (16.4). The query is used only when this token reads; the key and the value are used only when other tokens (or itself) read from it.

Now watch one reader do the whole lookup. Figure 3 runs Algorithm 1 (in the pseudo-code section below) on the sentence “the animal crossed the street because it was tired”, one line at a time, with real numbers. Each token has 6 hand-made features: noun, living thing, verb, function word, pronoun, adjective. The key matrix \(W_K\) copies two of them, “noun” and “living thing” (times 1.5), so the keys say “I am a noun” and “I am alive”. The query matrix \(W_Q\) lets verbs ask for nouns, the adjective ask for living things, and the pronoun ask for both. The value matrix \(W_V\) sends each token to a point in a 2-dimensional “value plane”, so we can draw the output as a point that moves.

query of the readerkeys (brighter = higher score)weights and the output \(\mathbf{o}\)values
  1. input: \(\mathbf{q} = \mathbf{x}_{\text{reader}} W_Q\); keys \(\mathbf{k}_j\), values \(\mathbf{v}_j\); \(\beta\) // (16.4)
  2. for \(j = 1 \dots n\): \(s_j \leftarrow \beta\, \mathbf{q} \cdot \mathbf{k}_j\) // (16.2)
  3. \(m \leftarrow \max_j s_j\) // allowed by (16.8)
  4. for \(j = 1 \dots n\): \(e_j \leftarrow \exp(s_j - m)\)
  5. \(a_j \leftarrow e_j / \sum_m e_m\) // sums to 1
  6. \(\mathbf{o} \leftarrow \mathbf{0}\); for \(j = 1 \dots n\): \(\mathbf{o} \leftarrow \mathbf{o} + a_j \mathbf{v}_j\) // (16.3)
  7. return \(\mathbf{o}\)
Figure 3. One query looks up every key, computed live in the browser. Left (on a phone: top): the 9 tokens with their keys (orange), the score of each key against the reader’s query, the softmax weights, and the values (green). Right (on a phone: below): the value plane. The blue arc shows which key the query is reading; the dashed path shows the output \(\mathbf{o}\) being built one term \(a_j \mathbf{v}_j\) at a time. The highlighted line in the algorithm box is the line being run; the box beside it shows the formula with the current numbers.
How to read Figure 3

Press Play, or Step through it. Steps 2 to 10 compute one score each: watch the keys of “animal” and “street” light up for the reader “it”, because they show “noun”, which “it” asks for. Steps 11 to 13 turn the scores into weights: “it” gives 0.845 to “animal”, 0.089 to “street” and 0.009 to each of the other 7 tokens. Steps 14 to 22 add the values one by one: the output point jumps toward “animal” at j = 2 and ends at (1.812, 1.662), close to the value of “animal”, (2, 2), and far from the value of “it” itself, (−2, 0). Now pick “crossed”: its query asks only for “noun”, so “animal” and “street” tie at 0.365 each and the output, (1.230, 0.615), lands between them. Pick “animal”: its query is zero, every weight is 1/9 = 0.111 and the output is the plain average of all values. Raise \(\beta\) to 2 and “it” gives 0.988 to “animal”: almost a hard lookup.

Put the pieces together for one reading token \(i\). The score with token \(j\) is the query of \(i\) times the key of \(j\), and by (16.4) it can be written with the raw token vectors:

\[ s_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j = \mathbf{x}_i\, W_Q W_K^\top\, \mathbf{x}_j^\top \tag{16.5} \]
Read the symbols
\(s_{ij}\)
how much reader \(i\) wants to read from token \(j\), before the softmax
\(W_Q W_K^\top\)
a \(d \times d\) matrix that defines the “compatibility” between two token vectors. It is learned, but only through its two thin factors.
Worked example: one attention step by hand

Three tokens, “the cat it”, with \(d = 3\) features (function word, noun, pronoun) and \(d_k = d_v = 2\). The token vectors are \(\mathbf{x}_{\text{the}} = (1, 0, 0)\), \(\mathbf{x}_{\text{cat}} = (0, 1, 0)\) and \(\mathbf{x}_{\text{it}} = (0, 0.5, 1)\) (“it” is mostly a pronoun and a bit noun-like). The three matrices, written by their rows, are

\(W_Q\): rows \((0, 0), (0, 1), (2, 0)\); \(\quad W_K\): rows \((0, 0), (2, 0), (0, 1)\); \(\quad W_V\): rows \((1, 0), (0, 2), (0, 0)\).

Project. \(\mathbf{q}_{\text{it}} = 0.5\cdot(0, 1) + 1\cdot(2, 0) = (2, 0.5)\). The keys are \(\mathbf{k}_{\text{the}} = (0, 0)\), \(\mathbf{k}_{\text{cat}} = (2, 0)\), \(\mathbf{k}_{\text{it}} = (1, 1)\). The values are \(\mathbf{v}_{\text{the}} = (1, 0)\), \(\mathbf{v}_{\text{cat}} = (0, 2)\), \(\mathbf{v}_{\text{it}} = (0, 1)\).

Score. \(\mathbf{q}_{\text{it}} \cdot \mathbf{k}_j = 0,\ 4,\ 2.5\) for the, cat, it.

Softmax. \(e^0 = 1\), \(e^4 = 54.60\), \(e^{2.5} = 12.18\); the sum is 67.78, so the weights are \(0.0148,\ 0.8055,\ 0.1797\). They add up to 1.

Mix. \(\mathbf{o}_{\text{it}} = 0.0148\,(1, 0) + 0.8055\,(0, 2) + 0.1797\,(0, 1) = (0.0148,\ 1.7908)\). The new vector for “it” is mostly the value of “cat”.

Without projections (query = key = token vector): the scores of “it” would be \(\mathbf{x}_{\text{it}} \cdot \mathbf{x}_j = 0,\ 0.5,\ 1.25\), and the largest weight, 0.5685, would go to “it” itself. The projections are what turn “it” toward “cat”.

  1. input: \(X\) (\(3 \times 3\)), \(W_Q, W_K, W_V\) (\(3 \times 2\))
  2. for \(t\): \(\mathbf{q}_t \leftarrow \mathbf{x}_t W_Q\); \(\mathbf{k}_t \leftarrow \mathbf{x}_t W_K\); \(\mathbf{v}_t \leftarrow \mathbf{x}_t W_V\) // (16.4)
  3. for each reader \(i\):
  4. for each \(j\): \(s_{ij} \leftarrow \mathbf{q}_i \cdot \mathbf{k}_j\) // (16.5)
  5. \(\mathbf{a}_i \leftarrow \softmax(\mathbf{s}_i)\) // (16.2)
  6. \(\mathbf{o}_i \leftarrow \sum_j a_{ij}\, \mathbf{v}_j\) // (16.3)
  7. return \(O\)
  8. compare: \(Q = K = X\), no projections
Figure 4. The worked example, computed one cell at a time: the token vectors, the three projections, then for each reader its three scores, its softmax and its weighted sum of values. Rows are readers (queries), columns are the tokens being read (keys and values). Cells not yet computed show a dot; the cells used in the current step are outlined.
How to read Figure 4

Press Play or Step. Blue cells are queries, orange cells keys, green cells values. The box under the tables writes out the formula of the current line with the numbers that go in. Follow reader “it” (steps 15 to 19): the scores 0, 4 and 2.5 appear one by one, the softmax turns them into 0.0148, 0.8055 and 0.1797, and the output is (0.0148, 1.7908). Check two things: every row of weights adds up to 1, and the row of “the” is flat (1/3 each) because its query is \((0, 0)\): a token with nothing to ask simply averages. The last step shows the “no projections” weights next to the real ones: there “it” reads itself most (0.5685).

Why three different projections

Why not skip the matrices and use the token vector itself as query, key and value? Or use one matrix for all three? Each simpler choice breaks something. Figure 5 lets you compare them on one sentence.

No projections: every token looks at itself

If \(\mathbf{q}_i = \mathbf{k}_i = \mathbf{x}_i\), the score of a token with itself is \(\mathbf{x}_i \cdot \mathbf{x}_i = \lVert\mathbf{x}_i\rVert^2\). By the Cauchy–Schwarz inequality, \(\mathbf{x}_i \cdot \mathbf{x}_j \le \lVert\mathbf{x}_i\rVert\,\lVert\mathbf{x}_j\rVert\), so when token vectors have similar lengths, the self-score is the largest score in the row. Each token mostly reads itself, plus tokens that are similar to it. But attention is useful exactly when a token must read something different from itself: “it” must read “animal”, a verb must find its subject, a question must find the answer in a document.

One shared matrix: scores are forced to be symmetric

With one matrix \(W\) for both queries and keys, every token is mapped to one vector \(\mathbf{a}_t = \mathbf{x}_t W\), and

\[ s_{ij} = \mathbf{a}_i \cdot \mathbf{a}_j = s_{ji} \qquad \text{(symmetric for every pair)} \tag{16.6} \]

So “how much ‘it’ wants to read ‘animal’” must equal “how much ‘animal’ wants to read ‘it’”. Language is full of one-way relations: a pronoun looks for its noun, but the noun has no reason to look for the pronoun; an adjective looks for the noun it describes; a closing bracket looks for the opening one. A symmetric score cannot express “A needs B, B does not need A”. The self-score problem also stays: \(s_{ii} = \lVert\mathbf{a}_i\rVert^2\) is still the largest unless another vector is longer.

Separate \(W_Q\) and \(W_K\): a learned, one-way compatibility

With two matrices, the score (16.5) uses the matrix \(M = W_Q W_K^\top\), which can be any \(d \times d\) matrix of rank at most \(d_k\), symmetric or not. The query side can ask about one set of features (“pronoun-ness”) while the key side answers with a different set (“noun-ness”, “living thing”). The self-score is no longer special.

The factored form is also cheap. A full \(d \times d\) matrix for GPT-2 small (\(d = 768\)) would have 589,824 numbers per head. The two factors with \(d_k = 64\) have \(2 \times 768 \times 64 = 98{,}304\). [Certain] The low rank is not a loss: one head only needs to match a few features, and session 19 runs many such heads side by side.

Separate \(W_V\): what you match on is not what you pass on

The key is an address; the value is the content. They are needed for different jobs. To be found by “it”, “animal” should advertise “noun, living thing”. But what “it” should copy is different: maybe the full meaning of “animal”, or only the fact that it is singular. A separate \(W_V\) lets the model choose what to send, independently of how it is found. This is the same split as in key–value memory networks, and it is why the KV cache (session 80) stores two tensors, not one.

Figure 5. The same sentence, “the animal crossed the street because it was tired”, under three choices of projection. Each token vector has the same 6 hand-made features as in Figure 3 (noun, living thing, verb, function word, pronoun, adjective). The separate projections use a 2-dimensional key space (noun, living thing): keys show these two features, and the queries of verbs, the pronoun and the adjective ask for them. The shared matrix is the average of \(W_Q\) and \(W_K\). Darker cells are larger weights; each row sums to 1.
How to read Figure 5

Row \(i\) is the attention distribution of reader \(i\) over the 9 tokens. Click a row (a word on the left) to see its weights in the readout. With no projections, the darkest cell in most rows is on the diagonal: each token reads itself, and “it” gives itself 0.254 and “animal” only 0.093. With one shared W, the matrix is mirror-symmetric, and “it” gives the same weight, 0.190, to itself and to “animal”. With separate projections, “it” puts 0.845 on “animal” and only 0.089 on “street”, “tired” finds “animal” too, while “animal” itself has nothing to ask and reads every token equally (1/9 each). The readout also reports the largest gap \(\lvert s_{ij} - s_{ji} \rvert\) between mirrored scores.

Attention weights are a probability distribution

For a fixed reader \(i\), the softmax turns its \(n\) scores into \(n\) weights. These weights have exactly the properties of a probability distribution over positions:

\[ a_{ij} \ge 0, \qquad \sum_{j=1}^{n} a_{ij} = 1, \qquad \mathbf{o}_i = \sum_{j=1}^{n} a_{ij}\,\mathbf{v}_j = \E_{j \sim \mathbf{a}_i}\!\left[\mathbf{v}_j\right] \tag{16.7} \]
Read the symbols
\(\mathbf{a}_i\)
row \(i\) of the attention matrix: the distribution of reader \(i\) over the positions \(j = 1, \dots, n\)
\(\E_{j \sim \mathbf{a}_i}[\mathbf{v}_j]\)
the expected value of \(\mathbf{v}_j\) when the position \(j\) is drawn at random with probabilities \(\mathbf{a}_i\). The output is this average.

Read it as a story: reader \(i\) picks one position at random, with the probabilities \(\mathbf{a}_i\), and takes its value. The output is the average of what it would get. Four consequences matter in practice.

1. The output stays inside the values. A weighted average with positive weights that sum to 1 lies in the convex hull of the values (the smallest shape that contains all of them). Attention can mix and select, but it cannot amplify: one head cannot make a value twice as large. Making new things is the job of the MLP that follows (session 21).

2. Only score differences matter. Adding the same number \(c\) to all scores of a row changes nothing, because the factor \(e^c\) cancels:

\[ \frac{\exp(s_j + c)}{\sum_m \exp(s_m + c)} = \frac{e^c \exp(s_j)}{e^c \sum_m \exp(s_m)} = a_j \tag{16.8} \]

A direct result: a bias on the keys is useless. If \(\mathbf{k}_j = \mathbf{x}_j W_K + \mathbf{b}\), then \(\mathbf{q}_i \cdot \mathbf{k}_j = \mathbf{q}_i \cdot \mathbf{x}_j W_K + \mathbf{q}_i \cdot \mathbf{b}\), and the second term is the same for every \(j\) in row \(i\), so it cancels. [Certain] Many modern models drop the biases in these projections; the Llama 3 8B configuration sets attention_bias to false. [Certain]

3. Sharpness: from a hard pick to a plain average. Multiply all scores by a number \(\beta \ge 0\) (an inverse temperature; a longer query does exactly this, as in Figure 1):

\[ \mathbf{a}_i(\beta) = \softmax(\beta\, \mathbf{s}_i), \qquad \beta \to 0:\ a_{ij} \to \tfrac{1}{n}, \qquad \beta \to \infty:\ \text{all weight on } \argmax_j s_{ij} \tag{16.9} \]

To measure how spread out a row is, use its entropy, and turn it into an effective number of positions:

\[ H(\mathbf{a}_i) = -\sum_{j=1}^{n} a_{ij} \log a_{ij} \in [0, \log n], \qquad n_{\text{eff}} = e^{H(\mathbf{a}_i)} \in [1, n] \tag{16.10} \]

\(n_{\text{eff}} = 1\) means the reader looks at one position (a hard lookup); \(n_{\text{eff}} = n\) means it averages all positions equally. In the worked example, the row of “it” has entropy 0.545 nats, so \(n_{\text{eff}} = 1.72\): mostly “cat”, a little “it”. The row of “the” has \(H = \log 3 = 1.099\) and \(n_{\text{eff}} = 3\). Why the transformer divides scores by \(\sqrt{d_k}\) is a question about \(\beta\): without it, scores in large dimensions are so big that every row becomes a hard pick. That is the subject of session 17.

4. Weights must go somewhere. The weights must sum to 1, even when nothing in the context is useful. Two effects follow. First, dilution: every extra position takes a share. In Figure 6 the row of “it” puts 0.845 on “animal” in the 9-token sentence; add 100 unrelated tokens with score 0 and the same scores give “animal” only 0.436; with 1,000 such tokens, 0.081. This is one reason long contexts are hard (session 28). Second, attention sinks: trained LLMs often put a large weight on the first token of the sequence even when it carries little meaning, as a place to park weight they do not need. Xiao et al. (2023) found that keeping the keys and values of these first tokens is what makes sliding-window generation work on very long streams. [Certain]

Figure 6. The row of “it” from Figure 5 (separate projections) as a probability distribution. Its scores are 4.5 for “animal”, 2.25 for “street” and 0 for the other 7 tokens. The sliders multiply all scores by \(\beta\) (formula 16.9), add unrelated tokens with score 0 (dilution), or add a constant to every score (formula 16.8).
How to read Figure 6

Each bar is one weight; the last bar sums all the extra tokens together. The readout shows the weight on “animal”, the entropy \(H\) and \(n_{\text{eff}}\) from (16.10), and the total, which is always 1. Move \(\beta\) to 0: all weights become \(1/9 = 0.111\) and \(n_{\text{eff}} = 9\). Move it to 3: “animal” gets 0.999. Move the constant \(c\): nothing changes at all. Add 100 extra tokens: “animal” drops to 0.436 although its score did not change; raise \(\beta\) to win it back. A sharper query is how a trained model fights dilution.

A distribution, not an explanation

Because attention weights look like probabilities over the input, it is tempting to read them as “what the model used”. Be careful. The output also depends on the values (a large weight on a small value matters little), there are many heads and layers mixing information, and Jain and Wallace (2019) showed that very different attention patterns can often give the same prediction. [Certain] Attention maps are a useful first look, not a proof. Better tools are covered in session 157.

All positions at once

A transformer does the soft lookup for every reader at the same time. Stack the token vectors as rows of \(X \in \R^{n \times d}\); then \(Q = XW_Q\), \(K = XW_K\), \(V = XW_V\), and one matrix product computes every score. The transformer also divides the scores by \(\sqrt{d_k}\):

\[ O = \softmax\!\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V, \qquad \text{softmax over each row} \tag{16.11} \]

Row \(i\) of \(O\) is formula (16.3) for reader \(i\). This is formula (2.20) from session 2, now with a meaning for each letter. Two more pieces turn it into what LLMs use: a causal mask so that a token cannot read the future (session 18), and many heads, each with its own \(W_Q, W_K, W_V\) (session 19). Real sizes:

ModelWidth \(d\)Heads\(d_k\) per headNotes
Transformer base (2017)512864\(d_k = d_v = d/h\) [Certain]
GPT-2 small7681264one fused \(768 \times 2304\) matrix computes Q, K and V together (\(3 \times 768 = 2304\)), with biases [Certain]
Llama 3 8B4,09632 query heads, 8 key/value heads128no projection biases; keys and values shared across groups of 4 query heads (session 26) [Certain]

Sizes from the original paper and the public model configurations.

Beyond one sentence: images, video and memory

Nothing in (16.2)–(16.3) says that queries, keys and values must come from the same sequence. That freedom is what makes attention the glue of multimodal models.

  • Self-attention: Q, K and V all come from the same tokens. This is what we used above, and what every LLM layer does.
  • Cross-attention: Q comes from one sequence and K, V from another. In a translation model, the decoder asks and the encoder answers (session 20). In a VLM with cross-attention layers, text tokens ask and image patches answer: the query of “red” is compared with the keys of all patches, and the values of the red patches flow into the text (session 179).
  • Images in self-attention: In the Vision Transformer (session 164) each \(16 \times 16\) patch is a token with its own query, key and value; a patch of a dog’s ear can read the patch of its nose. [Certain]
  • Video: Frames become many tokens. A text query such as “the moment the goal is scored” can be matched against keys of frame tokens, and the attention weights over time act as a soft pointer to the moment (session 207).
  • Memory: At generation time, the keys and values of past tokens do not change, so they are stored and reused; only the new token computes a fresh query. This is the KV cache (session 80): a literal dictionary of keys and values that grows with every token.

Where this is used

ApplicationQueryKeys and valuesLink
Every LLM layer (decoder self-attention)Each tokenAll earlier tokens (causal mask)session 18
Fast generationThe newest token onlyCached K, V of all earlier tokenssession 80
VLM with cross-attention layersText tokensImage patch featuressession 179
Image classification and detection (ViT)Each patch (or a class token)All patchessession 164
Video action recognitionEach space–time patchPatches in the same frame and in other framessession 202
Key-moment detection, temporal groundingA text description of the momentFrame or clip features over timesession 207
Video-RAG and video question answeringQuestion tokensRetrieved frames, captions and transcript chunkssession 210
Text-to-video retrieval (re-ranking)Caption tokensFrame tokens of a candidate clipsession 208

Pseudo-code

Algorithm 1: a soft dictionary lookup for one query
  1. input: query \(\mathbf{q}\), keys \(\mathbf{k}_1, \dots, \mathbf{k}_n\), values \(\mathbf{v}_1, \dots, \mathbf{v}_n\), sharpness \(\beta\)
  2. for \(j = 1\) to \(n\) do \(s_j \leftarrow \beta\, \mathbf{q} \cdot \mathbf{k}_j\) // compatibility, formula (16.2)
  3. \(m \leftarrow \max_j s_j\) // subtract the max for stability; allowed by (16.8)
  4. for \(j = 1\) to \(n\) do \(e_j \leftarrow \exp(s_j - m)\)
  5. \(a_j \leftarrow e_j / \sum_m e_m\) for all \(j\) // a distribution over the entries
  6. return \(\sum_j a_j \mathbf{v}_j\) // weighted average of values, formula (16.3)
Algorithm 2: one self-attention step for a sequence
  1. input: token vectors \(\mathbf{x}_1, \dots, \mathbf{x}_n\), matrices \(W_Q, W_K, W_V\)
  2. for \(t = 1\) to \(n\) do \(\mathbf{q}_t \leftarrow \mathbf{x}_t W_Q;\ \mathbf{k}_t \leftarrow \mathbf{x}_t W_K;\ \mathbf{v}_t \leftarrow \mathbf{x}_t W_V\) // formula (16.4)
  3. for each reader \(i = 1\) to \(n\) do
  4. \(\mathbf{o}_i \leftarrow\) Algorithm 1 with query \(\mathbf{q}_i\), all keys and values, \(\beta = 1/\sqrt{d_k}\) // formula (16.11)
  5. return \(\mathbf{o}_1, \dots, \mathbf{o}_n\) // each token now carries information from the tokens it chose

The same steps in PyTorch, with the numbers of the worked example. Each block checks one claim from this session:

import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)

# 1) the worked example: 3 tokens "the cat it", d = 3, d_k = 2
X  = torch.tensor([[1., 0, 0], [0, 1, 0], [0, 0.5, 1]])
WQ = torch.tensor([[0., 0], [0, 1], [2, 0]])
WK = torch.tensor([[0., 0], [2, 0], [0, 1]])
WV = torch.tensor([[1., 0], [0, 2], [0, 0]])
Q, K, V = X @ WQ, X @ WK, X @ WV                # (16.4)
S = Q @ K.T                                     # scores, (16.5)
A = torch.softmax(S, dim=-1)                    # each row sums to 1, (16.7)
O = A @ V                                       # weighted sum of values, (16.3)
print(S[2], A[2].round(decimals=4), O[2].round(decimals=4))
# tensor([0.0000, 4.0000, 2.5000]) tensor([0.0148, 0.8055, 0.1797]) tensor([0.0148, 1.7908])

# 2) the same with PyTorch's fused function (scale=1: no 1/sqrt(d_k) here)
print(torch.allclose(F.scaled_dot_product_attention(Q, K, V, scale=1.0), O))   # True

# 3) adding the same number to a whole row of scores changes nothing, (16.8)
print(torch.allclose(torch.softmax(S + 7.0, dim=-1), A))                      # True

# 4) a key bias is useless: it adds q_i . b to every score in row i
b = torch.randn(2)
print(torch.allclose(torch.softmax(Q @ (K + b).T, dim=-1), A, atol=1e-6))      # True

# 5) one shared matrix (W_Q = W_K) always gives a symmetric score matrix, (16.6)
W = torch.randn(3, 2)
S_shared = (X @ W) @ (X @ W).T
print(torch.allclose(S_shared, S_shared.T))                                   # True
print(torch.allclose(S, S.T))                                                 # False

# 6) a self-attention layer with learned projections (d = 16, d_k = d_v = 8)
class SoftLookup(nn.Module):
    def __init__(self, d, d_k, d_v):
        super().__init__()
        self.W_Q = nn.Linear(d, d_k, bias=False)
        self.W_K = nn.Linear(d, d_k, bias=False)
        self.W_V = nn.Linear(d, d_v, bias=False)
    def forward(self, x):                       # x: (T, d)
        q, k, v = self.W_Q(x), self.W_K(x), self.W_V(x)
        a = torch.softmax(q @ k.T, dim=-1)      # (T, T): one distribution per query
        return a @ v, a                         # (T, d_v), (T, T)
layer = SoftLookup(16, 8, 8)
out, a = layer(torch.randn(5, 16))
print(out.shape, a.shape, a.sum(-1))            # (5, 8) (5, 5) all ones
ent = -(a * a.log()).sum(-1)                    # entropy of each row, (16.10)
print(bool((ent >= 0).all() and (ent <= torch.log(torch.tensor(5.))).all()))  # True

Tested with PyTorch 2.8 on CPU. Block 6 leaves out the \(1/\sqrt{d_k}\) scale on purpose, to match formula (16.2); the next session adds it.

Common mistakes

  • Softmax over the wrong axis. The weights of one reader must sum to 1 over the keys, so the softmax runs over the last axis of \(QK^\top\) (dim=-1). A softmax over the queries gives columns that sum to 1, a different and wrong operation that still runs without error.
  • Mixing up the roles of K and V. Keys decide where to read; values decide what is read. Changing a value never changes the weights; changing a key never changes what is copied, only how much.
  • Thinking queries are only for “the current word”. In self-attention every token is a reader and an entry at the same time: each one has a query, a key and a value.
  • Using the token vectors directly as Q and K. The self-score \(\lVert\mathbf{x}_i\rVert^2\) then tends to win, and the pattern is forced to be symmetric. The projections are not decoration.
  • Expecting attention to amplify. The output is an average inside the convex hull of the values (16.7). If you need a larger or new feature, that comes from \(W_V\), the output projection and the MLP.
  • Forgetting dilution. The same scores give much smaller weights when more positions compete. A pattern that works at 512 tokens can fade at 32,000 tokens.
  • Reading attention maps as explanations. High weight does not prove that a token caused the output; values, other heads and later layers all matter.

Interview questions

Explain attention in one minute, using queries, keys and values.basic
Attention is a soft dictionary lookup. Each token makes a query (what it looks for), a key (the label it shows) and a value (what it gives if chosen), all by learned linear maps of its vector. A reader compares its query with every key by dot product, a softmax turns those scores into weights that are positive and sum to 1, and the output is the weighted average of the values (formulas 16.2 and 16.3). So each token’s new vector is a mix of information from the tokens whose keys match its question.
What is the difference between a key and a value? Why not use one vector for both?basic
The key is an address: it is only used to compute how much a reader attends to this token. The value is the content: it is only used in the weighted sum. The features that make a token easy to find (for “animal”: noun, living thing) are not the same as the information a reader should copy (its meaning, number, and so on). Separate \(W_K\) and \(W_V\) let the model choose both independently. Key–value memory networks made the same split before transformers.
Why do we need learned projections at all? What goes wrong with \(Q = K = V = X\)?core
The self-score \(\mathbf{x}_i \cdot \mathbf{x}_i = \lVert \mathbf{x}_i \rVert^2\) is, by Cauchy–Schwarz, at least as large as any \(\mathbf{x}_i \cdot \mathbf{x}_j\) when vectors have similar lengths, so tokens mostly attend to themselves and to similar tokens. Scores are also symmetric. And the value would be the raw vector, so the model could not choose what to send. Projections let the model decide which features to ask about, which to advertise and which to pass on; the same layer can then learn many different relations, one per head.
If \(W_Q = W_K\), what property do the scores have, and why is it a problem?core
The score matrix becomes symmetric: \(s_{ij} = (\mathbf{x}_i W)\cdot(\mathbf{x}_j W) = s_{ji}\) (formula 16.6), and \(W W^\top\) is positive semi-definite. Many relations in language are one-way: a pronoun needs its noun, an adjective needs its noun, a closing bracket needs its opening one, but not the reverse. A symmetric score cannot express that. Separate matrices give \(M = W_Q W_K^\top\), which can be any matrix of rank at most \(d_k\), non-symmetric included. (Note that the softmax still normalizes rows, so the weights \(a_{ij}\) are not exactly symmetric even with symmetric scores; but the preference structure is.)
Only \(W_Q W_K^\top\) enters the scores. Why store two matrices instead of one \(d \times d\) matrix?advanced
Three reasons. Cost: two \(d \times d_k\) factors have \(2 d d_k\) parameters, for \(d = 768\), \(d_k = 64\) that is 98,304 instead of 589,824 per head. Compute: with factors, each token computes its key once (\(O(n d d_k)\)) and scores are \(d_k\)-dimensional dot products; with a full \(M\) you would multiply by a \(d \times d\) matrix. Caching: at generation time the keys of past tokens are computed once and stored in the KV cache. The low rank is a feature too: each head matches a few features, and many heads together cover more. Also note the factorization is not unique: \(W_Q R\) and \(W_K R^{-\top}\) give the same product for any invertible \(R\).
Why do the attention weights of a token sum to one, and what are the consequences?core
Because they are a softmax over the keys: positive and normalized, a probability distribution over positions (16.7). Consequences: the output is an expected value of the values, inside their convex hull, so attention mixes but does not amplify; only differences of scores matter (16.8), so a key bias does nothing; adding positions dilutes the weights (0.845 becomes 0.436 with 100 extra zero-score tokens in Figure 6); and when nothing is relevant the weight must still go somewhere, which is one explanation of attention sinks on the first token.
Show that a bias in the key projection has no effect. Is the same true for a query bias or a value bias?advanced
With \(\mathbf{k}_j = \mathbf{x}_j W_K + \mathbf{b}_K\), the score is \(\mathbf{q}_i \cdot \mathbf{x}_j W_K + \mathbf{q}_i \cdot \mathbf{b}_K\); the second term is the same for all \(j\) in row \(i\) and cancels in the softmax (16.8). A query bias is different: \((\mathbf{x}_i W_Q + \mathbf{b}_Q)\cdot\mathbf{k}_j\) adds \(\mathbf{b}_Q \cdot \mathbf{k}_j\), which depends on \(j\): a fixed preference for some keys. A value bias \(\mathbf{b}_V\) adds \(\sum_j a_{ij}\mathbf{b}_V = \mathbf{b}_V\) to every output (the weights sum to 1), so it is a constant shift that the output projection’s bias could absorb.
How does the length of the query affect attention?core
Scores are \(\lVert\mathbf{q}\rVert\lVert\mathbf{k}_j\rVert\cos\theta_j\), so scaling the query by \(\beta\) scales all scores by \(\beta\): it is an inverse temperature (16.9). Short queries give near-uniform weights (an average over the context); long queries give a near-hard pick of the best key. The model learns this through \(W_Q\). It is also why the raw dot product needs the \(1/\sqrt{d_k}\) scale: in high dimensions the typical score size grows and softmax saturates, which makes gradients tiny (next session).
What is the entropy of an attention row, and what does it tell you?core
\(H = -\sum_j a_{ij}\log a_{ij}\), between 0 (all weight on one position) and \(\log n\) (uniform over \(n\) positions). \(e^H\) is the effective number of positions the reader looks at. It is a common diagnostic: heads with very low entropy act like pointers (for example, copy the previous token), heads with high entropy act like averages. Entropy that collapses to 0 early in training is a sign of attention logit growth and instability. [Likely] In the worked example, “it” has \(H = 0.545\) and \(n_{\text{eff}} = 1.72\).
In a VLM with cross-attention, which modality gives the queries and which gives the keys and values? Why?core
Usually the text (the language model’s hidden states) gives the queries and the image or video features give the keys and values. The language model is the one producing output, so it is the reader: each text token asks “which parts of the image matter for me?”, the patch keys answer, and the patch values flow into the text stream. The output has one vector per text token, so the sequence length of the LLM does not grow with the number of patches. Video works the same way, with keys and values from frame tokens across time.
Can we read attention weights as an explanation of the model’s decision?advanced
Only with care. The output depends on weight times value, so a high weight on a small or irrelevant value matters little; information is mixed across many heads and layers, so a token that gets no direct weight can still have flowed in through earlier layers; and Jain and Wallace (2019) showed that quite different attention distributions can give the same prediction. Use attention maps as hints, then check with interventions (remove or change a token and see the effect) or gradient-based and causal methods.

Exercises

  1. A reader has scores \((2, 0, 0)\) over three keys with values \(v = (10, 20, 30)\) (scalars). Compute the weights and the output.
  2. For the same scores, add 100 to each score. What are the new weights? What if you multiply each score by 2 instead?
  3. In Figure 1, the query has length 0. What is the output, and why does it not depend on the direction?
  4. Show that with \(W_Q = W_K = W\), the score matrix \(S = X W W^\top X^\top\) is symmetric and has non-negative diagonal.
  5. What are the smallest and largest possible entropies of an attention row over 9 positions? Give \(n_{\text{eff}}\) for both.
  6. In the worked example, change \(\mathbf{v}_{\text{cat}}\) to \((0, 4)\). Which numbers in the computation change and which do not?
  7. How many parameters do \(W_Q\), \(W_K\) and \(W_V\) have together in one head of GPT-2 small (\(d = 768\), \(d_k = d_v = 64\)), without biases? How many in one layer with 12 heads?
Answerscheck
1. \(e^2 = 7.389\), so the weights are \(7.389/9.389 = 0.787\) and \(1/9.389 = 0.107\) twice. Output \(= 0.787 \cdot 10 + 0.107 \cdot 20 + 0.107 \cdot 30 = 13.20\). 2. Adding 100 changes nothing (16.8): still \(0.787, 0.107, 0.107\). Doubling gives scores \((4, 0, 0)\): \(54.6/56.6 = 0.965\) and \(0.018\) twice, a sharper row. 3. All scores are \(0\), so every weight is \(1/5\) and the output is the plain average \((700 + 580 + 530 + 470 + 410)/5 = 538\) nm. A zero vector has no direction. 4. \(S^\top = (X W W^\top X^\top)^\top = X W W^\top X^\top = S\). The diagonal entries are \(\mathbf{a}_i \cdot \mathbf{a}_i = \lVert \mathbf{a}_i \rVert^2 \ge 0\) with \(\mathbf{a}_i = \mathbf{x}_i W\). 5. Smallest 0 (\(n_{\text{eff}} = 1\)), largest \(\log 9 = 2.197\) (\(n_{\text{eff}} = 9\)). 6. Only the values and the output change: the scores, the weights and the entropy depend on queries and keys only. The output of “it” becomes \(0.0148\,(1, 0) + 0.8055\,(0, 4) + 0.1797\,(0, 1) = (0.0148, 3.4017)\). 7. Each matrix is \(768 \times 64 = 49{,}152\), so three are 147,456 per head; 12 heads give 1,769,472, which is exactly GPT-2’s fused \(768 \times 2304\) matrix.

Key takeaways

  • Attention is a soft dictionary lookup: score every key against the query, turn the scores into weights with a softmax, and return the weighted average of the values (16.2, 16.3).
  • Each token plays three roles: query (what I look for), key (the label I show), value (what I give if chosen), made by three learned matrices from the same vector (16.4).
  • Without projections, tokens mostly attend to themselves; with one shared matrix, scores are symmetric (16.6). Separate \(W_Q, W_K\) give a learned, one-way, low-rank compatibility \(W_Q W_K^\top\) (16.5); a separate \(W_V\) decides what is passed on.
  • The weights of each reader are a probability distribution over positions (16.7): the output is an expected value inside the convex hull of the values, only score differences matter (16.8), and a key bias is useless.
  • Sharpness (query length, temperature) moves attention between a hard pick and a plain average (16.9); entropy and \(n_{\text{eff}}\) measure it (16.10). More positions dilute the weights.
  • The same pattern, with queries and keys/values from different sources, is cross-attention in VLMs, temporal grounding in video, and the KV cache in generation.

Key formulas

\[ \mathbf{q}_t = \mathbf{x}_t W_Q,\quad \mathbf{k}_t = \mathbf{x}_t W_K,\quad \mathbf{v}_t = \mathbf{x}_t W_V, \qquad a_{ij} = \frac{\exp(\mathbf{q}_i \cdot \mathbf{k}_j)}{\sum_m \exp(\mathbf{q}_i \cdot \mathbf{k}_m)} \]
\[ \mathbf{o}_i = \sum_j a_{ij}\, \mathbf{v}_j, \qquad O = \softmax\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V, \qquad H(\mathbf{a}_i) = -\sum_j a_{ij}\log a_{ij} \]

References for this session

  1. Vaswani et al. (2017). Attention Is All You Need
  2. Alammar (2018). The Illustrated Transformer (blog)
  3. Raschka, Build a Large Language Model (From Scratch), chapter 3: Coding attention mechanisms
  4. Bahdanau et al. (2014). Neural Machine Translation by Jointly Learning to Align and Translate
  5. Miller et al. (2016). Key-Value Memory Networks for Directly Reading Documents
  6. Xiao et al. (2023). Efficient Streaming Language Models with Attention Sinks
  7. Jain and Wallace (2019). Attention is not Explanation
  8. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale