LLM & VLM courseBlog
12Track 1: Foundations · session 12 of 15

Contextual embeddings, hidden states and the output layer

The embedding table gives every token one fixed vector, so “bank” by a river and “bank” with your money look the same. Each transformer layer then rewrites the vectors using the other tokens, and after a few layers every vector depends on its sentence. This session follows these hidden states layer by layer, shows how the output layer turns the last one into a score for every token in the vocabulary, and explains the three common ways to get one vector for a whole sentence.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 2, 3 and 11 (softmax, cross-entropy, embeddings)

What you will learn

  • Static versus contextual vectors: why the same token gets different hidden states in different sentences
  • The residual stream: what a hidden state is, how many there are, and what different layers tend to hold
  • The output layer (unembedding): from the last hidden state to logits, the final norm, weight tying, and what it costs
  • The geometry of logits: length acts like inverse temperature, shifts do not matter, and the low-rank limit of the softmax
  • The logit lens: reading what the model would predict at every layer
  • Sentence embeddings: CLS, mean pooling and last-token pooling, how padding breaks them, and why they need contrastive training
  • How the same choices appear in vision encoders and video retrieval

Why this matters

In Session 11 every token became one row of a table. That row is the same in every sentence. But words change meaning with their neighbors: “bank” in “the river bank” and “bank” in “a loan from the bank” are different things. A model that keeps one vector per token cannot tell them apart.

A transformer fixes this by rewriting the vectors, layer after layer, using the other tokens in the sentence. The vectors in the middle of the network are called hidden states. After a few layers each hidden state is a contextual embedding: a vector for one token in this sentence. At the very top, one more matrix, the output layer (also called the unembedding or LM head), turns the last hidden state into one score per vocabulary token. Softmax of those scores is the next-token distribution from session 6.

This session follows a token’s vector from the bottom of the model to the top. Four ideas come back again and again in this course:

  • Static versus contextual. The input embedding is static (one vector per token type). Hidden states are contextual (one vector per token per sentence). This is what made ELMo and BERT a big step over word2vec.
  • The residual stream. Each layer adds something to the hidden state. So the last hidden state is the input embedding plus everything all layers wrote. This view explains the logit lens, early exit and many interpretability tools.
  • The output layer is a big dot product. Logits are dot products of the hidden state with one vector per token. Its geometry explains why the length of the hidden state acts like a temperature and why some tokens can never win.
  • “The embedding of a sentence” is a choice, not a fact. You can take the CLS token, the mean over tokens, or the last token, from any layer. Each choice has rules (padding, causal masks) and none works well without training for similarity.

Interviewers ask about this often, because it connects many parts of the stack: “What is the difference between word2vec and BERT embeddings?”, “How do you get a sentence embedding from a decoder-only LLM?”, “Why mean pooling and not CLS?”, “What does the LM head compute and how much memory do the logits take?”, “What is the logit lens?”. Real systems depend on the answers: every semantic search engine, every RAG system and every text-to-video retrieval system turns hidden states into one vector per text.

Static and contextual vectors

The problem with one vector per token

The embedding table gives token \(i\) the row \(E_{i,:}\), whatever the sentence. A word with several meanings (a polysemous word, like “bank”, “bat”, “spring”) gets one vector that is a blend of all its meanings, weighted by how often each one appears in the training text. The same is true for subword tokens: the piece “ing” or “un” means different things in different words. The model must use the other tokens to decide what each one means.

Contextual vectors: let each token look at the others

The first popular contextual model was ELMo (Peters et al., 2018). It ran a two-layer bidirectional LSTM (session 7) over the sentence and used its internal states as word vectors. [Certain] One year later, BERT (Devlin et al., 2018) did the same with a transformer encoder, trained to fill in masked words (session 33). In both, the vector for “bank” is computed from the whole sentence, so it differs between sentences.

Let us write this down. The first hidden state of position \(t\) is its token embedding (plus a position signal, which we meet in session 13):

\[ \mathbf{h}^{(0)}_t = E_{\mathrm{id}_t,:} + \mathbf{p}_t \in \R^{d} \tag{12.1} \]

Each layer \(\ell = 1, \dots, L\) then computes a new hidden state for every position from the hidden states of all positions it may look at:

\[ \mathbf{h}^{(\ell)}_t = f_\ell\big(\mathbf{h}^{(\ell-1)}_1, \dots, \mathbf{h}^{(\ell-1)}_T\big)_t \tag{12.2} \]
Read the symbols
\(\mathbf{h}^{(\ell)}_t\)
the hidden state of position \(t\) after layer \(\ell\): a vector of \(d\) numbers. \(\ell = 0\) is the embedding output.
\(E_{\mathrm{id}_t,:}\)
the row of the embedding table for the token at position \(t\) (session 11)
\(\mathbf{p}_t\)
a position vector, or nothing if the model adds position inside attention (RoPE, session 14)
\(f_\ell\)
transformer block \(\ell\): attention, which mixes information between positions, then an MLP, which works on each position alone
\(T, L, d\)
sequence length, number of layers, hidden size (the same \(d\) as the embedding size)

The key part is the attention step. Here is its simplest form, which session 17 builds in full. Position \(t\) gives every position \(j\) a weight \(\alpha_{tj}\), and adds the weighted average of their vectors to its own:

\[ \alpha_{tj} = \softmax_j\big(\beta\, \mathbf{q}_t \cdot \mathbf{k}_j\big), \qquad \mathbf{h}^{\text{new}}_t = \mathbf{h}_t + \sum_{j} \alpha_{tj}\, \mathbf{v}_j \tag{12.3} \]
Read the symbols
\(\mathbf{q}_t, \mathbf{k}_j, \mathbf{v}_j\)
query, key and value: three vectors made from the hidden states by learned matrices. The query of “bank” asks “which neighbors tell me my meaning?”; the keys answer; the values carry the content that is copied.
\(\alpha_{tj}\)
how much position \(t\) takes from position \(j\). For each \(t\) the weights are positive and sum to 1.
\(\beta\)
a scale; real attention uses \(1/\sqrt{d_k}\)

Because the weights depend on the neighbors, the sum is different in every sentence. In “we fished from the river bank”, “bank” gives large weight to “fished” and “river”, and its vector moves toward the water meaning. In “she got a loan from the bank”, it moves toward money.

Worked example: one mixing step for “bank”

Take a toy model with three features per word: (water, money, other). The static vectors, scaled to length 1, are bank \(= (0.667, 0.667, 0.333)\), fished \(= (0.930, 0.116, 0.349)\) and river \(= (0.981, 0, 0.196)\). Function words like “we”, “from”, “the” are \((0, 0, 1)\). In this toy the query, key and value only use the two topic features, and \(\beta = 4\).

Scores of “bank” in “we fished from the river bank”: \(4(0.667 \cdot 0.930 + 0.667 \cdot 0.116) = 2.790\) for “fished”, \(2.615\) for “river”, \(3.556\) for itself, and 0 for the three function words. Softmax gives the weights \((0.015, 0.240, 0.015, 0.015, 0.201, 0.515)\).

The weighted sum of the topic parts is \((0.763, 0.371, 0)\). Adding it to the old vector gives \((1.430, 1.038, 0.333)\), and rescaling to length 1 gives \((0.795, 0.577, 0.185)\). The angle in the (water, money) plane went from 45° (exactly between the meanings) to 36.0°: one step toward “water”. In the loan sentence the same step moves “bank” to 50.3°, toward “money”. The two “bank” vectors had cosine 1 before the layer and 0.970 after it.

Attention weights of “bank” in sentence 1, at this layer
… and in sentence 2
Figure 1. From static to contextual. A toy model with 3 features per word applies the mixing step (12.3) again and again (one step per “layer”, each followed by rescaling to length 1). The plot shows the vector of “bank” in two sentences, projected on the (water, money) plane. The grey dot is the static embedding, the same in every sentence; the paths show where each layer moves it.
How to read Figure 1

At layer 0 both sentences give the same point: the static vector at 45°, cosine 1 between them. Move the slider. In the river sentence, “bank” turns toward the water axis (about 19° after 4 layers); in the loan sentence it turns toward money (about 64°). The cosine between the two “bank” vectors falls from 1 to about 0.71. Choose “i walked to the bank” as sentence 2: it has no topic words, so “bank” stays at 45° and keeps its blended meaning. Choose the two money sentences: their “bank” vectors stay almost identical (cosine 1.000), because the contexts agree. The token chips show where “bank” looks at each layer.

How contextual are real models?

Ethayarajh (2019) measured this for ELMo, BERT and GPT-2. Two results are worth remembering. First, the same word in different sentences gets less similar vectors in higher layers: upper layers are more context-specific. Second, in all layers of the three models, on average less than 5% of the variance of a word’s contextual vectors can be explained by one static vector for that word. [Certain] So hidden states are not “the word vector plus a small correction”: most of what they hold comes from the context. The same paper found that the hidden states of all words lie in a narrow cone (anisotropy, as in Figure 4 of session 11), so raw cosines between hidden states are high even for unrelated words. [Certain]

Hidden states, layer by layer

The residual stream

In a real transformer block, attention and MLP do not replace the hidden state. They add to it. This design is called a residual connection, and the running vector is called the residual stream (session 22):

\[ \mathbf{h}^{(\ell)}_t = \mathbf{h}^{(\ell-1)}_t + \mathbf{a}^{(\ell)}_t + \mathbf{m}^{(\ell)}_t \tag{12.4} \]
Read the symbols
\(\mathbf{a}^{(\ell)}_t\)
what the attention part of block \(\ell\) writes at position \(t\): information copied from other positions
\(\mathbf{m}^{(\ell)}_t\)
what the MLP part writes: a function of this position’s vector alone. (In the common “pre-norm” design each part first normalizes its input; the stream itself is not normalized.)

Unroll the sum and the last hidden state is the first one plus everything every layer wrote:

\[ \mathbf{h}^{(L)}_t = \mathbf{h}^{(0)}_t + \sum_{\ell=1}^{L} \big(\mathbf{a}^{(\ell)}_t + \mathbf{m}^{(\ell)}_t\big) \tag{12.5} \]

This is a useful picture. All hidden states live in the same \(d\)-dimensional space, and each layer makes a small edit to a shared notebook. That is why we can compare hidden states of different layers, and why we can decode a middle layer with the output layer (the logit lens, below).

theriverbank embedding table E (lookup) h⁽⁰⁾h⁽⁰⁾h⁽⁰⁾h⁽¹⁾h⁽¹⁾h⁽¹⁾h⁽ᴸ⁾h⁽ᴸ⁾h⁽ᴸ⁾ block 1: h + attention + MLP ⋮ blocks 2 … L, each adds to the stream ⋮ final norm unembedding WU (|V| × d) logits z: one score per vocabulary token hidden_states[0]: embedding output hidden_states[1]: after block 1 hidden_states[L]: after block L (the L + 1 tensors, each T × d) orange: attention mixes positions (causal: only from the left) sentence embeddings are pooled from rows of some hidden_states[ℓ]
Figure 2. Where the hidden states are. A decoder-only model with \(L\) blocks has \(L + 1\) hidden states per position: the embedding output and one after each block. The last one goes through the final norm and the unembedding matrix to give logits. During generation only the last position’s logits are needed.

How many hidden states, and how big

Libraries return all of them if you ask. In Hugging Face Transformers, model(**inputs, output_hidden_states=True).hidden_states is a tuple of \(L + 1\) tensors of shape \((B, T, d)\); element 0 is the embedding output. [Certain] In the GPT-2 and Llama implementations the last element already has the final norm applied, while the others do not. [Likely] Check this for your model before you compare layers.

Worked example: the size of all hidden states

GPT-2 small has \(L = 12\) and \(d = 768\), so it has 13 hidden states. For one sequence of 1,024 tokens in float32 (4 bytes): \(13 \times 1024 \times 768 \times 4 = 40.9\) MB.

Llama 3 8B has \(L = 32\) and \(d = 4096\), so 33 hidden states. For one sequence of 8,192 tokens in bfloat16 (2 bytes): \(33 \times 8192 \times 4096 \times 2 = 2.21\) GB. Asking for all hidden states of a long batch can use more memory than the model’s KV cache. Keep only the layer you need.

What different layers hold

Probing studies train a small classifier on hidden states of each layer to see what information is easy to read out there. Tenney, Das and Pavlick (2019) found that BERT holds the steps of a classic NLP pipeline in the expected order: part-of-speech tags in lower layers, then parsing, named entities, semantic roles, and coreference highest. [Certain] A rough rule that many papers support [Likely]:

  • Low layers: the token itself, its spelling and its neighbors. Hidden states are still close to the input embedding.
  • Middle layers: syntax and meaning in context. These often give the best features for probes and for frozen-model embeddings.
  • Top layers: shaped by the training task. In a language model they turn toward “what is the next token?”, which is not the same as “what does this text mean?”. In BERT they turn toward the masked-word task.

This is why vision-language models often read a vision encoder at the second-to-last layer, not the last one: LLaVA uses the CLIP features “before the last layer”, and reports that the last layer works slightly worse (89.96% on ScienceQA, below its main result). [Certain]

What “the embedding of a sentence” can mean

People say “take the embedding of the sentence” as if there were one. In fact you must make four choices:

  1. Which model. A language model, a masked-language model such as BERT, or a model trained for similarity (SBERT, E5).
  2. Which layer. The last hidden state, a middle layer, or an average of several layers.
  3. Which positions. One special token (CLS), the mean of all tokens, the last token, or a learned weighting.
  4. What normalization. Usually the vector is scaled to length 1, so that dot product equals cosine.

The section “Sentence embeddings from a model” below works through choice 3 and shows that choice 1 matters most.

The output layer: from hidden state to logits

Unembedding

To predict the next token, the model must turn the last hidden state at the last position, a vector of \(d\) numbers, into a score for each of the \(|V|\) tokens. It does this with the final norm and one matrix:

\[ \mathbf{z}_t = W_U\, \mathrm{Norm}\big(\mathbf{h}^{(L)}_t\big) \in \R^{|V|}, \qquad p(\text{next} = j \mid \text{text}_{\le t}) = \softmax(\mathbf{z}_t)_j \tag{12.6} \]
Read the symbols
\(W_U\)
the unembedding matrix, \(|V| \times d\). Row \(j\), written \(\mathbf{w}_j\), is the “output vector” of token \(j\).
\(\mathrm{Norm}\)
the final normalization: LayerNorm in GPT-2 (ln_f), RMSNorm in Llama (norm). [Certain] It removes the size that the residual stream has grown to over the layers and applies a learned per-feature gain.
\(\mathbf{z}_t\)
the logits at position \(t\): \(z_{t,j} = \mathbf{w}_j \cdot \mathrm{Norm}(\mathbf{h}^{(L)}_t)\)
\(p(\cdot)\)
the next-token distribution. It is trained with cross-entropy (session 3).

The name “unembedding” shows the symmetry with the input. The embedding table maps a token to a vector; the unembedding maps a vector to a score for every token. With weight tying (session 11) the two are the same matrix, \(W_U = E\), so token \(j\) is predicted when the hidden state points in the direction of token \(j\)’s own input vector. GPT-2 ties; Llama 2 and Llama 3 8B do not. [Certain] Most LLMs have no bias in this layer. [Likely]

The geometry of logits

Each logit is a dot product, so we can write it with lengths and an angle:

\[ z_j = \lVert \mathbf{w}_j \rVert \, \lVert \mathbf{h} \rVert \cos\theta_j \tag{12.7} \]

(Here \(\mathbf{h}\) is the normalized last hidden state and \(\theta_j\) is its angle with \(\mathbf{w}_j\).) Three facts follow.

  1. Direction picks the token. The token whose output vector is best aligned with \(\mathbf{h}\) (weighted by its length) gets the highest logit.
  2. Length acts like inverse temperature. Doubling \(\lVert \mathbf{h} \rVert\) doubles every logit. That is the same as dividing by a temperature of 0.5: the distribution becomes sharper, and the argmax does not change. The final norm’s gain controls this length, so the model can learn how confident to be. (session 79 adds an explicit temperature on top.)
  3. Shifts do not matter. Adding the same number \(c\) to all logits gives the same softmax, because \(e^{z_j + c} / \sum_k e^{z_k + c} = e^{z_j} / \sum_k e^{z_k}\). Only differences of logits matter.
Worked example: logits by hand

Three tokens with output vectors \(\mathbf{w}_1 = (1, 0)\), \(\mathbf{w}_2 = (0, 1)\), \(\mathbf{w}_3 = (-1, 0)\), and hidden state \(\mathbf{h} = (2, 1)\). The logits are \(\mathbf{z} = (2, 1, -2)\). Softmax: \(e^{2} = 7.389\), \(e^{1} = 2.718\), \(e^{-2} = 0.135\), sum \(10.243\), so \(\mathbf{p} = (0.721, 0.265, 0.013)\).

Double the length, \(\mathbf{h} = (4, 2)\): logits \((4, 2, -4)\), \(\mathbf{p} = (0.881, 0.119, 0.000)\). Halve it, \(\mathbf{h} = (1, 0.5)\): \(\mathbf{p} = (0.574, 0.348, 0.078)\). Token 1 wins every time; only the confidence changes. Add 5 to all three logits: \(\mathbf{p}\) stays \((0.721, 0.265, 0.013)\).

softmax over the 6 logits
Figure 3. The output layer as geometry, with \(d = 2\) and six tokens. Each grey or colored arrow is a row \(\mathbf{w}_j\) of \(W_U\); the thick arrow is the hidden state \(\mathbf{h}\). Logits are the dot products (12.7); the bars are their softmax.
How to read Figure 3

Turn the direction slider: the token whose arrow points most like \(\mathbf{h}\) (and is long) wins. Now keep the direction and change the length: at length 0 every token gets 1/6 = 0.167; as the length grows the winner’s probability goes toward 1. Length is a confidence knob, not a choice of token. Finally, look at the short arrow “thing”. It lies inside the dashed convex hull of the other arrows, so for every direction some other token has a larger dot product: “thing” can never be the top token, and its probability never gets much above 1/6. This is a real effect of a low-dimensional output layer, explained in the next part.

The gradient that reaches the hidden state

With cross-entropy loss \(\mathcal{L} = -\log p_y\) for the true next token \(y\), the gradient with respect to the logits is \(\mathbf{p} - \mathbf{y}\) (session 3), where \(\mathbf{y}\) is the one-hot vector of \(y\). The chain rule through \(\mathbf{z} = W_U \mathbf{h}\) gives:

\[ \frac{\partial \mathcal{L}}{\partial \mathbf{h}} = W_U^\top (\mathbf{p} - \mathbf{y}) = \sum_j p_j \mathbf{w}_j - \mathbf{w}_y, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{w}_j} = (p_j - y_j)\, \mathbf{h} \tag{12.8} \]

Read it as a tug of war. Gradient descent moves \(\mathbf{h}\) toward the output vector of the correct token and away from the probability-weighted average of all output vectors. Every row \(\mathbf{w}_j\) gets a gradient at every step (all \(p_j \gt 0\)), so with tying, even rare tokens’ embeddings are updated through the output side. In the worked example with true token 2: \(\mathbf{p} - \mathbf{y} = (0.721, -0.735, 0.013)\), so \(\partial\mathcal{L}/\partial\mathbf{h} = (0.708, -0.735)\): the step moves \(\mathbf{h}\) away from \(\mathbf{w}_1\) and toward \(\mathbf{w}_2\). The loss is \(-\log 0.265 = 1.327\).

The cost of the output layer

The output layer is one of the largest single matrix products in the model. Per token it costs \(2|V|d\) FLOPs: \(2 \times 128{,}256 \times 4{,}096 = 1.05\) billion for Llama 3 8B, and 77 million for GPT-2 small. Its output is big too: the logits tensor has shape \((B, T, |V|)\).

Worked example: the memory of the logits

Training Llama 3 8B on one sequence of 8,192 tokens produces \(8192 \times 128{,}256 = 1.05\) billion logits. In float32 that is \(4.20\) GB; in bfloat16, 2.10 GB. With a batch of 8 sequences, float32 logits alone take 33.6 GB, before the softmax and its gradient, which have the same size. This is why training code often computes the loss in chunks, or fuses the output layer with the cross-entropy so that the full logits tensor is never stored. [Likely] At generation time only the last position is needed, so the cost is \(|V|\) numbers per sequence (session 78).

The softmax bottleneck

Collect the hidden states of \(N\) different contexts as the rows of \(H \in \R^{N \times d}\). Their logits are \(Z = H W_U^\top\), an \(N \times |V|\) matrix whose rank is at most \(d\). The log-probabilities subtract one number per row (the log of the softmax denominator), which adds at most 1 to the rank:

\[ \log P = H W_U^\top - \mathbf{c}\,\mathbf{1}^\top, \qquad \operatorname{rank}(\log P) \le d + 1 \tag{12.9} \]
Read the symbols
\(\log P\)
the \(N \times |V|\) table of log-probabilities: one row per context, one column per token
\(\mathbf{c}\)
the vector of log-sum-exp values, one per context; \(\mathbf{1}\) is a vector of ones

For GPT-2 small this rank is at most 769, while the table has 50,257 columns. Yang et al. (2017) called this the softmax bottleneck: if the true log-probability table of language has a higher rank, no hidden state can express all distributions. [Certain] Figure 3 shows one visible symptom: a token whose output vector lies inside the convex hull of the others can never be the argmax. Grivas et al. (2022) checked 150 public models and found such “unargmaxable” tokens in 13 of them, but rarely and with little effect on quality. [Certain] In large models, \(d\) in the thousands seems to be enough in practice. [Likely]

Reading the middle layers: the logit lens

Formula (12.5) says that the last hidden state is a sum of edits in one shared space. So we can stop at any layer \(\ell\) and decode the partial sum with the same final norm and unembedding. This trick is called the logit lens (nostalgebraist, 2020):

\[ \mathbf{p}^{(\ell)}_t = \softmax\Big(W_U\, \mathrm{Norm}\big(\mathbf{h}^{(\ell)}_t\big)\Big), \qquad \ell = 0, 1, \dots, L \tag{12.10} \]

For GPT-2, the post showed that the decoded guesses get closer to the final prediction layer by layer, often reaching the right token well before the top. [Certain] The method costs nothing and needs no training, but it is a rough tool: middle layers were never trained to be read by \(W_U\), and for some model families the decoded distributions look like noise. The tuned lens (Belrose et al., 2023) trains one small affine map per layer before the unembedding and is more reliable. [Certain] The same idea powers early exit: stop computing when a middle layer is already confident (session 121).

residual stream at the last position (5 features)
logit lens: \(\softmax\big(W_U\, \mathrm{Norm}(\mathbf{h})\big)\)
Figure 4. The logit lens on a hand-made toy model with \(d = 5\) features (function word, place, France, capital, UK), 7 tokens, and 4 layers. The prompt is “The Eiffel Tower is in the city of”. Each step adds one layer’s write (12.4) to the residual stream at the last position and decodes the partial sum with formula (12.10), using RMSNorm with gain 2.
How to read Figure 4

The layer outputs here are made up to show the idea, but every number on the screen is computed from them. Step 1 decodes the embedding of the last token “of”: it predicts “the”, a common next word after “of”, with no use of the context. Each layer then writes into the stream (dark part of the bars; a dashed outline marks a part the layer removed): layer 1 adds “place”, layer 2 brings in “France” from the word “Eiffel”, layer 3 adds “capital”. Watch the top guess go “the” \(\to\) “the” \(\to\) “France” \(\to\) “Paris”, and the entropy rise and then fall. In real models the path is noisier, but this pattern of a plausible early guess that is refined later is common.

Sentence embeddings from a model

Search, clustering, RAG and retrieval need one vector per text, not one per token. We must pool the \(T\) hidden states of one layer into one vector. Every method needs the attention mask \(m_t\), which is 1 for real tokens and 0 for padding. Batches mix texts of different lengths, so short texts are filled with pad tokens, and pad positions still get hidden states that mean nothing.

CLS pooling

BERT puts a special token [CLS] at the start of every input (session 9). Its last hidden state is used as the summary:

\[ \mathbf{s} = \mathbf{h}^{(L)}_{1} \quad \text{(position 1 holds [CLS])} \tag{12.11} \]

This works when the model was trained to put the summary there. BERT’s CLS vector was trained only through the “next sentence prediction” task (its pooler_output is \(\tanh(W\mathbf{h}_1 + \mathbf{b})\) on top of it). [Certain] After fine-tuning for classification, CLS works well for that task. Without fine-tuning, it is a poor general sentence vector (see the numbers below).

Mean pooling

Average the hidden states of the real tokens:

\[ \mathbf{s} = \frac{\sum_{t=1}^{T} m_t\, \mathbf{h}^{(\ell)}_t}{\sum_{t=1}^{T} m_t} \tag{12.12} \]
Read the symbols
\(m_t\)
the attention mask: 1 for a real token, 0 for padding. Forgetting it averages in the pad vectors.
\(\mathbf{h}^{(\ell)}_t\)
the hidden state of position \(t\) at the chosen layer, usually the last
\(\mathbf{s}\)
the sentence embedding, usually scaled to length 1 afterwards

Mean pooling uses every token, so it is robust, and it is the default in Sentence-BERT. [Certain] It suits bidirectional encoders, where every hidden state has seen the whole text.

Last-token pooling

In a decoder-only model, position \(t\) can only see positions \(1, \dots, t\). Only the last real token has seen the whole text, so it is the natural summary:

\[ \mathbf{s} = \mathbf{h}^{(L)}_{t^*}, \qquad t^* = \max\{\, t : m_t = 1 \,\} \tag{12.13} \]

The index \(t^*\) is the last real position. With right padding it is not the last column of the batch, and taking h[:, -1] picks a pad. With left padding (common for decoder models) the last column is right. E5-mistral appends an [EOS] token to every text and uses its last-layer hidden state, fine-tuning Mistral-7B for retrieval. [Certain] A mean over a causal model’s hidden states gives too much weight to early tokens, which saw little context; SGPT uses a position-weighted mean that gives later tokens more weight. [Likely] LLM2Vec goes further: it turns on bidirectional attention, trains with masked next-token prediction, then adds contrastive training. [Certain]

Worked example: mean pooling with and without the mask

The batch row for “the cat sat” is padded to 6 positions on the right. The real hidden states are \((0.2, 0.1, 0.9, 0)\), \((0.9, 0.3, 0.1, 0.2)\), \((0.4, 0.8, 0.2, 0.1)\). Masked mean (12.12): sum \((1.5, 1.2, 1.2, 0.3)\), divided by 3, gives \((0.5, 0.4, 0.4, 0.1)\). This equals the mean of the sentence alone, without padding, so the batch does not change the answer.

Without the mask we divide the sum of all 6 rows by 6. The three pad rows of Figure 5 sum to \((-1.7, 0.9, -0.1, 2.3)\), so the result is \((-0.033, 0.35, 0.183, 0.433)\), whose cosine with the correct vector is only 0.537. The same sentence would get a different embedding depending on what else is in its batch.

Figure 5. Pooling a padded batch. Row 2 holds the short sentence “the cat sat” and three pad positions. Each cell shows a 4-number hidden state (hover to see it) and, below the token, the weight the pooling method gives that position. The readout compares the pooled vector with the same method applied to the sentence alone, with no padding.
How to read Figure 5

A correct method gives cosine 1.000: padding must not change a text’s embedding. Masked mean and last real token are correct for both padding sides. “Mean, no mask” is wrong for both (cosine 0.537). “Last column” is wrong with right padding (it picks a pad) and correct with left padding. “First token” is correct with right padding and wrong with left padding, which is one reason BERT-style models pad on the right and decoder models used for last-token pooling often pad on the left. In this toy we assume the real tokens’ hidden states do not change with padding; a real model gives that only if its attention mask and position ids handle the pads correctly.

Without training for similarity, pooling is not enough

Here is the most important fact about sentence embeddings: the pooling method matters much less than whether the model was trained to make similar texts close. Sentence-BERT (Reimers and Gurevych, 2019) measured the Spearman correlation between cosine similarity and human similarity scores, averaged over seven semantic textual similarity (STS) tasks: [Certain]

Sentence vectorAverage STS (Spearman × 100)
BERT, CLS vector, no fine-tuning29.19
BERT, mean of last-layer hidden states, no fine-tuning54.81
Average of GloVe word vectors (static)61.32
SBERT-NLI-base (BERT fine-tuned on sentence pairs, mean pooling)74.89
SBERT-NLI-large76.55

Plain BERT vectors are worse than averaged static GloVe vectors. Fine-tuning on sentence pairs adds about 20 points. In the same paper the pooling ablation on the STS benchmark gave 87.44 for mean, 86.62 for CLS and 69.92 for max pooling (with a regression objective). [Certain] So mean pooling is a good default, but the training is what matters.

Modern embedding models (SimCSE, E5, and session 60) train with a contrastive loss: in a batch of \(N\) pairs of related texts \((x_i, x_i^+)\), each text must pick its partner among all partners in the batch:

\[ \mathcal{L}_i = -\log \frac{\exp\big(\cos(\mathbf{s}_i, \mathbf{s}_i^{+}) / \tau\big)}{\sum_{j=1}^{N} \exp\big(\cos(\mathbf{s}_i, \mathbf{s}_j^{+}) / \tau\big)} \tag{12.14} \]
Read the symbols
\(\mathbf{s}_i, \mathbf{s}_i^{+}\)
pooled embeddings of a text and of its related text (a paraphrase, an answer, a caption)
\(\mathbf{s}_j^{+}, j \ne i\)
the partners of the other texts in the batch: free negatives
\(\tau\)
a temperature, often around 0.05 [Likely]; small \(\tau\) makes the softmax sharp

This is the same “pull real pairs together, push the others apart” as negative sampling in word2vec (11.6) and as CLIP. It also fixes much of the anisotropy that makes raw hidden-state cosines all high.

Beyond text: images and video

The same choices appear in vision encoders, which are transformers over image patches (session 164):

  • CLS or mean. The original ViT classifies from a [CLS] token, like BERT. [Certain] CLIP’s image embedding is also read from that class token, then projected into the shared space (session 166). [Likely] Other encoders use the mean over patches or a small attention-pooling head (SigLIP). [Likely]
  • Patch hidden states, not one vector. A VLM such as LLaVA does not pool. It passes all patch hidden states (from the second-to-last layer, as above) through a projector into the LLM, so the LLM can look at every region. Pooling to one vector is for retrieval; generation keeps the grid.
  • Video: pooling over time. A clip becomes one embedding per frame (or per group of frames); a single clip vector for retrieval is often the mean over frames, and fingerprinting keeps per-frame vectors to match moments. Mean pooling over time loses order, so actions like “open a door” and “close a door” can get almost the same vector (session 202). [Likely]

And the output layer has a vision twin: a classifier head is a small unembedding over class names, and CLIP zero-shot classification uses the text embeddings of class names as the rows of \(W_U\).

Where this is used

ApplicationWhich vector, and howLink
Next-token prediction in every LLM and VLMLast hidden state at the last position \(\to\) final norm \(\to\) unembedding (12.6) \(\to\) softmaxsession 78
Semantic search and RAGMean or last-token pooling (12.12), (12.13) of a contrastively trained model (12.14); cosine rankingsession 150
Text classification and NERCLS vector (12.11) for the sentence, or every token’s hidden state for token labels, plus a small output layersession 55
Text-to-video retrievalText embedding of the query against clip embeddings pooled over frames, in one shared spacesession 208
Video fingerprinting and near-duplicatesPer-frame embeddings kept (not pooled) so that moments can be matched in ordersession 209
Action recognition and key-moment detectionPer-frame hidden states scored against text embeddings of action names; a moment is where the score peakssession 207
VLM connectorsAll patch hidden states from a chosen vision layer go through a projector into the LLMsession 179
Interpretability and early exitLogit lens (12.10) or tuned lens on middle layers; stop when a middle layer is confidentsession 157
DistillationMatch the student’s logits, or its hidden states through a small projection, to the teacher’ssession 119

Pseudo-code

Algorithm 1: forward pass that keeps all hidden states and the logits
  1. input: token IDs \(\mathrm{id}_{1..T}\), mask \(m_{1..T}\), table \(E\), blocks \(f_1, \dots, f_L\), final norm, \(W_U\)
  2. \(\mathbf{h}^{(0)}_t \leftarrow E_{\mathrm{id}_t,:} + \mathbf{p}_t\) for all \(t\) // formula (12.1), static
  3. \(\mathcal{H} \leftarrow [\mathbf{h}^{(0)}]\)
  4. for \(\ell = 1\) to \(L\) do
  5. \(\mathbf{h}^{(\ell)} \leftarrow \mathbf{h}^{(\ell-1)} + \mathrm{Attn}_\ell(\mathbf{h}^{(\ell-1)}, m) + \mathrm{MLP}_\ell(\cdot)\) // residual stream (12.4); attention mixes positions
  6. append \(\mathbf{h}^{(\ell)}\) to \(\mathcal{H}\)
  7. \(\mathbf{z}_t \leftarrow W_U\, \mathrm{Norm}(\mathbf{h}^{(L)}_t)\) // formula (12.6); at generation only for the last \(t\)
  8. return \(\mathcal{H}\) (\(L + 1\) tensors), \(\mathbf{z}\)
Algorithm 2: logit lens
  1. input: hidden states \(\mathcal{H}\) from Algorithm 1, position \(t\), final norm, \(W_U\)
  2. for \(\ell = 0\) to \(L\) do
  3. \(\mathbf{p}^{(\ell)} \leftarrow \softmax\big(W_U\, \mathrm{Norm}(\mathbf{h}^{(\ell)}_t)\big)\) // formula (12.10); skip Norm for the last one if it is already applied
  4. record \(\argmax \mathbf{p}^{(\ell)}\), its probability, and the entropy of \(\mathbf{p}^{(\ell)}\)
  5. return the per-layer guesses // compare with the final prediction
Algorithm 3: sentence embedding from a padded batch
  1. input: hidden states \(H \in \R^{B \times T \times d}\) of one layer, mask \(M \in \{0, 1\}^{B \times T}\), method
  2. if method = CLS then \(\mathbf{s}_b \leftarrow H_{b,1}\) // (12.11); requires right padding
  3. if method = mean then \(\mathbf{s}_b \leftarrow \sum_t M_{bt} H_{bt} / \sum_t M_{bt}\) // (12.12); never divide by T
  4. if method = last then \(t^*_b \leftarrow \max\{t : M_{bt} = 1\}\); \(\ \mathbf{s}_b \leftarrow H_{b, t^*_b}\) // (12.13); works for both padding sides
  5. \(\mathbf{s}_b \leftarrow \mathbf{s}_b / \lVert \mathbf{s}_b \rVert\) // so that dot product = cosine
  6. return \(\mathbf{s}_1, \dots, \mathbf{s}_B\)

The same ideas in PyTorch, with no downloads. A tiny decoder-only model is trained for a few seconds on four sentences, then each block checks one claim of this session. The printed values come from PyTorch 2.8 on CPU; other versions may differ in the last digits.

import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)

texts = ["we fished from the muddy river bank .", "they took a loan from the bank .",
         "we swam in the river .", "they paid the loan to the bank ."]
vocab = ["<pad>"] + sorted({w for t in texts for w in t.split()})
ix = {w: i for i, w in enumerate(vocab)}
V, d, L, T_MAX, PAD, H = len(vocab), 32, 3, 12, 0, 2

class TinyLM(nn.Module):
    """Decoder-only toy: embedding, L causal blocks, final norm, tied unembedding."""
    def __init__(self):
        super().__init__()
        self.tok = nn.Embedding(V, d)
        self.pos = nn.Embedding(T_MAX, d)
        self.blocks = nn.ModuleList(
            nn.TransformerEncoderLayer(d, H, 4 * d, dropout=0.0, batch_first=True, norm_first=True)
            for _ in range(L))
        self.norm = nn.LayerNorm(d)
    def forward(self, ids, mask):
        B, T = ids.shape
        pos = (mask.cumsum(1) - 1).clamp(min=0)            # positions from the mask: left padding is safe
        h = self.tok(ids) + self.pos(pos)                  # h^(0), formula (12.1)
        blocked = torch.triu(torch.ones(T, T, dtype=torch.bool), 1) | (mask == 0)[:, None, :]
        blocked &= ~torch.eye(T, dtype=torch.bool)         # a pad may see itself: avoids NaN rows
        blocked = blocked.repeat_interleave(H, 0)          # (B * heads, T, T)
        hs = [h]
        for blk in self.blocks:                            # residual stream, formula (12.4)
            h = blk(h, src_mask=blocked)
            hs.append(h)
        logits = self.norm(h) @ self.tok.weight.T          # tied unembedding, formula (12.6)
        return logits, hs

def encode(sents, side="right"):
    toks = [[ix[w] for w in s.split()] for s in sents]
    T = max(map(len, toks))
    ids = torch.full((len(toks), T), PAD); mask = torch.zeros(len(toks), T, dtype=torch.long)
    for b, t in enumerate(toks):
        sl = slice(0, len(t)) if side == "right" else slice(T - len(t), T)
        ids[b, sl] = torch.tensor(t); mask[b, sl] = 1
    return ids, mask

# a few hundred steps of next-token training on the four sentences
model = TinyLM()
opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
ids, mask = encode(texts)
for step in range(300):
    logits, _ = model(ids, mask)
    tgt = ids[:, 1:].masked_fill(mask[:, 1:] == 0, -100)   # ignore pads in the loss
    loss = F.cross_entropy(logits[:, :-1].reshape(-1, V), tgt.reshape(-1), ignore_index=-100)
    opt.zero_grad(); loss.backward(); opt.step()
model.eval()
print("final loss", round(loss.item(), 3))                # 0.107

with torch.no_grad():
    # 1) static vs contextual: "bank" is token 7 of both sentences
    ids, mask = encode(texts[:2])
    logits, hs = model(ids, mask)
    print(len(hs), tuple(hs[-1].shape), tuple(logits.shape))   # 4 (2, 8, 32) (2, 8, 17): L + 1 hidden states
    for l, h in enumerate(hs):
        print(l, round(F.cosine_similarity(h[0, 6], h[1, 6], dim=0).item(), 3))
    # 0 1.0 | 1 0.985 | 2 0.982 | 3 0.966: identical at layer 0, then the contexts pull them apart

    # 2) logit lens on "we fished from the", formula (12.10)
    p_ids, p_mask = encode(["we fished from the"])
    _, hs1 = model(p_ids, p_mask)
    for l, h in enumerate(hs1):
        p = F.softmax(model.norm(h[0, -1]) @ model.tok.weight.T, dim=-1)
        print(l, vocab[p.argmax()], round(p.max().item(), 3))
    # 0 the 1.0 | 1 muddy 0.925 | 2 muddy 0.989 | 3 muddy 0.999
    # with tied weights, layer 0 just "predicts" the input token itself

    # 3) pooling, formulas (12.12) and (12.13), and the padding test
    def mean_pool(h, mask):
        m = mask.unsqueeze(-1).float()
        return (h * m).sum(1) / m.sum(1).clamp(min=1)
    def last_token_pool(h, mask):
        last = mask.shape[1] - 1 - mask.flip(1).argmax(1)    # index of the last real token
        return h[torch.arange(h.shape[0]), last]
    short = "they took a loan"
    _, ha = model(*encode([short]))                         # the sentence alone, no padding
    one = torch.ones(1, 4, dtype=torch.long)
    c = lambda a, b: round(F.cosine_similarity(a, b, dim=0).item(), 4)
    for side in ["right", "left"]:
        ids_b, m_b = encode([texts[0], short], side)
        _, hb = model(ids_b, m_b)
        print(side,
              c(last_token_pool(hb[-1], m_b)[1], last_token_pool(ha[-1], one)[0]),  # last real token
              c(hb[-1][1, -1], ha[-1][0, -1]),                                   # last column
              c(mean_pool(hb[-1], m_b)[1], mean_pool(ha[-1], one)[0]),           # masked mean
              c(hb[-1][1].mean(0), ha[-1][0].mean(0)))                           # mean without mask
    # right 1.0 0.8884 1.0 0.8276
    # left  1.0 1.0    1.0 0.8348

# 4) the gradient that reaches the hidden state, formula (12.8), on the worked example
W = torch.tensor([[1., 0.], [0., 1.], [-1., 0.]])
h = torch.tensor([2., 1.], requires_grad=True)
loss = F.cross_entropy((W @ h).unsqueeze(0), torch.tensor([1]))
loss.backward()
print(round(loss.item(), 3), h.grad)                      # 1.327 tensor([ 0.7082, -0.7346])

Two details in this code are real-world bugs in miniature. The position ids come from the mask (cumsum), so a left-padded sentence gets positions 0, 1, 2, … for its real tokens, exactly as when it is alone. And each pad position is allowed to attend to itself: with left padding and a causal mask, a pad row would otherwise have no allowed key at all, the softmax would return NaN, and the NaN would spread into the real tokens through the next layer.

Common mistakes

  • Using the input embedding table as a “contextual” embedding. Rows of \(E\) (or hidden state 0) are static. For meaning in context you need hidden states after at least some layers.
  • Mean pooling without the mask. Dividing by \(T\) and averaging pad vectors makes a text’s embedding depend on the batch it is in (Figure 5: cosine 0.537 with the right answer).
  • Taking h[:, -1] as the last token with right padding. That position is a pad. Use the mask to find the last real token, or pad on the left.
  • Using raw BERT or raw LLM hidden states as sentence embeddings for search. Without contrastive or pair training, they can be worse than averaged GloVe vectors (29.19 for BERT CLS against 61.32 for GloVe on STS). Use a model trained for embeddings.
  • Comparing hidden states across layers without care. In many implementations the last element of hidden_states has the final norm applied and the others do not, and the residual stream grows in size with depth. Normalize before comparing or decoding.
  • Materializing full logits in training. \(B \times T \times |V|\) float32 logits for Llama 3 8B take 4.2 GB per 8k-token sequence. Compute the loss in chunks or with a fused kernel, and at generation compute logits only for the last position.
  • Untying by accident (or tying by accident). Loading weights, resizing the vocabulary, or wrapping a model can break the shared tensor so that the input table and the LM head drift apart. Check with model.lm_head.weight is model.get_input_embeddings().weight (Hugging Face) after every change.
  • Trusting the logit lens too much. Middle layers were not trained to be decoded by \(W_U\). A wrong early guess does not prove the model “believes” it; the tuned lens or probes are better evidence.

Interview questions

What is the difference between a static embedding and a contextual embedding?basic
A static embedding gives one vector per token type: word2vec, GloVe, or the input table \(E\) of an LLM. “Bank” has the same vector in every sentence. A contextual embedding is a hidden state after one or more transformer layers (or LSTM layers, as in ELMo): attention mixes in information from the other tokens (12.3), so “bank” in “river bank” and in “bank loan” get different vectors. Ethayarajh (2019) found that, on average, less than 5% of the variance of a word’s contextual vectors is explained by one static vector, and that upper layers are the most context-specific.
What is a hidden state, and how many does a model with L layers have per token?basic
A hidden state is the \(d\)-dimensional vector at one position between layers. There are \(L + 1\): the embedding output \(\mathbf{h}^{(0)}\) and one after each block. Hugging Face returns them as a tuple of \(L + 1\) tensors of shape \((B, T, d)\) with output_hidden_states=True. GPT-2 small has 13, Llama 3 8B has 33. Each block adds to the residual stream (12.4), so all hidden states live in the same space.
What does the LM head compute? Write the formula and give its cost.basic
\(\mathbf{z} = W_U\, \mathrm{Norm}(\mathbf{h}^{(L)})\) with \(W_U \in \R^{|V| \times d}\), then softmax gives the next-token distribution (12.6). Each logit is the dot product of the hidden state with one token’s output vector. Cost: \(2|V|d\) FLOPs per token (1.05 GFLOPs for Llama 3 8B) and a logits tensor of \(B \times T \times |V|\) numbers in training (4.2 GB in float32 for one 8k-token Llama 3 sequence). With weight tying, \(W_U = E\).
If you double the norm of the final hidden state, what happens to the output distribution?core
All logits double (12.7), which is the same as sampling at temperature 0.5: the argmax stays, the distribution gets sharper, entropy falls. Example: logits \((2, 1, -2)\) give \((0.721, 0.265, 0.013)\); \((4, 2, -4)\) give \((0.881, 0.119, 0.000)\). This is why the final norm’s learned gain acts as a confidence control. Adding a constant to all logits, in contrast, changes nothing.
How do you get a sentence embedding from BERT, and why is mean pooling often preferred to CLS?core
Options: the last hidden state of [CLS] (12.11), the masked mean of the token hidden states (12.12), or max pooling. BERT’s CLS was trained only through next-sentence prediction, so without fine-tuning it is a poor general summary: in the SBERT paper, raw CLS scored 29.19 average STS against 54.81 for raw mean pooling. Mean pooling uses all tokens and is more robust; it is SBERT’s default. After pair fine-tuning the gap is small (87.44 mean vs 86.62 CLS on STSb), and the fine-tuning itself matters far more than the pooling.
How do you get a sentence embedding from a decoder-only LLM?core
Because of the causal mask, only the last real token has seen the whole text, so the common choice is last-token pooling (12.13), often on an appended EOS token: E5-mistral appends [EOS] and uses its last-layer hidden state, fine-tuned contrastively. Find the last real position from the attention mask, or use left padding. Plain mean pooling over-weights early tokens that saw little context; SGPT uses a position-weighted mean. LLM2Vec instead enables bidirectional attention and retrains (masked next-token prediction, then contrastive learning). In all cases the model needs contrastive training (12.14) to be a good embedding model.
Your embeddings change when the same sentence is put in a different batch. What could be wrong?core
Padding is leaking in. Typical causes: mean pooling without the mask (dividing by \(T\)); taking the last column with right padding; CLS pooling with left padding; position ids not computed from the mask, so left padding shifts positions; or a missing padding mask in attention, so real tokens attend to pads. Test: embed a sentence alone and inside a padded batch, and check that the cosine is 1 up to float error (Figure 5 and the PyTorch code do exactly this).
What is the logit lens, and what are its limits?core
Decode a middle hidden state as if it were the last one: \(\softmax(W_U\,\mathrm{Norm}(\mathbf{h}^{(\ell)}))\) (12.10). It works because of the residual stream: every layer adds to one shared vector in the same space that \(W_U\) reads. For GPT-2 it shows guesses converging toward the final prediction. Limits: middle layers were never trained to be read that way, the basis can drift between layers, and for some models the outputs look like noise. The tuned lens trains one affine map per layer and is more reliable. Related: early-exit inference.
What is the softmax bottleneck?advanced
Stack hidden states of many contexts into \(H \in \R^{N \times d}\). The log-probability table is \(HW_U^\top - \mathbf{c}\mathbf{1}^\top\), so its rank is at most \(d + 1\) (12.9), while it has \(|V|\) columns (769 versus 50,257 for GPT-2 small). If real language needs a higher-rank table, a single softmax cannot express all the right distributions (Yang et al., 2017, proposed a mixture of softmaxes). A geometric symptom: a token whose output vector is inside the convex hull of others can never be the argmax. Grivas et al. (2022) found such tokens in 13 of 150 models, but rarely and with little effect.
Derive the gradient of the cross-entropy loss with respect to the last hidden state. What does it say?advanced
\(\partial\mathcal{L}/\partial\mathbf{z} = \mathbf{p} - \mathbf{y}\), and \(\mathbf{z} = W_U\mathbf{h}\), so \(\partial\mathcal{L}/\partial\mathbf{h} = W_U^\top(\mathbf{p} - \mathbf{y}) = \sum_j p_j\mathbf{w}_j - \mathbf{w}_y\) (12.8). Descent moves \(\mathbf{h}\) toward the correct token’s output vector and away from the probability-weighted mean of all output vectors. For the weights, \(\partial\mathcal{L}/\partial\mathbf{w}_j = (p_j - y_j)\mathbf{h}\): every row gets a gradient every step, the correct row is pulled toward \(\mathbf{h}\), and the others are pushed away in proportion to their probability. With tying this also updates rare tokens’ input embeddings.
Which layer of a vision encoder should feed a VLM or a retrieval index, and why?advanced
It depends on the use. The last layer of a contrastive encoder like CLIP is shaped for the image–text matching objective and, after pooling, is best for retrieval and zero-shot classification. For a VLM connector, a slightly lower layer often keeps more general, spatial detail: LLaVA uses the CLIP features before the last layer and found the last layer slightly worse. VLMs also keep all patch hidden states rather than pooling, so the LLM can attend to regions. For video, per-frame vectors are kept for fingerprinting and moment search; a mean over frames is enough for clip-level retrieval but loses order.

Exercises

  1. BERT-base has 12 layers and \(d = 768\). With output_hidden_states=True, how many tensors do you get, and how many float32 megabytes do they take for one sequence of 512 tokens?
  2. A model has \(|V| = 151{,}936\). How much memory do float32 logits take for one sequence of 4,096 tokens?
  3. Logits are \((3, 1, 0)\). Compute the softmax. What is the softmax of \((13, 11, 10)\)?
  4. Hidden states of a right-padded row are \((1, 0)\), \((0, 2)\), \((3, 1)\) with mask \((1, 1, 0)\). Compute the masked mean and the unmasked mean.
  5. For GPT-2 small (\(d = 768\), \(|V| = 50{,}257\), no output bias), what is the largest possible rank of the log-probability table over any set of contexts?
  6. A decoder-only embedding model uses last-token pooling. Your batch is right-padded and your code takes h[:, -1]. Which texts in the batch get a correct embedding?
  7. In Figure 1, why does “bank” in “i walked to the bank” not move at all? What would a real model need in order to resolve it?
Answerscheck
1. \(L + 1 = 13\) tensors of \(512 \times 768\); \(13 \times 512 \times 768 = 5{,}111{,}808\) numbers, \(\times 4\) bytes \(\approx 20.4\) MB. 2. \(4096 \times 151{,}936 \times 4\) bytes \(\approx 2.49\) GB. 3. \(e^3 = 20.09\), \(e^1 = 2.72\), \(e^0 = 1\), sum 23.81: \((0.844, 0.114, 0.042)\). Adding 10 to every logit changes nothing, so \((13, 11, 10)\) gives the same \((0.844, 0.114, 0.042)\). 4. Masked: \(((1, 0) + (0, 2))/2 = (0.5, 1.0)\). Unmasked: \((4, 3)/3 = (1.333, 1.0)\), which includes the pad. 5. At most \(d + 1 = 769\) (12.9), far below 50,257. 6. Only the longest texts, whose last real token is in the last column. Every shorter text gets a pad’s hidden state. 7. In the toy, “bank” only looks at topic words, and this sentence has none, so the weighted sum adds a vector in the same direction as “bank” itself and the rescaled vector does not turn. A real model would need more context (the next sentence, or a document) to choose; until then a blended hidden state is the honest answer.

Key takeaways

  • The input embedding \(\mathbf{h}^{(0)}\) is static; every hidden state after an attention layer is contextual, because attention mixes in the other tokens (12.2), (12.3). Real hidden states are mostly context: less than 5% of their variance is explained by one static vector per word.
  • A model with \(L\) layers has \(L + 1\) hidden states per position. Each block adds to the residual stream (12.4), so the last hidden state is the embedding plus all layer writes (12.5).
  • The output layer computes logits \(\mathbf{z} = W_U\,\mathrm{Norm}(\mathbf{h}^{(L)})\) (12.6): one dot product per vocabulary token. Direction chooses the token, length sets the confidence, shifts change nothing (12.7). It costs \(2|V|d\) FLOPs per token and a large logits tensor in training.
  • The gradient pulls \(\mathbf{h}\) toward the correct token’s output vector and away from the average (12.8). The log-probability table has rank at most \(d + 1\) (12.9), the softmax bottleneck.
  • The logit lens decodes middle layers with the same head (12.10); it is a cheap but rough window into the model.
  • Sentence embeddings pool hidden states: CLS (12.11), masked mean (12.12) or last real token (12.13). Padding must never change the result. Contrastive training (12.14) matters far more than the pooling choice.

Key formulas

\[ \mathbf{h}^{(\ell)}_t = \mathbf{h}^{(\ell-1)}_t + \mathbf{a}^{(\ell)}_t + \mathbf{m}^{(\ell)}_t, \qquad \mathbf{z}_t = W_U\,\mathrm{Norm}\big(\mathbf{h}^{(L)}_t\big), \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{h}} = W_U^\top(\mathbf{p} - \mathbf{y}) \]
\[ \mathbf{s}_{\text{mean}} = \frac{\sum_t m_t\,\mathbf{h}_t}{\sum_t m_t}, \qquad \mathbf{s}_{\text{last}} = \mathbf{h}_{t^*},\ t^* = \max\{t : m_t = 1\}, \qquad \operatorname{rank}(\log P) \le d + 1 \]

References for this session

  1. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 9: Masked Language Models
  2. Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  3. Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
  4. Peters, Neumann, Iyyer, Gardner, Clark, Lee, Zettlemoyer (2018). Deep Contextualized Word Representations
  5. Ethayarajh (2019). How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
  6. Tenney, Das, Pavlick (2019). BERT Rediscovers the Classical NLP Pipeline
  7. Press & Wolf (2017). Using the Output Embedding to Improve Language Models
  8. Yang, Dai, Salakhutdinov, Cohen (2017). Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
  9. Grivas, Bogoychev, Lopez (2022). Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practice
  10. nostalgebraist (2020). interpreting GPT: the logit lens
  11. Belrose, Ostrovsky, McKinney, Furman, Smith, Halawi, Biderman, Steinhardt (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens
  12. Gao et al. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings
  13. Wang, Yang, Huang, Yang, Majumder, Wei (2024). Improving Text Embeddings with Large Language Models
  14. Muennighoff (2022). SGPT: GPT Sentence Embeddings for Semantic Search
  15. BehnamGhader, Adlakha, Mosbach, Bahdanau, Chapados, Reddy (2024). LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
  16. Liu et al. (2023). Visual Instruction Tuning
  17. Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  18. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision