Interpretability: looking inside the model
A language model is a big stack of matrix multiplications, but inside it we can find readable structure: predictions that form layer by layer, attention heads that copy patterns, directions that encode concepts, and sparse features that the model uses as its vocabulary of ideas. This session teaches the four main tools for looking inside (the logit lens, attention-head analysis, probing classifiers and sparse autoencoders) plus activation patching, which tests whether what you found is really used. Every number in the figures comes from GPT-2 small or from a small toy that we explain fully.
What you will learn
- The residual stream view of a transformer: every layer reads from and writes into one shared vector, which is why we can inspect it anywhere
- The logit lens: decode the hidden state at every layer with the output layer, and what real GPT-2 numbers look like; the tuned lens as the fix for early layers
- Attention heads as QK and OV circuits, previous-token heads and induction heads, the induction score, and an ablation that shows the two heads work together
- Probing classifiers: what a linear probe can and cannot prove, control tasks and selectivity
- Superposition and sparse autoencoders: why single neurons are hard to read, how an SAE turns activations into sparse features, and the sparsity versus reconstruction trade-off
- Activation patching: moving from correlation to cause, with a real GPT-2 example, plus steering with feature directions
- Where these tools are used in practice: debugging, safety monitoring, model editing, and vision and video models
Why this matters
A trained LLM gives good answers, but nobody wrote down how it finds them. Its knowledge is spread over billions of numbers. Interpretability is the study of what those numbers do: which internal signals carry which information, and which parts of the network cause which behavior.
This is not only a research hobby. When a model answers wrongly, refuses a safe request, or follows a hidden instruction in a document, the outputs alone often cannot tell you why. Looking inside can. Real uses today include: checking which layer “knows” a fact before you edit it, finding a direction in the hidden state that turns refusal on or off, monitoring internal signals to catch harmful or injected inputs, and testing whether a vision-language model really looked at the image. The previous session (session 156) asked whether the model “knows that it does not know”. Interpretability gives tools to measure such internal states directly.
We use four tools, in order of how deep they go, plus one tool that checks the others:
- The logit lens: decode the hidden state of every layer with the model’s own output layer and watch the prediction form.
- Attention-head analysis: read the attention patterns and find heads with a clear job, such as induction heads, which copy patterns from earlier in the context.
- Probing classifiers: train a small classifier on hidden states to test whether some information (a part of speech, a truth value, a board position) can be read out.
- Sparse autoencoders (SAEs): break each hidden state into a small number of active “features” from a large learned dictionary, so that each feature means one thing.
- Activation patching: copy an internal value from one run into another and measure the change in the output. This tests cause, not only correlation.
All real-model numbers on this page come from GPT-2 small (12 layers, hidden size 768, 12 heads per layer, about 124M parameters), which we ran ourselves. We count layers and heads from 0, as most interpretability papers do: head “5.5” is head 5 in layer 5. [Certain]
The residual stream: one shared workspace
All the tools in this session rest on one picture of the transformer, from session 22 and from Elhage et al.’s “mathematical framework”. Each token has a vector, the residual stream. The embedding writes the first value. Then every attention layer and every MLP reads the stream (through a normalization) and adds its output back:
- Read the symbols
- \(\mathbf{h}_\ell \in \R^{d}\)
- the residual stream of one token after layer \(\ell\); \(d = 768\) for GPT-2 small
- \(\mathbf{h}_0\)
- the token embedding plus the position embedding
- \(\mathbf{a}_\ell, \mathbf{m}_\ell\)
- what the attention block and the MLP block of layer \(\ell\) add to the stream (in Pre-LN models the MLP reads the stream after the attention output was added)
- \(L\)
- the number of layers
The second form of (157.1) is the key idea: the final vector is a sum of everything every component wrote. Nothing is overwritten, only added. So we can ask of any partial sum “what does this already say?” (logit lens), “what information is in it?” (probes), “what is it made of?” (SAEs), and “what happens if I swap one piece?” (patching). Figure 1 shows where each tool looks.
The logit lens: decode every layer
At the end of the network, the model turns the last residual stream into next-token scores with a final normalization and the unembedding matrix \(W_U\) (the LM head; see session 12). The logit lens (nostalgebraist, 2020) applies the same two steps to the stream after any layer:
- Read the symbols
- \(W_U \in \R^{V \times d}\)
- the unembedding matrix; \(V = 50{,}257\) tokens for GPT-2
- \(\mathrm{Norm}\)
- the model’s final LayerNorm or RMSNorm, with its learned gain
- \(q_\ell\)
- the “early prediction” at layer \(\ell\): a probability for every token
No training is needed: we only reuse the model’s own weights. This works because of (157.1): the final stream is the stream at layer \(\ell\) plus the rest of the updates, and in many models the later layers mostly refine a direction that is already there. To measure how close an early prediction is to the final one, we use the KL divergence from session 3:
\(D_\ell\) is 0 at the last layer and shrinks as the prediction forms. Figure 2 shows the real logit lens of GPT-2 small on four prompts.
- for \(\ell =\) emb, L0, L1, …, L11:
- \(\mathbf{h} \leftarrow\) residual stream at the last position after \(\ell\)
- \(\mathbf{z} \leftarrow W_U \, \mathrm{Norm}(\mathbf{h})\); \(\;q_\ell \leftarrow \softmax(\mathbf{z})\) (157.2)
- record \(q_\ell(\text{target})\), its rank, and \(D_\ell = \KL(q_L \| q_\ell)\) (157.3)
- return the trajectory: where the answer appears, and how fast \(D_\ell\) falls
How to read Figure 2
Start with the Eiffel Tower prompt. Press Reset, then Play (or Step), or drag the slider from “emb” to “L11”. In the early layers the lens says “ the” or “ course”: generic continuations that fit almost any sentence. The rank of “ Paris” climbs from 22,680 at the embedding to 1,012 after layer 3, 10 after layer 8, and 2 after layer 9 (where “ London” is still first). After layer 10, “ Paris” is first. The KL line falls toward 0. Now try “Capital of Japan”: “ Tokyo” is first after layer 10 with \(p = 0.46\), but the final layer prefers “ the” (the model keeps the option “the capital of Japan is the city of…”). The lens shows that the answer was present inside the model even though the output did not show it first.
After layer 9, the lens gives p(“ London”) = 0.285 and p(“ Paris”) = 0.248: the model has decided “a European capital” but not yet which one. After layer 10, “ Paris” leads with 0.183. The final output gives “ Paris” only 0.070, still first but less sure. The KL to the final prediction falls from 31.4 nats at the embedding to 6.2 after layer 0, 2.4 after layer 7, and 0.86 after layer 10.
Two lessons. First, the answer appears suddenly in the last third of the network (layers 8 to 10), which is where other studies also find factual recall finishing. [Likely] Second, the last layer lowers the confidence. Final layers often spread probability out, so a middle layer can look more “certain” than the output. Do not read lens probabilities as calibrated. [Likely]
When the logit lens fails: the tuned lens
The logit lens assumes that every layer uses the same “coordinate system” as the last layer. That is roughly true for GPT-2, but not for every model. In some model families, the early-layer lens gives nonsense, and its predictions are biased toward certain tokens. [Likely] The tuned lens (Belrose et al., 2023) fixes this with a small learned affine map per layer, a “translator”, trained so that the decoded distribution matches the final one:
The model stays frozen; only the \(d \times d\) translator of each layer is trained. Belrose et al. report that the tuned lens predicts the final output better and is less biased than the logit lens on models up to 20B parameters, and that its layer-by-layer trajectories can help detect malicious inputs. [Certain] The cost: the translator could, in principle, add information of its own. A good translator starts near the identity, so it should change the coordinates, not the content.
Attention heads as circuits
The logit lens tells us when a prediction forms. To learn how, we look at individual attention heads. Elhage et al. (2021) write one head in a form that separates its two jobs. For a sequence of residual-stream vectors stacked as rows of \(X \in \R^{T \times d}\):
- Read the symbols
- \(W_Q W_K^\top \in \R^{d \times d}\)
- the QK circuit: it decides where each position looks (one low-rank \(d \times d\) matrix of rank at most \(d_h\))
- \(W_V W_O \in \R^{d \times d}\)
- the OV circuit: it decides what is copied from the attended position and written into the stream
- \(A \in \R^{T \times T}\)
- the attention pattern: row \(i\) is how query position \(i\) spreads its attention over key positions
- \(M\)
- the causal mask (\(-\infty\) above the diagonal)
- \(d_h\)
- the head size, 64 in GPT-2 small
This split matters because the two circuits can be studied separately. A head’s pattern \(A\) is easy to plot, and the OV matrix can be multiplied with the embedding and unembedding (\(W_U W_O^\top W_V^\top W_E^\top\), up to norms) to see which input tokens raise which output tokens. Heads in later layers can also read what earlier heads wrote. When a later head’s keys use an earlier head’s output, it is called K-composition; with queries, Q-composition; with values, V-composition. [Certain]
Induction heads
The best-understood circuit is the induction circuit, first introduced in session 143. It completes the pattern \([A][B] \dots [A] \to [B]\): if the current token appeared before, predict the token that came after it last time. Two heads work together:
- A previous-token head in an earlier layer. At each position \(j\) it attends to \(j - 1\) and writes “the token before me was \(t_{j-1}\)” into the stream at \(j\).
- An induction head in a later layer. Its query at position \(i\) asks “where was the token \(t_i\) before?”. Its keys read the previous-token information (K-composition), so the best match is the position \(j\) with \(t_{j-1} = t_i\). Its OV circuit copies \(t_j\), which raises the logit of \(t_j\) as the next token.
To find induction heads, feed a random token sequence of length \(T\) followed by the same sequence again. In the second copy, the right answer is always \(T - 1\) positions back, so we measure the average attention to that spot:
Here position 0 is a beginning-of-sequence token, the first copy fills positions 1 to \(T\), and the second copy fills \(T + 1\) to \(2T\). For query \(i\), its earlier copy sits at \(i - T\), and the token after that copy sits at \(i - T + 1\). An induction score near 1 means the head always looks exactly there.
We fed 50 random tokens, then the same 50 again. On the first copy the model cannot predict anything: the average loss is 12.33 nats (worse than the 10.82 of a uniform guess over 50,257 tokens, because the model expects real text). On the second copy the loss falls to 0.27 nats: the model copies almost perfectly. The heads with the highest induction score (157.6) are 7.10 (0.91), 5.1 (0.91), 5.5 (0.91), 6.9 (0.90) and 7.2 (0.84). Head 4.11 attends to the previous token 99% of the time: a previous-token head, one layer below the first induction heads, as the circuit needs. [Certain: we ran this; the code is below.]
How to read Figure 3
Head 4.11 draws a line just below the diagonal: every token looks at the token before it. Head 5.5 draws a line in the second copy, offset so that each query looks at the token after the earlier copy of itself (its score is 0.945). In the first copy it has nothing to match, so it rests on the beginning token. Now tick the ablation box. We set the outputs of the 11 heads in layers 2 to 4 whose previous-token score is above 0.3 to zero. The induction line of head 5.5 fades (score 0.309), and the loss on the second copy jumps from 0.54 to 3.61 nats. The induction head did not change; its keys lost the information that the previous-token heads used to write. That is K-composition, shown causally.
Olsson et al. (2022) found that induction heads appear suddenly during training, in a “phase change” that happens at the same time as a sharp rise in in-context learning, visible as a small bump in the training loss. They also gave evidence that induction-like heads produce a large part of in-context learning in small models, and they argued (with weaker evidence) that the same holds in large models. [Certain] A one-layer attention-only model cannot form induction heads, because the circuit needs one head to read another head’s output. [Certain]
Probing classifiers: is the information there?
The logit lens only asks about next-token predictions. Often we want to know about other information: does the hidden state encode the part of speech, the sentiment, whether a statement is true, where the pieces are on a game board? A probe is a small classifier trained on frozen hidden states to predict such a label (Alain and Bengio, 2016). The most common one is a linear probe:
- Read the symbols
- \(\mathbf{h}_\ell^{(n)}\)
- the hidden state at layer \(\ell\) for example \(n\) (one token, or a mean over tokens); the model is frozen
- \(\sigma\)
- the sigmoid; for many classes use a softmax
- \(\text{CE}\)
- cross-entropy between the true label and the probe’s probability
- \(\lambda\)
- weight decay that keeps the probe simple
If a linear probe reaches high accuracy on held-out data, the information is linearly decodable: there is a direction \(\mathbf{w}\) in the stream that separates the classes. Probing at every layer shows where the information appears. Real examples: true and false statements separate along one direction in LLaMA models (Marks and Tegmark, 2023); in a model trained on Othello moves, the board state can be decoded, first with nonlinear probes (Li et al., 2022) and then with linear ones once the board is written as “my piece / their piece” instead of “black / white” (Nanda et al., 2023). [Certain]
Training a probe is ordinary logistic regression: the model stays frozen, and only \(\mathbf{w}\) and \(b\) move. Figure 4 runs it live on a small toy (described in the next subsection): 96 training points in 2D, full-batch gradient descent, 400 steps. Each step has three parts: a forward pass, the loss and its gradient, and the update. The gradient of (157.7) has a simple form: \(\nabla_{\mathbf{w}} = \frac{1}{N}\sum_n (p_n - y_n)\, \mathbf{h}_n + 2\lambda \mathbf{w}\), so every misfit point pulls the boundary toward its correct side.
- \(\mathbf{w} \leftarrow \mathbf{0}, \; b \leftarrow 0\)
- for \(t = 1 \dots 400\): (full batch, \(N = 96\))
- \(z_n \leftarrow \mathbf{w}^\top \mathbf{h}_n + b, \; p_n \leftarrow \sigma(z_n)\) for every \(n\)
- \(\mathcal{L} \leftarrow \frac{1}{N}\sum_n \text{CE}(y_n, p_n) + \lambda \lVert \mathbf{w} \rVert^2\) (157.7)
- \(\mathbf{g} \leftarrow \frac{1}{N}\sum_n (p_n - y_n)\,\mathbf{h}_n + 2\lambda\mathbf{w}\)
- \(g_b \leftarrow \frac{1}{N}\sum_n (p_n - y_n)\)
- \(\mathbf{w} \leftarrow \mathbf{w} - \eta\,\mathbf{g}; \; b \leftarrow b - \eta\, g_b\)
- return \(\mathbf{w}, b\); report accuracy on held-out points
How to read Figure 4
Keep layer 4, real labels and \(\eta = 0.5\), and press Play. At step 1 the gradient is \(\mathbf{g} = (-0.584, -0.387)\), so the first update moves \(\mathbf{w}\) from \((0, 0)\) to \((0.292, 0.194)\), and the loss falls from \(\log 2 = 0.693\) to 0.493. After 400 steps the loss is 0.213, \(\mathbf{w} = (2.246, 1.322)\), and the probe gets 0.938 on the training points and 0.948 on held-out points. Now pick \(\eta = 0.05\): after 400 steps the loss is still 0.223 (too slow). With \(\eta = 20\) the steps overshoot: the loss jumps around and ends at 0.231, and held-out accuracy drops to 0.896. Switch to the control task: the loss stops far higher and held-out accuracy is only 0.708. Move the layer slider down to 0: with real labels the probe now stops at 0.531, near chance. The probe can only find what the layer already contains.
The trap: a probe can learn the task by itself
A probe is a model too. A flexible probe (a deep MLP, or nearest neighbors) can reach high accuracy just by memorizing word identities, even when the network does not represent the concept in any useful way. Hewitt and Liang (2019) proposed control tasks: give each word type a random label, and train the same probe on it. Only the probe can “know” those labels, so the control accuracy measures how much the probe memorizes. A good probe has high selectivity:
Figure 5 makes this concrete with a toy. There are 32 “word types”, each a small cloud of points in 2D (think of a 2D slice of the hidden state). Half the types have the concept (label 1). As the layer grows, the model pushes the two groups apart along one direction. We train a linear probe (logistic regression, formula 157.7, 400 steps of gradient descent) and a 1-nearest-neighbor probe, on both the real labels and a control task.
How to read Figure 5
At layer 0 the concept is not in the representation. The linear probe gets 53% on real labels: chance. But the 1-nearest-neighbor probe already gets 82%, because the types are tight clouds and it simply remembers which type has which label. Switch to the control task: 1-NN gets 88% on random labels. Its selectivity is near 0 at every layer, so its accuracy tells us nothing about the model. The linear probe is different: on the control task it stays near 70% (some random labelings of 32 clouds are partly separable by a line), while on the real task it rises to 95% at layer 4. Its selectivity grows from −0.13 at layer 0 to 0.26 at layers 5 and 6. That growth is the real signal.
Linear probe: real-task test accuracy 0.948, control-task test accuracy 0.708, so selectivity = 0.948 − 0.708 = 0.240. The 1-NN probe: 0.958 and 0.927, selectivity 0.031. The 1-NN probe has the higher task accuracy and the lower selectivity. If you reported only task accuracy, you would pick the wrong probe and the wrong conclusion.
Even a selective probe shows only that information is present, not that the model uses it. The concept may sit in the stream as a side effect. To test use, change the representation along the probe direction and see whether behavior changes. Li et al. did this for Othello: editing the decoded board changed the model’s predicted legal moves. [Certain] That brings us to causal methods, after one more tool.
Superposition: why neurons are hard to read
The simplest hope would be: one neuron, one concept. In practice, most neurons are polysemantic: in the one-layer model of Bricken et al., a single MLP unit fires for academic citations, English dialogue, HTTP requests and Korean text. [Likely] The reason is superposition (Elhage et al., “Toy Models of Superposition”, 2022): a model wants to store more features than it has dimensions, so it gives features directions that are almost orthogonal, not exactly. The activation is a sum of active features:
- Read the symbols
- \(\mathbf{x} \in \R^{d}\)
- an activation vector (an MLP layer or the residual stream)
- \(\mathbf{d}_i\)
- the unit direction of feature \(i\)
- \(f_i\)
- how strongly feature \(i\) is present; for one input, only a few are non-zero (features are sparse)
- \(n\)
- the number of features, much larger than the dimension \(d\)
Why does this work? In \(d\) dimensions only \(d\) directions can be exactly orthogonal, but very many can be nearly orthogonal. If features rarely appear together, small overlaps cause little harm, because the interference only appears when two overlapping features are active at the same time. In the toy models, 5 features stored in 2 dimensions settle into a regular pentagon. [Certain]
Sparse autoencoders: finding the features
If activations are sparse sums of feature directions (157.9), we can try to recover the directions with dictionary learning. A sparse autoencoder (SAE) maps each activation to a much wider, mostly-zero code, then reconstructs the activation from it:
- Read the symbols
- \(W_e \in \R^{m \times d}, \ W_d \in \R^{d \times m}\)
- encoder and decoder; \(m\) is the dictionary size, often 8 to 256 times \(d\) or more
- \(W_d[:, i]\)
- column \(i\) of the decoder: the learned direction of feature \(i\) (the estimate of \(\mathbf{d}_i\))
- \(\mathbf{b}_d, \mathbf{b}_e\)
- biases; subtracting \(\mathbf{b}_d\) centers the input
- \(\lambda\)
- sparsity weight: larger \(\lambda\) means fewer active features but worse reconstruction
- \(\lVert W_d[:, i] \rVert\)
- multiplying the L1 term by the decoder norm stops the SAE from cheating by making codes tiny and decoder columns huge
Each feature is then interpreted by looking at the inputs that activate it most, and checked by asking whether those inputs share one meaning. Bricken et al. (2023) trained SAEs on the 512-neuron MLP layer of a one-layer transformer, with dictionaries from 512 to 131,072 features, and studied a run with 4,096 features in detail. They found features for DNA sequences, legal language, HTTP requests, Arabic script and more, features that are much more monosemantic than the neurons. [Certain] Templeton et al. (2024) scaled this to the middle-layer residual stream of Claude 3 Sonnet with about 1M, 4M and 34M features, and found abstract features that respond to the same concept in many languages and in images, including safety-relevant ones (deception, code vulnerabilities, bias). Clamping a “Golden Gate Bridge” feature to a high value made the model bring up the bridge in unrelated answers. [Certain]
The ReLU plus L1 recipe has a known weakness: the L1 term pulls every active code toward 0 (shrinkage), so features are underestimated. Newer SAEs replace it. TopK SAEs keep only the \(k\) largest pre-activations, so the number of active features is fixed directly (Gao et al., 2024, who trained a 16M-latent SAE on GPT-4 activations). JumpReLU SAEs use a learned threshold: a pre-activation below \(\theta\) becomes 0, above it passes unchanged (used for Gemma Scope, open SAEs on Gemma 2). [Certain] Figure 6 shows the three choices on the pentagon toy.
- \(\mathbf{x} \leftarrow \sum_{i \in \text{active}} \mathbf{d}_i\) (157.9, every \(f_i = 1\))
- for \(i = 0 \dots 4\): \(\; p_i \leftarrow \mathbf{d}_i^\top \mathbf{x}\) (tied encoder \(W_e = W_d^\top\))
- \(\hat f_i \leftarrow\) linear: \(p_i\); ReLU: \(\max(0, p_i - \theta)\); JumpReLU: \(p_i \cdot [p_i \gt \theta]\)
- \(\hat{\mathbf{x}} \leftarrow \mathbf{0}\); for \(i = 0 \dots 4\): \(\; \hat{\mathbf{x}} \leftarrow \hat{\mathbf{x}} + \hat f_i \, \mathbf{d}_i\) (157.10)
- error \(\leftarrow \lVert \mathbf{x} - \hat{\mathbf{x}} \rVert^2\); \(\;L_0 \leftarrow\) number of \(\hat f_i \ne 0\)
How to read Figure 6
Press Reset, then Step through the encoder. With only feature 0 active, the projections onto the five directions are 1, 0.309, −0.809, −0.809 and 0.309 (cosines of 0°, 72° and 144°). The linear encoder keeps all of this interference: 5 active codes and an error of 2.25. ReLU with bias 0.4 removes the interference but shrinks the true code to 0.6 (error 0.16). JumpReLU with \(\theta = 0.4\) gives exactly 1 and error 0. Now turn on features 0 and 2 together. Their sum points exactly along \(\mathbf{d}_1\) (because \(\mathbf{d}_0 + \mathbf{d}_2 = 2\cos 72° \, \mathbf{d}_1 = 0.618\,\mathbf{d}_1\)), so JumpReLU reports feature 1 with value 0.618 and a perfect reconstruction: zero error, wrong explanation. This is why superposition only works when features are sparse, and why reconstruction error alone does not prove an SAE is right.
We hid 8 sparse features in 4 dimensions (each feature on with probability 0.05, so 0.41 active per input on average) and trained an SAE with 16 latents using (157.10) and (157.11). For three values of \(\lambda\):
| \(\lambda\) | Mean best cosine to a true feature | Active latents per input (L0) | Unexplained variance |
|---|---|---|---|
| 0.01 | 0.897 | 1.97 | 0.001 |
| 0.10 | 0.944 | 1.07 | 0.020 |
| 0.30 | 0.985 | 0.83 | 0.098 |
More sparsity gives features closer to the true directions and fewer active latents, but worse reconstruction. Real SAE papers report this as a curve of reconstruction quality against L0, and also check how much the language-model loss rises when the SAE’s reconstruction replaces the real activation.
From correlation to cause: activation patching
A lens, a probe or an SAE feature can show that information is there. To show that a component matters, we intervene. Activation patching (also called causal tracing, Meng et al., 2022; Wang et al., 2022) uses two inputs that differ in one detail:
- Clean: “When Mary and John went to the store, John gave a drink to” \(\to\) the model should say “ Mary”.
- Corrupted: the same, but with “Mary gave a drink to” \(\to\) now “ John” is the right answer.
We measure the logit difference \(\text{LD} = z_{\text{Mary}} - z_{\text{John}}\) at the last position. Then we run the corrupted input but replace one activation (one layer, one position) with its value from the clean run. If that single value restores the clean behavior, it carries the information that matters:
A value of 1 means the patch fully restores clean behavior; 0 means no effect. Figure 7 shows the real result on GPT-2 small, patching the residual stream after each layer at two positions: the second name (“ John” in the clean input) and the last token (“ to”).
- run clean \(x_c\); cache \(\mathbf{h}^c_\ell[p]\) for every \(\ell, p\); \(\;\text{LD}_c \leftarrow z_{\text{Mary}} - z_{\text{John}}\)
- run corrupted \(x_k\); \(\;\text{LD}_k \leftarrow z_{\text{Mary}} - z_{\text{John}}\)
- for \(\ell = 0 \dots 11\), for \(p\) in the chosen positions:
- run \(x_k\) with a hook: \(\mathbf{h}_\ell[p] \leftarrow \mathbf{h}^c_\ell[p]\)
- \(\text{LD} \leftarrow z_{\text{Mary}} - z_{\text{John}}\) at the last position
- \(R[\ell, p] \leftarrow (\text{LD} - \text{LD}_k) / (\text{LD}_c - \text{LD}_k)\) (157.12)
- return the map \(R\)
How to read Figure 7
Early on, all the needed information sits at the second-name position: patching it there after layers 0 to 3 recovers almost 100%. Patching at the last token does almost nothing until layer 6. Between layers 7 and 8 the two rows of the map trade places: the information has been moved from the name position to the last position, by attention heads in those layers (the “S-inhibition heads” of Wang et al. sit in layers 7 and 8). [Likely] After layer 8, patching the last token alone recovers 99.5%, and the name position no longer matters. This is how the indirect-object circuit of Wang et al. was found: patching narrows the search from “the whole model” to “a few heads in layers 7 to 10 that read the name positions”. [Certain for our numbers]
Patching the last token after layer 7 recovers 0.389. The gap between clean and corrupted is \(2.006 - (-1.789) = 3.795\). So the patched logit difference is \(-1.789 + 0.389 \times 3.795 \approx -0.31\): the model still prefers “ John”, but much less. One layer later, the patch recovers 0.995: the patched run prefers “ Mary” almost as strongly as the clean run.
Steering: using a direction
Once you have a direction that matters (a probe weight, a difference of mean activations between two prompt sets, or an SAE decoder column), you can add it to the stream during generation:
Here \(\hat{\mathbf{d}}\) is a unit direction and \(\alpha\) its strength (\(\alpha \lt 0\) pushes away from the concept). Turner et al. (2023) steer topic and sentiment this way with directions from pairs of prompts; Arditi et al. (2024) find that refusal in many open chat models is controlled by one direction: removing it from the stream stops refusals, adding it makes the model refuse harmless requests. [Certain] This is powerful for research and a real risk for deployed open-weight models, a topic for session 158.
Putting it together: algorithms and code
- input: model with \(L\) layers, final norm, \(W_U\); clean input \(x_c\), corrupted input \(x_k\) of the same length; answer tokens \(a\) (clean) and \(b\) (corrupted)
- run \(x_c\); store \(\mathbf{h}^{c}_\ell[p]\) for all layers \(\ell\) and positions \(p\)
- for \(\ell = 0 \dots L\): \(q_\ell \leftarrow \softmax(W_U \mathrm{Norm}(\mathbf{h}^c_\ell[\text{last}]))\); record top tokens, rank of \(a\), \(\KL(q_L \| q_\ell)\) // logit lens, (157.2), (157.3)
- \(\text{LD}_c \leftarrow z_a - z_b\) on \(x_c\); \(\;\text{LD}_k \leftarrow z_a - z_b\) on \(x_k\)
- for each layer \(\ell\) and position \(p\):
- run \(x_k\) with a hook that sets \(\mathbf{h}_\ell[p] \leftarrow \mathbf{h}^c_\ell[p]\)
- \(R[\ell, p] \leftarrow (\text{LD} - \text{LD}_k) / (\text{LD}_c - \text{LD}_k)\) // formula (157.12)
- return lens table and the map \(R\); zoom into the cells with large \(R\) by patching single heads
- input: a frozen model, a layer \(\ell\), a large text set, dictionary size \(m\), sparsity weight \(\lambda\)
- collect activations \(\mathbf{x}\) at layer \(\ell\) for many tokens (often hundreds of millions or more) // stream them, do not store all
- initialize \(W_d\) with unit columns, \(W_e \leftarrow W_d^\top\), biases 0
- repeat: \(\mathbf{f} \leftarrow \mathrm{ReLU}(W_e(\mathbf{x} - \mathbf{b}_d) + \mathbf{b}_e)\), \(\hat{\mathbf{x}} \leftarrow W_d \mathbf{f} + \mathbf{b}_d\); step on (157.11) with Adam
- track L0, reconstruction error, and dead latents (never active); re-initialize dead ones
- check: replace \(\mathbf{x}\) by \(\hat{\mathbf{x}}\) inside the model and measure the rise in LM loss
- for each feature \(i\): list the top-activating tokens and contexts; give it a label; check the label on new text
- test causally: add \(\alpha W_d[:, i]\) to the stream (157.13) or clamp \(f_i\), and see if behavior changes as the label predicts
The PyTorch code below reproduces the logit lens of Figure 2 and the induction scan from the worked example on GPT-2 small. It needs transformers and downloads the 124M model once.
import torch from transformers import GPT2LMHeadModel, GPT2TokenizerFast torch.manual_seed(0) tok = GPT2TokenizerFast.from_pretrained("gpt2") model = GPT2LMHeadModel.from_pretrained("gpt2", attn_implementation="eager").eval() # 1) Logit lens, formula (157.2): decode the last position of every layer ids = tok("The Eiffel Tower is in the city of", return_tensors="pt").input_ids paris = tok.encode(" Paris")[0] with torch.no_grad(): out = model(ids, output_hidden_states=True) hs = out.hidden_states # 13 tensors: embeddings + 12 layers for l, h in enumerate(hs): x = h[0, -1] if l < len(hs) - 1: # HF already applied ln_f to the last one x = model.transformer.ln_f(x) logp = torch.log_softmax(model.lm_head(x), -1) rank = int((logp > logp[paris]).sum()) + 1 print(f"layer {l:2d} top: {tok.decode(logp.argmax())!r:12} p(Paris) = {logp[paris].exp():.4f} rank {rank}") # 2) Induction score, formula (157.6): random tokens, repeated once T = 50 r = torch.randint(1000, 10000, (1, T), generator=torch.Generator().manual_seed(0)) seq = torch.cat([torch.tensor([[tok.bos_token_id]]), r, r], 1) with torch.no_grad(): o = model(seq, output_attentions=True) nll = torch.nn.functional.cross_entropy(o.logits[0, :-1], seq[0, 1:], reduction="none") print(f"loss on first copy {nll[1:T].mean():.2f}, on second copy {nll[T+1:].mean():.2f} nats") q = torch.arange(T + 1, 2 * T + 1) # queries in the second copy scores = sorted(((float(a[0, h, q, q - T + 1].mean()), f"{l}.{h}") for l, a in enumerate(o.attentions) for h in range(12)), reverse=True) print("top induction heads:", [(name, round(s, 2)) for s, name in scores[:5]]) # layer 10 top ' Paris' p = 0.1834 ... loss 12.33 then 0.27 ... [('7.10', 0.91), ('5.1', 0.91), ('5.5', 0.91), ...]
The second program needs only PyTorch. It hides 8 sparse features in 4 dimensions (formula 157.9), trains the SAE of (157.10) and (157.11) for three sparsity weights, and prints the table from the worked example. It runs in about 20 seconds on a laptop CPU and prints the same numbers every time.
import torch torch.manual_seed(0) # 1) Toy activations in superposition: 8 sparse features stored in 4 dimensions, formula (157.9) n_feat, d, m = 8, 4, 16 D_true = torch.nn.functional.normalize(torch.randn(n_feat, d), dim=1) # true feature directions def batch(B=1024, p=0.05): on = (torch.rand(B, n_feat) < p).float() f = on * torch.rand(B, n_feat) # sparse, non-negative feature values return f @ D_true, f # x = sum_i f_i d_i # 2) A sparse autoencoder, formulas (157.10) and (157.11) class SAE(torch.nn.Module): def __init__(self, d, m): super().__init__() self.W_e = torch.nn.Parameter(torch.randn(m, d) * 0.3) self.b_e = torch.nn.Parameter(torch.zeros(m)) self.W_d = torch.nn.Parameter(self.W_e.data.t().clone()) # (d, m), starts tied self.b_d = torch.nn.Parameter(torch.zeros(d)) def forward(self, x): f = torch.relu((x - self.b_d) @ self.W_e.t() + self.b_e) # sparse codes return f @ self.W_d.t() + self.b_d, f # reconstruction def train(lam, steps=10000): torch.manual_seed(1) sae = SAE(d, m); opt = torch.optim.Adam(sae.parameters(), lr=3e-3) for _ in range(steps): x, _ = batch() xh, f = sae(x) mse = ((x - xh) ** 2).sum(1).mean() l1 = (f * sae.W_d.norm(dim=0)).sum(1).mean() # L1 weighted by decoder column norms opt.zero_grad(); (mse + lam * l1).backward(); opt.step() return sae # 3) Sweep the sparsity weight: did the SAE find the true directions? torch.manual_seed(2); x_test, f_test = batch(8192) print("true L0: %.2f" % (f_test > 0).float().sum(1).mean()) for lam in [0.01, 0.1, 0.3]: sae = train(lam) with torch.no_grad(): xh, f = sae(x_test) Wd = torch.nn.functional.normalize(sae.W_d, dim=0) # unit decoder columns best = (D_true @ Wd).max(dim=1).values # best cosine per true feature fvu = ((x_test - xh) ** 2).sum() / ((x_test - x_test.mean(0)) ** 2).sum() print("lam %.2f mean best cosine %.3f L0 %.2f unexplained variance %.3f" % (lam, best.mean(), (f > 0).float().sum(1).mean(), fvu)) # lam 0.01 mean best cosine 0.897 L0 1.97 unexplained variance 0.001 # lam 0.10 mean best cosine 0.944 L0 1.07 unexplained variance 0.020 # lam 0.30 mean best cosine 0.985 L0 0.83 unexplained variance 0.098
For real work, libraries such as TransformerLens (hooks on every activation of many open models) and SAELens (training and loading SAEs, including Gemma Scope) save a lot of hook-writing. [Likely]
Where this is used
| Use | Tool | What it gives you |
|---|---|---|
| Debugging a wrong answer | Logit lens, patching | Was the right answer present in a middle layer and lost later, or never formed? Which position carried it? |
| Model editing (fix one fact) | Causal tracing, then a targeted edit (ROME) | The layers where a fact is stored at the subject token, so the edit touches only those weights (session 21) |
| Safety monitoring | Linear probes, SAE features | Cheap detectors on hidden states for deception, harmful intent or prompt injection, run alongside the model [Likely] |
| Controlling behavior | Steering vectors (157.13) | Turn a style, topic or refusal up or down without fine-tuning; also a way attackers remove safety from open weights |
| Understanding in-context learning | Attention patterns, induction score | Why few-shot prompts work, and why repeated text is predicted so well |
| Early exit and speculative decoding | Logit lens, tuned lens | Middle layers that already predict well can draft tokens or stop early (session 121) |
| Vision-language models | Logit lens on image tokens, attention to image patches, probes | Whether the answer used the image or only the text prior; which patches were read. A tool for checking visual hallucination [Likely] |
| Video models (action recognition, key-moment detection) | Probes per layer and per frame, attention over time, SAEs on vision encoders | Does the encoder represent motion and order, or only single-frame appearance? Which frames drove a detected key moment? [Speculation for SAEs on video encoders] |
| Video retrieval and Video-RAG | Probes and patching on the embedding model | Whether a retrieval embedding encodes the asked attribute (time, action, object count) before you trust it in a pipeline [Speculation] |
Common mistakes
- Treating an attention pattern as an explanation. High attention to a token does not mean that token’s information was used: the OV circuit may copy almost nothing, and many heads attend to the first token as a resting place. Check with patching or ablation.
- Forgetting the final norm in the logit lens, or applying it twice when the library already did (Hugging Face applies
ln_fto the last hidden state of GPT-2). - Reading lens probabilities as calibrated. In our Eiffel Tower example, layer 10 gives “ Paris” 0.183 and the output gives 0.070. Middle layers are not the model’s answer.
- Reporting probe accuracy without a baseline. Always compare with a control task (157.8), a random-weights model, and a majority-class baseline. A flexible probe can learn the task itself.
- Concluding “the model uses X” from a probe. Decodable is not the same as used. Intervene along the direction and check behavior.
- Trusting one-head ablations. Redundant and backup heads hide effects, as in Figure 3 where removing head 4.11 alone changed almost nothing. Ablate groups, and prefer patching with real activations from another input over setting values to zero (zero can push the model off its normal range).
- Judging an SAE by reconstruction only. Figure 6 shows zero error with the wrong features. Also measure L0, dead latents, the rise in LM loss when the reconstruction is used, and whether feature labels hold on new text.
- Mixing up layer numbering. Hidden state 0 is the embedding; hidden state \(\ell\) is the output of layer \(\ell - 1\) when layers are counted from 0. Write down your convention.
Interview questions
What is the residual stream, and why does it make interpretability possible?basic
Explain the logit lens. What are its limitations?basic
What are the QK and OV circuits of an attention head?core
What is an induction head? How would you find one in a model?core
Why do induction heads matter for in-context learning?core
A linear probe gets 95% accuracy at layer 20 for “the statement is true”. What can you conclude?core
What is superposition, and why does it make neurons polysemantic?core
How does a sparse autoencoder work, and how do you evaluate one?core
Why do TopK and JumpReLU SAEs exist? What is wrong with ReLU plus L1?advanced
Describe activation patching. Why patch with real activations instead of zeros?advanced
How could you use interpretability to monitor a deployed LLM application?advanced
Exercises
- In Figure 2 (Eiffel Tower), find the first layer where “ Paris” is in the top 10, and the first where it is top 1. What is the KL to the final prediction at each?
- For a repeated random sequence with \(T = 10\) (positions 0 to 20), which key position should an induction head attend to from query 15? Which token does it predict? Check it in Figure 3.
- In the pentagon toy (Figure 6), compute by hand the five projections when features 0 and 1 are both on. With JumpReLU at \(\theta = 0.4\), what is the reconstruction error? (Hint: \(\lVert \mathbf{d}_0 + \mathbf{d}_1 \rVert^2 = 2 + 2\cos 72°\).)
- A 1-NN probe reaches 0.90 on a task and 0.88 on its control task; a linear probe reaches 0.80 and 0.55. Which result tells you more about the model, and why?
- In Figure 7, the clean logit difference is 2.006 and the corrupted one −1.789. Patching the name position after layer 6 recovers 0.835. What is the patched logit difference?
- An SAE has \(m = 32{,}768\) latents on a \(d = 4{,}096\) residual stream. How many parameters do \(W_e\) and \(W_d\) have together? What is the expansion factor?
Answerscheck
Key takeaways
- The residual stream is a sum of everything each layer wrote (157.1), so it can be decoded, probed, decomposed and patched at any point.
- The logit lens (157.2) shows predictions forming layer by layer; in GPT-2 small, “ Paris” appears only in layers 8 to 10. The tuned lens (157.4) fixes early layers with a learned translator.
- Attention heads split into QK (where to look) and OV (what to copy) circuits (157.5). Induction heads, built from a previous-token head by K-composition, copy patterns and drive in-context learning; the induction score (157.6) finds them.
- Probes (157.7) show what is decodable, not what is used. Report selectivity against a control task (157.8).
- Superposition (157.9) makes neurons polysemantic; sparse autoencoders (157.10, 157.11) recover sparse, more monosemantic features, with a sparsity versus reconstruction trade-off.
- Activation patching (157.12) and steering (157.13) test cause: change an internal value, watch the output.
Key formulas
References for this session
- Elhage et al. (2021). A Mathematical Framework for Transformer Circuits
- Olsson et al. (2022). In-context Learning and Induction Heads
- Bricken et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 10: Interpretability
- Elhage et al. (2021). A Mathematical Framework for Transformer Circuits
- Olsson et al. (2022). In-context Learning and Induction Heads
- Bricken et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 10: Interpretability
- nostalgebraist (2020). interpreting GPT: the logit lens
- Belrose, Ostrovsky, McKinney, Furman, Smith, Halawi, Biderman, Steinhardt (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens
- Elhage et al. (2022). Toy Models of Superposition
- Alain & Bengio (2016). Understanding intermediate layers using linear classifier probes
- Hewitt & Liang (2019). Designing and Interpreting Probes with Control Tasks
- Belinkov (2021). Probing Classifiers: Promises, Shortcomings, and Advances
- Li et al. (2022). Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
- Nanda, Lee & Wattenberg (2023). Emergent Linear Representations in World Models of Self-Supervised Sequence Models
- Marks & Tegmark (2023). The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
- Cunningham et al. (2023). Sparse Autoencoders Find Highly Interpretable Features in Language Models
- Templeton et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
- Gao et al. (2024). Scaling and evaluating sparse autoencoders
- Lieberum et al. (2024). Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
- Meng, Bau, Andonian, Belinkov (2022). Locating and Editing Factual Associations in GPT
- Wang et al. (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
- Turner et al. (2023). Steering Language Models With Activation Engineering
- Arditi et al. (2024). Refusal in Language Models Is Mediated by a Single Direction