LLM & VLM courseBlog
157Track 6: LLM applications · session 15 of 20

Interpretability: looking inside the model

A language model is a big stack of matrix multiplications, but inside it we can find readable structure: predictions that form layer by layer, attention heads that copy patterns, directions that encode concepts, and sparse features that the model uses as its vocabulary of ideas. This session teaches the four main tools for looking inside (the logit lens, attention-head analysis, probing classifiers and sparse autoencoders) plus activation patching, which tests whether what you found is really used. Every number in the figures comes from GPT-2 small or from a small toy that we explain fully.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions 12, 19, 21, 22 and 143

What you will learn

  • The residual stream view of a transformer: every layer reads from and writes into one shared vector, which is why we can inspect it anywhere
  • The logit lens: decode the hidden state at every layer with the output layer, and what real GPT-2 numbers look like; the tuned lens as the fix for early layers
  • Attention heads as QK and OV circuits, previous-token heads and induction heads, the induction score, and an ablation that shows the two heads work together
  • Probing classifiers: what a linear probe can and cannot prove, control tasks and selectivity
  • Superposition and sparse autoencoders: why single neurons are hard to read, how an SAE turns activations into sparse features, and the sparsity versus reconstruction trade-off
  • Activation patching: moving from correlation to cause, with a real GPT-2 example, plus steering with feature directions
  • Where these tools are used in practice: debugging, safety monitoring, model editing, and vision and video models

Why this matters

A trained LLM gives good answers, but nobody wrote down how it finds them. Its knowledge is spread over billions of numbers. Interpretability is the study of what those numbers do: which internal signals carry which information, and which parts of the network cause which behavior.

This is not only a research hobby. When a model answers wrongly, refuses a safe request, or follows a hidden instruction in a document, the outputs alone often cannot tell you why. Looking inside can. Real uses today include: checking which layer “knows” a fact before you edit it, finding a direction in the hidden state that turns refusal on or off, monitoring internal signals to catch harmful or injected inputs, and testing whether a vision-language model really looked at the image. The previous session (session 156) asked whether the model “knows that it does not know”. Interpretability gives tools to measure such internal states directly.

We use four tools, in order of how deep they go, plus one tool that checks the others:

  1. The logit lens: decode the hidden state of every layer with the model’s own output layer and watch the prediction form.
  2. Attention-head analysis: read the attention patterns and find heads with a clear job, such as induction heads, which copy patterns from earlier in the context.
  3. Probing classifiers: train a small classifier on hidden states to test whether some information (a part of speech, a truth value, a board position) can be read out.
  4. Sparse autoencoders (SAEs): break each hidden state into a small number of active “features” from a large learned dictionary, so that each feature means one thing.
  5. Activation patching: copy an internal value from one run into another and measure the change in the output. This tests cause, not only correlation.

All real-model numbers on this page come from GPT-2 small (12 layers, hidden size 768, 12 heads per layer, about 124M parameters), which we ran ourselves. We count layers and heads from 0, as most interpretability papers do: head “5.5” is head 5 in layer 5. [Certain]

The residual stream: one shared workspace

All the tools in this session rest on one picture of the transformer, from session 22 and from Elhage et al.’s “mathematical framework”. Each token has a vector, the residual stream. The embedding writes the first value. Then every attention layer and every MLP reads the stream (through a normalization) and adds its output back:

\[ \mathbf{h}_\ell = \mathbf{h}_{\ell-1} + \mathbf{a}_\ell + \mathbf{m}_\ell, \qquad \mathbf{h}_L = \mathbf{h}_0 + \sum_{\ell=1}^{L} \left( \mathbf{a}_\ell + \mathbf{m}_\ell \right) \tag{157.1} \]
Read the symbols
\(\mathbf{h}_\ell \in \R^{d}\)
the residual stream of one token after layer \(\ell\); \(d = 768\) for GPT-2 small
\(\mathbf{h}_0\)
the token embedding plus the position embedding
\(\mathbf{a}_\ell, \mathbf{m}_\ell\)
what the attention block and the MLP block of layer \(\ell\) add to the stream (in Pre-LN models the MLP reads the stream after the attention output was added)
\(L\)
the number of layers

The second form of (157.1) is the key idea: the final vector is a sum of everything every component wrote. Nothing is overwritten, only added. So we can ask of any partial sum “what does this already say?” (logit lens), “what information is in it?” (probes), “what is it made of?” (SAEs), and “what happens if I swap one piece?” (patching). Figure 1 shows where each tool looks.

residual stream \(\mathbf{h}\) (one vector per token)attention blockMLP blocktool attached to the stream
Figure 1. The residual stream view. Blocks read the stream and add their outputs to it. The logit lens decodes the stream at any point; probes and SAEs read it; activation patching replaces a value with one from a different input and watches the output.

The logit lens: decode every layer

At the end of the network, the model turns the last residual stream into next-token scores with a final normalization and the unembedding matrix \(W_U\) (the LM head; see session 12). The logit lens (nostalgebraist, 2020) applies the same two steps to the stream after any layer:

\[ \mathbf{z}_\ell = W_U \, \mathrm{Norm}(\mathbf{h}_\ell), \qquad q_\ell = \softmax(\mathbf{z}_\ell) \tag{157.2} \]
Read the symbols
\(W_U \in \R^{V \times d}\)
the unembedding matrix; \(V = 50{,}257\) tokens for GPT-2
\(\mathrm{Norm}\)
the model’s final LayerNorm or RMSNorm, with its learned gain
\(q_\ell\)
the “early prediction” at layer \(\ell\): a probability for every token

No training is needed: we only reuse the model’s own weights. This works because of (157.1): the final stream is the stream at layer \(\ell\) plus the rest of the updates, and in many models the later layers mostly refine a direction that is already there. To measure how close an early prediction is to the final one, we use the KL divergence from session 3:

\[ D_\ell = \KL\!\left( q_L \,\|\, q_\ell \right) = \sum_{v} q_L(v) \log \frac{q_L(v)}{q_\ell(v)} \tag{157.3} \]

\(D_\ell\) is 0 at the last layer and shrinks as the prediction forms. Figure 2 shows the real logit lens of GPT-2 small on four prompts.

\(p(\text{target})\) by layer, log scaleKL to the final prediction (right axis, nats)top 5 tokens at the chosen layer
  1. for \(\ell =\) emb, L0, L1, …, L11:
  2.   \(\mathbf{h} \leftarrow\) residual stream at the last position after \(\ell\)
  3.   \(\mathbf{z} \leftarrow W_U \, \mathrm{Norm}(\mathbf{h})\); \(\;q_\ell \leftarrow \softmax(\mathbf{z})\) (157.2)
  4.   record \(q_\ell(\text{target})\), its rank, and \(D_\ell = \KL(q_L \| q_\ell)\) (157.3)
  5. return the trajectory: where the answer appears, and how fast \(D_\ell\) falls
Figure 2. The logit lens on GPT-2 small, computed with formula (157.2) at the last position of each prompt. Press Play to walk through the layers: each layer is one pass of the loop on the left (decode, then record); the curves grow one point at a time. The numbers are real GPT-2 outputs that we stored, so the page shows exactly what the loop returned. “emb” is the embedding \(\mathbf{h}_0\); “L5” means after layer 5; “L11” is the real output. The left plot shows the probability of the expected next token on a log scale. Tokens are shown in quotes; a leading space inside the quotes, as in " Paris", is part of the token.
How to read Figure 2

Start with the Eiffel Tower prompt. Press Reset, then Play (or Step), or drag the slider from “emb” to “L11”. In the early layers the lens says “ the” or “ course”: generic continuations that fit almost any sentence. The rank of “ Paris” climbs from 22,680 at the embedding to 1,012 after layer 3, 10 after layer 8, and 2 after layer 9 (where “ London” is still first). After layer 10, “ Paris” is first. The KL line falls toward 0. Now try “Capital of Japan”: “ Tokyo” is first after layer 10 with \(p = 0.46\), but the final layer prefers “ the” (the model keeps the option “the capital of Japan is the city of…”). The lens shows that the answer was present inside the model even though the output did not show it first.

Reading the numbers from Figure 2 (Eiffel Tower)

After layer 9, the lens gives p(“ London”) = 0.285 and p(“ Paris”) = 0.248: the model has decided “a European capital” but not yet which one. After layer 10, “ Paris” leads with 0.183. The final output gives “ Paris” only 0.070, still first but less sure. The KL to the final prediction falls from 31.4 nats at the embedding to 6.2 after layer 0, 2.4 after layer 7, and 0.86 after layer 10.

Two lessons. First, the answer appears suddenly in the last third of the network (layers 8 to 10), which is where other studies also find factual recall finishing. [Likely] Second, the last layer lowers the confidence. Final layers often spread probability out, so a middle layer can look more “certain” than the output. Do not read lens probabilities as calibrated. [Likely]

When the logit lens fails: the tuned lens

The logit lens assumes that every layer uses the same “coordinate system” as the last layer. That is roughly true for GPT-2, but not for every model. In some model families, the early-layer lens gives nonsense, and its predictions are biased toward certain tokens. [Likely] The tuned lens (Belrose et al., 2023) fixes this with a small learned affine map per layer, a “translator”, trained so that the decoded distribution matches the final one:

\[ q^{\text{tuned}}_\ell = \softmax\!\Big( W_U \, \mathrm{Norm}\big( A_\ell \mathbf{h}_\ell + \mathbf{b}_\ell \big) \Big), \qquad \min_{A_\ell, \mathbf{b}_\ell} \ \E_{x}\Big[ \KL\big( q_L \,\|\, q^{\text{tuned}}_\ell \big) \Big] \tag{157.4} \]

The model stays frozen; only the \(d \times d\) translator of each layer is trained. Belrose et al. report that the tuned lens predicts the final output better and is less biased than the logit lens on models up to 20B parameters, and that its layer-by-layer trajectories can help detect malicious inputs. [Certain] The cost: the translator could, in principle, add information of its own. A good translator starts near the identity, so it should change the coordinates, not the content.

Attention heads as circuits

The logit lens tells us when a prediction forms. To learn how, we look at individual attention heads. Elhage et al. (2021) write one head in a form that separates its two jobs. For a sequence of residual-stream vectors stacked as rows of \(X \in \R^{T \times d}\):

\[ A = \softmax\!\left( \frac{X W_Q W_K^\top X^\top}{\sqrt{d_h}} + M \right), \qquad \text{head}(X) = A \, X \, W_V W_O \tag{157.5} \]
Read the symbols
\(W_Q W_K^\top \in \R^{d \times d}\)
the QK circuit: it decides where each position looks (one low-rank \(d \times d\) matrix of rank at most \(d_h\))
\(W_V W_O \in \R^{d \times d}\)
the OV circuit: it decides what is copied from the attended position and written into the stream
\(A \in \R^{T \times T}\)
the attention pattern: row \(i\) is how query position \(i\) spreads its attention over key positions
\(M\)
the causal mask (\(-\infty\) above the diagonal)
\(d_h\)
the head size, 64 in GPT-2 small

This split matters because the two circuits can be studied separately. A head’s pattern \(A\) is easy to plot, and the OV matrix can be multiplied with the embedding and unembedding (\(W_U W_O^\top W_V^\top W_E^\top\), up to norms) to see which input tokens raise which output tokens. Heads in later layers can also read what earlier heads wrote. When a later head’s keys use an earlier head’s output, it is called K-composition; with queries, Q-composition; with values, V-composition. [Certain]

Induction heads

The best-understood circuit is the induction circuit, first introduced in session 143. It completes the pattern \([A][B] \dots [A] \to [B]\): if the current token appeared before, predict the token that came after it last time. Two heads work together:

  1. A previous-token head in an earlier layer. At each position \(j\) it attends to \(j - 1\) and writes “the token before me was \(t_{j-1}\)” into the stream at \(j\).
  2. An induction head in a later layer. Its query at position \(i\) asks “where was the token \(t_i\) before?”. Its keys read the previous-token information (K-composition), so the best match is the position \(j\) with \(t_{j-1} = t_i\). Its OV circuit copies \(t_j\), which raises the logit of \(t_j\) as the next token.

To find induction heads, feed a random token sequence of length \(T\) followed by the same sequence again. In the second copy, the right answer is always \(T - 1\) positions back, so we measure the average attention to that spot:

\[ \text{IS}(h) = \frac{1}{T} \sum_{i = T + 1}^{2T} A^{(h)}_{i,\; i - T + 1} \tag{157.6} \]

Here position 0 is a beginning-of-sequence token, the first copy fills positions 1 to \(T\), and the second copy fills \(T + 1\) to \(2T\). For query \(i\), its earlier copy sits at \(i - T\), and the token after that copy sits at \(i - T + 1\). An induction score near 1 means the head always looks exactly there.

Real numbers: induction in GPT-2 small

We fed 50 random tokens, then the same 50 again. On the first copy the model cannot predict anything: the average loss is 12.33 nats (worse than the 10.82 of a uniform guess over 50,257 tokens, because the model expects real text). On the second copy the loss falls to 0.27 nats: the model copies almost perfectly. The heads with the highest induction score (157.6) are 7.10 (0.91), 5.1 (0.91), 5.5 (0.91), 6.9 (0.90) and 7.2 (0.84). Head 4.11 attends to the previous token 99% of the time: a previous-token head, one layer below the first induction heads, as the circuit needs. [Certain: we ran this; the code is below.]

Figure 3. Real attention patterns of GPT-2 small on 10 random tokens repeated twice (21 positions with the beginning token). Rows are queries, columns are keys; darker means more attention. Click any row to select a query. Tokens are quoted as in Figure 2. The bars on the right show the model’s loss for predicting the next token after each row’s position.
How to read Figure 3

Head 4.11 draws a line just below the diagonal: every token looks at the token before it. Head 5.5 draws a line in the second copy, offset so that each query looks at the token after the earlier copy of itself (its score is 0.945). In the first copy it has nothing to match, so it rests on the beginning token. Now tick the ablation box. We set the outputs of the 11 heads in layers 2 to 4 whose previous-token score is above 0.3 to zero. The induction line of head 5.5 fades (score 0.309), and the loss on the second copy jumps from 0.54 to 3.61 nats. The induction head did not change; its keys lost the information that the previous-token heads used to write. That is K-composition, shown causally.

Olsson et al. (2022) found that induction heads appear suddenly during training, in a “phase change” that happens at the same time as a sharp rise in in-context learning, visible as a small bump in the training loss. They also gave evidence that induction-like heads produce a large part of in-context learning in small models, and they argued (with weaker evidence) that the same holds in large models. [Certain] A one-layer attention-only model cannot form induction heads, because the circuit needs one head to read another head’s output. [Certain]

Probing classifiers: is the information there?

The logit lens only asks about next-token predictions. Often we want to know about other information: does the hidden state encode the part of speech, the sentiment, whether a statement is true, where the pieces are on a game board? A probe is a small classifier trained on frozen hidden states to predict such a label (Alain and Bengio, 2016). The most common one is a linear probe:

\[ p(y = 1 \mid \mathbf{h}_\ell) = \sigma\!\left( \mathbf{w}^\top \mathbf{h}_\ell + b \right), \qquad \min_{\mathbf{w}, b} \ \frac{1}{N} \sum_{n=1}^{N} \text{CE}\big( y_n, p(y_n \mid \mathbf{h}_\ell^{(n)}) \big) + \lambda \lVert \mathbf{w} \rVert^2 \tag{157.7} \]
Read the symbols
\(\mathbf{h}_\ell^{(n)}\)
the hidden state at layer \(\ell\) for example \(n\) (one token, or a mean over tokens); the model is frozen
\(\sigma\)
the sigmoid; for many classes use a softmax
\(\text{CE}\)
cross-entropy between the true label and the probe’s probability
\(\lambda\)
weight decay that keeps the probe simple

If a linear probe reaches high accuracy on held-out data, the information is linearly decodable: there is a direction \(\mathbf{w}\) in the stream that separates the classes. Probing at every layer shows where the information appears. Real examples: true and false statements separate along one direction in LLaMA models (Marks and Tegmark, 2023); in a model trained on Othello moves, the board state can be decoded, first with nonlinear probes (Li et al., 2022) and then with linear ones once the board is written as “my piece / their piece” instead of “black / white” (Nanda et al., 2023). [Certain]

Training a probe is ordinary logistic regression: the model stays frozen, and only \(\mathbf{w}\) and \(b\) move. Figure 4 runs it live on a small toy (described in the next subsection): 96 training points in 2D, full-batch gradient descent, 400 steps. Each step has three parts: a forward pass, the loss and its gradient, and the update. The gradient of (157.7) has a simple form: \(\nabla_{\mathbf{w}} = \frac{1}{N}\sum_n (p_n - y_n)\, \mathbf{h}_n + 2\lambda \mathbf{w}\), so every misfit point pulls the boundary toward its correct side.

label 0label 1decision boundary and the \(-\eta\,\mathbf{g}\) arrow on \(\mathbf{w}\)training loss
  1. \(\mathbf{w} \leftarrow \mathbf{0}, \; b \leftarrow 0\)
  2. for \(t = 1 \dots 400\): (full batch, \(N = 96\))
  3.   \(z_n \leftarrow \mathbf{w}^\top \mathbf{h}_n + b, \; p_n \leftarrow \sigma(z_n)\) for every \(n\)
  4.   \(\mathcal{L} \leftarrow \frac{1}{N}\sum_n \text{CE}(y_n, p_n) + \lambda \lVert \mathbf{w} \rVert^2\) (157.7)
  5.   \(\mathbf{g} \leftarrow \frac{1}{N}\sum_n (p_n - y_n)\,\mathbf{h}_n + 2\lambda\mathbf{w}\)
  6.   \(g_b \leftarrow \frac{1}{N}\sum_n (p_n - y_n)\)
  7.   \(\mathbf{w} \leftarrow \mathbf{w} - \eta\,\mathbf{g}; \; b \leftarrow b - \eta\, g_b\)
  8. return \(\mathbf{w}, b\); report accuracy on held-out points
Figure 4. A linear probe trained live in your browser with formula (157.7), \(\lambda = 0.001\). Left: the 96 training points at the chosen layer (circles are classified correctly by the current probe, crosses wrongly), the decision boundary \(\mathbf{w}^\top \mathbf{h} + b = 0\), and the arrow \(-\mathbf{g}\) that the next update follows. The ringed point is example 1, whose numbers appear in the box. Right: the training loss so far and the current accuracies. At slow speeds each click shows one part of a step; at fast speeds one tick runs several whole steps.
How to read Figure 4

Keep layer 4, real labels and \(\eta = 0.5\), and press Play. At step 1 the gradient is \(\mathbf{g} = (-0.584, -0.387)\), so the first update moves \(\mathbf{w}\) from \((0, 0)\) to \((0.292, 0.194)\), and the loss falls from \(\log 2 = 0.693\) to 0.493. After 400 steps the loss is 0.213, \(\mathbf{w} = (2.246, 1.322)\), and the probe gets 0.938 on the training points and 0.948 on held-out points. Now pick \(\eta = 0.05\): after 400 steps the loss is still 0.223 (too slow). With \(\eta = 20\) the steps overshoot: the loss jumps around and ends at 0.231, and held-out accuracy drops to 0.896. Switch to the control task: the loss stops far higher and held-out accuracy is only 0.708. Move the layer slider down to 0: with real labels the probe now stops at 0.531, near chance. The probe can only find what the layer already contains.

The trap: a probe can learn the task by itself

A probe is a model too. A flexible probe (a deep MLP, or nearest neighbors) can reach high accuracy just by memorizing word identities, even when the network does not represent the concept in any useful way. Hewitt and Liang (2019) proposed control tasks: give each word type a random label, and train the same probe on it. Only the probe can “know” those labels, so the control accuracy measures how much the probe memorizes. A good probe has high selectivity:

\[ \text{selectivity} = \text{Acc}_{\text{task}} - \text{Acc}_{\text{control}} \tag{157.8} \]

Figure 5 makes this concrete with a toy. There are 32 “word types”, each a small cloud of points in 2D (think of a 2D slice of the hidden state). Half the types have the concept (label 1). As the layer grows, the model pushes the two groups apart along one direction. We train a linear probe (logistic regression, formula 157.7, 400 steps of gradient descent) and a 1-nearest-neighbor probe, on both the real labels and a control task.

label 0label 1linear decision boundary
Figure 5. Probing a toy representation. Points are held-out examples (3 per word type); circles are classified correctly, crosses wrongly. The readout below compares both probes on both tasks at the chosen layer.
How to read Figure 5

At layer 0 the concept is not in the representation. The linear probe gets 53% on real labels: chance. But the 1-nearest-neighbor probe already gets 82%, because the types are tight clouds and it simply remembers which type has which label. Switch to the control task: 1-NN gets 88% on random labels. Its selectivity is near 0 at every layer, so its accuracy tells us nothing about the model. The linear probe is different: on the control task it stays near 70% (some random labelings of 32 clouds are partly separable by a line), while on the real task it rises to 95% at layer 4. Its selectivity grows from −0.13 at layer 0 to 0.26 at layers 5 and 6. That growth is the real signal.

Selectivity at layer 4 (Figure 5)

Linear probe: real-task test accuracy 0.948, control-task test accuracy 0.708, so selectivity = 0.948 − 0.708 = 0.240. The 1-NN probe: 0.958 and 0.927, selectivity 0.031. The 1-NN probe has the higher task accuracy and the lower selectivity. If you reported only task accuracy, you would pick the wrong probe and the wrong conclusion.

Even a selective probe shows only that information is present, not that the model uses it. The concept may sit in the stream as a side effect. To test use, change the representation along the probe direction and see whether behavior changes. Li et al. did this for Othello: editing the decoded board changed the model’s predicted legal moves. [Certain] That brings us to causal methods, after one more tool.

Superposition: why neurons are hard to read

The simplest hope would be: one neuron, one concept. In practice, most neurons are polysemantic: in the one-layer model of Bricken et al., a single MLP unit fires for academic citations, English dialogue, HTTP requests and Korean text. [Likely] The reason is superposition (Elhage et al., “Toy Models of Superposition”, 2022): a model wants to store more features than it has dimensions, so it gives features directions that are almost orthogonal, not exactly. The activation is a sum of active features:

\[ \mathbf{x} \approx \sum_{i=1}^{n} f_i \, \mathbf{d}_i, \qquad n \gg d, \qquad f_i \ge 0 \text{ and mostly } 0 \tag{157.9} \]
Read the symbols
\(\mathbf{x} \in \R^{d}\)
an activation vector (an MLP layer or the residual stream)
\(\mathbf{d}_i\)
the unit direction of feature \(i\)
\(f_i\)
how strongly feature \(i\) is present; for one input, only a few are non-zero (features are sparse)
\(n\)
the number of features, much larger than the dimension \(d\)

Why does this work? In \(d\) dimensions only \(d\) directions can be exactly orthogonal, but very many can be nearly orthogonal. If features rarely appear together, small overlaps cause little harm, because the interference only appears when two overlapping features are active at the same time. In the toy models, 5 features stored in 2 dimensions settle into a regular pentagon. [Certain]

Sparse autoencoders: finding the features

If activations are sparse sums of feature directions (157.9), we can try to recover the directions with dictionary learning. A sparse autoencoder (SAE) maps each activation to a much wider, mostly-zero code, then reconstructs the activation from it:

\[ \mathbf{f}(\mathbf{x}) = \mathrm{ReLU}\!\big( W_e (\mathbf{x} - \mathbf{b}_d) + \mathbf{b}_e \big), \qquad \hat{\mathbf{x}} = W_d \, \mathbf{f}(\mathbf{x}) + \mathbf{b}_d \tag{157.10} \]
\[ \mathcal{L}_{\text{SAE}} = \big\lVert \mathbf{x} - \hat{\mathbf{x}} \big\rVert_2^2 + \lambda \sum_{i=1}^{m} f_i(\mathbf{x}) \, \big\lVert W_d[:, i] \big\rVert_2 \tag{157.11} \]
Read the symbols
\(W_e \in \R^{m \times d}, \ W_d \in \R^{d \times m}\)
encoder and decoder; \(m\) is the dictionary size, often 8 to 256 times \(d\) or more
\(W_d[:, i]\)
column \(i\) of the decoder: the learned direction of feature \(i\) (the estimate of \(\mathbf{d}_i\))
\(\mathbf{b}_d, \mathbf{b}_e\)
biases; subtracting \(\mathbf{b}_d\) centers the input
\(\lambda\)
sparsity weight: larger \(\lambda\) means fewer active features but worse reconstruction
\(\lVert W_d[:, i] \rVert\)
multiplying the L1 term by the decoder norm stops the SAE from cheating by making codes tiny and decoder columns huge

Each feature is then interpreted by looking at the inputs that activate it most, and checked by asking whether those inputs share one meaning. Bricken et al. (2023) trained SAEs on the 512-neuron MLP layer of a one-layer transformer, with dictionaries from 512 to 131,072 features, and studied a run with 4,096 features in detail. They found features for DNA sequences, legal language, HTTP requests, Arabic script and more, features that are much more monosemantic than the neurons. [Certain] Templeton et al. (2024) scaled this to the middle-layer residual stream of Claude 3 Sonnet with about 1M, 4M and 34M features, and found abstract features that respond to the same concept in many languages and in images, including safety-relevant ones (deception, code vulnerabilities, bias). Clamping a “Golden Gate Bridge” feature to a high value made the model bring up the bridge in unrelated answers. [Certain]

The ReLU plus L1 recipe has a known weakness: the L1 term pulls every active code toward 0 (shrinkage), so features are underestimated. Newer SAEs replace it. TopK SAEs keep only the \(k\) largest pre-activations, so the number of active features is fixed directly (Gao et al., 2024, who trained a 16M-latent SAE on GPT-4 activations). JumpReLU SAEs use a learned threshold: a pre-activation below \(\theta\) becomes 0, above it passes unchanged (used for Gemma Scope, open SAEs on Gemma 2). [Certain] Figure 6 shows the three choices on the pentagon toy.

Active features (value 1):
feature directions \(\mathbf{d}_0\) to \(\mathbf{d}_4\)activation \(\mathbf{x}\)reconstruction \(\hat{\mathbf{x}}\)recovered codes \(\hat f_i\)projection \(p_i = \mathbf{d}_i^\top \mathbf{x}\) (encode step)
  1. \(\mathbf{x} \leftarrow \sum_{i \in \text{active}} \mathbf{d}_i\) (157.9, every \(f_i = 1\))
  2. for \(i = 0 \dots 4\): \(\; p_i \leftarrow \mathbf{d}_i^\top \mathbf{x}\) (tied encoder \(W_e = W_d^\top\))
  3.   \(\hat f_i \leftarrow\) linear: \(p_i\); ReLU: \(\max(0, p_i - \theta)\); JumpReLU: \(p_i \cdot [p_i \gt \theta]\)
  4. \(\hat{\mathbf{x}} \leftarrow \mathbf{0}\); for \(i = 0 \dots 4\): \(\; \hat{\mathbf{x}} \leftarrow \hat{\mathbf{x}} + \hat f_i \, \mathbf{d}_i\) (157.10)
  5. error \(\leftarrow \lVert \mathbf{x} - \hat{\mathbf{x}} \rVert^2\); \(\;L_0 \leftarrow\) number of \(\hat f_i \ne 0\)
Figure 6. Five features stored in two dimensions as a pentagon (157.9). The SAE here uses the true directions as its tied encoder and decoder, so we can see only the effect of the activation function. Toggle features on and off and compare the three encoders. Press Play to run the SAE one operation at a time: build \(\mathbf{x}\), encode it feature by feature (the dashed line drops \(\mathbf{x}\) onto \(\mathbf{d}_i\)), then decode by adding the arrows \(\hat f_i \mathbf{d}_i\) head to tail.
How to read Figure 6

Press Reset, then Step through the encoder. With only feature 0 active, the projections onto the five directions are 1, 0.309, −0.809, −0.809 and 0.309 (cosines of 0°, 72° and 144°). The linear encoder keeps all of this interference: 5 active codes and an error of 2.25. ReLU with bias 0.4 removes the interference but shrinks the true code to 0.6 (error 0.16). JumpReLU with \(\theta = 0.4\) gives exactly 1 and error 0. Now turn on features 0 and 2 together. Their sum points exactly along \(\mathbf{d}_1\) (because \(\mathbf{d}_0 + \mathbf{d}_2 = 2\cos 72° \, \mathbf{d}_1 = 0.618\,\mathbf{d}_1\)), so JumpReLU reports feature 1 with value 0.618 and a perfect reconstruction: zero error, wrong explanation. This is why superposition only works when features are sparse, and why reconstruction error alone does not prove an SAE is right.

The sparsity trade-off, measured (toy SAE in the code below)

We hid 8 sparse features in 4 dimensions (each feature on with probability 0.05, so 0.41 active per input on average) and trained an SAE with 16 latents using (157.10) and (157.11). For three values of \(\lambda\):

\(\lambda\)Mean best cosine to a true featureActive latents per input (L0)Unexplained variance
0.010.8971.970.001
0.100.9441.070.020
0.300.9850.830.098

More sparsity gives features closer to the true directions and fewer active latents, but worse reconstruction. Real SAE papers report this as a curve of reconstruction quality against L0, and also check how much the language-model loss rises when the SAE’s reconstruction replaces the real activation.

From correlation to cause: activation patching

A lens, a probe or an SAE feature can show that information is there. To show that a component matters, we intervene. Activation patching (also called causal tracing, Meng et al., 2022; Wang et al., 2022) uses two inputs that differ in one detail:

  • Clean: “When Mary and John went to the store, John gave a drink to” \(\to\) the model should say “ Mary”.
  • Corrupted: the same, but with “Mary gave a drink to” \(\to\) now “ John” is the right answer.

We measure the logit difference \(\text{LD} = z_{\text{Mary}} - z_{\text{John}}\) at the last position. Then we run the corrupted input but replace one activation (one layer, one position) with its value from the clean run. If that single value restores the clean behavior, it carries the information that matters:

\[ \text{recovered}(\ell, p) = \frac{\text{LD}\big(\text{corrupted with } \mathbf{h}_\ell[p] \leftarrow \mathbf{h}^{\text{clean}}_\ell[p]\big) - \text{LD}_{\text{corrupted}}}{\text{LD}_{\text{clean}} - \text{LD}_{\text{corrupted}}} \tag{157.12} \]

A value of 1 means the patch fully restores clean behavior; 0 means no effect. Figure 7 shows the real result on GPT-2 small, patching the residual stream after each layer at two positions: the second name (“ John” in the clean input) and the last token (“ to”).

clean runcorrupted runpatched run, copied activationrecovered fraction (darker = more)
  1. run clean \(x_c\); cache \(\mathbf{h}^c_\ell[p]\) for every \(\ell, p\); \(\;\text{LD}_c \leftarrow z_{\text{Mary}} - z_{\text{John}}\)
  2. run corrupted \(x_k\); \(\;\text{LD}_k \leftarrow z_{\text{Mary}} - z_{\text{John}}\)
  3. for \(\ell = 0 \dots 11\), for \(p\) in the chosen positions:
  4.   run \(x_k\) with a hook: \(\mathbf{h}_\ell[p] \leftarrow \mathbf{h}^c_\ell[p]\)
  5.   \(\text{LD} \leftarrow z_{\text{Mary}} - z_{\text{John}}\) at the last position
  6.   \(R[\ell, p] \leftarrow (\text{LD} - \text{LD}_k) / (\text{LD}_c - \text{LD}_k)\) (157.12)
  7. return the map \(R\)
Figure 7. Activation patching in GPT-2 small (157.12). Clean logit difference 2.006, corrupted −1.789. Press Play to run the sweep of Algorithm 1: for each layer, the residual stream at one position is copied from the clean run (top row) into the corrupted run (second row); the marker on the number line shows the patched logit difference, and the cell of the map gets the recovered fraction. Click a cell to jump to it. The recovered fractions are real measurements on GPT-2 small; the patched logit difference shown is rebuilt from them with (157.12).
How to read Figure 7

Early on, all the needed information sits at the second-name position: patching it there after layers 0 to 3 recovers almost 100%. Patching at the last token does almost nothing until layer 6. Between layers 7 and 8 the two rows of the map trade places: the information has been moved from the name position to the last position, by attention heads in those layers (the “S-inhibition heads” of Wang et al. sit in layers 7 and 8). [Likely] After layer 8, patching the last token alone recovers 99.5%, and the name position no longer matters. This is how the indirect-object circuit of Wang et al. was found: patching narrows the search from “the whole model” to “a few heads in layers 7 to 10 that read the name positions”. [Certain for our numbers]

One patch, by hand

Patching the last token after layer 7 recovers 0.389. The gap between clean and corrupted is \(2.006 - (-1.789) = 3.795\). So the patched logit difference is \(-1.789 + 0.389 \times 3.795 \approx -0.31\): the model still prefers “ John”, but much less. One layer later, the patch recovers 0.995: the patched run prefers “ Mary” almost as strongly as the clean run.

Steering: using a direction

Once you have a direction that matters (a probe weight, a difference of mean activations between two prompt sets, or an SAE decoder column), you can add it to the stream during generation:

\[ \mathbf{h}_\ell \leftarrow \mathbf{h}_\ell + \alpha \, \hat{\mathbf{d}} \tag{157.13} \]

Here \(\hat{\mathbf{d}}\) is a unit direction and \(\alpha\) its strength (\(\alpha \lt 0\) pushes away from the concept). Turner et al. (2023) steer topic and sentiment this way with directions from pairs of prompts; Arditi et al. (2024) find that refusal in many open chat models is controlled by one direction: removing it from the stream stops refusals, adding it makes the model refuse harmless requests. [Certain] This is powerful for research and a real risk for deployed open-weight models, a topic for session 158.

Putting it together: algorithms and code

Algorithm 1: logit lens and a layer-by-position patching sweep
  1. input: model with \(L\) layers, final norm, \(W_U\); clean input \(x_c\), corrupted input \(x_k\) of the same length; answer tokens \(a\) (clean) and \(b\) (corrupted)
  2. run \(x_c\); store \(\mathbf{h}^{c}_\ell[p]\) for all layers \(\ell\) and positions \(p\)
  3. for \(\ell = 0 \dots L\): \(q_\ell \leftarrow \softmax(W_U \mathrm{Norm}(\mathbf{h}^c_\ell[\text{last}]))\); record top tokens, rank of \(a\), \(\KL(q_L \| q_\ell)\) // logit lens, (157.2), (157.3)
  4. \(\text{LD}_c \leftarrow z_a - z_b\) on \(x_c\); \(\;\text{LD}_k \leftarrow z_a - z_b\) on \(x_k\)
  5. for each layer \(\ell\) and position \(p\):
  6.   run \(x_k\) with a hook that sets \(\mathbf{h}_\ell[p] \leftarrow \mathbf{h}^c_\ell[p]\)
  7.   \(R[\ell, p] \leftarrow (\text{LD} - \text{LD}_k) / (\text{LD}_c - \text{LD}_k)\) // formula (157.12)
  8. return lens table and the map \(R\); zoom into the cells with large \(R\) by patching single heads
Algorithm 2: train an SAE and interpret its features
  1. input: a frozen model, a layer \(\ell\), a large text set, dictionary size \(m\), sparsity weight \(\lambda\)
  2. collect activations \(\mathbf{x}\) at layer \(\ell\) for many tokens (often hundreds of millions or more) // stream them, do not store all
  3. initialize \(W_d\) with unit columns, \(W_e \leftarrow W_d^\top\), biases 0
  4. repeat: \(\mathbf{f} \leftarrow \mathrm{ReLU}(W_e(\mathbf{x} - \mathbf{b}_d) + \mathbf{b}_e)\), \(\hat{\mathbf{x}} \leftarrow W_d \mathbf{f} + \mathbf{b}_d\); step on (157.11) with Adam
  5.   track L0, reconstruction error, and dead latents (never active); re-initialize dead ones
  6. check: replace \(\mathbf{x}\) by \(\hat{\mathbf{x}}\) inside the model and measure the rise in LM loss
  7. for each feature \(i\): list the top-activating tokens and contexts; give it a label; check the label on new text
  8. test causally: add \(\alpha W_d[:, i]\) to the stream (157.13) or clamp \(f_i\), and see if behavior changes as the label predicts

The PyTorch code below reproduces the logit lens of Figure 2 and the induction scan from the worked example on GPT-2 small. It needs transformers and downloads the 124M model once.

import torch
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
torch.manual_seed(0)
tok = GPT2TokenizerFast.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2", attn_implementation="eager").eval()

# 1) Logit lens, formula (157.2): decode the last position of every layer
ids = tok("The Eiffel Tower is in the city of", return_tensors="pt").input_ids
paris = tok.encode(" Paris")[0]
with torch.no_grad():
    out = model(ids, output_hidden_states=True)
    hs = out.hidden_states                       # 13 tensors: embeddings + 12 layers
    for l, h in enumerate(hs):
        x = h[0, -1]
        if l < len(hs) - 1:                      # HF already applied ln_f to the last one
            x = model.transformer.ln_f(x)
        logp = torch.log_softmax(model.lm_head(x), -1)
        rank = int((logp > logp[paris]).sum()) + 1
        print(f"layer {l:2d}  top: {tok.decode(logp.argmax())!r:12}  p(Paris) = {logp[paris].exp():.4f}  rank {rank}")

# 2) Induction score, formula (157.6): random tokens, repeated once
T = 50
r = torch.randint(1000, 10000, (1, T), generator=torch.Generator().manual_seed(0))
seq = torch.cat([torch.tensor([[tok.bos_token_id]]), r, r], 1)
with torch.no_grad():
    o = model(seq, output_attentions=True)
nll = torch.nn.functional.cross_entropy(o.logits[0, :-1], seq[0, 1:], reduction="none")
print(f"loss on first copy {nll[1:T].mean():.2f}, on second copy {nll[T+1:].mean():.2f} nats")
q = torch.arange(T + 1, 2 * T + 1)               # queries in the second copy
scores = sorted(((float(a[0, h, q, q - T + 1].mean()), f"{l}.{h}")
                 for l, a in enumerate(o.attentions) for h in range(12)), reverse=True)
print("top induction heads:", [(name, round(s, 2)) for s, name in scores[:5]])
# layer 10 top ' Paris' p = 0.1834 ... loss 12.33 then 0.27 ... [('7.10', 0.91), ('5.1', 0.91), ('5.5', 0.91), ...]

The second program needs only PyTorch. It hides 8 sparse features in 4 dimensions (formula 157.9), trains the SAE of (157.10) and (157.11) for three sparsity weights, and prints the table from the worked example. It runs in about 20 seconds on a laptop CPU and prints the same numbers every time.

import torch
torch.manual_seed(0)

# 1) Toy activations in superposition: 8 sparse features stored in 4 dimensions, formula (157.9)
n_feat, d, m = 8, 4, 16
D_true = torch.nn.functional.normalize(torch.randn(n_feat, d), dim=1)   # true feature directions
def batch(B=1024, p=0.05):
    on = (torch.rand(B, n_feat) < p).float()
    f = on * torch.rand(B, n_feat)               # sparse, non-negative feature values
    return f @ D_true, f                         # x = sum_i f_i d_i

# 2) A sparse autoencoder, formulas (157.10) and (157.11)
class SAE(torch.nn.Module):
    def __init__(self, d, m):
        super().__init__()
        self.W_e = torch.nn.Parameter(torch.randn(m, d) * 0.3)
        self.b_e = torch.nn.Parameter(torch.zeros(m))
        self.W_d = torch.nn.Parameter(self.W_e.data.t().clone())   # (d, m), starts tied
        self.b_d = torch.nn.Parameter(torch.zeros(d))
    def forward(self, x):
        f = torch.relu((x - self.b_d) @ self.W_e.t() + self.b_e)   # sparse codes
        return f @ self.W_d.t() + self.b_d, f                      # reconstruction

def train(lam, steps=10000):
    torch.manual_seed(1)
    sae = SAE(d, m); opt = torch.optim.Adam(sae.parameters(), lr=3e-3)
    for _ in range(steps):
        x, _ = batch()
        xh, f = sae(x)
        mse = ((x - xh) ** 2).sum(1).mean()
        l1 = (f * sae.W_d.norm(dim=0)).sum(1).mean()   # L1 weighted by decoder column norms
        opt.zero_grad(); (mse + lam * l1).backward(); opt.step()
    return sae

# 3) Sweep the sparsity weight: did the SAE find the true directions?
torch.manual_seed(2); x_test, f_test = batch(8192)
print("true L0: %.2f" % (f_test > 0).float().sum(1).mean())
for lam in [0.01, 0.1, 0.3]:
    sae = train(lam)
    with torch.no_grad():
        xh, f = sae(x_test)
        Wd = torch.nn.functional.normalize(sae.W_d, dim=0)   # unit decoder columns
        best = (D_true @ Wd).max(dim=1).values                # best cosine per true feature
        fvu = ((x_test - xh) ** 2).sum() / ((x_test - x_test.mean(0)) ** 2).sum()
    print("lam %.2f  mean best cosine %.3f  L0 %.2f  unexplained variance %.3f"
          % (lam, best.mean(), (f > 0).float().sum(1).mean(), fvu))
# lam 0.01  mean best cosine 0.897  L0 1.97  unexplained variance 0.001
# lam 0.10  mean best cosine 0.944  L0 1.07  unexplained variance 0.020
# lam 0.30  mean best cosine 0.985  L0 0.83  unexplained variance 0.098

For real work, libraries such as TransformerLens (hooks on every activation of many open models) and SAELens (training and loading SAEs, including Gemma Scope) save a lot of hook-writing. [Likely]

Where this is used

UseToolWhat it gives you
Debugging a wrong answerLogit lens, patchingWas the right answer present in a middle layer and lost later, or never formed? Which position carried it?
Model editing (fix one fact)Causal tracing, then a targeted edit (ROME)The layers where a fact is stored at the subject token, so the edit touches only those weights (session 21)
Safety monitoringLinear probes, SAE featuresCheap detectors on hidden states for deception, harmful intent or prompt injection, run alongside the model [Likely]
Controlling behaviorSteering vectors (157.13)Turn a style, topic or refusal up or down without fine-tuning; also a way attackers remove safety from open weights
Understanding in-context learningAttention patterns, induction scoreWhy few-shot prompts work, and why repeated text is predicted so well
Early exit and speculative decodingLogit lens, tuned lensMiddle layers that already predict well can draft tokens or stop early (session 121)
Vision-language modelsLogit lens on image tokens, attention to image patches, probesWhether the answer used the image or only the text prior; which patches were read. A tool for checking visual hallucination [Likely]
Video models (action recognition, key-moment detection)Probes per layer and per frame, attention over time, SAEs on vision encodersDoes the encoder represent motion and order, or only single-frame appearance? Which frames drove a detected key moment? [Speculation for SAEs on video encoders]
Video retrieval and Video-RAGProbes and patching on the embedding modelWhether a retrieval embedding encodes the asked attribute (time, action, object count) before you trust it in a pipeline [Speculation]

Common mistakes

  • Treating an attention pattern as an explanation. High attention to a token does not mean that token’s information was used: the OV circuit may copy almost nothing, and many heads attend to the first token as a resting place. Check with patching or ablation.
  • Forgetting the final norm in the logit lens, or applying it twice when the library already did (Hugging Face applies ln_f to the last hidden state of GPT-2).
  • Reading lens probabilities as calibrated. In our Eiffel Tower example, layer 10 gives “ Paris” 0.183 and the output gives 0.070. Middle layers are not the model’s answer.
  • Reporting probe accuracy without a baseline. Always compare with a control task (157.8), a random-weights model, and a majority-class baseline. A flexible probe can learn the task itself.
  • Concluding “the model uses X” from a probe. Decodable is not the same as used. Intervene along the direction and check behavior.
  • Trusting one-head ablations. Redundant and backup heads hide effects, as in Figure 3 where removing head 4.11 alone changed almost nothing. Ablate groups, and prefer patching with real activations from another input over setting values to zero (zero can push the model off its normal range).
  • Judging an SAE by reconstruction only. Figure 6 shows zero error with the wrong features. Also measure L0, dead latents, the rise in LM loss when the reconstruction is used, and whether feature labels hold on new text.
  • Mixing up layer numbering. Hidden state 0 is the embedding; hidden state \(\ell\) is the output of layer \(\ell - 1\) when layers are counted from 0. Write down your convention.

Interview questions

What is the residual stream, and why does it make interpretability possible?basic
It is the per-token vector that every layer reads from (through a norm) and adds its output to; nothing is overwritten. So the final hidden state is the sum of the embedding and all block outputs (157.1). Because it is a sum in one shared space, we can decode partial sums (logit lens), train readers on it (probes), decompose it into directions (SAEs), and swap pieces of it between inputs (patching).
Explain the logit lens. What are its limitations?basic
Apply the model’s final norm and unembedding to the hidden state of a middle layer, \(\softmax(W_U \mathrm{Norm}(\mathbf{h}_\ell))\), to see what the model would predict if it stopped there. It needs no training. Limits: early layers may use a different basis, so the lens can give nonsense or biased predictions (fixed by the tuned lens, a learned affine translator per layer); lens probabilities are not calibrated; and it only shows next-token information, not other features. In GPT-2 small, “ Paris” goes from rank 22,680 at the embedding to first place after layer 10.
What are the QK and OV circuits of an attention head?core
An attention head can be written with two \(d \times d\) low-rank matrices: \(W_Q W_K^\top\) decides the attention pattern (where to look) and \(W_V W_O\) decides what is read from the attended position and written into the stream. They can be analyzed separately: plot patterns for QK, and multiply the OV matrix with embedding and unembedding to see which tokens it copies or suppresses. Composition between heads happens when a later head’s query, key or value input reads an earlier head’s output.
What is an induction head? How would you find one in a model?core
A head that, at a token A, attends to the token right after an earlier A and copies it, completing \([A][B] \dots [A] \to [B]\). It relies on a previous-token head in an earlier layer through K-composition. To find it, feed a random sequence repeated twice and compute the induction score: mean attention from each second-copy position to the position after its earlier copy (157.6). In GPT-2 small, heads 5.1, 5.5, 6.9, 7.2 and 7.10 score 0.84 to 0.91, and the loss on the repeated half drops from 12.3 to 0.27 nats.
Why do induction heads matter for in-context learning?core
Olsson et al. showed that induction heads form in a sudden phase change during training, at the same time as a sharp improvement in in-context learning (loss on later tokens of a context falls much more than on early tokens), and that ablating them in small models removes much of that ability. They are a concrete mechanism for “use what appeared earlier in the prompt”: copying, and fuzzy versions of copying that support few-shot pattern following. The evidence for large models is more indirect.
A linear probe gets 95% accuracy at layer 20 for “the statement is true”. What can you conclude?core
Only that the label is linearly decodable from layer 20 on this data. Check first: held-out data from a different distribution, a control task or random-label baseline (selectivity, 157.8), a random-weights model, and whether a simpler feature (sentence length, a keyword) explains it. Even then, decodable does not mean used. To claim the model uses this direction, intervene: add or remove the direction and check whether the model’s true/false behavior changes as predicted.
What is superposition, and why does it make neurons polysemantic?core
A model can represent more features than it has dimensions by giving them nearly orthogonal directions, as long as features are sparse (rarely active together). Then the activation is a sparse sum of feature directions (157.9), and any single neuron (a basis direction) overlaps with many features, so it fires for several unrelated things. Interference only costs something when overlapping features co-occur, which is why sparsity makes superposition worthwhile.
How does a sparse autoencoder work, and how do you evaluate one?core
It encodes an activation into a wide code with ReLU (or TopK or JumpReLU), \(\mathbf{f} = \mathrm{ReLU}(W_e(\mathbf{x} - \mathbf{b}_d) + \mathbf{b}_e)\), and decodes \(\hat{\mathbf{x}} = W_d\mathbf{f} + \mathbf{b}_d\). Training minimizes reconstruction error plus an L1 penalty weighted by decoder column norms (157.11). Each decoder column is a feature direction. Evaluate with: L0 (active features per token), reconstruction error, the increase in LM loss when \(\hat{\mathbf{x}}\) replaces \(\mathbf{x}\), the number of dead latents, and human or automated checks that feature labels predict activations on new text, plus causal tests by steering with the feature.
Why do TopK and JumpReLU SAEs exist? What is wrong with ReLU plus L1?advanced
The L1 penalty pulls all active codes toward zero, so true feature magnitudes are underestimated (shrinkage), and the bias that removes interference also shrinks the real codes (Figure 6: 0.6 instead of 1 with bias 0.4). L1 also only indirectly controls sparsity. TopK keeps the \(k\) largest pre-activations, fixing L0 directly; JumpReLU uses a learned threshold and passes values above it unchanged. Both reduce shrinkage and give a better reconstruction-sparsity trade-off; TopK also makes the sparsity level easy to tune.
Describe activation patching. Why patch with real activations instead of zeros?advanced
Run a clean and a corrupted input that differ in one detail; run the corrupted one while copying one activation from the clean run; measure how much of the clean-vs-corrupted logit difference is restored (157.12). Sweeping layers and positions locates where the decisive information lives; in GPT-2 on the IOI prompt it moves from the name position to the last position between layers 7 and 8. Real activations from another input keep the model in its normal range, while zero or mean ablation can create out-of-distribution states and exaggerate or hide effects.
How could you use interpretability to monitor a deployed LLM application?advanced
Train linear probes or select SAE features on hidden states for the risks you care about (prompt injection, harmful intent, the model being unsure), validate them on held-out and adversarial data, and run them on every request; they cost one dot product per layer read. Tuned-lens trajectories have also been used to flag unusual inputs. Combine with output-based checks, monitor false positive rates, and remember that probes can be evaded by distribution shift or adaptive attacks. This needs access to hidden states, so it works with open or self-hosted models, not closed APIs.

Exercises

  1. In Figure 2 (Eiffel Tower), find the first layer where “ Paris” is in the top 10, and the first where it is top 1. What is the KL to the final prediction at each?
  2. For a repeated random sequence with \(T = 10\) (positions 0 to 20), which key position should an induction head attend to from query 15? Which token does it predict? Check it in Figure 3.
  3. In the pentagon toy (Figure 6), compute by hand the five projections when features 0 and 1 are both on. With JumpReLU at \(\theta = 0.4\), what is the reconstruction error? (Hint: \(\lVert \mathbf{d}_0 + \mathbf{d}_1 \rVert^2 = 2 + 2\cos 72°\).)
  4. A 1-NN probe reaches 0.90 on a task and 0.88 on its control task; a linear probe reaches 0.80 and 0.55. Which result tells you more about the model, and why?
  5. In Figure 7, the clean logit difference is 2.006 and the corrupted one −1.789. Patching the name position after layer 6 recovers 0.835. What is the patched logit difference?
  6. An SAE has \(m = 32{,}768\) latents on a \(d = 4{,}096\) residual stream. How many parameters do \(W_e\) and \(W_d\) have together? What is the expansion factor?
Answerscheck
1. After layer 8, rank 10 (\(p = 0.0255\), KL 1.77 nats); first place after layer 10 (\(p = 0.183\), KL 0.86). 2. Query 15 holds the 5th token of the second copy; its earlier copy is at position 5, so the head attends to position \(15 - 10 + 1 = 6\) and predicts the token at 6, which is also the true next token at 16. 3. \(\mathbf{x} = \mathbf{d}_0 + \mathbf{d}_1\); projections: \(\mathbf{d}_0\) and \(\mathbf{d}_1\): \(1 + 0.309 = 1.309\); \(\mathbf{d}_2\) and \(\mathbf{d}_4\): \(0.309 - 0.809 = -0.5\); \(\mathbf{d}_3\): \(-0.809 - 0.809 = -1.618\). JumpReLU keeps 1.309 and 1.309, so \(\hat{\mathbf{x}} = 1.309\,\mathbf{x}\) and the error is \(0.309^2 \times 2.618 = 0.250\). 4. The linear probe: selectivity 0.25 versus 0.02. The 1-NN probe learns almost everything by itself, so its accuracy says little about the representation. 5. \(-1.789 + 0.835 \times 3.795 \approx 1.38\). 6. \(2 \times 32{,}768 \times 4{,}096 = 268{,}435{,}456\) weights (about 268M), plus biases; the expansion factor is 8.

Key takeaways

  • The residual stream is a sum of everything each layer wrote (157.1), so it can be decoded, probed, decomposed and patched at any point.
  • The logit lens (157.2) shows predictions forming layer by layer; in GPT-2 small, “ Paris” appears only in layers 8 to 10. The tuned lens (157.4) fixes early layers with a learned translator.
  • Attention heads split into QK (where to look) and OV (what to copy) circuits (157.5). Induction heads, built from a previous-token head by K-composition, copy patterns and drive in-context learning; the induction score (157.6) finds them.
  • Probes (157.7) show what is decodable, not what is used. Report selectivity against a control task (157.8).
  • Superposition (157.9) makes neurons polysemantic; sparse autoencoders (157.10, 157.11) recover sparse, more monosemantic features, with a sparsity versus reconstruction trade-off.
  • Activation patching (157.12) and steering (157.13) test cause: change an internal value, watch the output.

Key formulas

\[ \mathbf{h}_L = \mathbf{h}_0 + \sum_\ell (\mathbf{a}_\ell + \mathbf{m}_\ell), \quad q_\ell = \softmax(W_U \mathrm{Norm}(\mathbf{h}_\ell)), \quad \text{IS} = \tfrac{1}{T}\sum_i A_{i,\,i-T+1} \]
\[ \mathbf{f} = \mathrm{ReLU}(W_e(\mathbf{x} - \mathbf{b}_d) + \mathbf{b}_e), \quad \mathcal{L} = \lVert \mathbf{x} - \hat{\mathbf{x}} \rVert^2 + \lambda \textstyle\sum_i f_i \lVert W_d[:, i] \rVert, \quad \text{recovered} = \frac{\text{LD}_{\text{patch}} - \text{LD}_{\text{corr}}}{\text{LD}_{\text{clean}} - \text{LD}_{\text{corr}}} \]

References for this session

  1. Elhage et al. (2021). A Mathematical Framework for Transformer Circuits
  2. Olsson et al. (2022). In-context Learning and Induction Heads
  3. Bricken et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
  4. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 10: Interpretability
  5. Elhage et al. (2021). A Mathematical Framework for Transformer Circuits
  6. Olsson et al. (2022). In-context Learning and Induction Heads
  7. Bricken et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
  8. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 10: Interpretability
  9. nostalgebraist (2020). interpreting GPT: the logit lens
  10. Belrose, Ostrovsky, McKinney, Furman, Smith, Halawi, Biderman, Steinhardt (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens
  11. Elhage et al. (2022). Toy Models of Superposition
  12. Alain & Bengio (2016). Understanding intermediate layers using linear classifier probes
  13. Hewitt & Liang (2019). Designing and Interpreting Probes with Control Tasks
  14. Belinkov (2021). Probing Classifiers: Promises, Shortcomings, and Advances
  15. Li et al. (2022). Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
  16. Nanda, Lee & Wattenberg (2023). Emergent Linear Representations in World Models of Self-Supervised Sequence Models
  17. Marks & Tegmark (2023). The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
  18. Cunningham et al. (2023). Sparse Autoencoders Find Highly Interpretable Features in Language Models
  19. Templeton et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
  20. Gao et al. (2024). Scaling and evaluating sparse autoencoders
  21. Lieberum et al. (2024). Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
  22. Meng, Bau, Andonian, Belinkov (2022). Locating and Editing Factual Associations in GPT
  23. Wang et al. (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
  24. Turner et al. (2023). Steering Language Models With Activation Engineering
  25. Arditi et al. (2024). Refusal in Language Models Is Mediated by a Single Direction