The residual stream and normalization
Every transformer block reads from one shared vector per token and adds a small edit back to it: the residual stream. Before each read, the vector is rescaled by a normalization layer. This session explains residual connections, LayerNorm and RMSNorm, why modern LLMs put the norm before each layer (Pre-LN) instead of after it (Post-LN), and how QK-norm stops attention logits from blowing up.
What you will learn
- Why a residual connection makes deep stacks trainable: the identity path in the gradient
- The residual stream view: layers read through a norm and linear maps, and write by adding
- Why the stream grows with depth, and why each layer needs a normalized copy
- LayerNorm vs RMSNorm: what each removes, why RMSNorm is cheaper, and the details that matter in real code (FP32 statistics, epsilon, gain conventions)
- Post-LN vs Pre-LN: gradients at initialization, warmup, and newer variants (sandwich norm, DeepNorm, scaled residual init)
- Growing attention logits and the fixes: QK-norm, soft-capping and z-loss, with real values from Qwen3 and Gemma 2
Why this matters
A modern LLM is a stack of 30 to 100 blocks. Each block has an attention layer and a feed-forward layer. Two small pieces of plumbing hold the stack together: the residual connection (add the layer’s output to its input) and normalization (rescale a vector before a layer reads it). Without them, a deep transformer does not train.
These pieces look boring, but they decide three things you meet in practice. First, whether training is stable: where you put the normalization changes whether you need a long warmup and whether the loss suddenly jumps to NaN. Second, how to think about what the model does: the residual connections create one shared vector per token, the residual stream, and every layer reads from it and writes into it. Interpretability work is built on this view. Third, speed and numerics: normalization runs dozens of times per token, so a cheaper version (RMSNorm) is used by almost every new open model, and its details matter for mixed precision and quantization.
In session 21 we built the second half of a block. In this session we connect the halves and the blocks to each other. In Session 23 we then assemble the whole model.
The residual connection
Take one layer \(F\) (attention or a feed-forward network). A plain network replaces its input: \(\mathbf{x}_{\text{new}} = F(\mathbf{x})\). A residual connection (also called a skip connection) keeps the input and adds the layer’s output to it:
- Read the symbols
- \(\mathbf{x}_l \in \R^{d}\)
- the vector of one token before sub-layer \(l\); \(d\) is the model width (4,096 in Llama 3 8B)
- \(F_l\)
- sub-layer number \(l\): an attention layer or an FFN, together with its normalization
- \(F_l(\mathbf{x}_l)\)
- the edit that this sub-layer adds; also called the residual branch
The idea comes from ResNet in computer vision (He et al., 2015), where it made networks with more than 100 layers trainable. [Certain] The layer no longer has to learn the full new vector. It only has to learn the change. If the best thing a layer can do is nothing, it can output zeros, and the input passes through untouched. In a plain network, “do nothing” means learning an exact identity matrix, which is hard.
Unrolling the stack
Apply (22.1) again and again, from the embedding \(\mathbf{x}_0\) up to the last sub-layer. Every edit stays in the sum:
The final vector is the embedding plus all edits. Nothing is ever overwritten; information is only added. (A layer can remove information by adding the negative of something, but it has to do this on purpose.)
Why gradients flow
Differentiate (22.1). The Jacobian (the matrix of partial derivatives) of one residual step is the identity plus the layer’s own Jacobian:
Expand the product and one term is just \(\partial \mathcal{L} / \partial \mathbf{x}_L\) times \(I\): the gradient from the loss reaches every layer directly, through the chain of identities, without being multiplied by any weight matrix. In a plain deep network, the gradient is a product of \(L\) weight Jacobians. If their typical size is a bit below 1, the product shrinks to almost zero (vanishing gradients); if a bit above 1, it explodes. The identity path is a gradient highway. This is why the session 7 session’s LSTM worked better than a plain RNN, and it is the same trick here, applied across depth instead of time.
The residual stream
Elhage et al. (2021) suggested a useful way to look at (22.2). [Certain] Do not think of the transformer as a pipe from layer to layer. Think of one residual stream per token: a vector of width \(d\) that runs from the embedding at the bottom to the output layer at the top. Each attention and FFN layer is a reader and writer attached to this stream:
- Read: the layer normalizes the stream and applies linear maps (for example \(W_Q, W_K, W_V\) in attention, or \(W_1\) in the FFN). Each linear map picks out some directions of the stream.
- Write: the layer’s output projection (\(W_O\), or \(W_2\) in the FFN) turns its result back into a \(d\)-dimensional vector, which is added to the stream.
Attention is the only part that moves information between token positions: it reads the streams of earlier tokens and writes into the stream of the current token. The FFN works on each position alone. At the top, a final normalization and the output matrix read the stream and give the next-token logits.
Three consequences of this view are worth remembering. (1) The stream is a shared memory with \(d\) dimensions. Many features must share it, so they are stored in many directions at once, not one per coordinate. (2) A layer far up can directly read what a layer far down wrote, because the write was added and never replaced. (3) Because the stream is linear (only sums), you can decompose the final vector into “contribution of each layer” and study each one. A popular trick, the logit lens, applies the output layer to the stream in the middle of the network to see what the model would predict at that depth. [Likely] Figure 1 walks one token through a small stack.
How to read Figure 1
Step forward and watch the bars. The edits are small compared to the stream, so the direction of the stream changes only a little at each step: the readout shows the cosine between the stream before and after the step, which stays near 0.97. Over 8 steps the small edits add up to a real change. The vector at the end still contains the embedding, plus every edit. The edits here are random numbers; in a trained model they are learned features. Real models are similar in one way: each layer’s write is usually much smaller than the stream itself. [Likely]
Why normalize at all?
By (22.2), the stream is a sum of many terms. Its size changes from layer to layer and from token to token. If each edit is a random vector with about unit variance per coordinate, the variances add up, and the stream grows like the square root of the number of edits:
(This assumes the embedding and every edit have unit variance per coordinate and are roughly uncorrelated, which is close to true at initialization.) For \(d = 64\) and \(L = 48\) this gives \(\sqrt{64 \cdot 49} = 56\); the simulation in Figure 4 measures 55.7. A layer that reads the stream directly would see inputs whose size depends on its depth, and on the token. Its weights would need a different scale at every depth, and its non-linearities (softmax, GELU) would work in different regimes. A normalization layer fixes this: it rescales the vector to a standard size before the layer reads it.
LayerNorm
Layer normalization (Ba et al., 2016) works on one token’s vector at a time, over its \(d\) features. [Certain] It removes the mean, divides by the standard deviation, and then applies a learned per-feature scale and shift:
- Read the symbols
- \(\mu, \sigma^2\)
- mean and variance of the \(d\) numbers of this one vector (not of the batch)
- \(\varepsilon\)
- a small constant (often \(10^{-5}\)) so we never divide by zero
- \(\gamma, \beta \in \R^d\)
- learned gain and bias, one per feature; they start at 1 and 0
Two properties follow. LayerNorm is shift invariant: adding the same constant to every coordinate does not change the output. It is scale invariant: multiplying the vector by any positive number does not change the output. Before \(\gamma\) and \(\beta\), the output always has mean 0 and variance 1. Its length is therefore always about \(\sqrt{d}\).
Why not batch normalization, which CNNs use? BatchNorm averages each feature over the examples of a batch. In language models, the batch has sequences of different lengths, the statistics during generation (one token at a time) differ from training, and the model would depend on what else is in the batch. LayerNorm uses only the token itself, so training and inference behave the same.
Take \(\mathbf{x} = (2, 4, 6, 8)\), with \(\gamma = 1\), \(\beta = 0\) and \(\varepsilon\) ignored.
LayerNorm: \(\mu = 5\). Deviations \((-3, -1, 1, 3)\), so \(\sigma^2 = (9 + 1 + 1 + 9)/4 = 5\) and \(\sigma = 2.236\). Output: \((-1.342, -0.447, 0.447, 1.342)\).
RMSNorm (next section): \(\text{RMS} = \sqrt{(4 + 16 + 36 + 64)/4} = \sqrt{30} = 5.477\). Output: \((0.365, 0.730, 1.095, 1.461)\).
Now add 10 to every entry: \((12, 14, 16, 18)\). LayerNorm gives exactly the same output as before. RMSNorm gives \((0.791, 0.923, 1.055, 1.187)\): different, because RMSNorm does not remove the mean. Multiply the original by 100 instead: both outputs stay the same. The PyTorch code later in this session prints these numbers.
RMSNorm: drop the mean
Zhang and Sennrich (2019) asked: which part of LayerNorm actually helps? Their answer was that the rescaling (scale invariance) matters and the re-centering (removing the mean) is not needed. [Certain] RMSNorm keeps only the division by the root mean square:
- Read the symbols
- \(\text{RMS}(\mathbf{x})\)
- root mean square: the typical size of one coordinate, without subtracting the mean
- \(\gamma \in \R^d\)
- learned gain, one per feature; there is no bias \(\beta\)
What each one removes, in one sentence each:
- LayerNorm removes the mean (one direction, the all-ones direction) and the overall size of the vector.
- RMSNorm removes only the overall size. The output always has RMS 1, so length \(\sqrt{d}\), but its mean can be anything.
Note that if the mean is already 0, the two are identical, since then \(\sigma = \text{RMS}\). Why is RMSNorm cheaper? LayerNorm needs two reductions over the vector (first the mean, then the variance of the centered values; or one fused pass that tracks both sums), a subtraction, and stores a bias. RMSNorm needs one reduction (sum of squares) and one multiply. Normalization does little arithmetic but reads and writes the whole activation, so on a GPU it is limited by memory traffic, and the saving comes from fewer passes and fewer parameters to read. The paper reports that RMSNorm “reduces the running time by 7%~64% on different models” compared with LayerNorm, with similar quality. [Certain] Nearly all recent open LLMs use RMSNorm: Llama, Mistral, Qwen, Gemma, DeepSeek. [Likely] GPT-2, BERT and most vision transformers use LayerNorm. [Certain] Figure 2 lets you see the difference.
How to read Figure 2
Move the scale: the gray bars grow, but both outputs stay fixed. Both norms are scale invariant. Move the shift: LayerNorm still does not change, but the RMSNorm bars all move toward the same value, because a large common offset dominates the root mean square. In a trained network this rarely matters: the next layer can learn to ignore a common offset. Now add an outlier: one huge feature takes over the RMS (and \(\sigma\)), and all the other features are squashed toward zero in both norms. This is the “massive activation” effect seen in real LLMs, and one reason why quantizing activations is hard (session 108).
Small details that matter in real code
- Compute the statistics in FP32. In BF16, the sum of 4,096 squares loses precision. The Hugging Face Llama code casts to FP32, computes the RMS, normalizes, and casts back. [Certain]
- \(\varepsilon\) is a real hyper-parameter. Llama 3.1 8B uses \(10^{-5}\); Qwen3 8B and Gemma 2 9B use \(10^{-6}\) (from their
config.jsonfiles). [Certain] Using the wrong value when you load a checkpoint changes the outputs slightly. - Gemma stores \(\gamma - 1\). Its RMSNorm multiplies by \((1 + w)\), with \(w\) initialized to 0. A converter that treats \(w\) as \(\gamma\) breaks the model. [Likely]
- No weight decay on \(\gamma\) and \(\beta\). Decaying a gain toward 0 shrinks the layer’s input, as discussed in session 4.
- Norms are tiny in parameters. Llama 3 8B has two RMSNorms per block plus a final one: \(32 \times 2 \times 4{,}096 + 4{,}096 = 266{,}240\) parameters, about 0.003% of the model.
Where to put the norm: Post-LN vs Pre-LN
There are two classic ways to combine a residual connection with a normalization. The original transformer (Vaswani et al., 2017) and BERT put the norm after the addition. This is Post-LN:
GPT-2 moved the norm to the input of each sub-layer and added one final norm after the last block. [Certain] This is Pre-LN, and almost every LLM since then uses it:
The difference looks small, but it changes the residual stream completely. In Post-LN, there is no clean stream: every block normalizes the sum, so the identity path passes through \(L\) normalizations. In Pre-LN, the stream \(\mathbf{x}_l\) is a pure sum (22.2) that is never normalized; only copies of it are normalized before each layer reads them. The gradient highway of (22.3) is intact. Figure 3 shows the two layouts, and a third one used by some newer models.
Why Post-LN needs warmup
The original transformer was trained with 4,000 warmup steps, and Post-LN models are known to diverge or train badly without warmup. [Certain] Xiong et al. (2020) analyzed this at initialization. They showed that in Post-LN, the expected gradients of the parameters near the output are large, so a normal learning rate makes the top layers take huge steps at the start. Warmup hides this by starting with a tiny learning rate. In Pre-LN, the gradients are well behaved and become smaller with depth, so Pre-LN models can be trained without warmup and with less tuning. [Certain] (In practice, LLMs still use a short warmup for other reasons, such as Adam’s statistics being noisy in the first steps. [Likely])
A second explanation (Liu et al., 2020) is the amplification effect: in Post-LN, each layer depends heavily on its residual branch, so small changes of the weights produce large changes of the output, and training becomes unstable. [Certain] Both views agree that Pre-LN is easier to train. Figure 4 is a small experiment you can run: a deep stack of random linear sub-layers, once in Post-LN and once in Pre-LN form, with the gradient of every layer measured by backpropagation.
How to read Figure 4
Look at the right end of the curves: the layers closest to the output. Post-LN (blue) has much larger gradients there than Pre-LN (orange). Increase \(L\): the Pre-LN top-layer gradient shrinks (about 0.74 at \(L = 6\), 0.30 at \(L = 24\), 0.24 at \(L = 48\) for the first seed), because the final norm divides by the stream’s size, which grows like \(\sqrt{L}\) by (22.4). The Post-LN top layer stays of order 1 (between about 0.8 and 1.5 at these depths), whatever the depth. This is the gap Xiong et al. describe, and why the same learning rate is too large for Post-LN at the start. The toy is not a real transformer, so do not over-read the exact shapes; inside the Pre-LN stack, for example, the lower layers get the larger gradients here. The readout also shows the size of the Pre-LN stream at the top next to formula (22.4).
Variants you will see
- Scaled residual initialization. GPT-2 scales the initial weights of the layers that write into the stream by \(1/\sqrt{N}\), where \(N\) is the number of residual layers. [Certain] Each edit then starts small, and the stream does not grow too fast with depth. Many training codes (nanoGPT, for example) do the same.
- Sandwich norm. Gemma 2 and Gemma 3 normalize both the input and the output of each sub-layer: \(\mathbf{x} + \text{Norm}(F(\text{Norm}(\mathbf{x})))\). [Likely] The extra norm limits how much any single layer can write.
- Norm on the output only. OLMo 2 moved the norm from the input to the output of the sub-layer, \(\mathbf{x} + \text{Norm}(F(\mathbf{x}))\), as part of its stability changes. [Likely]
- DeepNorm. Wang et al. (2022) kept Post-LN but scaled up the identity path, \(\text{LN}(\alpha \mathbf{x} + F(\mathbf{x}))\) with \(\alpha \gt 1\), plus a smaller initialization, and trained transformers with 1,000 layers. [Certain]
Growing attention logits and QK-norm
Pre-LN fixes the stream, but one more problem remains inside attention. The attention score between a query and a key is \(\mathbf{q}^\top \mathbf{k} / \sqrt{d_h}\) (session 17), with \(\mathbf{q} = W_Q \mathbf{h}\) and \(\mathbf{k} = W_K \mathbf{h}\) computed from the normalized input \(\mathbf{h}\). The norm controls \(\mathbf{h}\), but not \(W_Q\) and \(W_K\). During training these weights can grow. If both grow by a factor \(g\), the logits grow by \(g^2\):
A 10% growth of both matrices makes the logits 21% larger; doubling them makes the logits 4 times larger. Large logits make the softmax almost one-hot: attention puts all its weight on one key, its entropy collapses to zero, and the gradient through the softmax becomes tiny or unstable. Dehghani et al. (2023) saw this when scaling vision transformers toward 22B parameters: without a fix, training diverged, with “unstable attention values”. [Certain] Wortsman et al. (2023) reproduced “the growth of logits in attention layers” in small models trained with high learning rates. [Certain]
The fix, QK-norm, is simple: normalize the query and the key of each head (with LayerNorm or RMSNorm over the head dimension \(d_h\)) just before the dot product:
Now the size of the logits no longer depends on the size of \(W_Q\) and \(W_K\). With gains \(\gamma = 1\), \(\lVert \hat{\mathbf{q}} \rVert = \lVert \hat{\mathbf{k}}_j \rVert = \sqrt{d_h}\), so by the Cauchy–Schwarz inequality \(|z_j| \le \sqrt{d_h}\). The model can still make attention sharp, but only by growing the small gain vectors, which is slower and more controlled. QK-norm was used in ViT-22B [Certain], and is used in Qwen3 (which also removed the QKV bias of Qwen2) [Certain], OLMo 2 [Likely] and Gemma 3 [Likely]. In Qwen3 the norm is over the head dimension, with one gain vector of size 128 shared by all heads. [Certain] Figure 5 shows the effect.
How to read Figure 5
Start at \(g = 1\): the largest weight is 0.57 and the entropy is 1.49 nats (the maximum for 8 keys is \(\ln 8 = 2.08\)). Move \(g\) to 2: the largest logit goes from 1.9 to 7.6, the largest weight to 0.998, and the entropy to almost 0. Attention has collapsed onto one key. Now turn on QK-norm: moving \(g\) does nothing at all; the largest logit stays at 2.26. The gain \(\gamma\) is the only way left to sharpen attention, and the readout shows the bound \(\gamma^2 \sqrt{d_h}\) on every logit.
Other fixes for runaway logits
- Logit soft-capping (Gemma 2): squash the logits smoothly into \((-c, c)\) with
- Gemma 2 9B uses \(c = 50\) for attention logits and \(c = 30\) for the final output logits (
attn_logit_softcappingandfinal_logit_softcappingin its config). [Certain] A logit of 10 becomes 9.87 (almost unchanged), a logit of 100 becomes 48.2. A drawback: the tanh does not fit the fused FlashAttention kernels as easily as plain attention, which is one reason Gemma 3 switched to QK-norm. [Likely] - z-loss on the output logits: add a small penalty \(\alpha (\log Z)^2\), where \(Z\) is the softmax normalizer, to keep the final logits from drifting. PaLM used \(\alpha = 10^{-4}\). [Likely] Wortsman et al. show it fixes output logit divergence in their small proxies. [Likely]
- Lower learning rate, warmup, weight decay and gradient clipping also slow down weight growth; they are covered in session 36.
Putting it together: algorithms and code
Here is one Pre-LN transformer block, as used (with small differences) in Llama, Qwen and Mistral, and the normalization it uses.
- input: vector \(\mathbf{x} \in \R^d\) (any precision), gain \(\gamma \in \R^d\), \(\varepsilon\)
- \(\mathbf{x}_{32} \leftarrow\) cast \(\mathbf{x}\) to FP32 // statistics need full precision
- \(m \leftarrow \frac{1}{d} \sum_i x_{32,i}^2\) // one reduction; LayerNorm also needs the mean
- \(\mathbf{y} \leftarrow \mathbf{x}_{32} \cdot \frac{1}{\sqrt{m + \varepsilon}}\) // formula (22.6)
- return cast \(\mathbf{y}\) back to the input precision, times \(\gamma\)
- input: token IDs \(t_1, \dots, t_T\)
- \(\mathbf{x} \leftarrow\) embedding of each token // the residual stream starts here
- for each block \(b = 1, \dots, N\):
- \(\mathbf{h} \leftarrow \text{RMSNorm}_1(\mathbf{x})\) // read a normalized copy
- \(\mathbf{q}, \mathbf{k}, \mathbf{v} \leftarrow \mathbf{h} W_Q, \mathbf{h} W_K, \mathbf{h} W_V\), split into heads
- \(\mathbf{q}, \mathbf{k} \leftarrow \text{RMSNorm}_q(\mathbf{q}), \text{RMSNorm}_k(\mathbf{k})\) per head // QK-norm, formula (22.10)
- \(\mathbf{x} \leftarrow \mathbf{x} + \text{Attention}(\mathbf{q}, \mathbf{k}, \mathbf{v})\, W_O\) // write: add, formula (22.1)
- \(\mathbf{x} \leftarrow \mathbf{x} + \text{FFN}(\text{RMSNorm}_2(\mathbf{x}))\) // second sub-layer, same pattern
- \(\text{logits} \leftarrow \text{RMSNorm}_{\text{final}}(\mathbf{x})\, W_{\text{out}}\) // Pre-LN needs this final norm, formula (22.8)
The PyTorch version. It checks our RMSNorm and LayerNorm against the built-in ones, reproduces the worked example, and runs 8 Pre-LN blocks with QK-norm to show the stream growing while the final norm brings it back to RMS 1. It runs on a CPU in about a second (tested with PyTorch 2.8).
import math, torch import torch.nn as nn import torch.nn.functional as F torch.manual_seed(0) class RMSNorm(nn.Module): def __init__(self, d, eps=1e-5): super().__init__() self.eps = eps self.weight = nn.Parameter(torch.ones(d)) # gain only, no bias def forward(self, x): xf = x.float() # statistics in FP32 rms_inv = torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + self.eps) return (xf * rms_inv).to(x.dtype) * self.weight def layer_norm(x, w, b, eps=1e-5): mu = x.mean(-1, keepdim=True) var = (x - mu).pow(2).mean(-1, keepdim=True) # biased variance return (x - mu) / torch.sqrt(var + eps) * w + b # 1) check against PyTorch x = torch.tensor([[2., 4., 6., 8.]]) print(layer_norm(x, torch.ones(4), torch.zeros(4))) # [-1.3416, -0.4472, 0.4472, 1.3416] print(RMSNorm(4)(x)) # [0.3651, 0.7303, 1.0954, 1.4606] print(torch.allclose(RMSNorm(4)(x), nn.RMSNorm(4, eps=1e-5)(x), atol=1e-6)) # True print(torch.allclose(layer_norm(x, torch.ones(4), torch.zeros(4)), nn.LayerNorm(4)(x), atol=1e-6)) # True print(RMSNorm(4)(x + 10)) # [0.7913, 0.9231, 1.0550, 1.1869]: a shift changes RMSNorm # 2) a Pre-LN block with QK-norm class Block(nn.Module): def __init__(self, d, n_head, qk_norm=True): super().__init__() self.h, self.dh = n_head, d // n_head self.norm1, self.norm2 = RMSNorm(d), RMSNorm(d) self.qkv = nn.Linear(d, 3 * d, bias=False) self.out = nn.Linear(d, d, bias=False) self.mlp = nn.Sequential(nn.Linear(d, 4 * d, bias=False), nn.GELU(), nn.Linear(4 * d, d, bias=False)) self.qn = RMSNorm(self.dh) if qk_norm else nn.Identity() self.kn = RMSNorm(self.dh) if qk_norm else nn.Identity() def attn(self, x): B, T, D = x.shape q, k, v = self.qkv(x).view(B, T, 3, self.h, self.dh).unbind(2) q, k = self.qn(q), self.kn(k) # normalize each head's q and k y = F.scaled_dot_product_attention(q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), is_causal=True) return self.out(y.transpose(1, 2).reshape(B, T, D)) def forward(self, x): x = x + self.attn(self.norm1(x)) # read with a norm, write by adding x = x + self.mlp(self.norm2(x)) return x d, L = 64, 8 blocks = nn.ModuleList([Block(d, 4) for _ in range(L)]) x = torch.randn(2, 10, d) print("start", round(x.norm(dim=-1).mean().item(), 2)) # 8.13 (about sqrt(64)) for i, blk in enumerate(blocks): x = blk(x) print(i, round(x.norm(dim=-1).mean().item(), 2)) # 8.46, 8.8, ..., 10.42: the stream grows final_norm = RMSNorm(d) print(final_norm(x).pow(2).mean(-1).sqrt().mean().item()) # 0.99999...: RMS 1 after the final norm
The stream grows slowly here (8.1 to 10.4) because PyTorch’s default nn.Linear initialization makes each edit small. With the unit-variance edits of formula (22.4), it would grow much faster. Try qk_norm=False, multiply blk.qkv.weight by 4, and print the attention entropy: you will see the collapse of Figure 5.
Where this is used
| Model or setting | Norm and placement | Note |
|---|---|---|
| Original transformer (2017), BERT | LayerNorm, Post-LN | Needs warmup (4,000 steps in the original) [Certain] |
| GPT-2, GPT-3 | LayerNorm, Pre-LN with final norm, \(1/\sqrt{N}\) residual init | The layout most LLMs still follow [Certain for GPT-2] |
| Llama 2 and 3, Mistral, DeepSeek | RMSNorm, Pre-LN | \(\varepsilon = 10^{-5}\) in Llama 3.1 8B [Certain] |
| Qwen3 | RMSNorm, Pre-LN, plus QK-norm per head | QKV bias of Qwen2 removed [Certain] |
| Gemma 2, Gemma 3 | RMSNorm, sandwich (pre and post), soft-capping in Gemma 2, QK-norm in Gemma 3 | Caps 50 and 30 in Gemma 2 [Certain]; rest [Likely] |
| ViT, CLIP, SigLIP image encoders | LayerNorm, Pre-LN; ViT-22B adds QK-norm | The vision side of most VLMs (session 164) |
| Video transformers (ViViT, TimeSformer, VideoMAE) | LayerNorm, Pre-LN, as in ViT [Likely] | Backbones for action recognition, video retrieval and key-moment detection (session 202) |
| VLM connectors (projector from vision encoder to LLM) | Often a LayerNorm before or inside the MLP projector [Likely] | Brings image features to the scale of text embeddings in the LLM’s residual stream |
| Interpretability and model editing | Reads and edits the residual stream directly | Logit lens, probes, steering vectors (session 157) |
| Quantization of activations | Massive activations in the stream make outliers | A few features can be 100,000 times larger than typical ones (Sun et al., 2024) [Certain] |
Common mistakes
- Forgetting the final norm in a Pre-LN model. The stream at the top has a size that grows with depth (22.4). Feeding it straight into the output layer gives badly scaled logits.
- Normalizing over the wrong axis. LayerNorm and RMSNorm work over the feature dimension of each token (the last axis), never over the sequence or the batch.
- Computing the RMS in BF16 or FP16. The sum of thousands of squares loses precision or overflows in FP16. Upcast to FP32 for the statistics, as in Algorithm 1.
- Mixing up the \(\varepsilon\) or the gain convention when you port a checkpoint (\(10^{-5}\) vs \(10^{-6}\); Gemma’s \(1 + w\)). The model still runs, so the bug is silent.
- Thinking RMSNorm is “LayerNorm without centering, so it is worse”. In practice the quality is similar, and the next layer can absorb a common offset. It is chosen for speed and simplicity.
- Putting dropout or a norm on the identity path (the skip connection) by accident. The skip should be a plain addition; all extra operations belong inside the residual branch.
- Training a Post-LN model with a Pre-LN recipe. Without warmup and with a high learning rate, Post-LN often diverges early.
- Applying weight decay to norm gains, which pushes them toward zero.
Interview questions
Why do transformers use residual connections?basic
What is the residual stream?basic
What exactly does LayerNorm compute, and what does RMSNorm drop?core
Why is RMSNorm cheaper, and is it worse?core
Why not BatchNorm in a language model?core
Explain Post-LN vs Pre-LN. Which do modern LLMs use and why?core
Why does a Post-LN transformer need learning-rate warmup?advanced
Does the residual stream grow with depth? Is that a problem?advanced
What is QK-norm and what problem does it solve?advanced
Compare QK-norm, soft-capping and z-loss.advanced
Where would you look if a pretraining run suddenly spikes to NaN?core
Exercises
- Compute LayerNorm and RMSNorm of \(\mathbf{x} = (3, 4)\) by hand (\(\gamma = 1\), \(\beta = 0\), ignore \(\varepsilon\)).
- Show that LayerNorm of \(a\mathbf{x} + b\mathbf{1}\) equals LayerNorm of \(\mathbf{x}\) for any \(a \gt 0\) and any \(b\). Which part of the argument fails for RMSNorm?
- Count the normalization parameters of Qwen3 8B: 36 layers, \(d = 4{,}096\), two RMSNorms per block, QK-norm with one gain vector of size 128 for queries and one for keys per layer, and a final norm.
- Using (22.4), how large is the Pre-LN stream after 48 sub-layers with \(d = 64\)? Check it against the readout of Figure 4.
- Gemma 2 caps attention logits at \(c = 50\). What does a logit of 100 become? And a logit of 10? For a final-logit cap of 30, what does 40 become?
- If \(W_Q\) and \(W_K\) both grow by 10% during training, by how much do the attention logits grow without QK-norm? With QK-norm (gains fixed)?
- Write the gradient of \(\mathcal{L}\) with respect to \(\mathbf{x}_0\) for a 2-layer residual stack, and point to the term that does not involve any layer Jacobian.
Answerscheck
Key takeaways
- A residual connection adds each layer’s output to its input (22.1). The stack becomes a sum of edits (22.2), and the identity path carries the gradient to every layer (22.3).
- The residual stream is one shared vector per token. Layers read it through a norm and linear maps and write by adding. Attention moves information between tokens; the FFN works per token.
- The stream grows with depth (22.4), so each layer reads a normalized copy. LayerNorm (22.5) removes the mean and the scale; RMSNorm (22.6) removes only the scale, is cheaper, and is used by almost all new LLMs.
- Post-LN (22.7) normalizes the stream itself and needs warmup; Pre-LN (22.8) normalizes copies, keeps the stream clean, and needs a final norm. Modern LLMs use Pre-LN, sometimes with extra output norms.
- Attention logits grow like \(g^2\) when \(W_Q, W_K\) grow (22.9). QK-norm (22.10) bounds them; soft-capping (22.11) and z-loss are alternatives.
Key formulas
References for this session
- Zhang & Sennrich (2019). Root Mean Square Layer Normalization
- Xiong et al. (2020). On Layer Normalization in the Transformer Architecture
- Elhage et al. (2021). A Mathematical Framework for Transformer Circuits
- Zhang & Sennrich (2019). Root Mean Square Layer Normalization
- Xiong et al. (2020). On Layer Normalization in the Transformer Architecture
- Elhage et al. (2021). A Mathematical Framework for Transformer Circuits
- He et al. (2015). Deep Residual Learning for Image Recognition
- Vaswani et al. (2017). Attention Is All You Need
- Ba, Kiros, Hinton (2016). Layer Normalization
- Radford et al. (2019). Language Models are Unsupervised Multitask Learners
- Liu, Liu, Gao, Chen, Han (2020). Understanding the Difficulty of Training Transformers
- Wang, Ma, Dong, Huang, Zhang, Wei (2022). DeepNet: Scaling Transformers to 1,000 Layers
- Dehghani et al. (2023). Scaling Vision Transformers to 22 Billion Parameters
- Henry, Dachapally, Pawar, Chen (2020). Query-Key Normalization for Transformers
- Wortsman et al. (2023). Small-scale proxies for large-scale Transformer training instabilities
- Sun, Chen, Kolter, Liu (2024). Massive Activations in Large Language Models
- Qwen Team (2025). Qwen3 Technical Report
- Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size
- Gemma Team (2025). Gemma 3 Technical Report