LLM & VLM courseBlog
30Track 2: Attention and transformers · session 15 of 15

write a GPT from scratch and train it

In this lab you write a small GPT in about 100 lines of PyTorch: embeddings, causal multi-head attention, the feed-forward network, RMSNorm, residual connections and a tied output layer. You count its 812,288 parameters by formula, train it on Tiny Shakespeare on a laptop CPU, watch train and validation loss, and sample text with different temperatures. Then you break it on purpose (no positions, no square-root scaling, Post-LN) and measure what really fails.

Time: 4 to 5 hours (mostly hands-on; each training run takes about 15 minutes on a CPU)Level: Beginner · hands-on labBefore this: Sessions 16 to 25 (attention, masks, multi-head attention, FFN, residual stream and norms, counting), Python and basic PyTorch

What you will learn

  • How embeddings, causal multi-head attention, the FFN, RMSNorm, residual adds and a tied LM head fit together, with the tensor shape at every step
  • How to count the parameters of a GPT exactly, and when the 12·L·d² rule is accurate and when it is not
  • Three checks before training: loss near ln |V|, the causal test, overfitting one batch
  • A full training loop with AdamW, warmup and cosine decay, gradient clipping, and honest train and validation curves
  • How temperature changes the next-token distribution and the generated text
  • What happens when you remove positions, remove the square-root scaling, or switch to Post-LN, and why small runs can hide some failures

Why this matters

In the sessions before this one we took the transformer apart: tokens, embeddings, attention, masks, heads, the feed-forward network, the residual stream, normalization, and the counting rules. In this lab you put all the pieces back together. You write a small GPT (a decoder-only transformer, the family of GPT-2, Llama and Qwen) in about 100 lines of PyTorch, train it on the complete works of Shakespeare on a laptop CPU, and make it write new text.

Then you break it on purpose. You remove the position embeddings, remove the \(\sqrt{d}\) in attention, and move the normalization to the “wrong” place, and you measure what happens. Some of these breaks hurt a lot. Some hurt much less than the textbooks suggest, at this small size. Knowing which is which, and why, is exactly the kind of understanding that interviews test and that real debugging needs.

  • Interviews. “Implement causal self-attention” and “write a transformer block” are common coding questions for LLM roles. “What happens if you remove the scaling factor?” and “Why Pre-LN?” are common follow-ups.
  • Real systems. Karpathy’s nanoGPT (the reference for this lab), Raschka’s book chapter 4, and the model files in Hugging Face transformers all have the same skeleton you will write here. Once you have written it yourself, you can read the Llama or Qwen model code in one sitting.
  • Debugging. Large training runs fail in boring ways: a mask off by one, a missing scale, a wrong init. The checks in this lab (loss at step 0, the causal test, overfitting one batch) catch most of them in seconds.

Setup and plan

You need Python 3.9 or newer and PyTorch (version 2.x; we used 2.8.0 on CPU). No GPU is needed: every number on this page comes from runs on a laptop CPU, each about 15 minutes long.

pip install torch
curl -O https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

The data is Tiny Shakespeare: 1,115,394 characters of Shakespeare plays in one text file. We use characters as tokens, so the vocabulary is the set of distinct characters: 65 (letters, digits 3, punctuation, space and newline). A character-level tokenizer keeps this lab about the model; with a BPE tokenizer from session 10, only the vocabulary size changes. The first 90% of the text (1,003,854 characters) is the training set and the last 10% (111,540 characters) is the validation set. [Certain: computed]

SettingValueMeaning
\(|V|\)65vocabulary size (distinct characters)
\(T\)128context length (block size): how many characters the model sees
\(d\)128width of the residual stream (embedding size)
\(L\)4number of transformer blocks
\(h\), \(d_h\)4, 32attention heads, and the size of one head (\(d_h = d/h\))
\(B\)32batch size (windows per step), so 4,096 tokens per step
steps, optimizer5,000, AdamWpeak learning rate \(10^{-3}\), 200 warmup steps, cosine decay to \(10^{-4}\), weight decay 0.1, gradient clipping at 1.0

The plan has seven steps: (1) data and batches, (2) the model, (3) count the parameters, (4) check the model before training, (5) train and plot the losses, (6) sample with temperature, (7) break it on purpose.

Step 1: data and batches

The tokenizer is two dictionaries: stoi (string to integer) and itos (integer to string). Training examples are windows of \(T + 1\) characters cut at random places: the first \(T\) are the input, and the same window shifted by one is the target. One window therefore holds \(T = 128\) training examples at once, one per position, exactly as in session 6.

import torch
text = open("input.txt", encoding="utf-8").read()
chars = sorted(set(text)); V = len(chars)                 # 65
stoi = {c: i for i, c in enumerate(chars)}
encode = lambda s: [stoi[c] for c in s]
decode = lambda ids: "".join(chars[i] for i in ids)
data = torch.tensor(encode(text), dtype=torch.long)       # shape (1115394,)
n = int(0.9 * len(data)); train, val = data[:n], data[n:]

def get_batch(split, g, B=32, T=128):
    d = train if split == "train" else val
    ix = torch.randint(len(d) - T - 1, (B,), generator=g)
    x = torch.stack([d[i:i + T] for i in ix])            # (B, T) inputs
    y = torch.stack([d[i + 1:i + T + 1] for i in ix])    # (B, T) targets, shifted by one
    return x, y

Why split by position (first 90% / last 10%) and not by random windows? Random windows from the same text would overlap: the model could see the validation characters during training, and the validation loss would look too good. A contiguous split avoids this leak.

Step 2: the model, piece by piece

A GPT is a short pipeline. Ids become vectors; \(L\) identical blocks update the vectors; a final layer turns each vector into 65 scores. Figure 1 follows one batch through it and shows the tensor shape at every step.

shape of the tensor at this step (\(B = 32,\ T = 128,\ d = 128,\ h = 4,\ |V| = 65\))
Figure 1. The forward pass of the lab model, step by step, with the shape of the tensor and the parameters used at each step. The dashed box is one block; the dashed arrows are the residual (skip) paths.
How to read Figure 1

Press Next and read the shape line. Notice two things. First, the shape (32, 128, 128) goes into a block and the same shape comes out: that is why blocks can be stacked. Second, only one tensor has a \(T \times T\) part: the attention scores, (32, 4, 128, 128). It holds 2,097,152 numbers per layer, more than the 524,288 numbers of the residual stream. At long context this tensor is what costs memory, and it is what FlashAttention avoids storing.

Embeddings

Each id selects a row of the token table \(E\), and each position selects a row of a learned position table \(P\), as in GPT-2 (session 11, session 13):

\[ \mathbf{h}^{(0)}_t = E[x_t] + P[t], \qquad E \in \R^{|V| \times d}, \quad P \in \R^{T \times d} \tag{30.1} \]
Read the symbols
\(x_t\)
the token id at position \(t\) (here a character, from 0 to 64)
\(E[x_t]\)
row \(x_t\) of the token table: the vector for that token
\(P[t]\)
row \(t\) of the position table: a learned vector that means “I am at position \(t\)”
\(\mathbf{h}^{(0)}_t\)
the vector of position \(t\) at the start of the residual stream, size \(d\)

Modern models such as Llama use RoPE instead of \(P\) (rotate \(\mathbf{q}\) and \(\mathbf{k}\) inside attention, session 14). We keep the learned table because it is the simplest thing that works, and because removing it later is one line.

The block

Each block does two things in order: let positions talk to each other (attention), then let each position think alone (the FFN). Both are written into the residual stream with an addition. We use Pre-LN: normalize the input of each sub-layer, not the sum (session 22):

\[ \mathbf{a} = \mathbf{h} + \operatorname{Attn}\big(\operatorname{RMSNorm}_1(\mathbf{h})\big), \qquad \mathbf{h}' = \mathbf{a} + \operatorname{FFN}\big(\operatorname{RMSNorm}_2(\mathbf{a})\big) \tag{30.2} \]
\[ \operatorname{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2 + \varepsilon}} \odot \gamma, \qquad \operatorname{FFN}(\mathbf{x}) = W_2\, \operatorname{GELU}(W_1 \mathbf{x}) \tag{30.3} \]
Read the symbols
\(\mathbf{h}, \mathbf{h}'\)
the residual stream before and after the block, shape (B, T, d)
\(\varepsilon\)
a small number (\(10^{-5}\)) so we never divide by zero
\(\gamma\)
a learned gain vector of size \(d\), initialized to ones; \(\odot\) is element-wise product
\(W_1, W_2\)
the FFN matrices, \(d \to 4d\) and \(4d \to d\) (session 21)

Llama and Qwen use SwiGLU instead of GELU in the FFN (three matrices instead of two). We keep the GPT-2 style two-matrix FFN; exercise 5 asks you to swap it.

Causal multi-head attention

One linear layer makes queries, keys and values for all heads at once; then we split the last axis into \(h\) heads. For one head:

\[ \operatorname{Attn}(Q, K, V) = \softmax\!\Big(\frac{Q K^\top}{\sqrt{d_h}} + M\Big) V, \qquad M_{ts} = \begin{cases} 0 & s \le t \\ -\infty & s \gt t \end{cases} \tag{30.4} \]
Read the symbols
\(Q, K, V\)
queries, keys and values of one head, each \((T, d_h)\) for one sequence
\(d_h\)
the head size, \(d/h = 32\); dividing by \(\sqrt{d_h}\) keeps the scores at a stable size (Step 7)
\(M\)
the causal mask: position \(t\) may look at positions \(s \le t\) only (session 18)

The \(h\) head outputs are joined back to size \(d\) and multiplied by \(W_O\) (session 19). In real code you would call F.scaled_dot_product_attention(q, k, v, is_causal=True), which runs a fused kernel. We write the four lines by hand so that we can break them in Step 7.

The head and the loss

After the last block, one more RMSNorm, then the LM head turns each vector into \(|V|\) logits. We tie the head to the token table: the head matrix is \(E\) itself, so it adds no parameters (GPT-2 does the same [Certain]). The loss is the average cross-entropy over all \(B \cdot T\) positions:

\[ \mathbf{z}_t = E\, \operatorname{RMSNorm}_f\big(\mathbf{h}^{(L)}_t\big) \in \R^{|V|}, \qquad \mathcal{L} = -\frac{1}{BT} \sum_{b=1}^{B} \sum_{t=1}^{T} \log \softmax(\mathbf{z}_{b,t})_{y_{b,t}} \tag{30.5} \]
Read the symbols
\(\mathbf{z}_t\)
the logits (unnormalized scores) for the next token at position \(t\)
\(y_{b,t}\)
the true next token: the target at position \(t\) of window \(b\)
\(\mathcal{L}\)
the training loss in nats; \(e^{\mathcal{L}}\) is the perplexity
Algorithm 1: GPT forward pass
  1. input: ids \(x\) of shape (B, T); optional targets \(y\)
  2. \(h \leftarrow E[x] + P[0{:}T]\) // \((B, T, d)\), formula (30.1)
  3. for \(l = 1\) to \(L\) do
  4. \(u \leftarrow \text{RMSNorm}_1(h)\);  \(q, k, v \leftarrow \operatorname{split}(u W_{QKV})\) into \(h\) heads // \((B, h, T, d_h)\)
  5. \(s \leftarrow q k^\top / \sqrt{d_h}\);  \(s[\text{above diagonal}] \leftarrow -\infty\) // \((B, h, T, T)\)
  6. \(h \leftarrow h + \text{concat}(\softmax(s)\, v)\, W_O\) // residual add
  7. \(h \leftarrow h + W_2\, \text{GELU}(W_1\, \text{RMSNorm}_2(h))\) // formula (30.2)
  8. \(z \leftarrow \text{RMSNorm}_f(h)\, E^\top\) // \((B, T, |V|)\), tied head
  9. if \(y\) given then \(\text{loss} \leftarrow \operatorname{CE}\big(z \text{ as } (BT, |V|),\ y \text{ as } (BT)\big)\) // (30.5)
  10. return \(z\), loss

Step 3: count the parameters

In session 25 we derived that a block has about \(12d^2\) parameters: \(4d^2\) in attention (\(W_Q, W_K, W_V, W_O\)) and \(8d^2\) in the FFN. Our model has no biases, so we can write the exact count:

\[ N = \underbrace{|V|\,d}_{E} + \underbrace{T\,d}_{P} + L\,\big(\underbrace{4d^2}_{\text{attention}} + \underbrace{8d^2}_{\text{FFN}} + \underbrace{2d}_{\text{two norms}}\big) + \underbrace{d}_{\text{final norm}} \tag{30.6} \]
\[ N \approx 12\, L\, d^2 \quad \text{(blocks only, large } d\text{)} \tag{30.7} \]
Worked example: our model, \(|V| = 65,\ T = 128,\ d = 128,\ L = 4\)

Token table: \(65 \times 128 = 8{,}320\). Position table: \(128 \times 128 = 16{,}384\). One block: \(12 \times 128^2 + 2 \times 128 = 196{,}608 + 256 = 196{,}864\); four blocks: \(787{,}456\). Final norm: 128. Head: 0 (tied).

Total: \(8{,}320 + 16{,}384 + 787{,}456 + 128 = \mathbf{812{,}288}\). PyTorch agrees: sum(p.numel() for p in model.parameters()) prints 812288. [Certain: computed]

The rule \(12Ld^2 = 786{,}432\) gives 96.8% of it. The missing 3.2% is mostly the two embedding tables. For a tiny vocabulary the rule works well; with a 50,257-token vocabulary at \(d = 768,\ L = 12\) (the shape of GPT-2 small), the same formula gives 124,337,664 parameters, but \(12Ld^2\) gives only 84,934,656 (68%): the token table alone is 38.6 million. (The real GPT-2 small has 124,439,808 parameters, because it also has biases and LayerNorm shifts: \(12d^2 + 13d\) per block. [Certain: computed from its config.json shape])

Figure 2. The exact parameter count of formula (30.6) split into its parts, for the architecture of this lab (no biases, RMSNorm, learned positions, GELU FFN). The line under the bar shows how much of the total the block-only rule \(12Ld^2\) covers.
How to read Figure 2

Start from “This lab”: blocks are almost everything (812,288 in total). Click “GPT-2 small shape”: the token table jumps to 31% of the model, and \(12Ld^2\) covers only 68%. Untie the head and the two tables become 47.9% of the model. Now drag \(d\) or \(L\) up: the block share grows as \(d^2 L\) while the tables grow only as \(d\), so for large models the rule \(12Ld^2\) becomes accurate again. Small models with large vocabularies are the case where you must count the tables.

Step 4: check the model before training

Three checks take seconds and catch most bugs. Do them every time you write a new model.

Check 1: the loss at step 0. With small random weights, the logits are almost equal, so the softmax is almost uniform over the vocabulary, and the loss should be

\[ \mathcal{L}_0 \approx -\log \frac{1}{|V|} = \ln |V| = \ln 65 = 4.174 \ \text{nats} \tag{30.8} \]

Our model gives 4.185 (train) and 4.182 (validation) at step 0. [Certain: measured] If you see 5, 10 or 40 instead, the logits are too large at init: a wrong init, a missing final norm, or a missing scale. A model that starts confident and wrong wastes the first part of training on undoing it.

Check 2: the causal test. Change the input at position 100 and run the model again. The logits at positions 0 to 99 must not change at all; positions 100 and later may change. In our model the largest change before position 100 is exactly 0.0, and after it 15.3 (in logit units). [Certain: measured] If the first number is not (almost) zero, the model can see the future, and its training loss will be wonderful and useless.

Check 3: overfit one batch. Train for 200 steps on one fixed batch. The loss should fall toward zero. Ours (AdamW, learning rate \(10^{-3}\), no schedule) went \(4.188 \to 1.630\) at step 100 \(\to 0.064\) at step 200. [Certain: measured] If it cannot memorize 4,096 characters, something in the backward path is broken (a detached tensor, a parameter not passed to the optimizer, a learning rate of zero).

Step 5: train and watch both losses

The training loop is the one from session 4: sample a batch, forward, backward, clip the gradient, update with AdamW. The learning rate rises linearly for \(S_w = 200\) steps (warmup), then follows a cosine down to one tenth of the peak:

\[ \eta(s) = \begin{cases} \eta_{\max}\,\dfrac{s+1}{S_w} & s \lt S_w \\[6pt] \eta_{\min} + \tfrac{1}{2}(\eta_{\max} - \eta_{\min})\big(1 + \cos(\pi\,p)\big), \quad p = \dfrac{s - S_w}{S - S_w} & s \ge S_w \end{cases} \tag{30.9} \]
Read the symbols
\(s, S\)
the current step and the total number of steps (5,000)
\(\eta_{\max}, \eta_{\min}\)
the peak and final learning rates, \(10^{-3}\) and \(10^{-4}\)
\(p\)
how far we are through the decay phase, from 0 to 1
Algorithm 2: training loop with evaluation
  1. input: model, data, steps \(S\), schedule \(\eta(s)\), clip value \(c = 1\)
  2. params with 2 or more dims get weight decay 0.1; gains \(\gamma\) get none
  3. for \(s = 0\) to \(S - 1\) do
  4. if \(s \bmod 250 = 0\) then estimate train and val loss on 20 fixed batches each, in eval mode, without gradients
  5. set the learning rate to \(\eta(s)\) // formula (30.9)
  6. \(x, y \leftarrow \text{get\_batch}(\text{train})\);  \(\text{loss} \leftarrow \text{model}(x, y)\)
  7. zero the gradients;  loss.backward()
  8. clip the global gradient norm to \(c\);  if loss is NaN then stop and report \(s\)
  9. AdamW step
  10. return model, loss curves

How much compute is this? 5,000 steps of 4,096 tokens is \(D = 20{,}480{,}000\) tokens, about 20 passes over the training text. With the rule \(C \approx 6ND\) from the counting session, training costs \(6 \times 812{,}288 \times 20{,}480{,}000 \approx 1.0 \times 10^{14}\) FLOPs (plus the attention term, which is not small here because \(T\) is as large as \(d\)). [Certain: computed] A modern GPU does this in seconds; a laptop CPU took about 15 minutes.

Figure 3. Real loss curves of the lab model and its broken variants, measured every 250 steps on 20 fixed batches (PyTorch 2.8, CPU, seed 1337). Click runs to add or remove them. Dashed lines are training loss, solid lines are validation loss.
How to read Figure 3

Start with the baseline alone. Train and validation loss fall together for the first few hundred steps; then the dashed training line pulls below the validation line, and the gap keeps growing. Add “No positions”: it is worse at every step. Add “No \(\sqrt{d_h}\)”: it lies almost on top of the baseline. Add “Post-LN”: a little worse, but healthy. Now switch to the 12-layer runs. “12 layers, Post-LN, no warmup” and “12 layers, Post-LN, hot” drop to about 3.35 within 250 steps and stay flat for 2,000 steps, right on the gray line: the loss of a model that only knows how frequent each character is. Their Pre-LN twins train normally. The x-axis stretches to the longest run you have selected. The 12-layer runs use a 2,000-step schedule (their learning rate decays faster), so compare them with each other, not with the 4-layer runs.

What the baseline did, in numbers (all measured): the validation loss fell from 4.182 at step 0 to 2.159 at step 500, 1.874 at step 1,000 and 1.518 at step 5,000. A loss of 1.518 nats is a perplexity of \(e^{1.518} = 4.56\): on average the model is as unsure as a fair choice among about 4.6 characters, against 65 for a uniform guess. The training loss ended lower, at 1.271. The gap between the two grew from 0.164 at step 1,000 to 0.247 at step 5,000: after about 20 passes over one megabyte of text, the model has started to memorize. The validation loss was still falling at the end, so we stopped before it turned up. [Certain: measured]

Step 6: sample with temperature

To generate, we feed a prompt, take the logits at the last position, turn them into probabilities, draw one character, append it, and repeat. The context is cut to the last \(T = 128\) characters, because the position table has only 128 rows. Before the softmax we divide the logits by a temperature \(\tau\):

\[ p_i(\tau) = \frac{\exp(z_i / \tau)}{\sum_{j=1}^{|V|} \exp(z_j / \tau)} \tag{30.10} \]
Read the symbols
\(z_i\)
the logit of character \(i\) at the last position
\(\tau\)
the temperature: \(\tau \lt 1\) makes the distribution sharper, \(\tau \gt 1\) flatter, \(\tau \to 0\) is greedy decoding (always the top choice)

The ratio of two probabilities is \(p_i / p_j = \exp((z_i - z_j)/\tau)\). So halving \(\tau\) squares every ratio: a 3-to-1 preference becomes 9-to-1. Temperature does not change the order of the choices, only how strongly the top ones win.

Real samples after “ROMEO:↵”, same seed:
Figure 4. Top: the 12 most likely next characters (in percent) from the trained lab model, for four prompts. Each character is shown in quotes: " " is the space character, which is a token of its own in this character-level model, and "\n" is a line break. The logits are real; only the temperature is applied in your browser with formula (30.10). Bottom: 240 characters generated by the same model at three temperatures, with the same random seed.
How to read Figure 4

Pick “…winter of our disc”. At \(\tau = 1\) the model puts 0.635 on “o” (as in “discontent”), and the top 4 characters hold 90% of the probability. At \(\tau = 0.5\) the top choice rises to 0.930; at \(\tau = 1.5\) it falls to 0.438. Now pick “…that is the q”: the model is almost certain at any temperature (0.998 for “u” at \(\tau = 1\)), because in English q is nearly always followed by u. “First Citizen:↵” is the start of a new speech, where many first letters are possible: entropy 2.823 nats, as uncertain as a fair choice among 16.8 characters. In the samples, \(\tau = 0.5\) keeps one speaker and common words; \(\tau = 1.0\) is the model’s own distribution, with new speaker names and some invented words; \(\tau = 1.5\) breaks words apart (“EdwEmouG”, a speaker called “ELBR3”).

# sampling (inside the GPT class)
@torch.no_grad()
def generate(self, idx, n, temperature=1.0, gen=None):
    for _ in range(n):
        logits, _ = self(idx[:, -self.T:])                 # crop to the last T tokens
        p = F.softmax(logits[:, -1, :] / temperature, dim=-1)   # formula (30.10)
        idx = torch.cat([idx, torch.multinomial(p, 1, generator=gen)], dim=1)
    return idx

This loop is slow on purpose: it runs the whole context again for every new character, so generating \(n\) tokens costs about \(n \cdot T\) token-forwards. Real servers keep the keys and values of past positions in a KV cache and compute only the new position (session 26).

Step 7: break it on purpose

Now the fun part. Each break is one line of code. For each one, first predict what will happen, then look at Figure 3.

Break 1: remove the positions

Delete \(P[t]\) from formula (30.1). What does the model lose? Attention by itself treats its inputs as a set: if you shuffle the keys and values together, the weighted sum does not change. So in the first layer, the output at position \(t\) depends only on which characters came before, not on their order:

\[ \text{no positions, layer 1:}\quad \mathbf{o}_t\big(x_1, \dots, x_{t-1}, x_t\big) = \mathbf{o}_t\big(x_{\pi(1)}, \dots, x_{\pi(t-1)}, x_t\big) \ \text{for every permutation } \pi \tag{30.11} \]

We checked this with a one-layer model without positions: shuffling the first 12 characters of “First Citizen” changes the last logits by at most \(2.4 \times 10^{-7}\) (float rounding). With positions, the change is 0.011; with two layers and no positions, it is 0.089. [Certain: measured] Why does the second layer break the symmetry? Because of the causal mask: position \(s\) in layer 1 averages over \(s\) inputs, so its output depends on how many characters it saw, and that is position information. A causal transformer can partly recover positions on its own. This is known: decoder-only models without any position encoding (“NoPE”) still learn, and still learn some notion of position (Haviv et al., 2022). [Certain]

Result. The model without positions still trained, but clearly worse: validation loss 1.572 against 1.518 for the baseline after 5,000 steps (0.054 nats worse), and the gap was larger early on (2.023 against 1.874 at step 1,000). [Certain: measured] Character-level text depends heavily on exact order (“dog” and “god” have the same letters), and the first layer is blind to it. The later layers recover part of the order through the causal mask, which is why the damage is moderate rather than total.

Break 2: remove \(\sqrt{d_h}\)

If the entries of \(\mathbf{q}\) and \(\mathbf{k}\) are independent with mean 0 and variance 1, their dot product is a sum of \(d_h\) such products:

\[ \operatorname{Var}(\mathbf{q} \cdot \mathbf{k}) = \sum_{i=1}^{d_h} \operatorname{Var}(q_i k_i) = d_h, \qquad \operatorname{Var}\Big(\frac{\mathbf{q} \cdot \mathbf{k}}{\sqrt{d_h}}\Big) = 1 \tag{30.12} \]

Without the division, the scores have standard deviation \(\sqrt{d_h}\): about 5.7 for our heads, 8 for \(d_h = 64\), 11.3 for \(d_h = 128\). The softmax of numbers that far apart is almost one-hot: one key gets nearly all the weight, and the gradient through the softmax becomes tiny, because \(\partial p_i / \partial z_j = p_i(\delta_{ij} - p_j)\) is near zero when every \(p\) is near 0 or 1 (session 17). Figure 5 shows it with random unit-variance vectors.

Figure 5. Attention weights of one random query over 16 random keys, all with independent standard normal entries (seeded). The readout averages over 200 queries: the largest weight and the entropy of the weights. Uniform attention has entropy \(\ln 16 = 2.773\) nats; one-hot attention has entropy 0.
How to read Figure 5

Keep “Divide by \(\sqrt{d_h}\)” and move the slider. The measured score standard deviation grows like \(\sqrt{d_h}\) (1.88 at \(d_h = 4\), 22.65 at 512), but after the division the weights look the same for every size: the mean largest weight stays between 0.22 and 0.26 and the entropy between 2.33 and 2.41 nats. Now click “No scaling”. At \(d_h = 32\) (our heads) the mean largest weight jumps to 0.750 and the entropy falls to 0.667; at \(d_h = 128\) (the head size of Llama 3 8B) it is 0.885 and 0.286. Attention turns into a hard lookup of a single key.

Result. With our normal recipe, removing \(\sqrt{d_h}\) changed nothing we could measure: validation loss 1.517 against 1.518. [Certain: measured] Why so little? Formula (30.12) assumes that the entries of \(\mathbf{q}\) and \(\mathbf{k}\) have size about 1. At our initialization they are much smaller: the input of attention has RMS 1 after RMSNorm and the weights have standard deviation 0.02, so each entry of \(\mathbf{q}\) or \(\mathbf{k}\) has standard deviation about \(0.02 \times \sqrt{128} = 0.23\), and a raw score has standard deviation about \(0.23^2 \times \sqrt{32} \approx 0.29\). The scores start near zero with or without the division, and the model can simply learn a smaller \(W_Q\) or \(W_K\). The scale starts to matter when training pushes \(\mathbf{q}\) and \(\mathbf{k}\) to larger values. With the hot recipe, the unscaled model ended at 1.657 against 1.570 for the scaled one: 0.087 nats worse. [Certain: measured] In large models, attention logits really do grow during training, which is why some recent models also normalize \(\mathbf{q}\) and \(\mathbf{k}\) (QK-norm, see session 22).

Break 3: Post-LN

Move the norm after the addition, as in the original transformer:

\[ \text{Post-LN:}\quad \mathbf{a} = \operatorname{Norm}_1\big(\mathbf{h} + \operatorname{Attn}(\mathbf{h})\big), \qquad \mathbf{h}' = \operatorname{Norm}_2\big(\mathbf{a} + \operatorname{FFN}(\mathbf{a})\big) \tag{30.13} \]

In Post-LN the stream itself is normalized after every sub-layer, so there is no clean identity path from the output back to the input: the gradient must pass through \(2L\) normalizations. Xiong et al. (2020) showed that at initialization the gradients near the output are large in Post-LN, which is why Post-LN transformers need a learning-rate warmup and can diverge without one. [Certain] We trained Post-LN with 4 layers (with our normal recipe, and with a “hot” recipe: learning rate \(3 \times 10^{-3}\), no warmup, no clipping), and with 12 layers (2,387,200 parameters, 2,000 steps): with the normal 200-step warmup, with no warmup, and with the hot recipe. Every Post-LN run has a Pre-LN twin with the same settings.

Result. With 4 layers, Post-LN was not a failure. With the normal recipe it reached 1.571 against 1.518 for Pre-LN, and with the hot recipe it was the best run of the whole lab: 1.502 against 1.570 for Pre-LN. With 12 layers and warmup, Post-LN trained, but worse than Pre-LN: 1.677 against 1.595 after 2,000 steps. With 12 layers and no warmup, Post-LN failed completely, at both learning rates. Within 250 steps the loss fell to 3.30 (train) and 3.35 (validation), and then it stayed flat until step 2,000. Its Pre-LN twin with the same hot recipe trained normally (1.694). [Certain: measured]

What is a loss of 3.35? It is the loss of a unigram model: one that ignores the context and predicts every character with its overall frequency in the training text. On the validation text that gives 3.347 nats. [Certain: computed] So the deep Post-LN network learned the character frequencies in the first 50 steps and then stopped learning: its gradient norm fell from 1.04 at step 0 to about 0.19 at step 50 and stayed small. Our reading is that the large early steps, passing through 24 normalizations, put the blocks in a state where the output ignores them, and the small gradients never get them out. That is a plausible explanation, not something we proved. [Likely] The lesson matches the literature: the problems of Post-LN grow with depth, and warmup is what keeps it alive.

The full program

This is the whole model, the same file that produced every number on this page. Put the data code from Step 1 and the training loop below it into one file and run it. With the same PyTorch version, seed and thread count, you should get the same curves; across versions or machines, expect small differences in the last digits, because floating-point sums are done in a different order.

import math, torch
import torch.nn as nn
import torch.nn.functional as F

class RMSNorm(nn.Module):
    def __init__(self, d, eps=1e-5):
        super().__init__(); self.eps = eps; self.weight = nn.Parameter(torch.ones(d))
    def forward(self, x):                                     # formula (30.3)
        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.weight

class CausalSelfAttention(nn.Module):
    def __init__(self, d, h, T, scale=True):
        super().__init__(); assert d % h == 0
        self.h, self.dh, self.scale = h, d // h, scale
        self.qkv = nn.Linear(d, 3 * d, bias=False)             # W_Q, W_K, W_V in one matrix
        self.proj = nn.Linear(d, d, bias=False)                # W_O
        self.register_buffer("mask", torch.tril(torch.ones(T, T, dtype=torch.bool)))
    def forward(self, x):
        B, T, d = x.shape
        q, k, v = self.qkv(x).split(d, dim=2)                   # 3 x (B, T, d)
        q, k, v = (t.view(B, T, self.h, self.dh).transpose(1, 2) for t in (q, k, v))  # (B, h, T, dh)
        s = q @ k.transpose(-2, -1)                             # (B, h, T, T)
        if self.scale: s = s / math.sqrt(self.dh)              # Break 2: scale=False
        s = s.masked_fill(~self.mask[:T, :T], float("-inf"))  # formula (30.4)
        y = (F.softmax(s, dim=-1) @ v).transpose(1, 2).reshape(B, T, d)
        return self.proj(y)

class MLP(nn.Module):
    def __init__(self, d):
        super().__init__(); self.up = nn.Linear(d, 4 * d, bias=False); self.down = nn.Linear(4 * d, d, bias=False)
    def forward(self, x): return self.down(F.gelu(self.up(x)))

class Block(nn.Module):
    def __init__(self, d, h, T, scale=True, pre_ln=True):
        super().__init__(); self.pre_ln = pre_ln
        self.n1, self.attn = RMSNorm(d), CausalSelfAttention(d, h, T, scale)
        self.n2, self.mlp = RMSNorm(d), MLP(d)
    def forward(self, x):
        if self.pre_ln:                                         # formula (30.2)
            x = x + self.attn(self.n1(x))
            x = x + self.mlp(self.n2(x))
        else:                                                   # Break 3, formula (30.13)
            x = self.n1(x + self.attn(x))
            x = self.n2(x + self.mlp(x))
        return x

class GPT(nn.Module):
    def __init__(self, V, T, d=128, L=4, h=4, pos=True, scale=True, pre_ln=True):
        super().__init__(); self.T = T
        self.tok = nn.Embedding(V, d)
        self.pos = nn.Embedding(T, d) if pos else None            # Break 1: pos=False
        self.blocks = nn.ModuleList([Block(d, h, T, scale, pre_ln) for _ in range(L)])
        self.norm = RMSNorm(d) if pre_ln else nn.Identity()
        self.head = nn.Linear(d, V, bias=False)
        self.head.weight = self.tok.weight                        # weight tying: no new parameters
        self.apply(self._init)
        for n, p in self.named_parameters():                     # GPT-2: smaller init for layers that write to the stream
            if n.endswith("proj.weight") or n.endswith("down.weight"):
                nn.init.normal_(p, 0.0, 0.02 / math.sqrt(2 * L))
    def _init(self, m):
        if isinstance(m, (nn.Linear, nn.Embedding)): nn.init.normal_(m.weight, 0.0, 0.02)
    def forward(self, idx, targets=None):
        B, T = idx.shape
        x = self.tok(idx)                                         # (B, T, d)
        if self.pos is not None: x = x + self.pos(torch.arange(T))  # formula (30.1)
        for b in self.blocks: x = b(x)
        logits = self.head(self.norm(x))                          # (B, T, V)
        loss = None if targets is None else F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))
        return logits, loss
    # generate(...) as in Step 6

# ---- training (data code from Step 1 above) ----
torch.manual_seed(1337); torch.set_num_threads(3)
model = GPT(V, 128)
print(sum(p.numel() for p in model.parameters()))            # 812288, formula (30.6)
decay = [p for p in model.parameters() if p.dim() >= 2]
no_decay = [p for p in model.parameters() if p.dim() < 2]
opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.1},
                         {"params": no_decay, "weight_decay": 0.0}], lr=1e-3, betas=(0.9, 0.95))
S, Sw, lr_max, lr_min = 5000, 200, 1e-3, 1e-4
def lr_at(s):                                                   # formula (30.9)
    if s < Sw: return lr_max * (s + 1) / Sw
    p = (s - Sw) / (S - Sw)
    return lr_min + 0.5 * (lr_max - lr_min) * (1 + math.cos(math.pi * p))

@torch.no_grad()
def estimate():
    model.eval(); out = {}
    for split in ("train", "val"):
        g = torch.Generator().manual_seed(42)                  # the same 20 batches every time
        out[split] = sum(model(*get_batch(split, g))[1].item() for _ in range(20)) / 20
    model.train(); return out

g = torch.Generator().manual_seed(1337)
for s in range(S + 1):
    if s % 250 == 0: print(s, estimate())                      # step 0: about 4.18 = ln 65
    if s == S: break
    for group in opt.param_groups: group["lr"] = lr_at(s)
    x, y = get_batch("train", g)
    _, loss = model(x, y)
    opt.zero_grad(set_to_none=True); loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()
# final line: 5000 {'train': 1.271, 'val': 1.518}  (rounded; 925 s on 3 CPU threads)

Where this is used

System or taskWhat it shares with this lab
nanoGPT, GPT-2 reproductionThe same model file at larger \(d\), \(L\), \(T\), with a BPE tokenizer and F.scaled_dot_product_attention; GPT-2 small is \(d = 768\), \(L = 12\), \(h = 12\), \(T = 1024\) [Certain]
Llama, Qwen, Mistral model codeThe same block, with three swaps: RoPE instead of \(P\), SwiGLU instead of GELU, grouped-query attention instead of full multi-head (session 26)
Vision-language modelsThe language model of a VLM is this decoder; image patches become extra vectors in the same residual stream (Track 7)
Video classification and action recognitionA video transformer runs the same blocks over frame or tube tokens, usually without the causal mask, and puts a classifier on a pooled vector
Key-moment detection and video captioningA causal decoder like this one writes timestamps and captions token by token, conditioned on video tokens; temperature controls how varied the captions are
Video-RAG and video retrievalThe same decoder generates the answer from retrieved clip transcripts; the retrieval side uses a non-causal encoder with the same block
Debugging real training runsThe checks of Step 4 (loss at init near \(\ln|V|\), causal test, overfit one batch) are standard first steps for any new model

Common mistakes

  • Mask in the wrong direction or off by one. torch.triu instead of torch.tril, or masking the diagonal, lets the model see the future or hides the current token. The training loss then looks great or terrible. Run the causal test of Step 4.
  • Splitting heads with view in the wrong order. x.view(B, h, T, dh) mixes positions and features. The correct order is view(B, T, h, dh).transpose(1, 2). Shapes match either way, so nothing crashes.
  • Forgetting the shift between input and target. If y == x, the model only learns to copy, the loss goes to zero, and the samples are garbage.
  • Sampling from the wrong position. Use logits[:, -1, :], the last position. Taking position 0 predicts the second character of the prompt again and again.
  • Not cropping the context. With learned positions, feeding more than \(T\) tokens crashes (index out of range in \(P\)). Crop to idx[:, -T:].
  • Weight decay on gains and embeddings by accident. Decaying \(\gamma\) toward 0 shrinks the inputs of every sub-layer. Use parameter groups.
  • Evaluating in train mode or with gradients. If you add dropout, model.eval() matters; and without torch.no_grad(), evaluation stores activations and wastes memory.
  • Comparing runs with different seeds or different evaluation batches. Differences of 0.01 nats can be noise. Fix the seed and the evaluation batches, as we did, before you claim that a change helps.

Interview questions

Write causal self-attention for input of shape \((B, T, d)\) with \(h\) heads. Say the shapes.basic
One linear layer gives \((B, T, 3d)\); split into \(q, k, v\) of \((B, T, d)\); reshape each to \((B, T, h, d_h)\) and transpose to \((B, h, T, d_h)\). Scores q @ k.transpose(-2, -1) / sqrt(dh) are \((B, h, T, T)\). Fill positions above the diagonal with \(-\infty\), softmax over the last axis, multiply by \(v\) to get \((B, h, T, d_h)\), transpose back to \((B, T, h, d_h)\), reshape to \((B, T, d)\), and apply \(W_O\). In production use F.scaled_dot_product_attention(..., is_causal=True).
What loss do you expect at step 0, and what does it tell you if it is different?basic
About \(\ln |V|\): 4.174 for 65 characters, 10.82 for a 50,257-token vocabulary. That is the loss of a uniform guess. Our model gives 4.185. A much higher value means the initial logits are large and confident (bad init, missing final norm, a missing scale); the model must first unlearn that. A much lower value at step 0 usually means a leak, such as targets equal to inputs or a broken causal mask.
Count the parameters of a GPT with \(|V| = 65,\ T = 128,\ d = 128,\ L = 4\), no biases, tied head.core
Token table 8,320; position table 16,384; each block \(12d^2 + 2d = 196{,}864\), so 787,456 for four; final norm 128; tied head 0. Total 812,288. The rule \(12Ld^2 = 786{,}432\) covers 96.8%. With a large vocabulary the tables matter much more: at GPT-2 small shape, the token table is 31% of our formula’s count.
Why do we divide by \(\sqrt{d_h}\)? What happens without it?core
For unit-variance entries, \(\operatorname{Var}(\mathbf{q}\cdot\mathbf{k}) = d_h\). Without the scale, scores have standard deviation \(\sqrt{d_h}\), the softmax becomes nearly one-hot, and its Jacobian \(p_i(\delta_{ij} - p_j)\) is nearly zero, so attention learns slowly or gets stuck on one key. In practice the effect depends on how large q and k are: in our small model with initialization std 0.02, the scores start tiny, so removing the scale behaved more like a larger learning rate for attention than like a failure (validation loss 1.517 against 1.518 with our normal recipe, but 1.657 against 1.570 with a hot learning rate). In large models, logit growth is a real problem; QK-norm is one modern fix.
What happens if you remove the position embeddings from a decoder-only model?core
The first attention layer becomes order-blind over the prefix: shuffling earlier tokens does not change its output (we measured a change of \(2.4 \times 10^{-7}\)). But the causal mask still breaks the symmetry from layer 2 on, because each position averages over a different number of tokens, so the model can partly recover order (“NoPE”, Haviv et al., 2022). In our lab the validation loss got worse by 0.054 (1.572 against 1.518 after 5,000 steps) nats. An encoder without a causal mask and without positions is truly a bag of tokens.
Pre-LN vs Post-LN: what is the difference, and why do LLMs use Pre-LN?core
Post-LN normalizes the sum (\(\text{LN}(x + F(x))\)); Pre-LN normalizes only the sub-layer input (\(x + F(\text{LN}(x))\)) and adds one final norm. Pre-LN keeps a clean identity path from the loss to every layer, so gradients are well behaved at init and training is stable without long warmup, especially when deep. Post-LN needs warmup and can diverge (Xiong et al., 2020). Post-LN can reach slightly better quality when it trains well, which is why variants like DeepNorm and sandwich norms exist. In our lab, with 4 layers Post-LN was fine (with a hot learning rate it was even the best run), but with 12 layers and no warmup it got stuck at 3.35 nats, the loss of a unigram model, while Pre-LN trained normally.
Why tie the input embedding and the LM head? When would you not?core
Both map between token ids and the \(d\)-dimensional space, so sharing one \(|V| \times d\) matrix saves parameters (38.6M at GPT-2 small shape) and gives rare tokens a better output vector, because the input side trains it too. Large models often untie (Llama 2 and Llama 3 8B are untied [Likely]) because the table is a small share of the model and separate matrices give more freedom. Small models with big vocabularies usually tie.
Your validation loss stops falling while training loss keeps falling. What do you do?core
That is overfitting: the model memorizes the training text. In our baseline, after about 20 passes over 1 MB of text, the gap is 0.164 nats at step 1,000 and 0.247 at step 5,000 (train 1.271, val 1.518); validation loss is still falling, but slowly. Options: more data (the best fix), a smaller model or fewer steps (stop at the best validation loss), dropout, stronger weight decay. For LLM pretraining the usual answer is more data: runs see most tokens once, so train and validation loss stay close.
What does temperature do to the distribution? What do \(\tau \to 0\) and \(\tau \to \infty\) give?basic
Logits are divided by \(\tau\) before the softmax. Ratios become \(\exp((z_i - z_j)/\tau)\): small \(\tau\) sharpens, large \(\tau\) flattens, the order never changes. \(\tau \to 0\) gives greedy decoding (argmax), which can loop on repeated phrases; \(\tau \to \infty\) gives the uniform distribution, which is random characters. Temperature is often combined with top-k, top-p or min-p, which cut off the long tail.
How would you check that your model is truly causal, without reading the code?advanced
Perturb the input at one position \(t\) and compare logits: all positions \(\lt t\) must be bit-for-bit (or within float noise) unchanged. Or compute the gradient of the logit at position \(t\) with respect to the input embeddings and check that it is zero for all positions \(\gt t\). We measured a largest change of exactly 0.0 before the perturbed position. Also check the training loss: a leaky model reaches a suspiciously low loss very fast.
Estimate the training FLOPs of this lab, and why the \(6ND\) rule underestimates it here.advanced
\(N = 812{,}288\), \(D = 5{,}000 \times 4{,}096 = 20.48\)M tokens, so \(6ND \approx 1.0 \times 10^{14}\) FLOPs. The rule counts only the weight matrices. The attention scores and the weighted sum add about \(12LTd\) FLOPs per token in training; here \(T = d = 128\), so this term is \(12 \cdot 4 \cdot 128 \cdot 128 = 786{,}432\) per token, compared with \(6N \approx 4.9\)M: about 16% extra. The tied LM head is a real \(d \times |V|\) matrix multiply; since it shares \(E\), the term \(6N\) counts that matrix once, which is right for the head (the embedding lookup itself costs almost nothing).

Exercises

  1. Compute by hand the number of parameters for \(|V| = 65,\ T = 256,\ d = 192,\ L = 6\) (tied head). Check it with Figure 2 and in PyTorch.
  2. What loss do you expect at step 0 for a GPT-2 tokenizer (\(|V| = 50{,}257\))? For \(|V| = 128{,}256\)?
  3. Replace the hand-written attention with F.scaled_dot_product_attention(q, k, v, is_causal=True). Check that the logits agree to about \(10^{-6}\) and measure the speed-up.
  4. Replace the learned position table with RoPE (session 14). Does the validation loss change? Can you now generate more than 128 characters of context?
  5. Replace the GELU FFN with SwiGLU and choose the hidden size so that the parameter count stays the same (hint: \(3 d\, d_{ff} = 8 d^2\)).
  6. Train the no-position model and the baseline, then feed both the string “ROMEO:” and a shuffled version “EORMO:”. Which model notices the difference in its first layer?
  7. A model gives three possible next characters with probabilities 0.6, 0.2 and 0.2 at \(\tau = 1\). What are they at \(\tau = 0.5\)? (Hint: work with the logits \(z_i = \ln p_i\).)
Answerscheck
1. \(65 \cdot 192 + 256 \cdot 192 + 6\,(12 \cdot 192^2 + 2 \cdot 192) + 192 = 12{,}480 + 49{,}152 + 6 \cdot 442{,}752 + 192 = 2{,}718{,}336\). 2. \(\ln 50{,}257 = 10.825\) and \(\ln 128{,}256 = 11.762\) nats. 3. They agree up to float rounding (a different summation order); the fused kernel is faster and does not store the (B, h, T, T) tensor. 4. Expect a similar loss at \(T = 128\); with RoPE there is no table to index, so the code runs on longer inputs, but quality beyond the training length is not guaranteed (session 28). 5. \(d_{ff} = 8d/3 \approx 341\) for \(d = 128\); round to a multiple of 8 or 64 in practice, which changes the count slightly. 6. Only the baseline: without positions, the first layer’s output at the last position is the same for any order of the earlier characters (30.11). 7. At \(\tau = 0.5\), \(p_i \propto e^{2 z_i} = p_i^2\): 0.36, 0.04, 0.04, which sum to 0.44, so the answer is 0.818, 0.091, 0.091. The top-to-second ratio went from 3 to 9.

Key takeaways

  • A GPT is embeddings (30.1), \(L\) Pre-LN blocks of causal attention and an FFN on a residual stream (30.2–30.4), a final norm and a tied LM head with cross-entropy (30.5). About 100 lines of PyTorch.
  • Count exactly with (30.6): our model has 812,288 parameters; \(12Ld^2\) (30.7) covers 96.8% here, but much less when the vocabulary is large and the model is small.
  • Check before training: loss near \(\ln |V|\) (30.8), the causal test, overfit one batch. Then watch train and validation loss together.
  • Temperature (30.10) divides the logits; it sharpens or flattens without changing the order.
  • Breaking things teaches more than building them. No positions makes layer 1 order-blind (30.11) and cost 0.054 nats; no \(\sqrt{d_h}\) lets scores grow as \(\sqrt{d_h}\) (30.12) and hurt only with a hot learning rate; Post-LN (30.13) was fine at 4 layers but, at 12 layers without warmup, got stuck at the unigram loss. Always report the scale at which you saw an effect.

Key formulas

\[ \mathbf{h}^{(0)}_t = E[x_t] + P[t], \qquad \mathbf{h}' = \mathbf{a} + \operatorname{FFN}(\operatorname{RMSNorm}(\mathbf{a})),\ \ \mathbf{a} = \mathbf{h} + \operatorname{Attn}(\operatorname{RMSNorm}(\mathbf{h})) \]
\[ N = |V|d + Td + L(12d^2 + 2d) + d, \qquad \mathcal{L}_0 \approx \ln|V|, \qquad p_i(\tau) \propto e^{z_i/\tau}, \qquad \operatorname{Var}(\mathbf{q}\cdot\mathbf{k}) = d_h \]

References for this session

  1. Karpathy (2023). Neural Networks: Zero to Hero (video course)
  2. Raschka, Build a Large Language Model (From Scratch), chapter 4: Implementing a GPT model from scratch
  3. Karpathy (2023). nanoGPT (code)
  4. Karpathy (2023). nanoGPT (code)
  5. Karpathy (2023). Neural Networks: Zero to Hero (video course)
  6. Raschka, Build a Large Language Model (From Scratch), chapter 4: Implementing a GPT model from scratch
  7. Vaswani et al. (2017). Attention Is All You Need
  8. Zhang & Sennrich (2019). Root Mean Square Layer Normalization
  9. Xiong et al. (2020). On Layer Normalization in the Transformer Architecture
  10. Haviv, Ram, Press, Izsak & Levy (2022). Transformer Language Models without Positional Encodings Still Learn Positional Information
  11. Karpathy (2015). Tiny Shakespeare dataset (char-rnn repository)
  12. Radford et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2)