write a GPT from scratch and train it
In this lab you write a small GPT in about 100 lines of PyTorch: embeddings, causal multi-head attention, the feed-forward network, RMSNorm, residual connections and a tied output layer. You count its 812,288 parameters by formula, train it on Tiny Shakespeare on a laptop CPU, watch train and validation loss, and sample text with different temperatures. Then you break it on purpose (no positions, no square-root scaling, Post-LN) and measure what really fails.
What you will learn
- How embeddings, causal multi-head attention, the FFN, RMSNorm, residual adds and a tied LM head fit together, with the tensor shape at every step
- How to count the parameters of a GPT exactly, and when the 12·L·d² rule is accurate and when it is not
- Three checks before training: loss near ln |V|, the causal test, overfitting one batch
- A full training loop with AdamW, warmup and cosine decay, gradient clipping, and honest train and validation curves
- How temperature changes the next-token distribution and the generated text
- What happens when you remove positions, remove the square-root scaling, or switch to Post-LN, and why small runs can hide some failures
Why this matters
In the sessions before this one we took the transformer apart: tokens, embeddings, attention, masks, heads, the feed-forward network, the residual stream, normalization, and the counting rules. In this lab you put all the pieces back together. You write a small GPT (a decoder-only transformer, the family of GPT-2, Llama and Qwen) in about 100 lines of PyTorch, train it on the complete works of Shakespeare on a laptop CPU, and make it write new text.
Then you break it on purpose. You remove the position embeddings, remove the \(\sqrt{d}\) in attention, and move the normalization to the “wrong” place, and you measure what happens. Some of these breaks hurt a lot. Some hurt much less than the textbooks suggest, at this small size. Knowing which is which, and why, is exactly the kind of understanding that interviews test and that real debugging needs.
- Interviews. “Implement causal self-attention” and “write a transformer block” are common coding questions for LLM roles. “What happens if you remove the scaling factor?” and “Why Pre-LN?” are common follow-ups.
- Real systems. Karpathy’s nanoGPT (the reference for this lab), Raschka’s book chapter 4, and the model files in Hugging Face
transformersall have the same skeleton you will write here. Once you have written it yourself, you can read the Llama or Qwen model code in one sitting. - Debugging. Large training runs fail in boring ways: a mask off by one, a missing scale, a wrong init. The checks in this lab (loss at step 0, the causal test, overfitting one batch) catch most of them in seconds.
Setup and plan
You need Python 3.9 or newer and PyTorch (version 2.x; we used 2.8.0 on CPU). No GPU is needed: every number on this page comes from runs on a laptop CPU, each about 15 minutes long.
pip install torch curl -O https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
The data is Tiny Shakespeare: 1,115,394 characters of Shakespeare plays in one text file. We use characters as tokens, so the vocabulary is the set of distinct characters: 65 (letters, digits 3, punctuation, space and newline). A character-level tokenizer keeps this lab about the model; with a BPE tokenizer from session 10, only the vocabulary size changes. The first 90% of the text (1,003,854 characters) is the training set and the last 10% (111,540 characters) is the validation set. [Certain: computed]
| Setting | Value | Meaning |
|---|---|---|
| \(|V|\) | 65 | vocabulary size (distinct characters) |
| \(T\) | 128 | context length (block size): how many characters the model sees |
| \(d\) | 128 | width of the residual stream (embedding size) |
| \(L\) | 4 | number of transformer blocks |
| \(h\), \(d_h\) | 4, 32 | attention heads, and the size of one head (\(d_h = d/h\)) |
| \(B\) | 32 | batch size (windows per step), so 4,096 tokens per step |
| steps, optimizer | 5,000, AdamW | peak learning rate \(10^{-3}\), 200 warmup steps, cosine decay to \(10^{-4}\), weight decay 0.1, gradient clipping at 1.0 |
The plan has seven steps: (1) data and batches, (2) the model, (3) count the parameters, (4) check the model before training, (5) train and plot the losses, (6) sample with temperature, (7) break it on purpose.
Step 1: data and batches
The tokenizer is two dictionaries: stoi (string to integer) and itos (integer to string). Training examples are windows of \(T + 1\) characters cut at random places: the first \(T\) are the input, and the same window shifted by one is the target. One window therefore holds \(T = 128\) training examples at once, one per position, exactly as in session 6.
import torch text = open("input.txt", encoding="utf-8").read() chars = sorted(set(text)); V = len(chars) # 65 stoi = {c: i for i, c in enumerate(chars)} encode = lambda s: [stoi[c] for c in s] decode = lambda ids: "".join(chars[i] for i in ids) data = torch.tensor(encode(text), dtype=torch.long) # shape (1115394,) n = int(0.9 * len(data)); train, val = data[:n], data[n:] def get_batch(split, g, B=32, T=128): d = train if split == "train" else val ix = torch.randint(len(d) - T - 1, (B,), generator=g) x = torch.stack([d[i:i + T] for i in ix]) # (B, T) inputs y = torch.stack([d[i + 1:i + T + 1] for i in ix]) # (B, T) targets, shifted by one return x, y
Why split by position (first 90% / last 10%) and not by random windows? Random windows from the same text would overlap: the model could see the validation characters during training, and the validation loss would look too good. A contiguous split avoids this leak.
Step 2: the model, piece by piece
A GPT is a short pipeline. Ids become vectors; \(L\) identical blocks update the vectors; a final layer turns each vector into 65 scores. Figure 1 follows one batch through it and shows the tensor shape at every step.
How to read Figure 1
Press Next and read the shape line. Notice two things. First, the shape (32, 128, 128) goes into a block and the same shape comes out: that is why blocks can be stacked. Second, only one tensor has a \(T \times T\) part: the attention scores, (32, 4, 128, 128). It holds 2,097,152 numbers per layer, more than the 524,288 numbers of the residual stream. At long context this tensor is what costs memory, and it is what FlashAttention avoids storing.
Embeddings
Each id selects a row of the token table \(E\), and each position selects a row of a learned position table \(P\), as in GPT-2 (session 11, session 13):
- Read the symbols
- \(x_t\)
- the token id at position \(t\) (here a character, from 0 to 64)
- \(E[x_t]\)
- row \(x_t\) of the token table: the vector for that token
- \(P[t]\)
- row \(t\) of the position table: a learned vector that means “I am at position \(t\)”
- \(\mathbf{h}^{(0)}_t\)
- the vector of position \(t\) at the start of the residual stream, size \(d\)
Modern models such as Llama use RoPE instead of \(P\) (rotate \(\mathbf{q}\) and \(\mathbf{k}\) inside attention, session 14). We keep the learned table because it is the simplest thing that works, and because removing it later is one line.
The block
Each block does two things in order: let positions talk to each other (attention), then let each position think alone (the FFN). Both are written into the residual stream with an addition. We use Pre-LN: normalize the input of each sub-layer, not the sum (session 22):
- Read the symbols
- \(\mathbf{h}, \mathbf{h}'\)
- the residual stream before and after the block, shape (B, T, d)
- \(\varepsilon\)
- a small number (\(10^{-5}\)) so we never divide by zero
- \(\gamma\)
- a learned gain vector of size \(d\), initialized to ones; \(\odot\) is element-wise product
- \(W_1, W_2\)
- the FFN matrices, \(d \to 4d\) and \(4d \to d\) (session 21)
Llama and Qwen use SwiGLU instead of GELU in the FFN (three matrices instead of two). We keep the GPT-2 style two-matrix FFN; exercise 5 asks you to swap it.
Causal multi-head attention
One linear layer makes queries, keys and values for all heads at once; then we split the last axis into \(h\) heads. For one head:
- Read the symbols
- \(Q, K, V\)
- queries, keys and values of one head, each \((T, d_h)\) for one sequence
- \(d_h\)
- the head size, \(d/h = 32\); dividing by \(\sqrt{d_h}\) keeps the scores at a stable size (Step 7)
- \(M\)
- the causal mask: position \(t\) may look at positions \(s \le t\) only (session 18)
The \(h\) head outputs are joined back to size \(d\) and multiplied by \(W_O\) (session 19). In real code you would call F.scaled_dot_product_attention(q, k, v, is_causal=True), which runs a fused kernel. We write the four lines by hand so that we can break them in Step 7.
The head and the loss
After the last block, one more RMSNorm, then the LM head turns each vector into \(|V|\) logits. We tie the head to the token table: the head matrix is \(E\) itself, so it adds no parameters (GPT-2 does the same [Certain]). The loss is the average cross-entropy over all \(B \cdot T\) positions:
- Read the symbols
- \(\mathbf{z}_t\)
- the logits (unnormalized scores) for the next token at position \(t\)
- \(y_{b,t}\)
- the true next token: the target at position \(t\) of window \(b\)
- \(\mathcal{L}\)
- the training loss in nats; \(e^{\mathcal{L}}\) is the perplexity
- input: ids \(x\) of shape (B, T); optional targets \(y\)
- \(h \leftarrow E[x] + P[0{:}T]\) // \((B, T, d)\), formula (30.1)
- for \(l = 1\) to \(L\) do
- \(u \leftarrow \text{RMSNorm}_1(h)\); \(q, k, v \leftarrow \operatorname{split}(u W_{QKV})\) into \(h\) heads // \((B, h, T, d_h)\)
- \(s \leftarrow q k^\top / \sqrt{d_h}\); \(s[\text{above diagonal}] \leftarrow -\infty\) // \((B, h, T, T)\)
- \(h \leftarrow h + \text{concat}(\softmax(s)\, v)\, W_O\) // residual add
- \(h \leftarrow h + W_2\, \text{GELU}(W_1\, \text{RMSNorm}_2(h))\) // formula (30.2)
- \(z \leftarrow \text{RMSNorm}_f(h)\, E^\top\) // \((B, T, |V|)\), tied head
- if \(y\) given then \(\text{loss} \leftarrow \operatorname{CE}\big(z \text{ as } (BT, |V|),\ y \text{ as } (BT)\big)\) // (30.5)
- return \(z\), loss
Step 3: count the parameters
In session 25 we derived that a block has about \(12d^2\) parameters: \(4d^2\) in attention (\(W_Q, W_K, W_V, W_O\)) and \(8d^2\) in the FFN. Our model has no biases, so we can write the exact count:
Token table: \(65 \times 128 = 8{,}320\). Position table: \(128 \times 128 = 16{,}384\). One block: \(12 \times 128^2 + 2 \times 128 = 196{,}608 + 256 = 196{,}864\); four blocks: \(787{,}456\). Final norm: 128. Head: 0 (tied).
Total: \(8{,}320 + 16{,}384 + 787{,}456 + 128 = \mathbf{812{,}288}\). PyTorch agrees: sum(p.numel() for p in model.parameters()) prints 812288. [Certain: computed]
The rule \(12Ld^2 = 786{,}432\) gives 96.8% of it. The missing 3.2% is mostly the two embedding tables. For a tiny vocabulary the rule works well; with a 50,257-token vocabulary at \(d = 768,\ L = 12\) (the shape of GPT-2 small), the same formula gives 124,337,664 parameters, but \(12Ld^2\) gives only 84,934,656 (68%): the token table alone is 38.6 million. (The real GPT-2 small has 124,439,808 parameters, because it also has biases and LayerNorm shifts: \(12d^2 + 13d\) per block. [Certain: computed from its config.json shape])
How to read Figure 2
Start from “This lab”: blocks are almost everything (812,288 in total). Click “GPT-2 small shape”: the token table jumps to 31% of the model, and \(12Ld^2\) covers only 68%. Untie the head and the two tables become 47.9% of the model. Now drag \(d\) or \(L\) up: the block share grows as \(d^2 L\) while the tables grow only as \(d\), so for large models the rule \(12Ld^2\) becomes accurate again. Small models with large vocabularies are the case where you must count the tables.
Step 4: check the model before training
Three checks take seconds and catch most bugs. Do them every time you write a new model.
Check 1: the loss at step 0. With small random weights, the logits are almost equal, so the softmax is almost uniform over the vocabulary, and the loss should be
Our model gives 4.185 (train) and 4.182 (validation) at step 0. [Certain: measured] If you see 5, 10 or 40 instead, the logits are too large at init: a wrong init, a missing final norm, or a missing scale. A model that starts confident and wrong wastes the first part of training on undoing it.
Check 2: the causal test. Change the input at position 100 and run the model again. The logits at positions 0 to 99 must not change at all; positions 100 and later may change. In our model the largest change before position 100 is exactly 0.0, and after it 15.3 (in logit units). [Certain: measured] If the first number is not (almost) zero, the model can see the future, and its training loss will be wonderful and useless.
Check 3: overfit one batch. Train for 200 steps on one fixed batch. The loss should fall toward zero. Ours (AdamW, learning rate \(10^{-3}\), no schedule) went \(4.188 \to 1.630\) at step 100 \(\to 0.064\) at step 200. [Certain: measured] If it cannot memorize 4,096 characters, something in the backward path is broken (a detached tensor, a parameter not passed to the optimizer, a learning rate of zero).
Step 5: train and watch both losses
The training loop is the one from session 4: sample a batch, forward, backward, clip the gradient, update with AdamW. The learning rate rises linearly for \(S_w = 200\) steps (warmup), then follows a cosine down to one tenth of the peak:
- Read the symbols
- \(s, S\)
- the current step and the total number of steps (5,000)
- \(\eta_{\max}, \eta_{\min}\)
- the peak and final learning rates, \(10^{-3}\) and \(10^{-4}\)
- \(p\)
- how far we are through the decay phase, from 0 to 1
- input: model, data, steps \(S\), schedule \(\eta(s)\), clip value \(c = 1\)
- params with 2 or more dims get weight decay 0.1; gains \(\gamma\) get none
- for \(s = 0\) to \(S - 1\) do
- if \(s \bmod 250 = 0\) then estimate train and val loss on 20 fixed batches each, in eval mode, without gradients
- set the learning rate to \(\eta(s)\) // formula (30.9)
- \(x, y \leftarrow \text{get\_batch}(\text{train})\); \(\text{loss} \leftarrow \text{model}(x, y)\)
- zero the gradients; loss.backward()
- clip the global gradient norm to \(c\); if loss is NaN then stop and report \(s\)
- AdamW step
- return model, loss curves
How much compute is this? 5,000 steps of 4,096 tokens is \(D = 20{,}480{,}000\) tokens, about 20 passes over the training text. With the rule \(C \approx 6ND\) from the counting session, training costs \(6 \times 812{,}288 \times 20{,}480{,}000 \approx 1.0 \times 10^{14}\) FLOPs (plus the attention term, which is not small here because \(T\) is as large as \(d\)). [Certain: computed] A modern GPU does this in seconds; a laptop CPU took about 15 minutes.
How to read Figure 3
Start with the baseline alone. Train and validation loss fall together for the first few hundred steps; then the dashed training line pulls below the validation line, and the gap keeps growing. Add “No positions”: it is worse at every step. Add “No \(\sqrt{d_h}\)”: it lies almost on top of the baseline. Add “Post-LN”: a little worse, but healthy. Now switch to the 12-layer runs. “12 layers, Post-LN, no warmup” and “12 layers, Post-LN, hot” drop to about 3.35 within 250 steps and stay flat for 2,000 steps, right on the gray line: the loss of a model that only knows how frequent each character is. Their Pre-LN twins train normally. The x-axis stretches to the longest run you have selected. The 12-layer runs use a 2,000-step schedule (their learning rate decays faster), so compare them with each other, not with the 4-layer runs.
What the baseline did, in numbers (all measured): the validation loss fell from 4.182 at step 0 to 2.159 at step 500, 1.874 at step 1,000 and 1.518 at step 5,000. A loss of 1.518 nats is a perplexity of \(e^{1.518} = 4.56\): on average the model is as unsure as a fair choice among about 4.6 characters, against 65 for a uniform guess. The training loss ended lower, at 1.271. The gap between the two grew from 0.164 at step 1,000 to 0.247 at step 5,000: after about 20 passes over one megabyte of text, the model has started to memorize. The validation loss was still falling at the end, so we stopped before it turned up. [Certain: measured]
Step 6: sample with temperature
To generate, we feed a prompt, take the logits at the last position, turn them into probabilities, draw one character, append it, and repeat. The context is cut to the last \(T = 128\) characters, because the position table has only 128 rows. Before the softmax we divide the logits by a temperature \(\tau\):
- Read the symbols
- \(z_i\)
- the logit of character \(i\) at the last position
- \(\tau\)
- the temperature: \(\tau \lt 1\) makes the distribution sharper, \(\tau \gt 1\) flatter, \(\tau \to 0\) is greedy decoding (always the top choice)
The ratio of two probabilities is \(p_i / p_j = \exp((z_i - z_j)/\tau)\). So halving \(\tau\) squares every ratio: a 3-to-1 preference becomes 9-to-1. Temperature does not change the order of the choices, only how strongly the top ones win.
" " is the space character, which is a token of its own in this character-level model, and "\n" is a line break. The logits are real; only the temperature is applied in your browser with formula (30.10). Bottom: 240 characters generated by the same model at three temperatures, with the same random seed.How to read Figure 4
Pick “…winter of our disc”. At \(\tau = 1\) the model puts 0.635 on “o” (as in “discontent”), and the top 4 characters hold 90% of the probability. At \(\tau = 0.5\) the top choice rises to 0.930; at \(\tau = 1.5\) it falls to 0.438. Now pick “…that is the q”: the model is almost certain at any temperature (0.998 for “u” at \(\tau = 1\)), because in English q is nearly always followed by u. “First Citizen:↵” is the start of a new speech, where many first letters are possible: entropy 2.823 nats, as uncertain as a fair choice among 16.8 characters. In the samples, \(\tau = 0.5\) keeps one speaker and common words; \(\tau = 1.0\) is the model’s own distribution, with new speaker names and some invented words; \(\tau = 1.5\) breaks words apart (“EdwEmouG”, a speaker called “ELBR3”).
# sampling (inside the GPT class) @torch.no_grad() def generate(self, idx, n, temperature=1.0, gen=None): for _ in range(n): logits, _ = self(idx[:, -self.T:]) # crop to the last T tokens p = F.softmax(logits[:, -1, :] / temperature, dim=-1) # formula (30.10) idx = torch.cat([idx, torch.multinomial(p, 1, generator=gen)], dim=1) return idx
This loop is slow on purpose: it runs the whole context again for every new character, so generating \(n\) tokens costs about \(n \cdot T\) token-forwards. Real servers keep the keys and values of past positions in a KV cache and compute only the new position (session 26).
Step 7: break it on purpose
Now the fun part. Each break is one line of code. For each one, first predict what will happen, then look at Figure 3.
Break 1: remove the positions
Delete \(P[t]\) from formula (30.1). What does the model lose? Attention by itself treats its inputs as a set: if you shuffle the keys and values together, the weighted sum does not change. So in the first layer, the output at position \(t\) depends only on which characters came before, not on their order:
We checked this with a one-layer model without positions: shuffling the first 12 characters of “First Citizen” changes the last logits by at most \(2.4 \times 10^{-7}\) (float rounding). With positions, the change is 0.011; with two layers and no positions, it is 0.089. [Certain: measured] Why does the second layer break the symmetry? Because of the causal mask: position \(s\) in layer 1 averages over \(s\) inputs, so its output depends on how many characters it saw, and that is position information. A causal transformer can partly recover positions on its own. This is known: decoder-only models without any position encoding (“NoPE”) still learn, and still learn some notion of position (Haviv et al., 2022). [Certain]
Result. The model without positions still trained, but clearly worse: validation loss 1.572 against 1.518 for the baseline after 5,000 steps (0.054 nats worse), and the gap was larger early on (2.023 against 1.874 at step 1,000). [Certain: measured] Character-level text depends heavily on exact order (“dog” and “god” have the same letters), and the first layer is blind to it. The later layers recover part of the order through the causal mask, which is why the damage is moderate rather than total.
Break 2: remove \(\sqrt{d_h}\)
If the entries of \(\mathbf{q}\) and \(\mathbf{k}\) are independent with mean 0 and variance 1, their dot product is a sum of \(d_h\) such products:
Without the division, the scores have standard deviation \(\sqrt{d_h}\): about 5.7 for our heads, 8 for \(d_h = 64\), 11.3 for \(d_h = 128\). The softmax of numbers that far apart is almost one-hot: one key gets nearly all the weight, and the gradient through the softmax becomes tiny, because \(\partial p_i / \partial z_j = p_i(\delta_{ij} - p_j)\) is near zero when every \(p\) is near 0 or 1 (session 17). Figure 5 shows it with random unit-variance vectors.
How to read Figure 5
Keep “Divide by \(\sqrt{d_h}\)” and move the slider. The measured score standard deviation grows like \(\sqrt{d_h}\) (1.88 at \(d_h = 4\), 22.65 at 512), but after the division the weights look the same for every size: the mean largest weight stays between 0.22 and 0.26 and the entropy between 2.33 and 2.41 nats. Now click “No scaling”. At \(d_h = 32\) (our heads) the mean largest weight jumps to 0.750 and the entropy falls to 0.667; at \(d_h = 128\) (the head size of Llama 3 8B) it is 0.885 and 0.286. Attention turns into a hard lookup of a single key.
Result. With our normal recipe, removing \(\sqrt{d_h}\) changed nothing we could measure: validation loss 1.517 against 1.518. [Certain: measured] Why so little? Formula (30.12) assumes that the entries of \(\mathbf{q}\) and \(\mathbf{k}\) have size about 1. At our initialization they are much smaller: the input of attention has RMS 1 after RMSNorm and the weights have standard deviation 0.02, so each entry of \(\mathbf{q}\) or \(\mathbf{k}\) has standard deviation about \(0.02 \times \sqrt{128} = 0.23\), and a raw score has standard deviation about \(0.23^2 \times \sqrt{32} \approx 0.29\). The scores start near zero with or without the division, and the model can simply learn a smaller \(W_Q\) or \(W_K\). The scale starts to matter when training pushes \(\mathbf{q}\) and \(\mathbf{k}\) to larger values. With the hot recipe, the unscaled model ended at 1.657 against 1.570 for the scaled one: 0.087 nats worse. [Certain: measured] In large models, attention logits really do grow during training, which is why some recent models also normalize \(\mathbf{q}\) and \(\mathbf{k}\) (QK-norm, see session 22).
Break 3: Post-LN
Move the norm after the addition, as in the original transformer:
In Post-LN the stream itself is normalized after every sub-layer, so there is no clean identity path from the output back to the input: the gradient must pass through \(2L\) normalizations. Xiong et al. (2020) showed that at initialization the gradients near the output are large in Post-LN, which is why Post-LN transformers need a learning-rate warmup and can diverge without one. [Certain] We trained Post-LN with 4 layers (with our normal recipe, and with a “hot” recipe: learning rate \(3 \times 10^{-3}\), no warmup, no clipping), and with 12 layers (2,387,200 parameters, 2,000 steps): with the normal 200-step warmup, with no warmup, and with the hot recipe. Every Post-LN run has a Pre-LN twin with the same settings.
Result. With 4 layers, Post-LN was not a failure. With the normal recipe it reached 1.571 against 1.518 for Pre-LN, and with the hot recipe it was the best run of the whole lab: 1.502 against 1.570 for Pre-LN. With 12 layers and warmup, Post-LN trained, but worse than Pre-LN: 1.677 against 1.595 after 2,000 steps. With 12 layers and no warmup, Post-LN failed completely, at both learning rates. Within 250 steps the loss fell to 3.30 (train) and 3.35 (validation), and then it stayed flat until step 2,000. Its Pre-LN twin with the same hot recipe trained normally (1.694). [Certain: measured]
What is a loss of 3.35? It is the loss of a unigram model: one that ignores the context and predicts every character with its overall frequency in the training text. On the validation text that gives 3.347 nats. [Certain: computed] So the deep Post-LN network learned the character frequencies in the first 50 steps and then stopped learning: its gradient norm fell from 1.04 at step 0 to about 0.19 at step 50 and stayed small. Our reading is that the large early steps, passing through 24 normalizations, put the blocks in a state where the output ignores them, and the small gradients never get them out. That is a plausible explanation, not something we proved. [Likely] The lesson matches the literature: the problems of Post-LN grow with depth, and warmup is what keeps it alive.
The full program
This is the whole model, the same file that produced every number on this page. Put the data code from Step 1 and the training loop below it into one file and run it. With the same PyTorch version, seed and thread count, you should get the same curves; across versions or machines, expect small differences in the last digits, because floating-point sums are done in a different order.
import math, torch import torch.nn as nn import torch.nn.functional as F class RMSNorm(nn.Module): def __init__(self, d, eps=1e-5): super().__init__(); self.eps = eps; self.weight = nn.Parameter(torch.ones(d)) def forward(self, x): # formula (30.3) return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.weight class CausalSelfAttention(nn.Module): def __init__(self, d, h, T, scale=True): super().__init__(); assert d % h == 0 self.h, self.dh, self.scale = h, d // h, scale self.qkv = nn.Linear(d, 3 * d, bias=False) # W_Q, W_K, W_V in one matrix self.proj = nn.Linear(d, d, bias=False) # W_O self.register_buffer("mask", torch.tril(torch.ones(T, T, dtype=torch.bool))) def forward(self, x): B, T, d = x.shape q, k, v = self.qkv(x).split(d, dim=2) # 3 x (B, T, d) q, k, v = (t.view(B, T, self.h, self.dh).transpose(1, 2) for t in (q, k, v)) # (B, h, T, dh) s = q @ k.transpose(-2, -1) # (B, h, T, T) if self.scale: s = s / math.sqrt(self.dh) # Break 2: scale=False s = s.masked_fill(~self.mask[:T, :T], float("-inf")) # formula (30.4) y = (F.softmax(s, dim=-1) @ v).transpose(1, 2).reshape(B, T, d) return self.proj(y) class MLP(nn.Module): def __init__(self, d): super().__init__(); self.up = nn.Linear(d, 4 * d, bias=False); self.down = nn.Linear(4 * d, d, bias=False) def forward(self, x): return self.down(F.gelu(self.up(x))) class Block(nn.Module): def __init__(self, d, h, T, scale=True, pre_ln=True): super().__init__(); self.pre_ln = pre_ln self.n1, self.attn = RMSNorm(d), CausalSelfAttention(d, h, T, scale) self.n2, self.mlp = RMSNorm(d), MLP(d) def forward(self, x): if self.pre_ln: # formula (30.2) x = x + self.attn(self.n1(x)) x = x + self.mlp(self.n2(x)) else: # Break 3, formula (30.13) x = self.n1(x + self.attn(x)) x = self.n2(x + self.mlp(x)) return x class GPT(nn.Module): def __init__(self, V, T, d=128, L=4, h=4, pos=True, scale=True, pre_ln=True): super().__init__(); self.T = T self.tok = nn.Embedding(V, d) self.pos = nn.Embedding(T, d) if pos else None # Break 1: pos=False self.blocks = nn.ModuleList([Block(d, h, T, scale, pre_ln) for _ in range(L)]) self.norm = RMSNorm(d) if pre_ln else nn.Identity() self.head = nn.Linear(d, V, bias=False) self.head.weight = self.tok.weight # weight tying: no new parameters self.apply(self._init) for n, p in self.named_parameters(): # GPT-2: smaller init for layers that write to the stream if n.endswith("proj.weight") or n.endswith("down.weight"): nn.init.normal_(p, 0.0, 0.02 / math.sqrt(2 * L)) def _init(self, m): if isinstance(m, (nn.Linear, nn.Embedding)): nn.init.normal_(m.weight, 0.0, 0.02) def forward(self, idx, targets=None): B, T = idx.shape x = self.tok(idx) # (B, T, d) if self.pos is not None: x = x + self.pos(torch.arange(T)) # formula (30.1) for b in self.blocks: x = b(x) logits = self.head(self.norm(x)) # (B, T, V) loss = None if targets is None else F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1)) return logits, loss # generate(...) as in Step 6 # ---- training (data code from Step 1 above) ---- torch.manual_seed(1337); torch.set_num_threads(3) model = GPT(V, 128) print(sum(p.numel() for p in model.parameters())) # 812288, formula (30.6) decay = [p for p in model.parameters() if p.dim() >= 2] no_decay = [p for p in model.parameters() if p.dim() < 2] opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.1}, {"params": no_decay, "weight_decay": 0.0}], lr=1e-3, betas=(0.9, 0.95)) S, Sw, lr_max, lr_min = 5000, 200, 1e-3, 1e-4 def lr_at(s): # formula (30.9) if s < Sw: return lr_max * (s + 1) / Sw p = (s - Sw) / (S - Sw) return lr_min + 0.5 * (lr_max - lr_min) * (1 + math.cos(math.pi * p)) @torch.no_grad() def estimate(): model.eval(); out = {} for split in ("train", "val"): g = torch.Generator().manual_seed(42) # the same 20 batches every time out[split] = sum(model(*get_batch(split, g))[1].item() for _ in range(20)) / 20 model.train(); return out g = torch.Generator().manual_seed(1337) for s in range(S + 1): if s % 250 == 0: print(s, estimate()) # step 0: about 4.18 = ln 65 if s == S: break for group in opt.param_groups: group["lr"] = lr_at(s) x, y = get_batch("train", g) _, loss = model(x, y) opt.zero_grad(set_to_none=True); loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) opt.step() # final line: 5000 {'train': 1.271, 'val': 1.518} (rounded; 925 s on 3 CPU threads)
Where this is used
| System or task | What it shares with this lab |
|---|---|
| nanoGPT, GPT-2 reproduction | The same model file at larger \(d\), \(L\), \(T\), with a BPE tokenizer and F.scaled_dot_product_attention; GPT-2 small is \(d = 768\), \(L = 12\), \(h = 12\), \(T = 1024\) [Certain] |
| Llama, Qwen, Mistral model code | The same block, with three swaps: RoPE instead of \(P\), SwiGLU instead of GELU, grouped-query attention instead of full multi-head (session 26) |
| Vision-language models | The language model of a VLM is this decoder; image patches become extra vectors in the same residual stream (Track 7) |
| Video classification and action recognition | A video transformer runs the same blocks over frame or tube tokens, usually without the causal mask, and puts a classifier on a pooled vector |
| Key-moment detection and video captioning | A causal decoder like this one writes timestamps and captions token by token, conditioned on video tokens; temperature controls how varied the captions are |
| Video-RAG and video retrieval | The same decoder generates the answer from retrieved clip transcripts; the retrieval side uses a non-causal encoder with the same block |
| Debugging real training runs | The checks of Step 4 (loss at init near \(\ln|V|\), causal test, overfit one batch) are standard first steps for any new model |
Common mistakes
- Mask in the wrong direction or off by one.
torch.triuinstead oftorch.tril, or masking the diagonal, lets the model see the future or hides the current token. The training loss then looks great or terrible. Run the causal test of Step 4. - Splitting heads with
viewin the wrong order.x.view(B, h, T, dh)mixes positions and features. The correct order isview(B, T, h, dh).transpose(1, 2). Shapes match either way, so nothing crashes. - Forgetting the shift between input and target. If
y == x, the model only learns to copy, the loss goes to zero, and the samples are garbage. - Sampling from the wrong position. Use
logits[:, -1, :], the last position. Taking position 0 predicts the second character of the prompt again and again. - Not cropping the context. With learned positions, feeding more than \(T\) tokens crashes (index out of range in \(P\)). Crop to
idx[:, -T:]. - Weight decay on gains and embeddings by accident. Decaying \(\gamma\) toward 0 shrinks the inputs of every sub-layer. Use parameter groups.
- Evaluating in train mode or with gradients. If you add dropout,
model.eval()matters; and withouttorch.no_grad(), evaluation stores activations and wastes memory. - Comparing runs with different seeds or different evaluation batches. Differences of 0.01 nats can be noise. Fix the seed and the evaluation batches, as we did, before you claim that a change helps.
Interview questions
Write causal self-attention for input of shape \((B, T, d)\) with \(h\) heads. Say the shapes.basic
q @ k.transpose(-2, -1) / sqrt(dh) are \((B, h, T, T)\). Fill positions above the diagonal with \(-\infty\), softmax over the last axis, multiply by \(v\) to get \((B, h, T, d_h)\), transpose back to \((B, T, h, d_h)\), reshape to \((B, T, d)\), and apply \(W_O\). In production use F.scaled_dot_product_attention(..., is_causal=True).What loss do you expect at step 0, and what does it tell you if it is different?basic
Count the parameters of a GPT with \(|V| = 65,\ T = 128,\ d = 128,\ L = 4\), no biases, tied head.core
Why do we divide by \(\sqrt{d_h}\)? What happens without it?core
What happens if you remove the position embeddings from a decoder-only model?core
Pre-LN vs Post-LN: what is the difference, and why do LLMs use Pre-LN?core
Why tie the input embedding and the LM head? When would you not?core
Your validation loss stops falling while training loss keeps falling. What do you do?core
What does temperature do to the distribution? What do \(\tau \to 0\) and \(\tau \to \infty\) give?basic
How would you check that your model is truly causal, without reading the code?advanced
Estimate the training FLOPs of this lab, and why the \(6ND\) rule underestimates it here.advanced
Exercises
- Compute by hand the number of parameters for \(|V| = 65,\ T = 256,\ d = 192,\ L = 6\) (tied head). Check it with Figure 2 and in PyTorch.
- What loss do you expect at step 0 for a GPT-2 tokenizer (\(|V| = 50{,}257\))? For \(|V| = 128{,}256\)?
- Replace the hand-written attention with
F.scaled_dot_product_attention(q, k, v, is_causal=True). Check that the logits agree to about \(10^{-6}\) and measure the speed-up. - Replace the learned position table with RoPE (session 14). Does the validation loss change? Can you now generate more than 128 characters of context?
- Replace the GELU FFN with SwiGLU and choose the hidden size so that the parameter count stays the same (hint: \(3 d\, d_{ff} = 8 d^2\)).
- Train the no-position model and the baseline, then feed both the string “ROMEO:” and a shuffled version “EORMO:”. Which model notices the difference in its first layer?
- A model gives three possible next characters with probabilities 0.6, 0.2 and 0.2 at \(\tau = 1\). What are they at \(\tau = 0.5\)? (Hint: work with the logits \(z_i = \ln p_i\).)
Answerscheck
Key takeaways
- A GPT is embeddings (30.1), \(L\) Pre-LN blocks of causal attention and an FFN on a residual stream (30.2–30.4), a final norm and a tied LM head with cross-entropy (30.5). About 100 lines of PyTorch.
- Count exactly with (30.6): our model has 812,288 parameters; \(12Ld^2\) (30.7) covers 96.8% here, but much less when the vocabulary is large and the model is small.
- Check before training: loss near \(\ln |V|\) (30.8), the causal test, overfit one batch. Then watch train and validation loss together.
- Temperature (30.10) divides the logits; it sharpens or flattens without changing the order.
- Breaking things teaches more than building them. No positions makes layer 1 order-blind (30.11) and cost 0.054 nats; no \(\sqrt{d_h}\) lets scores grow as \(\sqrt{d_h}\) (30.12) and hurt only with a hot learning rate; Post-LN (30.13) was fine at 4 layers but, at 12 layers without warmup, got stuck at the unigram loss. Always report the scale at which you saw an effect.
Key formulas
References for this session
- Karpathy (2023). Neural Networks: Zero to Hero (video course)
- Raschka, Build a Large Language Model (From Scratch), chapter 4: Implementing a GPT model from scratch
- Karpathy (2023). nanoGPT (code)
- Karpathy (2023). nanoGPT (code)
- Karpathy (2023). Neural Networks: Zero to Hero (video course)
- Raschka, Build a Large Language Model (From Scratch), chapter 4: Implementing a GPT model from scratch
- Vaswani et al. (2017). Attention Is All You Need
- Zhang & Sennrich (2019). Root Mean Square Layer Normalization
- Xiong et al. (2020). On Layer Normalization in the Transformer Architecture
- Haviv, Ram, Press, Izsak & Levy (2022). Transformer Language Models without Positional Encodings Still Learn Positional Information
- Karpathy (2015). Tiny Shakespeare dataset (char-rnn repository)
- Radford et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2)