PyTorch mental model and a tiny bigram language model
Time to build. In this lab you write the smallest real language model: it predicts the next character of a name from the current one. First as a table of counts, then as a tiny neural network trained with the same loss as GPT. Along the way you learn how PyTorch thinks: tensors, autograd, modules and a clean, reproducible training loop.
What you will learn
- The PyTorch mental model: shape, dtype and device of a tensor; views; the autograd graph; nn.Module; train and eval mode
- How to make a training run reproducible
- A bigram model by counting, and why counting is exactly maximum likelihood
- Smoothing, and why it is the same idea as weight decay in the neural version
- A neural bigram model trained with cross-entropy and gradient descent, reaching the same loss as counting
- Train/validation splits, perplexity, sampling with temperature, and the famous broadcasting bug
What you will build, and why
A bigram model predicts the next symbol from only the one symbol before it. It is the simplest language model there is. But it uses the same ideas as GPT: a probability for every next token, the cross-entropy loss, sampling, and perplexity.
We work with characters, not words. Our data is names.txt: 32,033 English first names (from Andrej Karpathy’s makemore). The model learns to write new names that look real. You will build it twice:
- By counting: how often does each letter follow each other letter? Turn counts into probabilities.
- As a neural network: one weight matrix, softmax, cross-entropy and gradient descent.
Then you will see that the two give the same answer. That is Session 3’s maximum-likelihood idea, made real. Every figure on this page runs a real model in your browser on 3,000 of the names.
Setup and reproducibility
You need Python 3.10 or newer and PyTorch. A CPU is enough for this lab.
pip install torch numpy matplotlib curl -O https://raw.githubusercontent.com/karpathy/makemore/master/names.txt
Results should be the same every time you run them. Randomness enters in three places: weight initialization, data order and sampling. Control all of them:
import random, numpy as np, torch def set_seed(seed=1337): random.seed(seed); np.random.seed(seed); torch.manual_seed(seed) set_seed() torch.use_deterministic_algorithms(True) # raise an error if an op cannot be deterministic device = "cuda" if torch.cuda.is_available() else "cpu" g = torch.Generator().manual_seed(2147483647) # a private random stream for sampling
The PyTorch mental model
A tensor has a shape, a dtype and a device
Every tensor answers three questions. Shape: how many numbers along each axis, for example (32, 27). Dtype: what kind of number, which sets memory and precision. Device: where it lives, cpu or cuda. Operations need matching devices and compatible dtypes and shapes; most errors you will see are about one of these three.
| dtype | Bytes | Typical use |
|---|---|---|
float32 | 4 | default for weights on CPU; optimizer states |
bfloat16 / float16 | 2 | weights and activations in mixed-precision training and inference (session 37) |
int64 (long) | 8 | token IDs and class labels (what F.cross_entropy expects as targets) |
int8, uint8 | 1 | quantized weights, images |
bool | 1 | masks |
Memory of a tensor \(=\) number of elements \(\times\) bytes per element. A \(4096 \times 4096\) matrix is \(16.8\) million numbers: 67 MB in float32, 34 MB in bfloat16.
Views share memory
Many operations do not copy data; they return a view: a new shape over the same memory. x.view(...), x.transpose(...), x[:, 0] and x.T are views. Changing a view changes the original. view only works on memory laid out in order (“contiguous”); after a transpose, use reshape (which copies when it must) or call .contiguous() first.
Autograd records a graph
When a tensor has requires_grad=True, every operation on it is recorded. Each result remembers the function that made it (grad_fn). Calling loss.backward() walks this graph backward with the chain rule (Session 4) and adds each gradient into the .grad field of the leaf tensors.
w = torch.tensor([0.5], requires_grad=True); b = torch.tensor([-0.5], requires_grad=True) z = w * 2.0 + b # z.grad_fn is AddBackward0 a = torch.sigmoid(z) # SigmoidBackward0 L = (a - 1.0) ** 2 # PowBackward0 L.backward() print(L.item(), w.grad.item(), b.grad.item()) # 0.1425, -0.3549, -0.1774: exactly Session 4, Figure 1
- Gradients accumulate: call
opt.zero_grad()(or set.grad = None) before each new backward. - Inside
with torch.no_grad():(ortorch.inference_mode()) nothing is recorded: use it for evaluation, sampling and manual weight updates. x.detach()returns the same data without the graph: gradients stop there.
Modules, parameters, modes
An nn.Module bundles parameters (tensors with requires_grad=True, listed by model.parameters()) with a forward function. model.state_dict() holds every parameter and buffer by name; that is what you save. model.train() and model.eval() switch layers such as dropout and batch norm between training and inference behavior. Forgetting eval() at evaluation time is a classic silent bug.
The data: names as characters
The vocabulary is the 26 letters plus one special token, written ., that marks both the start and the end of a name. So \(|V| = 27\). The name “emma” becomes the bigrams
A name of length \(n\) gives \(n + 1\) bigrams. Over all 32,033 names (average length 6.12) that is 228,146 training examples. The start token is how the model learns which letters names begin with; the end token is how it learns when to stop. This is the same role that BOS and EOS play in an LLM (session 9).
Model 1: counting
Make a \(27 \times 27\) table \(N\), where \(N_{ij}\) is how many times character \(j\) came right after character \(i\). Divide each row by its sum to get probabilities:
This is the maximum-likelihood estimate from Session 3: for a categorical variable, the best probability is the frequency in the data. The most common pairs in the full data are n. (6,763 names end in “n”), a. (6,640) and an (5,438). A name starts with “a” with probability 0.138, and after an “n” the name ends with probability 0.369.
How to read Figure 1
Look at the first row: which letters start names. The first column: which letters end names (“a”, “n”, “e”, “y” stand out). Switch to row probabilities: each row now sums to 1, and this table is the model. White cells are pairs that never appeared. Their probability is 0, which causes a problem we fix next.
Sampling a name
Start at ., draw the next character from its row, move to that character’s row, and repeat until you draw . again. This is autoregressive generation from Session 1, with a context of one character. Temperature works as before: raise each probability to the power \(1/\tau\) and renormalize, which is the same as dividing log-probabilities by \(\tau\).
Measuring the model
The quality of the model is the average negative log-likelihood of the data, exactly the cross-entropy loss of Session 3:
A uniform guess over 27 characters gives \(\ln 27 = 3.296\) nats. The counting model gives 2.454 nats, a perplexity of \(e^{2.454} \approx 11.6\): on average the model is as unsure as a fair choice among about 12 characters instead of 27.
The zero problem, and smoothing
102 of the 729 pairs never appear in the data. Their probability is 0, so any new name that contains one of them gets \(\log 0 = -\infty\): an infinite loss. The fix is add-\(k\) smoothing: pretend every pair was seen \(k\) extra times.
With \(k = 1\) the training loss rises only a little, from 2.4540 to 2.4546, and nothing is ever impossible. Larger \(k\) pushes every row toward uniform. On an 80/10/10 split of the names, \(k = 1\) gives 2.4548 on training and 2.4533 on validation; \(k = 10\) gives 2.4627 on validation, so too much smoothing hurts. The bigram model has only 729 numbers, so it barely overfits.
Model 2: a neural bigram
Now the same task with a neural network, trained the way GPT is trained. The input character \(i\) becomes a one-hot vector \(\mathbf{x} \in \R^{27}\). One weight matrix \(W \in \R^{27 \times 27}\) maps it to 27 logits, and softmax gives probabilities:
Multiplying a one-hot vector by \(W\) just picks row \(i\): this is an embedding lookup (Session 1, formula 1.4). We train \(W\) with the cross-entropy loss (5.2) and gradient descent. From Session 2, the gradient for row \(i\) is “probabilities minus targets”, summed over all examples that start with \(i\):
The gradient is zero exactly when \(P_W(j \mid i) = N_{ij} / n_i\), the counting answer. So training the network converges to the counting model, and the best weights are the log-counts: \(W_{ij} = \log N_{ij} + c_i\). Two methods, one answer, because both are maximum likelihood.
Regularization is smoothing
Add a small penalty on the weights, \(\lambda \cdot \operatorname{mean}(W^2)\). It pulls every \(W_{ij}\) toward 0, and \(W = 0\) means uniform probabilities. So the penalty pushes rows toward uniform, exactly like add-\(k\) smoothing. Weight decay in an LLM (Session 4) is the same idea at a much larger scale.
How to read Figure 3
The first step starts near 3.7, worse than uniform (3.30), because the random weights are confidently wrong. A large learning rate works here because the problem is simple and the gradient (5.5) is small. Raise \(\lambda\) to 1 or 2 and the loss can no longer reach the dashed line: the penalty pulls the model toward uniform, like heavy smoothing. The names under the chart are sampled from the current weights.
The full PyTorch program
Here is the whole lab in one script. It is the reference solution; type it yourself first.
import torch, torch.nn.functional as F torch.manual_seed(1337) words = open("names.txt").read().split() chars = sorted(set("".join(words))) stoi = {c: i + 1 for i, c in enumerate(chars)}; stoi["."] = 0 itos = {i: c for c, i in stoi.items()} V = len(stoi) # 27 # ---- data: (current char, next char) pairs, with "." as start and end ---- xs, ys = [], [] for w in words: cs = ["."] + list(w) + ["."] for a, b in zip(cs, cs[1:]): xs.append(stoi[a]); ys.append(stoi[b]) xs, ys = torch.tensor(xs), torch.tensor(ys) # int64, 228,146 each # ---- model 1: counting, formulas (5.1) and (5.3) ---- N = torch.zeros((V, V), dtype=torch.int64) N.index_put_((xs, ys), torch.ones_like(xs), accumulate=True) P = (N + 1).float() P = P / P.sum(dim=1, keepdim=True) # keepdim=True is essential (see Common mistakes) nll = -P[xs, ys].log().mean() print(f"counting model NLL: {nll.item():.4f}") # 2.4546 # ---- sampling ---- g = torch.Generator().manual_seed(2147483647) for _ in range(5): ix, out = 0, [] while True: ix = torch.multinomial(P[ix], num_samples=1, generator=g).item() if ix == 0: break out.append(itos[ix]) print("".join(out)) # ---- model 2: neural bigram, formula (5.4), trained with cross-entropy ---- W = torch.randn((V, V), generator=g, requires_grad=True) for step in range(300): logits = W[xs] # same as F.one_hot(xs, V).float() @ W loss = F.cross_entropy(logits, ys) + 0.01 * (W ** 2).mean() W.grad = None loss.backward() with torch.no_grad(): W -= 50 * W.grad # plain gradient descent if step % 50 == 0: print(step, round(loss.item(), 4)) # falls from about 3.7 toward 2.48
Our NumPy replica of this exact training loop (same initialization scheme, learning rate 50, \(\lambda = 0.01\)) gave losses 3.75, 2.68, 2.51, 2.49 and 2.48 at steps 0, 10, 50, 100 and 300. Your numbers will be close but not identical, because the random stream differs. The loss keeps creeping toward the counting model’s value; the small gap left is the regularization term.
Pseudo-code
- input: words, smoothing \(k\)
- \(N \leftarrow\) zeros\((27, 27)\)
- for each word \(w\) do
- \(c \leftarrow [\,.\,] + \text{chars}(w) + [\,.\,]\)
- for each pair \((a, b)\) of neighbors in \(c\) do \(N_{ab} \leftarrow N_{ab} + 1\)
- \(P_{ij} \leftarrow (N_{ij} + k) / \sum_m (N_{im} + k)\) // formula (5.3): divide each ROW
- return \(P\)
- input: pairs \((x_n, y_n)\), learning rate \(\eta\), penalty \(\lambda\), steps \(S\)
- \(W \leftarrow\) random normal \((27, 27)\)
- for \(s = 1\) to \(S\) do
- \(\mathcal{L} \leftarrow -\frac{1}{n}\sum_n \log \softmax(W[x_n])_{y_n} + \lambda\,\operatorname{mean}(W^2)\)
- \(G \leftarrow \nabla_W \mathcal{L}\) // formula (5.5) plus \(2\lambda W / 729\)
- \(W \leftarrow W - \eta\, G\)
- return \(W\) // \(\softmax(W[i]) \approx\) row \(i\) of the counting model
Where this is used
- A baseline for every language model. Before training a large model, compute the unigram and bigram losses on your data. If your big model does not beat them quickly, something is wrong.
- Character-level priors for OCR. Reading jersey numbers or player names from video frames gives noisy characters. A small character or n-gram model (or a fixed roster, which is a much stronger prior) can rescore the OCR candidates.
- Classic speech recognition and keyboards used n-gram models for decades; they are still used where memory is tiny.
- Data quality. Text that a simple n-gram model finds very surprising is often noise or another language; this is the idea behind perplexity filtering (session 34).
Common mistakes
- The broadcasting bug.
P = N / N.sum(1)withoutkeepdim=True: the sum has shape(27,), which broadcasts as a row(1, 27), so you divide each column by the wrong number. The code runs, the rows do not sum to 1, and the model is silently wrong. Always checkP.sum(1)equals 1. - Forgetting the start and end token. Without it, the model cannot learn how names begin or when to stop.
- Using float targets.
F.cross_entropywants class indices asint64, not one-hot floats (unless you pass probabilities on purpose). - Not zeroing gradients. They accumulate across
backward()calls. - Updating weights inside the graph. A manual update must be inside
torch.no_grad(), or autograd records it. - Sampling without a generator. Results then depend on everything else that used the global random state.
- Measuring on training data only. Keep a validation split, even for small models.
Interview questions
Why does the neural bigram converge to the counting model?core
What is the effect of L2 regularization on the neural bigram?core
What does loss.backward() actually do in PyTorch?basic
loss back to the leaf tensors with requires_grad=True, applying each node’s backward function (the chain rule), and accumulates the results into each leaf’s .grad. By default it then frees the graph’s saved tensors, so you cannot call it twice without retain_graph=True.What is the difference between view and reshape?basic
view always returns a view sharing memory and fails if the tensor is not contiguous in a compatible way. reshape returns a view when possible and copies otherwise. After transpose or permute, a tensor is usually non-contiguous, so view raises an error.Explain the N / N.sum(1) bug.core
N.sum(1) has shape (27,). Broadcasting aligns from the right, so it acts like (1, 27) and divides column \(j\) by the sum of row \(j\), not row \(i\). Use keepdim=True to get shape (27, 1), which broadcasts down the rows. Check by asserting each row sums to 1.How do you make a PyTorch training run reproducible?core
torch.Generator objects for sampling and data shuffling; set worker seeds in DataLoaders; turn on torch.use_deterministic_algorithms(True) (and the cuBLAS workspace setting on CUDA); pin library versions. Even then, results can differ across GPU types or library versions.Why is the bigram loss 2.45 and not lower? How would you lower it?basic
Exercises
- Build the trigram model (condition on the previous two characters). Report training and validation loss with add-1 smoothing. Is it better than 2.454? Does smoothing matter more or less than for bigrams, and why?
- Find the best \(k\) for add-\(k\) smoothing on the validation split.
- Replace
W[xs]withF.one_hot(xs, 27).float() @ Wand check the loss is identical. Which is faster, and why? - Train the neural bigram with
torch.optim.AdamWinstead of manual updates. What learning rate do you need, and why is it so different from 50? - Write the broadcasting bug on purpose, print
P.sum(1), and compute the loss. How much worse is it? - Sample 1,000 names at temperatures 0.5, 1 and 1.5. Measure how many are in the training set. What do you see?
Hintscheck
Key takeaways
- Every tensor has a shape, a dtype and a device; views share memory; autograd records operations and
backward()accumulates gradients into leaves. - The counting bigram (5.1) is the maximum-likelihood model; smoothing (5.3) removes zero probabilities.
- The neural bigram (5.4) with one-hot input is an embedding lookup; its gradient (5.5) is zero exactly at the counting answer, so both reach about 2.45 nats (perplexity about 11.6, against 27 for uniform).
- Weight regularization plays the role of smoothing. Always check shapes (
keepdim=True) and keep a validation split.
References for this session
- Karpathy (2023). Neural Networks: Zero to Hero (video course)
- Raschka, Build a Large Language Model (From Scratch), chapter A: Introduction to PyTorch
- Karpathy (2023). Neural Networks: Zero to Hero (video course)
- Raschka, Build a Large Language Model (From Scratch), chapter A: Introduction to PyTorch
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 3: N-gram Language Models