LLM & VLM courseBlog
5Track 1: Foundations · session 5 of 15

PyTorch mental model and a tiny bigram language model

Time to build. In this lab you write the smallest real language model: it predicts the next character of a name from the current one. First as a table of counts, then as a tiny neural network trained with the same loss as GPT. Along the way you learn how PyTorch thinks: tensors, autograd, modules and a clean, reproducible training loop.

Time: 3 to 5 hours (mostly hands-on)Level: Beginner · hands-on labBefore this: Sessions 1 to 4, Python

What you will learn

  • The PyTorch mental model: shape, dtype and device of a tensor; views; the autograd graph; nn.Module; train and eval mode
  • How to make a training run reproducible
  • A bigram model by counting, and why counting is exactly maximum likelihood
  • Smoothing, and why it is the same idea as weight decay in the neural version
  • A neural bigram model trained with cross-entropy and gradient descent, reaching the same loss as counting
  • Train/validation splits, perplexity, sampling with temperature, and the famous broadcasting bug

What you will build, and why

A bigram model predicts the next symbol from only the one symbol before it. It is the simplest language model there is. But it uses the same ideas as GPT: a probability for every next token, the cross-entropy loss, sampling, and perplexity.

We work with characters, not words. Our data is names.txt: 32,033 English first names (from Andrej Karpathy’s makemore). The model learns to write new names that look real. You will build it twice:

  1. By counting: how often does each letter follow each other letter? Turn counts into probabilities.
  2. As a neural network: one weight matrix, softmax, cross-entropy and gradient descent.

Then you will see that the two give the same answer. That is Session 3’s maximum-likelihood idea, made real. Every figure on this page runs a real model in your browser on 3,000 of the names.

Setup and reproducibility

You need Python 3.10 or newer and PyTorch. A CPU is enough for this lab.

pip install torch numpy matplotlib
curl -O https://raw.githubusercontent.com/karpathy/makemore/master/names.txt

Results should be the same every time you run them. Randomness enters in three places: weight initialization, data order and sampling. Control all of them:

import random, numpy as np, torch
def set_seed(seed=1337):
    random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)
set_seed()
torch.use_deterministic_algorithms(True)   # raise an error if an op cannot be deterministic
device = "cuda" if torch.cuda.is_available() else "cpu"
g = torch.Generator().manual_seed(2147483647)   # a private random stream for sampling

The PyTorch mental model

A tensor has a shape, a dtype and a device

Every tensor answers three questions. Shape: how many numbers along each axis, for example (32, 27). Dtype: what kind of number, which sets memory and precision. Device: where it lives, cpu or cuda. Operations need matching devices and compatible dtypes and shapes; most errors you will see are about one of these three.

dtypeBytesTypical use
float324default for weights on CPU; optimizer states
bfloat16 / float162weights and activations in mixed-precision training and inference (session 37)
int64 (long)8token IDs and class labels (what F.cross_entropy expects as targets)
int8, uint81quantized weights, images
bool1masks

Memory of a tensor \(=\) number of elements \(\times\) bytes per element. A \(4096 \times 4096\) matrix is \(16.8\) million numbers: 67 MB in float32, 34 MB in bfloat16.

Views share memory

Many operations do not copy data; they return a view: a new shape over the same memory. x.view(...), x.transpose(...), x[:, 0] and x.T are views. Changing a view changes the original. view only works on memory laid out in order (“contiguous”); after a transpose, use reshape (which copies when it must) or call .contiguous() first.

Autograd records a graph

When a tensor has requires_grad=True, every operation on it is recorded. Each result remembers the function that made it (grad_fn). Calling loss.backward() walks this graph backward with the chain rule (Session 4) and adds each gradient into the .grad field of the leaf tensors.

w = torch.tensor([0.5], requires_grad=True); b = torch.tensor([-0.5], requires_grad=True)
z = w * 2.0 + b                  # z.grad_fn is AddBackward0
a = torch.sigmoid(z)             # SigmoidBackward0
L = (a - 1.0) ** 2               # PowBackward0
L.backward()
print(L.item(), w.grad.item(), b.grad.item())   # 0.1425, -0.3549, -0.1774: exactly Session 4, Figure 1
  • Gradients accumulate: call opt.zero_grad() (or set .grad = None) before each new backward.
  • Inside with torch.no_grad(): (or torch.inference_mode()) nothing is recorded: use it for evaluation, sampling and manual weight updates.
  • x.detach() returns the same data without the graph: gradients stop there.

Modules, parameters, modes

An nn.Module bundles parameters (tensors with requires_grad=True, listed by model.parameters()) with a forward function. model.state_dict() holds every parameter and buffer by name; that is what you save. model.train() and model.eval() switch layers such as dropout and batch norm between training and inference behavior. Forgetting eval() at evaluation time is a classic silent bug.

The data: names as characters

The vocabulary is the 26 letters plus one special token, written ., that marks both the start and the end of a name. So \(|V| = 27\). The name “emma” becomes the bigrams

\[ (\texttt{.},\texttt{e}),\ (\texttt{e},\texttt{m}),\ (\texttt{m},\texttt{m}),\ (\texttt{m},\texttt{a}),\ (\texttt{a},\texttt{.}) \]

A name of length \(n\) gives \(n + 1\) bigrams. Over all 32,033 names (average length 6.12) that is 228,146 training examples. The start token is how the model learns which letters names begin with; the end token is how it learns when to stop. This is the same role that BOS and EOS play in an LLM (session 9).

Model 1: counting

Make a \(27 \times 27\) table \(N\), where \(N_{ij}\) is how many times character \(j\) came right after character \(i\). Divide each row by its sum to get probabilities:

\[ P(j \mid i) = \frac{N_{ij}}{\sum_{k} N_{ik}} \tag{5.1} \]

This is the maximum-likelihood estimate from Session 3: for a categorical variable, the best probability is the frequency in the data. The most common pairs in the full data are n. (6,763 names end in “n”), a. (6,640) and an (5,438). A name starts with “a” with probability 0.138, and after an “n” the name ends with probability 0.369.

Click any cell.
Figure 1. The bigram table for 3,000 names, computed in your browser. Row = current character, column = next character. Darker = more common. The first row and column are the start/end token “.”.
How to read Figure 1

Look at the first row: which letters start names. The first column: which letters end names (“a”, “n”, “e”, “y” stand out). Switch to row probabilities: each row now sums to 1, and this table is the model. White cells are pairs that never appeared. Their probability is 0, which causes a problem we fix next.

Sampling a name

Start at ., draw the next character from its row, move to that character’s row, and repeat until you draw . again. This is autoregressive generation from Session 1, with a context of one character. Temperature works as before: raise each probability to the power \(1/\tau\) and renormalize, which is the same as dividing log-probabilities by \(\tau\).

Figure 2. Names sampled from the counting model (with add-1 smoothing). Under each name: its average negative log-likelihood per character under the model. Real names in the data average about 2.45.

Measuring the model

The quality of the model is the average negative log-likelihood of the data, exactly the cross-entropy loss of Session 3:

\[ \mathcal{L} = -\frac{1}{|D|} \sum_{(i,j) \in D} \log P(j \mid i) \tag{5.2} \]
Real numbers on the full dataset

A uniform guess over 27 characters gives \(\ln 27 = 3.296\) nats. The counting model gives 2.454 nats, a perplexity of \(e^{2.454} \approx 11.6\): on average the model is as unsure as a fair choice among about 12 characters instead of 27.

The zero problem, and smoothing

102 of the 729 pairs never appear in the data. Their probability is 0, so any new name that contains one of them gets \(\log 0 = -\infty\): an infinite loss. The fix is add-\(k\) smoothing: pretend every pair was seen \(k\) extra times.

\[ P_k(j \mid i) = \frac{N_{ij} + k}{\sum_{m} \left(N_{im} + k\right)} = \frac{N_{ij} + k}{\sum_m N_{im} + 27k} \tag{5.3} \]

With \(k = 1\) the training loss rises only a little, from 2.4540 to 2.4546, and nothing is ever impossible. Larger \(k\) pushes every row toward uniform. On an 80/10/10 split of the names, \(k = 1\) gives 2.4548 on training and 2.4533 on validation; \(k = 10\) gives 2.4627 on validation, so too much smoothing hurts. The bigram model has only 729 numbers, so it barely overfits.

Model 2: a neural bigram

Now the same task with a neural network, trained the way GPT is trained. The input character \(i\) becomes a one-hot vector \(\mathbf{x} \in \R^{27}\). One weight matrix \(W \in \R^{27 \times 27}\) maps it to 27 logits, and softmax gives probabilities:

\[ \mathbf{z} = \mathbf{x}\, W = W[i, :], \qquad P_W(j \mid i) = \softmax(\mathbf{z})_j = \frac{e^{W_{ij}}}{\sum_m e^{W_{im}}} \tag{5.4} \]

Multiplying a one-hot vector by \(W\) just picks row \(i\): this is an embedding lookup (Session 1, formula 1.4). We train \(W\) with the cross-entropy loss (5.2) and gradient descent. From Session 2, the gradient for row \(i\) is “probabilities minus targets”, summed over all examples that start with \(i\):

\[ \frac{\partial \mathcal{L}}{\partial W_{ij}} = \frac{1}{|D|} \Big( n_i \, P_W(j \mid i) - N_{ij} \Big), \qquad n_i = \sum_m N_{im} \tag{5.5} \]

The gradient is zero exactly when \(P_W(j \mid i) = N_{ij} / n_i\), the counting answer. So training the network converges to the counting model, and the best weights are the log-counts: \(W_{ij} = \log N_{ij} + c_i\). Two methods, one answer, because both are maximum likelihood.

Regularization is smoothing

Add a small penalty on the weights, \(\lambda \cdot \operatorname{mean}(W^2)\). It pulls every \(W_{ij}\) toward 0, and \(W = 0\) means uniform probabilities. So the penalty pushes rows toward uniform, exactly like add-\(k\) smoothing. Weight decay in an LLM (Session 4) is the same idea at a much larger scale.

Figure 3. Training the neural bigram on 3,000 names in your browser, with full-batch gradient descent on formula (5.5). The dashed line is the loss of the counting model with add-1 smoothing on the same data: the network approaches it.
How to read Figure 3

The first step starts near 3.7, worse than uniform (3.30), because the random weights are confidently wrong. A large learning rate works here because the problem is simple and the gradient (5.5) is small. Raise \(\lambda\) to 1 or 2 and the loss can no longer reach the dashed line: the penalty pulls the model toward uniform, like heavy smoothing. The names under the chart are sampled from the current weights.

The full PyTorch program

Here is the whole lab in one script. It is the reference solution; type it yourself first.

import torch, torch.nn.functional as F
torch.manual_seed(1337)
words = open("names.txt").read().split()
chars = sorted(set("".join(words)))
stoi = {c: i + 1 for i, c in enumerate(chars)}; stoi["."] = 0
itos = {i: c for c, i in stoi.items()}
V = len(stoi)                                         # 27

# ---- data: (current char, next char) pairs, with "." as start and end ----
xs, ys = [], []
for w in words:
    cs = ["."] + list(w) + ["."]
    for a, b in zip(cs, cs[1:]):
        xs.append(stoi[a]); ys.append(stoi[b])
xs, ys = torch.tensor(xs), torch.tensor(ys)            # int64, 228,146 each

# ---- model 1: counting, formulas (5.1) and (5.3) ----
N = torch.zeros((V, V), dtype=torch.int64)
N.index_put_((xs, ys), torch.ones_like(xs), accumulate=True)
P = (N + 1).float()
P = P / P.sum(dim=1, keepdim=True)                    # keepdim=True is essential (see Common mistakes)
nll = -P[xs, ys].log().mean()
print(f"counting model NLL: {nll.item():.4f}")         # 2.4546

# ---- sampling ----
g = torch.Generator().manual_seed(2147483647)
for _ in range(5):
    ix, out = 0, []
    while True:
        ix = torch.multinomial(P[ix], num_samples=1, generator=g).item()
        if ix == 0: break
        out.append(itos[ix])
    print("".join(out))

# ---- model 2: neural bigram, formula (5.4), trained with cross-entropy ----
W = torch.randn((V, V), generator=g, requires_grad=True)
for step in range(300):
    logits = W[xs]                                     # same as F.one_hot(xs, V).float() @ W
    loss = F.cross_entropy(logits, ys) + 0.01 * (W ** 2).mean()
    W.grad = None
    loss.backward()
    with torch.no_grad():
        W -= 50 * W.grad                               # plain gradient descent
    if step % 50 == 0:
        print(step, round(loss.item(), 4))             # falls from about 3.7 toward 2.48

Our NumPy replica of this exact training loop (same initialization scheme, learning rate 50, \(\lambda = 0.01\)) gave losses 3.75, 2.68, 2.51, 2.49 and 2.48 at steps 0, 10, 50, 100 and 300. Your numbers will be close but not identical, because the random stream differs. The loss keeps creeping toward the counting model’s value; the small gap left is the regularization term.

Pseudo-code

Algorithm 1: counting bigram model
  1. input: words, smoothing \(k\)
  2. \(N \leftarrow\) zeros\((27, 27)\)
  3. for each word \(w\) do
  4. \(c \leftarrow [\,.\,] + \text{chars}(w) + [\,.\,]\)
  5. for each pair \((a, b)\) of neighbors in \(c\) do \(N_{ab} \leftarrow N_{ab} + 1\)
  6. \(P_{ij} \leftarrow (N_{ij} + k) / \sum_m (N_{im} + k)\) // formula (5.3): divide each ROW
  7. return \(P\)
Algorithm 2: neural bigram, full-batch gradient descent
  1. input: pairs \((x_n, y_n)\), learning rate \(\eta\), penalty \(\lambda\), steps \(S\)
  2. \(W \leftarrow\) random normal \((27, 27)\)
  3. for \(s = 1\) to \(S\) do
  4. \(\mathcal{L} \leftarrow -\frac{1}{n}\sum_n \log \softmax(W[x_n])_{y_n} + \lambda\,\operatorname{mean}(W^2)\)
  5. \(G \leftarrow \nabla_W \mathcal{L}\) // formula (5.5) plus \(2\lambda W / 729\)
  6. \(W \leftarrow W - \eta\, G\)
  7. return \(W\) // \(\softmax(W[i]) \approx\) row \(i\) of the counting model

Where this is used

  • A baseline for every language model. Before training a large model, compute the unigram and bigram losses on your data. If your big model does not beat them quickly, something is wrong.
  • Character-level priors for OCR. Reading jersey numbers or player names from video frames gives noisy characters. A small character or n-gram model (or a fixed roster, which is a much stronger prior) can rescore the OCR candidates.
  • Classic speech recognition and keyboards used n-gram models for decades; they are still used where memory is tiny.
  • Data quality. Text that a simple n-gram model finds very surprising is often noise or another language; this is the idea behind perplexity filtering (session 34).

Common mistakes

  • The broadcasting bug. P = N / N.sum(1) without keepdim=True: the sum has shape (27,), which broadcasts as a row (1, 27), so you divide each column by the wrong number. The code runs, the rows do not sum to 1, and the model is silently wrong. Always check P.sum(1) equals 1.
  • Forgetting the start and end token. Without it, the model cannot learn how names begin or when to stop.
  • Using float targets. F.cross_entropy wants class indices as int64, not one-hot floats (unless you pass probabilities on purpose).
  • Not zeroing gradients. They accumulate across backward() calls.
  • Updating weights inside the graph. A manual update must be inside torch.no_grad(), or autograd records it.
  • Sampling without a generator. Results then depend on everything else that used the global random state.
  • Measuring on training data only. Keep a validation split, even for small models.

Interview questions

Why does the neural bigram converge to the counting model?core
Both minimize the same negative log-likelihood. With one-hot inputs, each row of \(W\) is free, and the gradient for row \(i\) is \(n_i P_W(\cdot\mid i) - N_{i\cdot}\), which is zero exactly when \(P_W\) equals the empirical frequencies. So the optimum is \(\softmax(W_i) = N_i/n_i\), the counting estimate; the weights are log-counts up to a constant per row.
What is the effect of L2 regularization on the neural bigram?core
It pulls the logits toward 0, which corresponds to uniform probabilities, so it acts like smoothing: it adds probability to unseen pairs and flattens rows. Stronger \(\lambda\) is like larger \(k\) in add-\(k\) smoothing. It trades a little training loss for robustness on unseen data.
What does loss.backward() actually do in PyTorch?basic
It traverses the recorded graph from loss back to the leaf tensors with requires_grad=True, applying each node’s backward function (the chain rule), and accumulates the results into each leaf’s .grad. By default it then frees the graph’s saved tensors, so you cannot call it twice without retain_graph=True.
What is the difference between view and reshape?basic
view always returns a view sharing memory and fails if the tensor is not contiguous in a compatible way. reshape returns a view when possible and copies otherwise. After transpose or permute, a tensor is usually non-contiguous, so view raises an error.
Explain the N / N.sum(1) bug.core
N.sum(1) has shape (27,). Broadcasting aligns from the right, so it acts like (1, 27) and divides column \(j\) by the sum of row \(j\), not row \(i\). Use keepdim=True to get shape (27, 1), which broadcasts down the rows. Check by asserting each row sums to 1.
How do you make a PyTorch training run reproducible?core
Seed Python, NumPy and torch; use explicit torch.Generator objects for sampling and data shuffling; set worker seeds in DataLoaders; turn on torch.use_deterministic_algorithms(True) (and the cuBLAS workspace setting on CUDA); pin library versions. Even then, results can differ across GPU types or library versions.
Why is the bigram loss 2.45 and not lower? How would you lower it?basic
The model sees only one previous character, so it cannot use longer patterns (“-ian”, “-elle”). The data’s entropy given one character is the floor for any bigram model. To go lower, condition on more context: trigrams (sparser tables), an MLP over several character embeddings, or a transformer. That is the path of the next sessions.

Exercises

  1. Build the trigram model (condition on the previous two characters). Report training and validation loss with add-1 smoothing. Is it better than 2.454? Does smoothing matter more or less than for bigrams, and why?
  2. Find the best \(k\) for add-\(k\) smoothing on the validation split.
  3. Replace W[xs] with F.one_hot(xs, 27).float() @ W and check the loss is identical. Which is faster, and why?
  4. Train the neural bigram with torch.optim.AdamW instead of manual updates. What learning rate do you need, and why is it so different from 50?
  5. Write the broadcasting bug on purpose, print P.sum(1), and compute the loss. How much worse is it?
  6. Sample 1,000 names at temperatures 0.5, 1 and 1.5. Measure how many are in the training set. What do you see?
Hintscheck
1. The table has \(27^3 = 19{,}683\) cells, most of them rare or empty, so smoothing matters much more; the loss should fall below the bigram value on training data, and validation tells you whether it really helps. 3. Indexing avoids a \(n \times 27\) matrix multiplication with mostly zeros. 4. Adam’s step size is in weight units (Session 4), so values around 0.1 to 1 work, not 50. 6. Low temperature produces common, repeated names; high temperature produces more new and stranger ones.

Key takeaways

  • Every tensor has a shape, a dtype and a device; views share memory; autograd records operations and backward() accumulates gradients into leaves.
  • The counting bigram (5.1) is the maximum-likelihood model; smoothing (5.3) removes zero probabilities.
  • The neural bigram (5.4) with one-hot input is an embedding lookup; its gradient (5.5) is zero exactly at the counting answer, so both reach about 2.45 nats (perplexity about 11.6, against 27 for uniform).
  • Weight regularization plays the role of smoothing. Always check shapes (keepdim=True) and keep a validation split.

References for this session

  1. Karpathy (2023). Neural Networks: Zero to Hero (video course)
  2. Raschka, Build a Large Language Model (From Scratch), chapter A: Introduction to PyTorch
  3. Karpathy (2023). Neural Networks: Zero to Hero (video course)
  4. Raschka, Build a Large Language Model (From Scratch), chapter A: Introduction to PyTorch
  5. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, chapter 3: N-gram Language Models