Track 1 · sessions 1–15
Foundations
Math, PyTorch, next-token prediction, tokens, embeddings and positions.
Start here: math and PyTorch toolkit
Refresh the math and PyTorch you need, and see the whole course once.
1Ready
What is a large language model? The big picture
- An LLM is a function that gives a probability for the next token
- The life of a model: pretrain → post-train → serve → use
- Why one model can do many tasks: classification, summarization, chat
- Where images and video join the same picture
2Ready
Math you need, part 1: vectors, dot products, matrices and softmax
- Vectors as lists of features; the dot product as a similarity score
- Matrix multiply = many dot products at once; the shape rule (B, T, d)
- Softmax turns scores into probabilities; temperature makes it sharp or flat
- Numerical stability: subtract the max before exp
3Ready
Math you need, part 2: probability, cross-entropy, KL divergence and perplexity
- Maximum likelihood = minimize the negative log-likelihood
- Cross-entropy between data and model; KL as the extra cost of a wrong model
- Perplexity = exp(average loss)
- Nats vs bits; a loss of 2.0 nats means perplexity ≈ 7.4
4Ready
Optimization: backpropagation, SGD, Adam and AdamW, learning-rate schedules
- Backprop = the chain rule run backwards on a graph
- SGD vs momentum vs Adam; why AdamW separates weight decay
- Warmup + cosine decay, and warmup-stable-decay (WSD)
- How batch size, learning rate and gradient noise interact
5Ready
PyTorch mental model and a tiny bigram language model
- Tensors, dtypes, devices, the autograd graph, nn.Module
- A clean, deterministic training loop: seeds, eval mode, no_grad
- Train a bigram and a small MLP language model on short text
- Read the loss curve, sample text, compute perplexity
Language modeling and tokens
How text becomes numbers: next-token prediction, tokenizers and special tokens.
6Ready
Language modeling: next-token prediction, from n-grams to neural models
- Chain rule: the probability of a text is a product of next-token probabilities
- Autoregressive generation: predict, append, repeat
- n-gram models as the simple baseline, and why they fail (sparsity, short memory)
- Teacher forcing in training vs free running at inference
7Ready
Before transformers: RNN, LSTM, seq2seq and the first attention
- RNN hidden state as a running summary; vanishing gradients
- LSTM gates as memory control
- Seq2seq encoder–decoder and its one-vector bottleneck
- Bahdanau attention: the decoder looks back at all encoder states. Transformers keep this idea.
8Ready
What is a token? Subword tokenization (BPE, WordPiece, Unigram)
- Characters vs words vs subwords: vocabulary size vs sequence length
- Byte-level BPE step by step: count pairs, merge the most common, repeat
- WordPiece and Unigram (SentencePiece): how they differ
- Effects of vocabulary size: embedding size, output layer cost, tokens per word
9Ready
Special tokens, chat templates and tokenizer pitfalls
- BOS, EOS, PAD, UNK; BERT’s CLS, SEP, MASK: what each one is for
- Chat templates: role tokens (system, user, assistant), tool-call tokens, thinking tokens
- Placeholder tokens for images, audio and video in multimodal models
- Pitfalls: numbers, spaces, glitch tokens, and multilingual cost (Farsi or Arabic often need more tokens per word than English)
10Ready
build a byte-level BPE tokenizer from scratch
- Train BPE on a small corpus; check that encode then decode gives the same text
- Compare with tiktoken and a Hugging Face tokenizer
- Measure tokens per word in English and in another language you speak
- Read a real chat template and find its special tokens
Embeddings and positions
How tokens become vectors, and how the model knows word order.
11Ready
Embeddings: turning token IDs into vectors
- Embedding layer = one-hot × matrix = table lookup
- word2vec intuition: words in similar contexts get similar vectors
- Cosine similarity and the geometry of meaning, and its limits
- Embedding size and parameter count: |V| × d
13Ready
Position, part 1: why order needs help; absolute and relative positions
- Self-attention ignores order: without positions, “dog bites man” = “man bites dog”
- Sinusoidal and learned absolute positions
- Relative positions: T5 bias buckets and ALiBi linear bias
- What happens past the training length
14Ready
Position, part 2: RoPE in depth, and a first look at 2D and 3D RoPE
- Rotate each pair of dimensions by an angle that grows with position
- Why the score depends only on the distance m − n
- Frequencies θ_{i}, the base (10000 or larger), long and short wavelengths
- Preview: 2D RoPE for image patches and M-RoPE for video (time, height, width)
15Ready
embeddings and RoPE by hand
- Plot trained embeddings with PCA; find nearest neighbors
- Implement RoPE; test with numbers that scores depend only on relative position
- Plot RoPE frequencies; see which dimensions handle long and short distances