LLM & VLM courseBlog
Track 1 · sessions 1–15

Foundations

Math, PyTorch, next-token prediction, tokens, embeddings and positions.

Start here: math and PyTorch toolkit

Refresh the math and PyTorch you need, and see the whole course once.

1

What is a large language model? The big picture

  • An LLM is a function that gives a probability for the next token
  • The life of a model: pretrain → post-train → serve → use
  • Why one model can do many tasks: classification, summarization, chat
  • Where images and video join the same picture
Ready
2

Math you need, part 1: vectors, dot products, matrices and softmax

  • Vectors as lists of features; the dot product as a similarity score
  • Matrix multiply = many dot products at once; the shape rule (B, T, d)
  • Softmax turns scores into probabilities; temperature makes it sharp or flat
  • Numerical stability: subtract the max before exp
Ready
3

Math you need, part 2: probability, cross-entropy, KL divergence and perplexity

  • Maximum likelihood = minimize the negative log-likelihood
  • Cross-entropy between data and model; KL as the extra cost of a wrong model
  • Perplexity = exp(average loss)
  • Nats vs bits; a loss of 2.0 nats means perplexity ≈ 7.4
Ready
4

Optimization: backpropagation, SGD, Adam and AdamW, learning-rate schedules

  • Backprop = the chain rule run backwards on a graph
  • SGD vs momentum vs Adam; why AdamW separates weight decay
  • Warmup + cosine decay, and warmup-stable-decay (WSD)
  • How batch size, learning rate and gradient noise interact
Ready
5

PyTorch mental model and a tiny bigram language model

  • Tensors, dtypes, devices, the autograd graph, nn.Module
  • A clean, deterministic training loop: seeds, eval mode, no_grad
  • Train a bigram and a small MLP language model on short text
  • Read the loss curve, sample text, compute perplexity
Ready

Language modeling and tokens

How text becomes numbers: next-token prediction, tokenizers and special tokens.

6

Language modeling: next-token prediction, from n-grams to neural models

  • Chain rule: the probability of a text is a product of next-token probabilities
  • Autoregressive generation: predict, append, repeat
  • n-gram models as the simple baseline, and why they fail (sparsity, short memory)
  • Teacher forcing in training vs free running at inference
Ready
7

Before transformers: RNN, LSTM, seq2seq and the first attention

  • RNN hidden state as a running summary; vanishing gradients
  • LSTM gates as memory control
  • Seq2seq encoder–decoder and its one-vector bottleneck
  • Bahdanau attention: the decoder looks back at all encoder states. Transformers keep this idea.
Ready
8

What is a token? Subword tokenization (BPE, WordPiece, Unigram)

  • Characters vs words vs subwords: vocabulary size vs sequence length
  • Byte-level BPE step by step: count pairs, merge the most common, repeat
  • WordPiece and Unigram (SentencePiece): how they differ
  • Effects of vocabulary size: embedding size, output layer cost, tokens per word
Ready
9

Special tokens, chat templates and tokenizer pitfalls

  • BOS, EOS, PAD, UNK; BERT’s CLS, SEP, MASK: what each one is for
  • Chat templates: role tokens (system, user, assistant), tool-call tokens, thinking tokens
  • Placeholder tokens for images, audio and video in multimodal models
  • Pitfalls: numbers, spaces, glitch tokens, and multilingual cost (Farsi or Arabic often need more tokens per word than English)
Ready
10

build a byte-level BPE tokenizer from scratch

  • Train BPE on a small corpus; check that encode then decode gives the same text
  • Compare with tiktoken and a Hugging Face tokenizer
  • Measure tokens per word in English and in another language you speak
  • Read a real chat template and find its special tokens
Ready

Embeddings and positions

How tokens become vectors, and how the model knows word order.

11

Embeddings: turning token IDs into vectors

  • Embedding layer = one-hot × matrix = table lookup
  • word2vec intuition: words in similar contexts get similar vectors
  • Cosine similarity and the geometry of meaning, and its limits
  • Embedding size and parameter count: |V| × d
Ready
12

Contextual embeddings, hidden states and the output layer

  • Static vs contextual: “bank” (river) and “bank” (money) get different hidden states
  • Hidden states layer by layer; what “the embedding of a sentence” can mean
  • Unembedding: hidden state → logits over the vocabulary; weight tying
  • Sentence embeddings from a model: CLS, mean pooling, last token
Ready
13

Position, part 1: why order needs help; absolute and relative positions

  • Self-attention ignores order: without positions, “dog bites man” = “man bites dog”
  • Sinusoidal and learned absolute positions
  • Relative positions: T5 bias buckets and ALiBi linear bias
  • What happens past the training length
Ready
14

Position, part 2: RoPE in depth, and a first look at 2D and 3D RoPE

  • Rotate each pair of dimensions by an angle that grows with position
  • Why the score depends only on the distance m − n
  • Frequencies θ_{i}, the base (10000 or larger), long and short wavelengths
  • Preview: 2D RoPE for image patches and M-RoPE for video (time, height, width)
Ready
15

embeddings and RoPE by hand

  • Plot trained embeddings with PCA; find nearest neighbors
  • Implement RoPE; test with numbers that scores depend only on relative position
  • Plot RoPE frequencies; see which dimensions handle long and short distances
Ready