Track 2 · sessions 16–30
Attention and transformers
Query, key and value; masks; multi-head attention; the full transformer; MoE; long context.
Attention and the transformer
Query, key and value; every attention type; the full transformer; MoE and long context.
16Ready
Attention intuition: query, key, value
- Attention as a soft dictionary lookup
- Query = what I look for; key = the label I show; value = what I give if chosen
- Why three different projections W_{Q}, W_{K}, W_{V}
- Attention weights as a probability distribution over positions
17Ready
Scaled dot-product attention: the math and the shapes
- Q = XW_{Q}, K = XW_{K}, V = XW_{V}, each of shape (T, d_{k})
- Scores QKᵀ, scale by √d_{k}, softmax over keys, weighted sum of V
- Why √d_{k}: the variance of a dot product grows with d_{k} and saturates softmax
- Cost: O(T²·d) compute and O(T²) memory for the score matrix
18Ready
Masks: causal, padding, bidirectional, prefix and packing masks
- Causal mask: a token sees only itself and the past
- Padding mask for batches with different lengths
- Bidirectional (BERT) vs causal (GPT) vs prefix-LM (partly bidirectional)
- Document masks when many samples are packed into one sequence
19Ready
Multi-head attention
- Split d_{model} into h heads of size d_{k} = d_{model}/h
- Each head can learn a different relation: syntax, coreference, position
- Concatenate the heads, then the output projection W_{O}
- Parameters: 4·d_{model}² per attention layer (without biases)
20Ready
Self-attention vs cross-attention
- Self-attention: Q, K and V come from the same sequence
- Cross-attention: Q from one sequence (text), K and V from another (encoder output, image, audio)
- Encoder–decoder attention in translation and speech recognition
- Why cross-attention is the base of some VLM designs (Flamingo, Llama 3.2 Vision)
21Ready
The feed-forward network (MLP) inside each block
- Expand (d → 4d), non-linearity, project back (4d → d)
- GELU, and gated units (GLU, SwiGLU) in modern LLMs
- The FFN as a key–value memory where many facts seem to be stored
- About 2/3 of the parameters of a dense block are in the FFN
22Ready
The residual stream and normalization
- Residual connections: each layer adds a small edit to a shared stream
- LayerNorm vs RMSNorm: what each removes, and why RMSNorm is cheaper
- Post-LN (original) vs Pre-LN (modern): stability and warmup
- QK-norm and other fixes for growing attention logits
23Ready
The full transformer and its three families
- Encoder-only (BERT): understanding tasks, embeddings, classification
- Decoder-only (GPT, Llama, Qwen): generation and chat; almost all modern LLMs
- Encoder–decoder (T5, Whisper): input-to-output tasks like translation and speech
- How to choose for a task, and why decoder-only won at scale
24Ready
Encoder–decoder in depth: T5 span corruption, sentinel tokens, BART denoising
- T5: every task is text-to-text; span corruption with sentinel tokens <X>, <Y>
- How the decoder uses cross-attention to the encoder at every step
- BART: corrupt the text (mask, delete, shuffle), then rebuild it
- When encoder–decoders still win: translation, summarization, speech
25Ready
Counting: parameters, FLOPs and memory of a transformer
- Parameters per layer ≈ 12·d² (attention 4d² + FFN 8d²), plus embeddings
- Forward ≈ 2N FLOPs per token; training ≈ 6N FLOPs per token, so 6ND in total
- The extra attention term grows with context length
- Worked example: estimate an 8B model, then check it against its config file
26Ready
Attention variants 1: MHA, MQA, GQA, MLA and the KV cache
- MQA: all heads share one K and one V
- GQA: groups of heads share K and V; the standard choice today
- MLA (DeepSeek): compress K and V into a small latent vector
- Effect on KV-cache memory and on quality
27Ready
Attention variants 2: a first look at sliding windows, linear attention and SSMs
- Sliding-window attention (Mistral) and local + global patterns
- Linear attention: the kernel trick, O(T) cost, and what it loses
- State space models (Mamba) as a recurrent alternative
- Attention sinks. All of these get a full deep dive in Part C.
28Ready
What “context” means, and how to make it longer
- Context window vs training length vs effective (really usable) context
- Extending RoPE: position interpolation, NTK-aware scaling, YaRN
- Lost in the middle; needle-in-a-haystack and RULER tests
- Context vs memory vs retrieval: three ways to give a model more information
29Ready
Mixture of Experts (MoE)
- A router picks the top-k experts for each token; only those FFNs run
- Total vs active parameters: cheap per token, heavy in memory
- Load-balancing loss, capacity factor, dropped tokens, shared experts
- A first look at expert parallelism
30Ready
write a GPT from scratch and train it
- Build embeddings, causal multi-head attention, FFN, RMSNorm, residuals, LM head
- Count the parameters and compare with the formula from session 24
- Train on a small corpus; plot train and validation loss; sample with temperature
- Break it on purpose: remove positions, remove √d, use Post-LN. Explain what fails.