LLM & VLM courseBlog
Track 2 · sessions 16–30

Attention and transformers

Query, key and value; masks; multi-head attention; the full transformer; MoE; long context.

Attention and the transformer

Query, key and value; every attention type; the full transformer; MoE and long context.

16

Attention intuition: query, key, value

  • Attention as a soft dictionary lookup
  • Query = what I look for; key = the label I show; value = what I give if chosen
  • Why three different projections W_{Q}, W_{K}, W_{V}
  • Attention weights as a probability distribution over positions
Ready
17

Scaled dot-product attention: the math and the shapes

  • Q = XW_{Q}, K = XW_{K}, V = XW_{V}, each of shape (T, d_{k})
  • Scores QKᵀ, scale by √d_{k}, softmax over keys, weighted sum of V
  • Why √d_{k}: the variance of a dot product grows with d_{k} and saturates softmax
  • Cost: O(T²·d) compute and O(T²) memory for the score matrix
Ready
18

Masks: causal, padding, bidirectional, prefix and packing masks

  • Causal mask: a token sees only itself and the past
  • Padding mask for batches with different lengths
  • Bidirectional (BERT) vs causal (GPT) vs prefix-LM (partly bidirectional)
  • Document masks when many samples are packed into one sequence
Ready
19

Multi-head attention

  • Split d_{model} into h heads of size d_{k} = d_{model}/h
  • Each head can learn a different relation: syntax, coreference, position
  • Concatenate the heads, then the output projection W_{O}
  • Parameters: 4·d_{model}² per attention layer (without biases)
Ready
20

Self-attention vs cross-attention

  • Self-attention: Q, K and V come from the same sequence
  • Cross-attention: Q from one sequence (text), K and V from another (encoder output, image, audio)
  • Encoder–decoder attention in translation and speech recognition
  • Why cross-attention is the base of some VLM designs (Flamingo, Llama 3.2 Vision)
Ready
21

The feed-forward network (MLP) inside each block

  • Expand (d → 4d), non-linearity, project back (4d → d)
  • GELU, and gated units (GLU, SwiGLU) in modern LLMs
  • The FFN as a key–value memory where many facts seem to be stored
  • About 2/3 of the parameters of a dense block are in the FFN
Ready
22

The residual stream and normalization

  • Residual connections: each layer adds a small edit to a shared stream
  • LayerNorm vs RMSNorm: what each removes, and why RMSNorm is cheaper
  • Post-LN (original) vs Pre-LN (modern): stability and warmup
  • QK-norm and other fixes for growing attention logits
Ready
23

The full transformer and its three families

  • Encoder-only (BERT): understanding tasks, embeddings, classification
  • Decoder-only (GPT, Llama, Qwen): generation and chat; almost all modern LLMs
  • Encoder–decoder (T5, Whisper): input-to-output tasks like translation and speech
  • How to choose for a task, and why decoder-only won at scale
Ready
24

Encoder–decoder in depth: T5 span corruption, sentinel tokens, BART denoising

  • T5: every task is text-to-text; span corruption with sentinel tokens <X>, <Y>
  • How the decoder uses cross-attention to the encoder at every step
  • BART: corrupt the text (mask, delete, shuffle), then rebuild it
  • When encoder–decoders still win: translation, summarization, speech
Ready
25

Counting: parameters, FLOPs and memory of a transformer

  • Parameters per layer ≈ 12·d² (attention 4d² + FFN 8d²), plus embeddings
  • Forward ≈ 2N FLOPs per token; training ≈ 6N FLOPs per token, so 6ND in total
  • The extra attention term grows with context length
  • Worked example: estimate an 8B model, then check it against its config file
Ready
26

Attention variants 1: MHA, MQA, GQA, MLA and the KV cache

  • MQA: all heads share one K and one V
  • GQA: groups of heads share K and V; the standard choice today
  • MLA (DeepSeek): compress K and V into a small latent vector
  • Effect on KV-cache memory and on quality
Ready
27

Attention variants 2: a first look at sliding windows, linear attention and SSMs

  • Sliding-window attention (Mistral) and local + global patterns
  • Linear attention: the kernel trick, O(T) cost, and what it loses
  • State space models (Mamba) as a recurrent alternative
  • Attention sinks. All of these get a full deep dive in Part C.
Ready
28

What “context” means, and how to make it longer

  • Context window vs training length vs effective (really usable) context
  • Extending RoPE: position interpolation, NTK-aware scaling, YaRN
  • Lost in the middle; needle-in-a-haystack and RULER tests
  • Context vs memory vs retrieval: three ways to give a model more information
Ready
29

Mixture of Experts (MoE)

  • A router picks the top-k experts for each token; only those FFNs run
  • Total vs active parameters: cheap per token, heavy in memory
  • Load-balancing loss, capacity factor, dropped tokens, shared experts
  • A first look at expert parallelism
Ready
30

write a GPT from scratch and train it

  • Build embeddings, causal multi-head attention, FFN, RMSNorm, residuals, LM head
  • Count the parameters and compare with the formula from session 24
  • Train on a small corpus; plot train and validation loss; sample with temperature
  • Break it on purpose: remove positions, remove √d, use Post-LN. Explain what fails.
Ready