LLM & VLM courseBlog
Track 3 · sessions 31–52

Pretraining and scale

Objectives, data, scaling laws, number formats, GPU memory, FSDP, tensor and pipeline parallelism.

Pretraining

Objectives, data, scaling laws, number formats and memory math.

31

Pretraining objectives

  • Causal LM (next token) and its loss
  • Masked LM (BERT): predict hidden tokens using both sides
  • Span corruption (T5) and prefix-LM
  • Why next-token prediction learns so much: the compression view
Ready
32

One training step, end to end: forward, loss, backward, update

  • Follow one batch: tokens → embeddings → blocks → logits → loss
  • Backward: where every gradient comes from, layer by layer
  • Optimizer update, gradient clipping, learning-rate schedule
  • What lives in GPU memory at each moment of the step
Ready
33

Encoders for understanding: BERT, RoBERTa, ALBERT and the CLS token

  • BERT input: [CLS] sentence A [SEP] sentence B [SEP]; MLM + next-sentence prediction
  • RoBERTa: more data, dynamic masking, no NSP, bigger batches
  • ALBERT: cross-layer parameter sharing and factorized embeddings
  • The CLS token vs mean pooling; a classification head is one linear layer + softmax
Ready
34

Pretraining data: from web crawl to training mixture

  • Sources: web, books, code, papers, math, many languages
  • Cleaning and filtering: language ID, quality classifiers, rules
  • Deduplication: exact hashes and MinHash for near-duplicates
  • Data mixture and curriculum; synthetic data; contamination risk
Ready
35

Scaling laws

  • Loss falls as a power law in parameters, data and compute
  • Kaplan vs Chinchilla: why the best tokens-per-parameter changed (about 20)
  • Compute-optimal vs inference-optimal: train small models on much more data
  • Using scaling laws to plan a training run
Ready
36

Training recipe and stability

  • Initialization, and why scale matters
  • Warmup, cosine and warmup-stable-decay schedules
  • Loss spikes: causes and fixes (gradient clipping, z-loss, QK-norm, lower LR)
  • μP: tune on a small model, transfer the settings to a big one
Ready
37

Numbers on GPUs: FP32, TF32, FP16, BF16, FP8 and mixed precision

  • Sign, exponent and mantissa bits: range vs precision
  • Why FP16 needs loss scaling and BF16 usually does not
  • Mixed precision: low-precision math, FP32 master weights, FP32 accumulation
  • FP8 training basics (E4M3 and E5M2) and scaling factors
Ready
38

Memory math for training

  • Weights, gradients, optimizer states: 16 bytes per parameter for mixed-precision Adam
  • Activation memory: how it grows with batch, sequence length and layers
  • Why full fine-tuning of a 7B model does not fit on one 24 GB GPU
  • Where memory really goes: fragmentation, buffers, temporary tensors
Ready
39

Gradient accumulation and gradient checkpointing

  • Gradient accumulation: many small batches, one optimizer step; the effective batch size
  • Scale the loss correctly across steps (and the token-count trap)
  • Gradient (activation) checkpointing: drop activations, recompute them in backward
  • Selective recomputation; the cost is about one extra forward pass
Ready
40

Evaluating base models

  • Perplexity is only comparable with the same tokenizer and data
  • Benchmarks (MMLU, HellaSwag, GSM8K) and how scoring works: log-likelihood vs generation
  • Contamination: when test data leaks into training
  • Using an evaluation harness correctly
Ready
41

measure memory, accumulation and checkpointing

  • Profile peak memory of a ~1B model at different batch sizes
  • Turn on BF16, checkpointing and accumulation; record speed and memory
  • Check that accumulation gives the same gradients as one big batch
  • Write a memory calculator and compare it with your measurements
Ready

GPUs and distributed training

How GPUs work, and how training is split over many of them: DDP, ZeRO, FSDP, FSDP2, TP, PP, CP.

42

GPU architecture for ML

  • SMs, warps and tensor cores
  • Memory hierarchy: registers, shared memory (SRAM), L2, HBM or GDDR
  • The roofline: compute-bound vs memory-bound; arithmetic intensity
  • MFU: how much of the GPU you really use
Ready
43

Collective communication: all-reduce, reduce-scatter, all-gather

  • What each collective does, with pictures
  • Ring all-reduce = reduce-scatter + all-gather; its cost
  • NCCL; NVLink vs PCIe vs InfiniBand and Ethernet
  • Overlapping communication with compute
Ready
44

Data parallelism (DDP)

  • Copy the model to every GPU, split the batch, average the gradients
  • Gradient bucketing and overlap with the backward pass
  • The limit: every GPU holds the full model and optimizer state
  • Seeds and determinism across ranks
Ready
45

ZeRO stages 1, 2 and 3

  • Stage 1 shards optimizer states; stage 2 adds gradients; stage 3 adds parameters
  • Memory per GPU for each stage
  • Communication cost vs memory saved
  • Offloading to CPU or NVMe
Ready
46

FSDP (version 1)

  • FlatParameter: flatten and shard each wrapped unit
  • Forward: all-gather → compute → free. Backward: all-gather → gradients → reduce-scatter
  • Wrapping policy, prefetching, sharding strategies (FULL_SHARD, SHARD_GRAD_OP, HYBRID_SHARD)
  • Mixed precision and CPU offload options
Ready
47

FSDP2: per-parameter sharding with DTensor

  • The fully_shard API: each parameter is a DTensor sharded on dim 0
  • Why it is simpler than FlatParameter: clean state dicts, easy freezing and LoRA
  • MixedPrecisionPolicy, reshard_after_forward, prefetching; works with torch.compile and tensor parallel
  • Moving from FSDP1 to FSDP2; common bugs
Ready
48

Tensor parallelism and sequence parallelism

  • Megatron-style column-parallel and row-parallel linear layers
  • Attention heads split across GPUs; two all-reduces per layer in forward (after attention, after MLP)
  • Sequence parallelism for the LayerNorm and dropout regions
  • Why TP needs fast NVLink and usually stays inside one node
Ready
49

Pipeline, context and expert parallelism; 3D and 4D parallel

  • Pipeline parallel: split layers into stages; micro-batches; GPipe vs 1F1B; bubbles
  • Context parallel: split the sequence (ring attention, Ulysses)
  • Expert parallel for MoE (all-to-all)
  • How big runs combine DP × TP × PP × CP
Ready
50

Making training fast: compile, CUDA graphs, fused kernels, profiling

  • torch.compile: graph capture, fusion, recompiles and graph breaks
  • CUDA graphs remove kernel-launch overhead
  • Fused kernels: attention, RMSNorm, cross-entropy
  • Reading a PyTorch profiler or Nsight Systems trace: idle gaps and memory copies
Ready
51

Distributed checkpoints, fault tolerance and MFU

  • Sharded (distributed) checkpoints: save and load with many GPUs
  • Failures at scale: restart, elastic training, silent data corruption
  • Model FLOPs utilization (MFU): how to compute and read it
  • Overlapping communication with compute
Ready
52

DDP vs FSDP2 and reading a profile

  • Train the same model with DDP and with FSDP2 (2 or more GPUs; or a small simulation)
  • Compare memory per GPU, step time and communication
  • Capture a profiler trace and find the top 3 time sinks
  • Combine activation checkpointing with FSDP2
Ready