Track 3 · sessions 31–52
Pretraining and scale
Objectives, data, scaling laws, number formats, GPU memory, FSDP, tensor and pipeline parallelism.
Pretraining
Objectives, data, scaling laws, number formats and memory math.
31Ready
Pretraining objectives
- Causal LM (next token) and its loss
- Masked LM (BERT): predict hidden tokens using both sides
- Span corruption (T5) and prefix-LM
- Why next-token prediction learns so much: the compression view
32Ready
One training step, end to end: forward, loss, backward, update
- Follow one batch: tokens → embeddings → blocks → logits → loss
- Backward: where every gradient comes from, layer by layer
- Optimizer update, gradient clipping, learning-rate schedule
- What lives in GPU memory at each moment of the step
33Ready
Encoders for understanding: BERT, RoBERTa, ALBERT and the CLS token
- BERT input: [CLS] sentence A [SEP] sentence B [SEP]; MLM + next-sentence prediction
- RoBERTa: more data, dynamic masking, no NSP, bigger batches
- ALBERT: cross-layer parameter sharing and factorized embeddings
- The CLS token vs mean pooling; a classification head is one linear layer + softmax
34Ready
Pretraining data: from web crawl to training mixture
- Sources: web, books, code, papers, math, many languages
- Cleaning and filtering: language ID, quality classifiers, rules
- Deduplication: exact hashes and MinHash for near-duplicates
- Data mixture and curriculum; synthetic data; contamination risk
35Ready
Scaling laws
- Loss falls as a power law in parameters, data and compute
- Kaplan vs Chinchilla: why the best tokens-per-parameter changed (about 20)
- Compute-optimal vs inference-optimal: train small models on much more data
- Using scaling laws to plan a training run
36Ready
Training recipe and stability
- Initialization, and why scale matters
- Warmup, cosine and warmup-stable-decay schedules
- Loss spikes: causes and fixes (gradient clipping, z-loss, QK-norm, lower LR)
- μP: tune on a small model, transfer the settings to a big one
37Ready
Numbers on GPUs: FP32, TF32, FP16, BF16, FP8 and mixed precision
- Sign, exponent and mantissa bits: range vs precision
- Why FP16 needs loss scaling and BF16 usually does not
- Mixed precision: low-precision math, FP32 master weights, FP32 accumulation
- FP8 training basics (E4M3 and E5M2) and scaling factors
38Ready
Memory math for training
- Weights, gradients, optimizer states: 16 bytes per parameter for mixed-precision Adam
- Activation memory: how it grows with batch, sequence length and layers
- Why full fine-tuning of a 7B model does not fit on one 24 GB GPU
- Where memory really goes: fragmentation, buffers, temporary tensors
39Ready
Gradient accumulation and gradient checkpointing
- Gradient accumulation: many small batches, one optimizer step; the effective batch size
- Scale the loss correctly across steps (and the token-count trap)
- Gradient (activation) checkpointing: drop activations, recompute them in backward
- Selective recomputation; the cost is about one extra forward pass
40Ready
Evaluating base models
- Perplexity is only comparable with the same tokenizer and data
- Benchmarks (MMLU, HellaSwag, GSM8K) and how scoring works: log-likelihood vs generation
- Contamination: when test data leaks into training
- Using an evaluation harness correctly
41Ready
measure memory, accumulation and checkpointing
- Profile peak memory of a ~1B model at different batch sizes
- Turn on BF16, checkpointing and accumulation; record speed and memory
- Check that accumulation gives the same gradients as one big batch
- Write a memory calculator and compare it with your measurements
GPUs and distributed training
How GPUs work, and how training is split over many of them: DDP, ZeRO, FSDP, FSDP2, TP, PP, CP.
42Ready
GPU architecture for ML
- SMs, warps and tensor cores
- Memory hierarchy: registers, shared memory (SRAM), L2, HBM or GDDR
- The roofline: compute-bound vs memory-bound; arithmetic intensity
- MFU: how much of the GPU you really use
43Ready
Collective communication: all-reduce, reduce-scatter, all-gather
- What each collective does, with pictures
- Ring all-reduce = reduce-scatter + all-gather; its cost
- NCCL; NVLink vs PCIe vs InfiniBand and Ethernet
- Overlapping communication with compute
44Ready
Data parallelism (DDP)
- Copy the model to every GPU, split the batch, average the gradients
- Gradient bucketing and overlap with the backward pass
- The limit: every GPU holds the full model and optimizer state
- Seeds and determinism across ranks
45Ready
ZeRO stages 1, 2 and 3
- Stage 1 shards optimizer states; stage 2 adds gradients; stage 3 adds parameters
- Memory per GPU for each stage
- Communication cost vs memory saved
- Offloading to CPU or NVMe
46Ready
FSDP (version 1)
- FlatParameter: flatten and shard each wrapped unit
- Forward: all-gather → compute → free. Backward: all-gather → gradients → reduce-scatter
- Wrapping policy, prefetching, sharding strategies (FULL_SHARD, SHARD_GRAD_OP, HYBRID_SHARD)
- Mixed precision and CPU offload options
47Ready
FSDP2: per-parameter sharding with DTensor
- The fully_shard API: each parameter is a DTensor sharded on dim 0
- Why it is simpler than FlatParameter: clean state dicts, easy freezing and LoRA
- MixedPrecisionPolicy, reshard_after_forward, prefetching; works with torch.compile and tensor parallel
- Moving from FSDP1 to FSDP2; common bugs
48Ready
Tensor parallelism and sequence parallelism
- Megatron-style column-parallel and row-parallel linear layers
- Attention heads split across GPUs; two all-reduces per layer in forward (after attention, after MLP)
- Sequence parallelism for the LayerNorm and dropout regions
- Why TP needs fast NVLink and usually stays inside one node
49Ready
Pipeline, context and expert parallelism; 3D and 4D parallel
- Pipeline parallel: split layers into stages; micro-batches; GPipe vs 1F1B; bubbles
- Context parallel: split the sequence (ring attention, Ulysses)
- Expert parallel for MoE (all-to-all)
- How big runs combine DP × TP × PP × CP
50Ready
Making training fast: compile, CUDA graphs, fused kernels, profiling
- torch.compile: graph capture, fusion, recompiles and graph breaks
- CUDA graphs remove kernel-launch overhead
- Fused kernels: attention, RMSNorm, cross-entropy
- Reading a PyTorch profiler or Nsight Systems trace: idle gaps and memory copies
51Ready
Distributed checkpoints, fault tolerance and MFU
- Sharded (distributed) checkpoints: save and load with many GPUs
- Failures at scale: restart, elastic training, silent data corruption
- Model FLOPs utilization (MFU): how to compute and read it
- Overlapping communication with compute
52Ready
DDP vs FSDP2 and reading a profile
- Train the same model with DDP and with FSDP2 (2 or more GPUs; or a small simulation)
- Compare memory per GPU, step time and communication
- Capture a profiler trace and find the top 3 time sinks
- Combine activation checkpointing with FSDP2