Track 4 · sessions 53–77
Post-training and RL
SFT, LoRA and PEFT, distillation, RLHF, PPO, DPO, GRPO and reasoning models.
Post-training 1: SFT and PEFT
Turn a base model into a useful one: SFT, task fine-tuning, LoRA, QLoRA, distillation, embedding models.
53Ready
From base model to assistant
- What a base model does vs what an assistant does
- The stages: SFT → preference tuning → RL
- Instruction data: human-written, synthetic (self-instruct), distilled
- Chat templates, and why a wrong template breaks a model
54Ready
SFT in detail
- Loss only on the assistant tokens (loss masking)
- Packing many examples into one sequence with document masks
- Epochs, learning rate, sequence length; signs of overfitting
- Data quality vs quantity; diversity
55Ready
Fine-tuning for classic NLP tasks
- Sequence classification: sentiment (positive or negative documents), topic, intent
- Token classification: named entities, tagging
- Sequence-to-sequence: summarization, translation
- Encoder or decoder for each task; class imbalance; metrics (F1, ROUGE)
56Ready
The PEFT family: adapters, BitFit, prefix tuning, prompt tuning, P-Tuning v2
- Adapters: small bottleneck layers inside each block (Houlsby)
- BitFit: train only the biases
- Prompt tuning and prefix tuning: learn virtual tokens or per-layer key/value prefixes; P-Tuning and P-Tuning v2
- How to choose: prompting vs linear probe vs PEFT vs full fine-tuning (data size, latency, number of tasks)
57Ready
LoRA and other PEFT methods
- LoRA: freeze W and learn a low-rank update BA
- Rank r, the α/r scale, which layers to adapt (attention, MLP, all linear layers)
- Merging adapters for zero extra inference cost; serving many adapters
- DoRA, prefix tuning, prompt tuning: when they help
58Ready
QLoRA and training with quantized weights
- NF4 4-bit base weights with BF16 compute
- Double quantization of the scale factors
- Paged optimizers to survive memory spikes
- Quality vs memory, and when QLoRA is not enough
59Ready
Knowledge distillation
- Soft targets and temperature
- Logit distillation (KL) vs sequence-level distillation (train on teacher outputs)
- Distilling reasoning traces into small models
- When the student cannot copy the teacher (the capacity gap)
60Ready
Training embedding and retrieval models
- Contrastive learning: pull positives together, push negatives apart
- InfoNCE and temperature; in-batch negatives; hard negatives
- Bi-encoder training (Sentence-BERT, E5) and how to evaluate it
- The same idea as CLIP: our bridge to vision
61Ready
Forgetting, data mixing and model merging
- Catastrophic forgetting: why fine-tuning can break general skills
- Mix in general data, use a lower LR, or use LoRA as a regularizer
- Model merging: weight averaging, model soups, task vectors
- How to measure regressions
62Ready
LoRA vs QLoRA vs full fine-tuning
- Fine-tune a small LM for sentiment (positive vs negative documents)
- Fine-tune it for summarization; evaluate with ROUGE and an LLM judge
- Compare memory, speed and quality for full FT, LoRA and QLoRA
- Write down what you would ship, and why
Post-training 2: alignment and reinforcement learning
Policies, rewards and preferences: PPO, RLHF, DPO and its family, GRPO, reasoning models.
63Ready
Reinforcement learning words, for LLMs
- Agent, environment, state, action, reward, return, episode
- Text generation as RL: state = prompt + text so far, action = next token, policy = the LLM
- Sparse rewards that come only at the end of an answer
- Why RL and not only SFT: learn from a score, not from a copy
64Ready
Policy gradient and REINFORCE
- The log-derivative trick: ∇E[R] = E[R ∇log π]
- REINFORCE for a whole generated answer
- High variance, and baselines that add no bias
- Advantage = how much better than expected
65Ready
Value functions, advantage and GAE
- State value V(s) and action value Q(s, a)
- The TD error; the bias–variance trade-off
- Generalized Advantage Estimation with γ and λ
- Actor–critic: a policy plus a learned critic
66Ready
PPO
- The importance ratio between the new and old policy
- The clipped objective keeps each update small
- KL penalty to a frozen reference model; value loss; entropy bonus
- Four models in memory: policy, reference, reward model, critic
67Ready
RLHF: preference data and reward models
- Collect pairs: which answer is better?
- The Bradley–Terry reward model
- PPO against the reward model, with a KL leash
- Reward hacking and over-optimization (Goodhart’s law)
68Ready
DPO: alignment without a separate reward model
- Start from the KL-regularized RL objective
- Its closed-form best policy turns the reward into a log-ratio
- Put that into Bradley–Terry and you get the DPO loss
- β, the reference model, and what DPO can and cannot do (offline data)
69Ready
The DPO family: IPO, KTO, ORPO, SimPO
- IPO: fixes overfitting when preferences are almost certain
- KTO: works with single thumbs-up or thumbs-down labels
- ORPO: SFT and preference in one loss, no reference model
- SimPO: length-normalized reward, no reference model
70Ready
GRPO and RL with verifiable rewards (RLVR)
- Sample a group of answers for each prompt; normalize rewards inside the group
- No critic model: a big memory saving
- Verifiable rewards: math answers, unit tests, format checks
- From DeepSeekMath to DeepSeek-R1; known issues and fixes (length bias, std normalization)
71Ready
Policy learning words: on-policy, off-policy, online, offline
- On-policy: learn from samples of the current policy (PPO, GRPO)
- Off-policy and offline: learn from old or fixed data (DPO on a fixed set)
- Reference policy, behavior policy, rollout, trajectory
- Online and iterative DPO: mixing the two worlds
72Ready
Reasoning models and test-time compute
- Chain-of-thought; why thinking tokens help
- Self-consistency, best-of-N, majority vote
- Scaling test-time compute vs scaling model size
- Thinking mode in chat models; budget control; the latency cost
73Ready
Reward models, deeper: outcome rewards, process rewards, verifiers
- Outcome reward models vs process reward models
- LLM-as-judge as a reward; rubric rewards
- Verifiers for code and math
- Calibration and robustness of reward models
74Ready
Safety alignment: RLAIF and Constitutional AI
- AI feedback instead of human feedback
- Constitutional AI: rules → critiques → revisions → preferences
- Helpful vs harmless; over-refusal
- Red-teaming basics
75Ready
RL infrastructure
- Rollout generation is the bottleneck, so use a fast inference engine (vLLM, SGLang)
- Weight sync between trainer and generator; shared vs separate GPUs
- Synchronous vs asynchronous RL; off-policy drift
- Libraries: TRL, verl, OpenRLHF, and what each is good for
76Ready
DPO on a small model
- Build a preference dataset (chosen and rejected answers)
- Train with TRL’s DPOTrainer; tune β
- Measure win rate with an LLM judge; check for length bias
- Compare with SFT only
77Ready
GRPO with a verifiable reward
- A small model and math word problems with exact answers
- Write reward functions for correctness and format
- Train with GRPO; watch reward, answer length and KL
- Read samples before and after: what did the model learn?