LLM & VLM courseBlog
Track 4 · sessions 53–77

Post-training and RL

SFT, LoRA and PEFT, distillation, RLHF, PPO, DPO, GRPO and reasoning models.

Post-training 1: SFT and PEFT

Turn a base model into a useful one: SFT, task fine-tuning, LoRA, QLoRA, distillation, embedding models.

53

From base model to assistant

  • What a base model does vs what an assistant does
  • The stages: SFT → preference tuning → RL
  • Instruction data: human-written, synthetic (self-instruct), distilled
  • Chat templates, and why a wrong template breaks a model
Ready
54

SFT in detail

  • Loss only on the assistant tokens (loss masking)
  • Packing many examples into one sequence with document masks
  • Epochs, learning rate, sequence length; signs of overfitting
  • Data quality vs quantity; diversity
Ready
55

Fine-tuning for classic NLP tasks

  • Sequence classification: sentiment (positive or negative documents), topic, intent
  • Token classification: named entities, tagging
  • Sequence-to-sequence: summarization, translation
  • Encoder or decoder for each task; class imbalance; metrics (F1, ROUGE)
Ready
56

The PEFT family: adapters, BitFit, prefix tuning, prompt tuning, P-Tuning v2

  • Adapters: small bottleneck layers inside each block (Houlsby)
  • BitFit: train only the biases
  • Prompt tuning and prefix tuning: learn virtual tokens or per-layer key/value prefixes; P-Tuning and P-Tuning v2
  • How to choose: prompting vs linear probe vs PEFT vs full fine-tuning (data size, latency, number of tasks)
Ready
57

LoRA and other PEFT methods

  • LoRA: freeze W and learn a low-rank update BA
  • Rank r, the α/r scale, which layers to adapt (attention, MLP, all linear layers)
  • Merging adapters for zero extra inference cost; serving many adapters
  • DoRA, prefix tuning, prompt tuning: when they help
Ready
58

QLoRA and training with quantized weights

  • NF4 4-bit base weights with BF16 compute
  • Double quantization of the scale factors
  • Paged optimizers to survive memory spikes
  • Quality vs memory, and when QLoRA is not enough
Ready
59

Knowledge distillation

  • Soft targets and temperature
  • Logit distillation (KL) vs sequence-level distillation (train on teacher outputs)
  • Distilling reasoning traces into small models
  • When the student cannot copy the teacher (the capacity gap)
Ready
60

Training embedding and retrieval models

  • Contrastive learning: pull positives together, push negatives apart
  • InfoNCE and temperature; in-batch negatives; hard negatives
  • Bi-encoder training (Sentence-BERT, E5) and how to evaluate it
  • The same idea as CLIP: our bridge to vision
Ready
61

Forgetting, data mixing and model merging

  • Catastrophic forgetting: why fine-tuning can break general skills
  • Mix in general data, use a lower LR, or use LoRA as a regularizer
  • Model merging: weight averaging, model soups, task vectors
  • How to measure regressions
Ready
62

LoRA vs QLoRA vs full fine-tuning

  • Fine-tune a small LM for sentiment (positive vs negative documents)
  • Fine-tune it for summarization; evaluate with ROUGE and an LLM judge
  • Compare memory, speed and quality for full FT, LoRA and QLoRA
  • Write down what you would ship, and why
Ready

Post-training 2: alignment and reinforcement learning

Policies, rewards and preferences: PPO, RLHF, DPO and its family, GRPO, reasoning models.

63

Reinforcement learning words, for LLMs

  • Agent, environment, state, action, reward, return, episode
  • Text generation as RL: state = prompt + text so far, action = next token, policy = the LLM
  • Sparse rewards that come only at the end of an answer
  • Why RL and not only SFT: learn from a score, not from a copy
Ready
64

Policy gradient and REINFORCE

  • The log-derivative trick: ∇E[R] = E[R ∇log π]
  • REINFORCE for a whole generated answer
  • High variance, and baselines that add no bias
  • Advantage = how much better than expected
Ready
65

Value functions, advantage and GAE

  • State value V(s) and action value Q(s, a)
  • The TD error; the bias–variance trade-off
  • Generalized Advantage Estimation with γ and λ
  • Actor–critic: a policy plus a learned critic
Ready
66

PPO

  • The importance ratio between the new and old policy
  • The clipped objective keeps each update small
  • KL penalty to a frozen reference model; value loss; entropy bonus
  • Four models in memory: policy, reference, reward model, critic
Ready
67

RLHF: preference data and reward models

  • Collect pairs: which answer is better?
  • The Bradley–Terry reward model
  • PPO against the reward model, with a KL leash
  • Reward hacking and over-optimization (Goodhart’s law)
Ready
68

DPO: alignment without a separate reward model

  • Start from the KL-regularized RL objective
  • Its closed-form best policy turns the reward into a log-ratio
  • Put that into Bradley–Terry and you get the DPO loss
  • β, the reference model, and what DPO can and cannot do (offline data)
Ready
69

The DPO family: IPO, KTO, ORPO, SimPO

  • IPO: fixes overfitting when preferences are almost certain
  • KTO: works with single thumbs-up or thumbs-down labels
  • ORPO: SFT and preference in one loss, no reference model
  • SimPO: length-normalized reward, no reference model
Ready
70

GRPO and RL with verifiable rewards (RLVR)

  • Sample a group of answers for each prompt; normalize rewards inside the group
  • No critic model: a big memory saving
  • Verifiable rewards: math answers, unit tests, format checks
  • From DeepSeekMath to DeepSeek-R1; known issues and fixes (length bias, std normalization)
Ready
71

Policy learning words: on-policy, off-policy, online, offline

  • On-policy: learn from samples of the current policy (PPO, GRPO)
  • Off-policy and offline: learn from old or fixed data (DPO on a fixed set)
  • Reference policy, behavior policy, rollout, trajectory
  • Online and iterative DPO: mixing the two worlds
Ready
72

Reasoning models and test-time compute

  • Chain-of-thought; why thinking tokens help
  • Self-consistency, best-of-N, majority vote
  • Scaling test-time compute vs scaling model size
  • Thinking mode in chat models; budget control; the latency cost
Ready
73

Reward models, deeper: outcome rewards, process rewards, verifiers

  • Outcome reward models vs process reward models
  • LLM-as-judge as a reward; rubric rewards
  • Verifiers for code and math
  • Calibration and robustness of reward models
Ready
74

Safety alignment: RLAIF and Constitutional AI

  • AI feedback instead of human feedback
  • Constitutional AI: rules → critiques → revisions → preferences
  • Helpful vs harmless; over-refusal
  • Red-teaming basics
Ready
75

RL infrastructure

  • Rollout generation is the bottleneck, so use a fast inference engine (vLLM, SGLang)
  • Weight sync between trainer and generator; shared vs separate GPUs
  • Synchronous vs asynchronous RL; off-policy drift
  • Libraries: TRL, verl, OpenRLHF, and what each is good for
Ready
76

DPO on a small model

  • Build a preference dataset (chosen and rejected answers)
  • Train with TRL’s DPOTrainer; tune β
  • Measure win rate with an LLM judge; check for length bias
  • Compare with SFT only
Ready
77

GRPO with a verifiable reward

  • A small model and math word problems with exact answers
  • Write reward functions for correctness and format
  • Train with GRPO; watch reward, answer length and KL
  • Read samples before and after: what did the model learn?
Ready