LLM & VLM courseBlog
Track 9 · sessions 222–233

Generative models

VAEs, diffusion, guidance, latent diffusion, DiT, flow matching, few-step and video generation.

Generative models

From VAEs to diffusion, guidance, latent diffusion, DiT, flow matching, few-step models and video generation.

222

The map of generative models: autoregressive, VAE, GAN, diffusion, flow

  • What each family learns and how it samples
  • Likelihood, sample quality and speed: the trade-off triangle
  • Where LLMs (autoregressive) and image generators (diffusion) meet
  • Metrics: FID, CLIP score, human preference
Ready
223

Variational autoencoders and latent spaces

  • Encoder, decoder and the reparameterization trick
  • The ELBO: reconstruction plus a KL term
  • Why latent diffusion works in a VAE latent space
  • Posterior collapse and blurry samples
Ready
224

Visual tokenizers for generation

  • VQ-VAE: map image patches to entries of a codebook (discrete tokens)
  • VQGAN: sharper codes with adversarial and perceptual losses
  • Discrete vs continuous visual tokens: understanding vs generation
  • Why this matters for any-to-any models
Ready
225

Diffusion 1: the forward noising process and the reverse denoiser (DDPM)

  • Add Gaussian noise step by step; the closed form for any step t
  • Train a network to predict the noise
  • Sampling: start from pure noise and denoise step by step
  • Noise schedules and why they matter
Ready
226

Diffusion 2: score matching, DDIM and fast samplers

  • The score: the direction that makes data more likely
  • Diffusion as a differential equation (score SDE and probability-flow ODE)
  • DDIM: deterministic sampling with far fewer steps
  • Modern solvers and the step-count trade-off
Ready
227

Guidance and conditioning: classifier guidance, classifier-free guidance, ControlNet

  • Conditioning on text through cross-attention
  • Classifier guidance and classifier-free guidance; the guidance scale
  • Why high guidance gives sharp but less varied images
  • ControlNet: add spatial control (edges, pose, depth)
Ready
228

Latent diffusion and diffusion transformers (DiT)

  • Diffuse in a compressed latent space, not in pixels
  • The U-Net vs the transformer backbone (DiT)
  • Patchify latents; adaLN conditioning
  • Scaling behavior of DiT models
Ready
229

Flow matching and rectified flow

  • Learn a velocity field that moves noise to data
  • Straight paths are easier to integrate: rectified flow
  • The simple regression loss of flow matching
  • Rectified-flow transformers in modern image models
Ready
230

Few-step generation: consistency models and distillation

  • Why 50 steps is too slow for products
  • Progressive distillation: halve the steps again and again
  • Consistency models: jump from any noise level to the clean image
  • Quality vs steps vs latency
Ready
231

Video generation: temporal coherence, video DiTs and evaluation

  • From images to video: space-time latents and attention
  • Keeping objects consistent across frames
  • Evaluation: FVD, human preference, physics and motion checks
  • Cost: tokens, memory and latency for video
Ready
232

Native multimodal and any-to-any models

  • One shared vocabulary of discrete image tokens (Chameleon, Emu3)
  • Hybrid: next-token text + diffusion for images in one model (Transfusion)
  • Separate encoders for understanding and for generation (Janus)
  • Monolithic vs factorized output heads; balancing tasks in the data
Ready
233

train a tiny diffusion model and a tiny flow-matching model

  • Train both on 2D toy data, then on small images
  • Plot sample paths: curved (diffusion) vs straight (rectified flow)
  • Compare quality vs number of sampling steps
Ready