Track 9 · sessions 222–233
Generative models
VAEs, diffusion, guidance, latent diffusion, DiT, flow matching, few-step and video generation.
Generative models
From VAEs to diffusion, guidance, latent diffusion, DiT, flow matching, few-step models and video generation.
222Ready
The map of generative models: autoregressive, VAE, GAN, diffusion, flow
- What each family learns and how it samples
- Likelihood, sample quality and speed: the trade-off triangle
- Where LLMs (autoregressive) and image generators (diffusion) meet
- Metrics: FID, CLIP score, human preference
223Ready
Variational autoencoders and latent spaces
- Encoder, decoder and the reparameterization trick
- The ELBO: reconstruction plus a KL term
- Why latent diffusion works in a VAE latent space
- Posterior collapse and blurry samples
224Ready
Visual tokenizers for generation
- VQ-VAE: map image patches to entries of a codebook (discrete tokens)
- VQGAN: sharper codes with adversarial and perceptual losses
- Discrete vs continuous visual tokens: understanding vs generation
- Why this matters for any-to-any models
225Ready
Diffusion 1: the forward noising process and the reverse denoiser (DDPM)
- Add Gaussian noise step by step; the closed form for any step t
- Train a network to predict the noise
- Sampling: start from pure noise and denoise step by step
- Noise schedules and why they matter
226Ready
Diffusion 2: score matching, DDIM and fast samplers
- The score: the direction that makes data more likely
- Diffusion as a differential equation (score SDE and probability-flow ODE)
- DDIM: deterministic sampling with far fewer steps
- Modern solvers and the step-count trade-off
227Ready
Guidance and conditioning: classifier guidance, classifier-free guidance, ControlNet
- Conditioning on text through cross-attention
- Classifier guidance and classifier-free guidance; the guidance scale
- Why high guidance gives sharp but less varied images
- ControlNet: add spatial control (edges, pose, depth)
228Ready
Latent diffusion and diffusion transformers (DiT)
- Diffuse in a compressed latent space, not in pixels
- The U-Net vs the transformer backbone (DiT)
- Patchify latents; adaLN conditioning
- Scaling behavior of DiT models
229Ready
Flow matching and rectified flow
- Learn a velocity field that moves noise to data
- Straight paths are easier to integrate: rectified flow
- The simple regression loss of flow matching
- Rectified-flow transformers in modern image models
230Ready
Few-step generation: consistency models and distillation
- Why 50 steps is too slow for products
- Progressive distillation: halve the steps again and again
- Consistency models: jump from any noise level to the clean image
- Quality vs steps vs latency
231Ready
Video generation: temporal coherence, video DiTs and evaluation
- From images to video: space-time latents and attention
- Keeping objects consistent across frames
- Evaluation: FVD, human preference, physics and motion checks
- Cost: tokens, memory and latency for video
232Ready
Native multimodal and any-to-any models
- One shared vocabulary of discrete image tokens (Chameleon, Emu3)
- Hybrid: next-token text + diffusion for images in one model (Transfusion)
- Separate encoders for understanding and for generation (Janus)
- Monolithic vs factorized output heads; balancing tasks in the data
233Ready
train a tiny diffusion model and a tiny flow-matching model
- Train both on 2D toy data, then on small images
- Plot sample paths: curved (diffusion) vs straight (rectified flow)
- Compare quality vs number of sampling steps