The map of generative models: autoregressive, VAE, GAN, diffusion, flow
An LLM and an image generator solve the same problem: learn a probability distribution from examples, then draw new samples from it. This session puts the five big families side by side (autoregressive, VAE, GAN, diffusion, flow), shows what each one learns and how it samples, and explains the trade-off between exact likelihood, sample quality and speed. We end with how generated images are measured: FID, CLIP score and human preference.
What you will learn
- What it means to “learn a distribution”, and why maximum likelihood is the same as minimizing a KL divergence
- What each of the five families learns, how it is trained, and how it draws a sample, with one formula each
- Which families give an exact likelihood, a lower bound, or none at all, and why that matters
- The trade-off triangle: likelihood, sample quality and sampling speed, with real network-call counts
- Where LLMs and image generators meet: image tokens, diffusion transformers and hybrid models such as Transfusion
- How FID, CLIP score and human-preference models work, and how each of them can fool you
Why this matters
A chat model writing an answer and an image model painting a cat look like very different machines. Under the hood they solve the same problem: learn a probability distribution from examples, then draw new samples from it. They only differ in how they do it.
A generative model is a model that can produce new data (text, images, audio, video) that looks like its training data. There are five big families: autoregressive models (every LLM), variational autoencoders (VAEs), generative adversarial networks (GANs), diffusion models, and flow models. Each one makes a different bet about what to learn and how to sample, and each bet has a price. Some give you an exact probability for any input but sample slowly. Some sample in one network call but give no probability at all. Some make sharp images but forget whole parts of the data.
This session is the map for the whole generative track. Each later session zooms into one region of it: session 223, session 224, diffusion in two parts (session 225 and session 226), session 227, session 228, session 229, fast samplers in session 230, session 231 and finally models that do everything at once in session 232. Real systems mix the families: Stable Diffusion 3 is a VAE plus a rectified-flow transformer; Parti and LlamaGen are LLM-style autoregressive models over image tokens; Transfusion trains one transformer with a next-token loss on text and a diffusion loss on images. [Certain] By the end of this session you will be able to place any of them on the map.
What does it mean to learn a distribution?
Think of all possible \(256 \times 256\) color images. Real photos are a tiny, strangely shaped region of that huge space. There is some unknown probability distribution \(p_{\text{data}}(x)\) that says how likely each image is to appear as a real photo. We never see this distribution. We only see samples from it: our training set.
A generative model is a family of distributions \(p_\theta(x)\) with weights \(\theta\). Training should move \(p_\theta\) close to \(p_{\text{data}}\). The classic way is maximum likelihood: make the training examples as probable as possible under the model.
- Read the symbols
- \(p_{\text{data}}(x)\)
- the true (unknown) distribution of real data
- \(p_\theta(x)\)
- the model’s distribution; \(\theta\) are its weights
- \(\log p_\theta(x)\)
- the log-likelihood of one example: higher means the model finds it more probable
- \(\KL(p \,\|\, q)\)
- the Kullback–Leibler divergence, a measure of how different \(q\) is from \(p\); it is 0 only when they are equal
The two sides are equal because \(\KL(p_{\text{data}} \| p_\theta) = \E[\log p_{\text{data}}] - \E[\log p_\theta]\), and the first term does not depend on \(\theta\). This is exactly the cross-entropy loss you know from session 3. An LLM trained with next-token cross-entropy is doing maximum likelihood.
A trained generative model can be asked two different questions, and the families differ in which ones they can answer:
- Density (scoring): “How probable is this \(x\)?” Give a number \(p_\theta(x)\). Useful for compression, anomaly detection and comparing models.
- Sampling: “Give me a new \(x\).” Draw \(x \sim p_\theta\). This is what users see.
There is one hard part shared by all families: \(p_\theta(x)\) must sum (or integrate) to 1 over all possible \(x\). For images, that is a sum over more possible images than atoms in the universe. Each family is, at heart, a trick to avoid computing that sum.
The five families at a glance
Figure 1 is the map. Click a family to see what its network learns, how it is trained, how it draws a sample, and what kind of likelihood it gives. The sections after it explain each family with one key formula.
How to read Figure 1
Look at three things for each family. What is random: the autoregressive model draws a random choice at every token; the others draw a random vector \(z\) once (and diffusion adds noise along the way if you use a stochastic sampler). How many network calls per sample: shown in the readout. What the loss is: only autoregressive and flow models train on the exact likelihood; VAE and diffusion train on a bound; GANs train against a second network.
Family 1: autoregressive models
Split the data into a sequence of pieces \(x = (x_1, x_2, \dots, x_n)\): words, pixels, audio samples, or image tokens. The chain rule of probability says any joint distribution can be written, exactly, as a product of one-step conditionals:
The network only has to output one small distribution at a time, over the vocabulary for the next piece. A softmax over 50,000 tokens is easy to normalize, so the hard “sum over all images” problem disappears. This is the model you already know from session 6. Training is parallel (all positions at once with a causal mask); sampling is sequential: piece 1, then piece 2 given piece 1, and so on.
- Learns: the next-piece distribution \(p_\theta(x_i \mid x_{\lt i})\).
- Likelihood: exact, by formula (222.2). That is why LLMs report perplexity.
- Samples: \(n\) sequential network calls for \(n\) pieces. With a KV cache each call only processes one new piece (session 80).
- Examples: GPT and Llama (text tokens); PixelRNN/PixelCNN (pixels); WaveNet (audio samples); Parti and LlamaGen (image tokens from a learned tokenizer). [Certain]
Pixels are a poor choice of piece: a \(256 \times 256\) RGB image has \(256 \cdot 256 \cdot 3 = 196{,}608\) sub-pixel values, so sampling one image would take 196,608 sequential steps. Modern autoregressive image models first compress the image into a short grid of discrete image tokens with a VQ tokenizer (session 224). LlamaGen’s tokenizer downsamples by 16 in each direction, so a \(256 \times 256\) image becomes \(16 \times 16 = 256\) tokens. [Certain]
Figure 2 runs this on the smallest possible “image”: four black-or-white pixels. The data are mostly vertical or horizontal bars.
How to read Figure 2
The orange bar is the probability that the current pixel is white, given the pixels already chosen. Watch how it changes: once the first pixel is white and the second is black, the third pixel is white with probability 0.949, because “left bar” is now the most likely explanation. At the last step the readout shows the sum of log-probabilities, which equals the log of the joint probability of the finished image: that is formula (222.2). “New sample” uses the next seed of a fixed random generator, so results repeat.
Take the image “left bar”: pixels \((1, 0, 1, 0)\). From the data distribution of Figure 2, the four conditionals are \(p(x_1 = 1) = 0.5\), \(p(x_2 = 0 \mid 1) = 0.65\), \(p(x_3 = 1 \mid 1, 0) = 0.949\) and \(p(x_4 = 0 \mid 1, 0, 1) = 0.973\).
Sum of logs: \(\ln 0.5 + \ln 0.65 + \ln 0.949 + \ln 0.973 = -0.693 - 0.431 - 0.053 - 0.027 = -1.204\). And indeed \(\ln 0.30 = -1.204\): the chain rule is exact. A noise pattern such as \((1, 1, 1, 0)\) has \(p = 0.1/12 = 0.0083\), \(\ln p = -4.79\). The model scores both exactly.
Family 2: variational autoencoders (VAEs)
A VAE assumes each data point is made from a hidden latent vector \(z\) (a short code, for example 16 numbers, or a small grid for images). Sampling is simple: draw \(z\) from a standard Gaussian \(p(z) = \mathcal{N}(0, I)\), then pass it through a decoder network that outputs \(x\). The probability of an image then needs an integral over all codes, \(p_\theta(x) = \int p_\theta(x \mid z)\, p(z)\, dz\), which we cannot compute. The VAE trick is to add an encoder \(q_\phi(z \mid x)\) that guesses which codes could have made \(x\), and to optimize a lower bound on the log-likelihood, the evidence lower bound (ELBO):
- Read the symbols
- \(z\)
- the latent code; much smaller than \(x\)
- \(q_\phi(z \mid x)\)
- the encoder: a Gaussian over codes, with mean and variance predicted from \(x\)
- \(p_\theta(x \mid z)\)
- the decoder: how likely \(x\) is given code \(z\); with a Gaussian decoder, \(-\log p_\theta(x \mid z)\) is a squared error
- \(p(z)\)
- the prior, a standard Gaussian; sampling starts here
- Learns: an encoder and a decoder, trained together.
- Likelihood: a lower bound (the ELBO), not the exact value.
- Samples: one decoder call. Very fast.
- Weakness: samples are often blurry. The squared-error reconstruction term is minimized by predicting the average of all images that fit a code, and an average of sharp images is a blurry image.
Today the VAE’s most important job is as a compressor for other models. Latent diffusion (Stable Diffusion) runs diffusion inside a VAE’s latent space: a \(512 \times 512 \times 3\) image becomes a \(64 \times 64 \times 4\) latent, 48 times fewer numbers. [Certain] The full story, including the reparameterization trick and posterior collapse, is in session 223.
Family 3: generative adversarial networks (GANs)
A GAN gives up on likelihood completely. A generator \(G\) turns noise \(z\) into an image \(G(z)\). A second network, the discriminator \(D\), sees real images and generated images and outputs the probability that its input is real. They play a game:
The discriminator learns to tell real from fake; the generator learns to fool it. If \(D\) were perfect, this game would minimize the Jensen–Shannon divergence between \(p_{\text{data}}\) and the generator’s distribution (the value is \(2\,\mathrm{JSD} - \log 4\)). [Certain] In practice neither network is perfect, and training is a balancing act.
- Learns: a generator (kept) and a discriminator (thrown away after training).
- Likelihood: none. The generator defines a distribution only through its samples; you cannot ask “how probable is this image?”.
- Samples: one generator call, and samples are sharp. StyleGAN faces are the classic example.
- Weaknesses: unstable training, and mode collapse: the generator finds a few outputs that fool \(D\) and stops producing the rest of the data. Nothing in the loss punishes it for ignoring part of the data, because there is no likelihood term.
GANs are no longer the main tool for text-to-image, but the adversarial loss lives on as a helper: VQGAN tokenizers and many VAE decoders add a discriminator loss to make reconstructions sharp, and several few-step diffusion distillation methods use one too. [Likely]
Family 4: diffusion models
Diffusion splits the hard job “noise to image” into many easy jobs “slightly less noisy image”. First define a fixed forward process that slowly destroys data by adding Gaussian noise. At noise level \(t\) (from 0, clean, to 1, pure noise), a noisy version of a data point \(x_0\) is
where \(\bar\alpha_t\) goes from 1 (no noise) to about 0 (all noise). A network \(\varepsilon_\theta\) is trained to look at \(x_t\) and guess the noise that was added:
- Read the symbols
- \(x_0\)
- a clean training example
- \(x_t\)
- the same example after adding noise of level \(t\)
- \(\bar\alpha_t\)
- the noise schedule: how much of the signal survives at level \(t\)
- \(\varepsilon_\theta(x_t, t)\)
- the denoising network’s guess of the noise; from it you also get a guess of \(x_0\)
To sample, start from pure noise and apply the denoiser again and again, each time removing a little noise. DDPM used \(T = 1000\) steps and reached a then state-of-the-art FID of 3.17 on CIFAR-10. [Certain] Faster samplers such as DDIM take larger steps; 20 to 50 steps are common, and distillation brings it down to 1 to 4 (session 226, session 230).
- Learns: a denoiser (equivalently, the score \(\nabla_x \log p_t(x)\), the direction toward more probable data at each noise level).
- Likelihood: a lower bound (an ELBO, like the VAE), or an exact value through an ODE that is expensive to compute.
- Samples: many network calls, each on the full image. Slow, but samples are sharp and diverse.
The DDPM paper itself notes that its sampling can be seen as a generalization of autoregressive decoding: instead of “one more token”, each step adds “one more level of detail”. [Certain] Figure 3 lets you see why one big step is not enough.
How to read Figure 3
Move \(t\) to 1: the top curve becomes the standard Gaussian, which is where sampling starts. Now set \(S = 1\): every sample lands on the same point, 0.45, the average of the data. One step from pure noise can only predict the mean, the same reason VAE samples are blurry. With 2, 3, 5 steps the two bumps appear; by 20 to 50 steps the histogram matches the data, the fraction of samples left of 0 is close to 0.30, and the 1-D Fréchet distance (formula 222.10, explained later) is near 0.
Family 5: flow models
“Flow” names two related ideas. Both move samples from a simple distribution (a Gaussian) to the data distribution.
Normalizing flows
A normalizing flow is an invertible network \(f\) that maps data \(x\) to a code \(z = f(x)\) of the same size, where \(z\) should look Gaussian. Because \(f\) is invertible, the change-of-variables rule gives the exact likelihood:
The determinant term measures how much \(f\) stretches or squeezes space near \(x\). For a general network it would cost \(O(d^3)\) for \(d\) dimensions, so flow layers are designed to have cheap determinants (RealNVP, Glow use triangular “coupling” layers). [Certain] Sampling is one pass of \(f^{-1}\). The price: every layer must be invertible and keep the full size of \(x\), which limits the architecture, and image quality fell behind GANs and diffusion. [Likely]
Flow matching and rectified flow
The modern meaning of “flow” is closer to diffusion. Pick a noise sample \(x_0 \sim \mathcal{N}(0, I)\) and a data sample \(x_1\), and draw a straight line between them. A network \(v_\theta\) learns the velocity that moves points along such lines:
To sample, solve the ordinary differential equation \(dx/dt = v_\theta(x, t)\) from \(t = 0\) to \(1\) with a few Euler steps. Note that here \(t = 0\) is noise and \(t = 1\) is data, the opposite of formula (222.5); papers use both conventions. Stable Diffusion 3 is a rectified-flow transformer. [Certain] In this course “diffusion” and “flow matching” are close cousins: both learn a vector field from noise to data and both need several network calls. Details in session 229.
The trade-off triangle: likelihood, quality, speed
Put the five families next to each other and a pattern appears. We would like three things:
- Likelihood: an exact or tight \(\log p_\theta(x)\), so that we can score inputs, compress data, and know the model covers all of the data.
- Sample quality: sharp, realistic, diverse samples.
- Speed: few sequential network calls per sample.
No classic family gets all three. Xiao et al. call this the generative learning trilemma, with “mode coverage” in place of likelihood: GANs are fast and sharp but miss modes; VAEs and normalizing flows are fast and cover modes but have lower quality; diffusion has quality and coverage but is slow. [Certain] Autoregressive models add a fourth corner: exact likelihood and very high quality, paid for with one network call per token.
How to read Figure 4
The two bars measure different things. Sequential calls set the latency floor: call \(k+1\) cannot start before call \(k\) ends. Total work counts how many times the network sees the whole image, which sets the compute bill. An autoregressive model with a KV cache makes \(L\) calls but each call processes one new token, so its total work is about one full pass (plus attention to the cache). A diffusion model makes only \(S\) calls, but each one processes all \(L\) positions, so its work is \(S\) passes. Change \(L\): only the autoregressive latency grows. Change \(S\): only diffusion and flow matching change.
Take a \(256 \times 256\) image represented as 256 positions (a LlamaGen-style \(16 \times 16\) token grid, or a DiT-XL/2 latent: \(32 \times 32 \times 4\) with \(2 \times 2\) patches gives \(16 \times 16 = 256\) tokens [Certain]).
Autoregressive: 256 sequential calls, each processing 1 new token: \(256 \times 1 = 256\) token-positions in total, about one full pass.
Diffusion, 50 steps: 50 sequential calls, each processing 256 positions: \(50 \times 256 = 12{,}800\) token-positions, 50 times the autoregressive work. With classifier-free guidance each step runs two passes (with and without the prompt): 25,600 token-positions (and 512 for an autoregressive model that also uses guidance).
So diffusion is compute-heavy but parallel, and autoregression is compute-light but serial. Small-batch autoregressive decoding is limited by memory bandwidth, not FLOPs (session 82), which is why serving tricks from LLMs (KV cache, batching, speculative decoding) also speed up autoregressive image models: LlamaGen reports a 326% to 414% speedup from an LLM serving framework. [Certain]
Why likelihood and sample quality pull apart
Maximum likelihood (formula 222.1) minimizes \(\KL(p_{\text{data}} \| p_\theta)\). This divergence is huge wherever the data has mass and the model has almost none. So a likelihood-trained model is punished hard for missing any part of the data, and only mildly for putting a little mass in places where there is no data. The result is mass-covering: it spreads out, which can look like blur or odd samples. A GAN has no such term, so it can drop whole modes and still produce perfect-looking samples (mode-seeking). Theis et al. showed that log-likelihood and visual sample quality are largely independent in high dimensions: one does not predict the other. [Certain] Figure 5, in the metrics section, lets you see this with numbers.
Where LLMs and image generators meet
For a few years text used autoregressive transformers and images used diffusion U-Nets. Today the two worlds share almost everything:
- Images as tokens, generated by an LLM. A VQ tokenizer turns an image into discrete tokens; a decoder-only transformer predicts them one by one, exactly like text. Parti (up to 20B parameters, an encoder-decoder), LlamaGen (a plain Llama architecture, 2.18 FID on class-conditional ImageNet \(256 \times 256\)) and Chameleon (one model, one token stream for text and images) work this way. [Certain]
- Diffusion with a transformer backbone. DiT replaced the U-Net with a ViT-style transformer over latent patches; SD3’s MM-DiT also attends jointly over text and image tokens. [Certain] The scaling and serving know-how from LLMs carries over.
- One model, two losses. Transfusion trains a single transformer with next-token prediction on text and a diffusion loss on image patches, in one sequence. [Certain]
- Not strictly left to right. MaskGIT predicts many masked image tokens in parallel and refines them over a few steps; VAR predicts the image “next scale” by next scale, coarse to fine. [Certain] In the other direction, masked diffusion language models such as LLaDA generate text by iterative unmasking instead of left to right. [Certain]
- The language model as the brain. In most text-to-image systems, a text encoder (CLIP or a T5-style LLM) turns the prompt into vectors that condition the diffusion model (session 227). [Likely]
Measuring generated images: FID, CLIP score, human preference
For text LLMs we have perplexity (likelihood) and task benchmarks. For image generators the main question is “are the samples good?”, and there is no exact answer. Three kinds of metric are used, each measuring something different.
Likelihood in bits per dimension
Models with a likelihood (autoregressive, flows, VAE and diffusion bounds) report the negative log-likelihood in bits per dimension, so that images of different sizes can be compared:
Here \(D\) is the number of values in the image (\(32 \times 32 \times 3 = 3{,}072\) for CIFAR-10) and \(\ln 2\) converts nats to bits. Example: 3.75 bpd on CIFAR-10 means \(-\log p = 3.75 \times 3072 = 11{,}520\) bits, or \(11{,}520 \times \ln 2 = 7{,}985\) nats per image. This tells you how well the model would compress images, not how good its samples look.
FID: Fréchet Inception Distance
FID compares the set of generated images with a set of real images. Both sets are passed through an Inception-v3 classifier; each image becomes a 2,048-dimensional feature vector. Fit a Gaussian (mean \(\mu\), covariance \(\Sigma\)) to each set of features and compute the Fréchet distance between the two Gaussians:
- Read the symbols
- \(\mu_r, \Sigma_r\)
- mean and covariance of the features of real images
- \(\mu_g, \Sigma_g\)
- the same for generated images
- \(\operatorname{Tr}\)
- the trace: the sum of the diagonal of a matrix
- \((\cdot)^{1/2}\)
- the matrix square root
Lower is better; 0 means the two Gaussians are identical. The first term catches samples that look different on average (low quality); the second catches a wrong spread, including missing diversity. FID needs many samples (50,000 is a common choice [Likely]) and it is biased upward when the sample count is small, so FIDs computed with different sample counts or different implementations are not comparable. [Certain]
In 1-D the formula becomes \((\mu_r - \mu_g)^2 + (\sigma_r - \sigma_g)^2\), because \(\sigma_r^2 + \sigma_g^2 - 2\sigma_r\sigma_g = (\sigma_r - \sigma_g)^2\).
Real features: \(\mu_r = 0\), \(\sigma_r = 1\). Generated: \(\mu_g = 0.5\), \(\sigma_g = 0.6\). Then \(\text{FID} = 0.5^2 + 0.4^2 = 0.25 + 0.16 = 0.41\). The generator loses 0.25 for being shifted and 0.16 for having too little spread.
CLIP score: does the image match the prompt?
FID ignores the prompt. For text-to-image, we also want alignment: does the image show what was asked? CLIP (session 166) embeds the image and the prompt in the same space; their cosine similarity is a measure of match. CLIPScore defines
where \(\mathbf{e}_I\) and \(\mathbf{e}_T\) are the CLIP image and text embeddings. [Likely: the value of \(w\) is from the paper body] Many text-to-image papers instead report \(100 \times \cos\). CLIP score is cheap and needs no reference images, but CLIP is weak at counting, spatial relations and attribute binding (“a red cube on a blue sphere”). Object-focused tests such as GenEval check those with a detector. [Certain]
Human preference
In the end, people decide. Human raters compare two images for the same prompt and pick the better one. Since this is slow and costly, preference models are trained on many such votes: PickScore (from the Pick-a-Pic dataset), HPSv2 and ImageReward. [Certain] They are trained like the reward models of RLHF (session 67), with a pairwise model of the form
where \(r\) is the learned score and \(\sigma\) the sigmoid. [Likely: the exact training losses differ by paper] These scores correlate better with people than FID does on modern text-to-image models, but they can be hacked: if you optimize a generator against one of them, it learns that model’s taste and quirks. Report several metrics, and always keep a human study for final claims.
How to read Figure 5
Press “Mode collapse”: the model puts everything on the right bump. Its samples are exactly as realistic as real data (realism −1.419, same as the perfect model), but the likelihood explodes to 16.7 nats because half of the real data now has almost zero probability. FID sees it too (6.44), mostly through the shifted mean. Press “Blurry model”: likelihood stays reasonable (2.03) but samples land between the bumps, so realism drops to −4.39. Now set \(\sigma = 0.2\) with \(\pi = 0.5\): samples are more “realistic” than real data and FID is only 0.003, but likelihood gets worse (3.13). Each metric can be fooled; together they tell the story.
Putting it together: one training step and one sample per family
- input: a batch of real data \(x\)
- autoregressive: loss \(= -\sum_i \log p_\theta(x_i \mid x_{\lt i})\) // formula (222.2), all positions in parallel with a causal mask
- VAE: \(\mu, \sigma \leftarrow \text{enc}(x)\); \(z \leftarrow \mu + \sigma \odot \varepsilon\); loss \(= -\log p_\theta(x \mid z) + \KL\) // formula (222.3)
- GAN: update \(D\) to raise \(\log D(x) + \log(1 - D(G(z)))\); then update \(G\) to raise \(\log D(G(z))\) // formula (222.4), the common “non-saturating” form for G
- diffusion: \(t \sim U(0,1)\), \(\varepsilon \sim \mathcal{N}(0, I)\), \(x_t \leftarrow \sqrt{\bar\alpha_t} x + \sqrt{1 - \bar\alpha_t}\, \varepsilon\); loss \(= \lVert \varepsilon - \varepsilon_\theta(x_t, t) \rVert^2\) // formulas (222.5), (222.6)
- flow matching: \(x_0 \sim \mathcal{N}(0, I)\), \(t \sim U(0,1)\), \(x_t \leftarrow (1-t)x_0 + t x\); loss \(= \lVert v_\theta(x_t, t) - (x - x_0) \rVert^2\) // formula (222.8)
- backward pass and optimizer step, as in any network
- autoregressive: for \(i = 1 \dots n\): \(x_i \sim p_\theta(\cdot \mid x_{\lt i})\) // \(n\) sequential calls, KV cache
- VAE: \(z \sim \mathcal{N}(0, I)\); \(x \leftarrow \text{dec}(z)\) // 1 call
- GAN: \(z \sim \mathcal{N}(0, I)\); \(x \leftarrow G(z)\) // 1 call
- normalizing flow: \(z \sim \mathcal{N}(0, I)\); \(x \leftarrow f^{-1}(z)\) // 1 call
- diffusion (DDIM): \(x \sim \mathcal{N}(0, I)\); for \(k = 1 \dots S\): \(\hat x_0 \leftarrow\) denoise\((x, t_k)\); \(x \leftarrow\) re-noise \(\hat x_0\) to level \(t_{k+1}\) // \(S\) calls
- flow matching: \(x \sim \mathcal{N}(0, I)\); for \(k = 0 \dots S-1\): \(x \leftarrow x + \tfrac{1}{S} v_\theta(x, k/S)\) // \(S\) Euler steps
The PyTorch version trains a diffusion model and a flow-matching model on the 1-D data of Figure 3 and samples both with 1, 5 and 50 steps. It runs on a CPU in under a minute.
import math, torch torch.manual_seed(0) # data: 1-D mixture, 30% near -2 and 70% near +1.5 (the data of Figure 3) def sample_data(n): left = torch.rand(n) < 0.3 return torch.where(left, -2.0 + 0.4 * torch.randn(n), 1.5 + 0.6 * torch.randn(n)) def mlp(): # input (x, t) -> one number return torch.nn.Sequential(torch.nn.Linear(2, 128), torch.nn.SiLU(), torch.nn.Linear(128, 128), torch.nn.SiLU(), torch.nn.Linear(128, 1)) abar = lambda t: torch.cos(math.pi * t / 2) ** 2 # noise schedule: abar(0) = 1, abar(1) = 0 def train(net, loss_fn, steps=2000): opt = torch.optim.Adam(net.parameters(), lr=2e-3) for _ in range(steps): loss = loss_fn(net, sample_data(512)) opt.zero_grad(); loss.backward(); opt.step() return net # 1) diffusion: noise the data (formula 222.5), learn to undo it. # DDPM predicts the noise eps; here the net predicts the clean x0, an equivalent # target that is more stable at t = 1 (x0 = (xt - sqrt(1-abar) eps) / sqrt(abar)). def diff_loss(net, x0): t, eps = torch.rand_like(x0), torch.randn_like(x0) xt = abar(t).sqrt() * x0 + (1 - abar(t)).sqrt() * eps return ((net(torch.stack([xt, t], 1)).squeeze(1) - x0) ** 2).mean() # 2) flow matching: predict the velocity x1 - x0 (formula 222.8); x0 = noise, x1 = data def fm_loss(net, x1): x0, t = torch.randn_like(x1), torch.rand_like(x1) xt = (1 - t) * x0 + t * x1 return ((net(torch.stack([xt, t], 1)).squeeze(1) - (x1 - x0)) ** 2).mean() dnet, fnet = train(mlp(), diff_loss), train(mlp(), fm_loss) @torch.no_grad() def sample_ddim(net, n, S): # deterministic DDIM, S network calls x = torch.randn(n); ts = torch.linspace(1.0, 0.0, S + 1) for t, tn in zip(ts[:-1], ts[1:]): x0 = net(torch.stack([x, t.expand(n)], 1)).squeeze(1) eps = (x - abar(t).sqrt() * x0) / (1 - abar(t)).sqrt() x = abar(tn).sqrt() * x0 + (1 - abar(tn)).sqrt() * eps return x @torch.no_grad() def sample_fm(net, n, S): # Euler steps on dx/dt = v(x, t) x = torch.randn(n) for k in range(S): t = torch.full((n,), k / S) x = x + net(torch.stack([x, t], 1)).squeeze(1) / S return x def report(name, x): # 1-D Frechet distance (formula 222.10 with d = 1) real = sample_data(100_000) fd = (x.mean() - real.mean()) ** 2 + (x.std() - real.std()) ** 2 print(f"{name:22s} left-mode fraction {(x < 0).float().mean():.3f} 1-D FD {fd:.4f}") report("data", sample_data(20_000)) for S in (1, 5, 50): report(f"diffusion DDIM S={S}", sample_ddim(dnet, 20_000, S)) report(f"flow matching S={S}", sample_fm(fnet, 20_000, S))
In our run (PyTorch 2.8 on a CPU) one step gives a left-mode fraction of 0.029 for diffusion and 0.223 for flow matching, with 1-D FD above 2 for both: one step mostly lands near the data mean, as in Figure 3. With 50 steps the fractions are 0.276 and 0.315, close to the true 0.30, and the FD drops to 0.106 and 0.007. Exact values change with hardware and library version; the trend does not. The diffusion model is a little worse here only because this tiny network was trained for 2,000 steps; Figure 3 uses the exact denoiser.
Where this is used
| Application | Family used | Why that family |
|---|---|---|
| Chat, code and reasoning LLMs | Autoregressive over text tokens | Exact likelihood, simple training, text is naturally a sequence |
| Text-to-image (Stable Diffusion 3, DiT-based systems) | VAE latent + diffusion or rectified-flow transformer | Best quality and diversity; the VAE makes each step cheap (48× fewer values at 512 px) |
| Autoregressive image generation (Parti, LlamaGen) and native multimodal LLMs (Chameleon) | VQ tokenizer + autoregressive transformer | Reuses LLM training and serving stacks; one token stream for text and images |
| Image tokenizers and VAE decoders (VQGAN, SD’s VAE) | Autoencoder trained with reconstruction + adversarial (GAN) loss [Likely for every specific model] | The GAN loss restores sharp detail that a squared error would blur |
| Speech and audio | Autoregressive (WaveNet on raw samples; modern models over audio-codec tokens) [Likely for the modern part] | Audio is a sequence; codec tokens make it short |
| Video generation | Diffusion or flow-matching transformers over space-time latents (session 231) | Parallel generation of all frames keeps them coherent |
| Video understanding: action recognition, video retrieval, key-moment detection | Generated clips as extra training data; video-generation models used as feature extractors [Speculation for how much this helps] | Rare actions and events are hard to collect; generated data can fill gaps |
| Anomaly detection in video or images | Likelihood models (autoregressive, flows) or reconstruction error (VAE) | Low likelihood suggests unusual input, but see the warning in Common mistakes |
| Robot control (vision-language-action models) | Flow matching or diffusion over action chunks, or autoregressive action tokens | Several good actions may exist; a generative model can represent all of them instead of averaging them |
Common mistakes
- Treating high likelihood as good samples, or the reverse. They are largely independent in high dimensions (Theis et al.). A model can have great bits per dimension and poor samples, and a GAN with great samples has no likelihood at all.
- Using likelihood for out-of-distribution detection without checking. Flow, VAE and autoregressive models trained on CIFAR-10 can give higher likelihood to SVHN house numbers than to CIFAR test images (Nalisnick et al.). [Certain] Low-level statistics such as smoothness dominate the likelihood.
- Comparing FIDs that are not comparable. Different sample counts, image resizing, JPEG settings or Inception implementations change FID. Compare only numbers computed with the same code and the same number of samples.
- Counting steps instead of network calls. With classifier-free guidance a 50-step diffusion sampler makes 100 network evaluations (or 50 calls with a doubled batch). Report both.
- Mixing up the two meanings of “flow”. A normalizing flow gives an exact likelihood in one pass; a flow-matching model needs an ODE solver.
- Judging text-to-image alignment with CLIP score alone. CLIP misses counts and spatial relations, and if CLIP was used to guide generation, CLIP score is no longer an independent judge.
- Calling autoregressive models “slow” without saying which kind of slow. Their total compute is small; their latency is high because the calls are sequential. Diffusion is the opposite.
Interview questions
Name the five families of generative models. For each, what does the network learn and how do you sample?basic
Which families give an exact likelihood, which give a bound, and which give none?core
Why are VAE samples often blurry?core
What is mode collapse, and why do likelihood-trained models not suffer from it in the same way?advanced
Why is diffusion sampling slow, and what are the main ways to speed it up?core
Compare the inference cost of an autoregressive image model and a diffusion model for a 256-token image.advanced
What does FID measure, and what are its main problems?core
CLIP score and human-preference models: what do they measure and how can they mislead?core
Where do LLMs and image diffusion models meet today?advanced
A model reports 3.0 bits per dimension on CIFAR-10. What is its negative log-likelihood per image in nats?basic
Why does a GAN with a perfect discriminator minimize the Jensen–Shannon divergence, and why is that not the full story in practice?advanced
Exercises
- In Figure 2, compute the probability of the image “top bar” \((1, 1, 0, 0)\) with the chain rule. Use \(p(x_1 = 1) = 0.5\), \(p(x_2 = 1 \mid 1) = 0.35\), \(p(x_3 = 0 \mid 1, 1) = 0.905\), \(p(x_4 = 0 \mid 1, 1, 0) = 0.947\).
- A \(512 \times 512\) image goes through a VAE with downsampling factor 8 and 4 latent channels, then a DiT with \(2 \times 2\) patches. How many tokens does the DiT see? How many sequential calls does a 30-step sampler with classifier-free guidance make, and how many network evaluations?
- Compute the 1-D FID between real features \(\mathcal{N}(1, 2^2)\) and generated features \(\mathcal{N}(1, 1^2)\). What failure does this number describe?
- In Figure 5, set \(\pi = 0.25\), \(\sigma = 0.5\). Read the three metrics. Which ones notice that the left mode is under-represented?
- Explain in two sentences why one DDIM step from pure noise gives the data mean in Figure 3.
Answerscheck
Key takeaways
- Every generative model approximates \(p_{\text{data}}\); maximum likelihood (222.1) is the same as minimizing \(\KL(p_{\text{data}} \| p_\theta)\), the LLM training loss.
- Autoregressive models factor the likelihood exactly (222.2) and sample one piece at a time; VAEs optimize a lower bound (222.3) and sample in one call but blur; GANs (222.4) are sharp and fast with no likelihood and risk mode collapse.
- Diffusion (222.5, 222.6) and flow matching (222.8) learn a path from noise to data and need several steps; normalizing flows (222.7) give exact likelihood with constrained architectures.
- The trade-off: likelihood, sample quality and speed. Autoregression is compute-light but serial; diffusion is compute-heavy but parallel.
- Measure with several metrics: bits per dimension (222.9) for likelihood, FID (222.10) for quality and diversity, CLIP score (222.11) for prompt match, and preference models (222.12) plus human studies for taste.
Key formulas
References for this session
- Kingma & Welling (2013). Auto-Encoding Variational Bayes
- Goodfellow et al. (2014). Generative Adversarial Networks
- Ho et al. (2020). Denoising Diffusion Probabilistic Models
- Lipman et al. (2022). Flow Matching for Generative Modeling
- Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
- Peebles & Xie (2022). Scalable Diffusion Models with Transformers
- Esser et al. (2020). Taming Transformers for High-Resolution Image Synthesis
- Zhou et al. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
- Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models
- Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
- Theis, van den Oord & Bethge (2015). A note on the evaluation of generative models
- Xiao, Kreis & Vahdat (2021). Tackling the Generative Learning Trilemma with Denoising Diffusion GANs
- van den Oord, Kalchbrenner & Kavukcuoglu (2016). Pixel Recurrent Neural Networks
- van den Oord et al. (2016). WaveNet: A Generative Model for Raw Audio
- Dinh, Sohl-Dickstein & Bengio (2016). Density estimation using Real NVP
- Kingma & Dhariwal (2018). Glow: Generative Flow with Invertible 1x1 Convolutions
- Karras, Laine & Aila (2018). A Style-Based Generator Architecture for Generative Adversarial Networks
- Heusel et al. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (FID)
- Salimans et al. (2016). Improved Techniques for Training GANs (Inception Score)
- Hessel et al. (2021). CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- Kirstain et al. (2023). Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation (PickScore)
- Wu et al. (2023). Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
- Xu et al. (2023). ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
- Ghosh, Hajishirzi & Schmidt (2023). GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
- Yu et al. (2022). Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (Parti)
- Sun et al. (2024). Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (LlamaGen)
- Tian et al. (2024). Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
- Chang et al. (2022). MaskGIT: Masked Generative Image Transformer
- Nie et al. (2025). Large Language Diffusion Models
- Jayasumana et al. (2024). Rethinking FID: Towards a Better Evaluation Metric for Image Generation (CMMD)
- Kynkäänniemi et al. (2019). Improved Precision and Recall Metric for Assessing Generative Models
- Nalisnick et al. (2018). Do Deep Generative Models Know What They Don't Know?
- Chong & Forsyth (2019). Effectively Unbiased FID and Inception Score and where to find them