LLM & VLM courseBlog
222Track 9: Generative models · session 1 of 12

The map of generative models: autoregressive, VAE, GAN, diffusion, flow

An LLM and an image generator solve the same problem: learn a probability distribution from examples, then draw new samples from it. This session puts the five big families side by side (autoregressive, VAE, GAN, diffusion, flow), shows what each one learns and how it samples, and explains the trade-off between exact likelihood, sample quality and speed. We end with how generated images are measured: FID, CLIP score and human preference.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Session 3 (probability, cross-entropy and KL divergence), session 6 (next-token prediction); session 166 (CLIP) helps for the metrics part

What you will learn

  • What it means to “learn a distribution”, and why maximum likelihood is the same as minimizing a KL divergence
  • What each of the five families learns, how it is trained, and how it draws a sample, with one formula each
  • Which families give an exact likelihood, a lower bound, or none at all, and why that matters
  • The trade-off triangle: likelihood, sample quality and sampling speed, with real network-call counts
  • Where LLMs and image generators meet: image tokens, diffusion transformers and hybrid models such as Transfusion
  • How FID, CLIP score and human-preference models work, and how each of them can fool you

Why this matters

A chat model writing an answer and an image model painting a cat look like very different machines. Under the hood they solve the same problem: learn a probability distribution from examples, then draw new samples from it. They only differ in how they do it.

A generative model is a model that can produce new data (text, images, audio, video) that looks like its training data. There are five big families: autoregressive models (every LLM), variational autoencoders (VAEs), generative adversarial networks (GANs), diffusion models, and flow models. Each one makes a different bet about what to learn and how to sample, and each bet has a price. Some give you an exact probability for any input but sample slowly. Some sample in one network call but give no probability at all. Some make sharp images but forget whole parts of the data.

This session is the map for the whole generative track. Each later session zooms into one region of it: session 223, session 224, diffusion in two parts (session 225 and session 226), session 227, session 228, session 229, fast samplers in session 230, session 231 and finally models that do everything at once in session 232. Real systems mix the families: Stable Diffusion 3 is a VAE plus a rectified-flow transformer; Parti and LlamaGen are LLM-style autoregressive models over image tokens; Transfusion trains one transformer with a next-token loss on text and a diffusion loss on images. [Certain] By the end of this session you will be able to place any of them on the map.

What does it mean to learn a distribution?

Think of all possible \(256 \times 256\) color images. Real photos are a tiny, strangely shaped region of that huge space. There is some unknown probability distribution \(p_{\text{data}}(x)\) that says how likely each image is to appear as a real photo. We never see this distribution. We only see samples from it: our training set.

A generative model is a family of distributions \(p_\theta(x)\) with weights \(\theta\). Training should move \(p_\theta\) close to \(p_{\text{data}}\). The classic way is maximum likelihood: make the training examples as probable as possible under the model.

\[ \theta^\star = \argmax_\theta \; \E_{x \sim p_{\text{data}}} \big[ \log p_\theta(x) \big] = \argmin_\theta \; \KL\big(p_{\text{data}} \,\|\, p_\theta\big) \tag{222.1} \]
Read the symbols
\(p_{\text{data}}(x)\)
the true (unknown) distribution of real data
\(p_\theta(x)\)
the model’s distribution; \(\theta\) are its weights
\(\log p_\theta(x)\)
the log-likelihood of one example: higher means the model finds it more probable
\(\KL(p \,\|\, q)\)
the Kullback–Leibler divergence, a measure of how different \(q\) is from \(p\); it is 0 only when they are equal

The two sides are equal because \(\KL(p_{\text{data}} \| p_\theta) = \E[\log p_{\text{data}}] - \E[\log p_\theta]\), and the first term does not depend on \(\theta\). This is exactly the cross-entropy loss you know from session 3. An LLM trained with next-token cross-entropy is doing maximum likelihood.

A trained generative model can be asked two different questions, and the families differ in which ones they can answer:

  1. Density (scoring): “How probable is this \(x\)?” Give a number \(p_\theta(x)\). Useful for compression, anomaly detection and comparing models.
  2. Sampling: “Give me a new \(x\).” Draw \(x \sim p_\theta\). This is what users see.

There is one hard part shared by all families: \(p_\theta(x)\) must sum (or integrate) to 1 over all possible \(x\). For images, that is a sum over more possible images than atoms in the universe. Each family is, at heart, a trick to avoid computing that sum.

The five families at a glance

Figure 1 is the map. Click a family to see what its network learns, how it is trained, how it draws a sample, and what kind of likelihood it gives. The sections after it explain each family with one key formula.

Figure 1. The map of generative models. Each schematic shows the path from random noise (or nothing) to a sample. Orange marks the part that is trained; dashed arrows are used only during training.
How to read Figure 1

Look at three things for each family. What is random: the autoregressive model draws a random choice at every token; the others draw a random vector \(z\) once (and diffusion adds noise along the way if you use a stochastic sampler). How many network calls per sample: shown in the readout. What the loss is: only autoregressive and flow models train on the exact likelihood; VAE and diffusion train on a bound; GANs train against a second network.

Family 1: autoregressive models

Split the data into a sequence of pieces \(x = (x_1, x_2, \dots, x_n)\): words, pixels, audio samples, or image tokens. The chain rule of probability says any joint distribution can be written, exactly, as a product of one-step conditionals:

\[ p_\theta(x) = \prod_{i=1}^{n} p_\theta\big(x_i \mid x_1, \dots, x_{i-1}\big), \qquad \log p_\theta(x) = \sum_{i=1}^{n} \log p_\theta\big(x_i \mid x_{\lt i}\big) \tag{222.2} \]

The network only has to output one small distribution at a time, over the vocabulary for the next piece. A softmax over 50,000 tokens is easy to normalize, so the hard “sum over all images” problem disappears. This is the model you already know from session 6. Training is parallel (all positions at once with a causal mask); sampling is sequential: piece 1, then piece 2 given piece 1, and so on.

  • Learns: the next-piece distribution \(p_\theta(x_i \mid x_{\lt i})\).
  • Likelihood: exact, by formula (222.2). That is why LLMs report perplexity.
  • Samples: \(n\) sequential network calls for \(n\) pieces. With a KV cache each call only processes one new piece (session 80).
  • Examples: GPT and Llama (text tokens); PixelRNN/PixelCNN (pixels); WaveNet (audio samples); Parti and LlamaGen (image tokens from a learned tokenizer). [Certain]

Pixels are a poor choice of piece: a \(256 \times 256\) RGB image has \(256 \cdot 256 \cdot 3 = 196{,}608\) sub-pixel values, so sampling one image would take 196,608 sequential steps. Modern autoregressive image models first compress the image into a short grid of discrete image tokens with a VQ tokenizer (session 224). LlamaGen’s tokenizer downsamples by 16 in each direction, so a \(256 \times 256\) image becomes \(16 \times 16 = 256\) tokens. [Certain]

Figure 2 runs this on the smallest possible “image”: four black-or-white pixels. The data are mostly vertical or horizontal bars.

Figure 2. Autoregressive sampling of a 2×2 binary image in raster order (top-left, top-right, bottom-left, bottom-right). The data distribution: left bar 30%, right bar 30%, top bar 15%, bottom bar 15%, and the other 12 patterns share the last 10%. At each step the model gives \(p(x_i = 1 \mid x_{\lt i})\), we roll a die, and the log-probabilities add up.
How to read Figure 2

The orange bar is the probability that the current pixel is white, given the pixels already chosen. Watch how it changes: once the first pixel is white and the second is black, the third pixel is white with probability 0.949, because “left bar” is now the most likely explanation. At the last step the readout shows the sum of log-probabilities, which equals the log of the joint probability of the finished image: that is formula (222.2). “New sample” uses the next seed of a fixed random generator, so results repeat.

Worked example: the likelihood of one image

Take the image “left bar”: pixels \((1, 0, 1, 0)\). From the data distribution of Figure 2, the four conditionals are \(p(x_1 = 1) = 0.5\), \(p(x_2 = 0 \mid 1) = 0.65\), \(p(x_3 = 1 \mid 1, 0) = 0.949\) and \(p(x_4 = 0 \mid 1, 0, 1) = 0.973\).

Sum of logs: \(\ln 0.5 + \ln 0.65 + \ln 0.949 + \ln 0.973 = -0.693 - 0.431 - 0.053 - 0.027 = -1.204\). And indeed \(\ln 0.30 = -1.204\): the chain rule is exact. A noise pattern such as \((1, 1, 1, 0)\) has \(p = 0.1/12 = 0.0083\), \(\ln p = -4.79\). The model scores both exactly.

Family 2: variational autoencoders (VAEs)

A VAE assumes each data point is made from a hidden latent vector \(z\) (a short code, for example 16 numbers, or a small grid for images). Sampling is simple: draw \(z\) from a standard Gaussian \(p(z) = \mathcal{N}(0, I)\), then pass it through a decoder network that outputs \(x\). The probability of an image then needs an integral over all codes, \(p_\theta(x) = \int p_\theta(x \mid z)\, p(z)\, dz\), which we cannot compute. The VAE trick is to add an encoder \(q_\phi(z \mid x)\) that guesses which codes could have made \(x\), and to optimize a lower bound on the log-likelihood, the evidence lower bound (ELBO):

\[ \log p_\theta(x) \;\ge\; \underbrace{\E_{z \sim q_\phi(z \mid x)} \big[ \log p_\theta(x \mid z) \big]}_{\text{reconstruction}} \;-\; \underbrace{\KL\big( q_\phi(z \mid x) \,\|\, p(z) \big)}_{\text{keep codes near } \mathcal{N}(0, I)} \tag{222.3} \]
Read the symbols
\(z\)
the latent code; much smaller than \(x\)
\(q_\phi(z \mid x)\)
the encoder: a Gaussian over codes, with mean and variance predicted from \(x\)
\(p_\theta(x \mid z)\)
the decoder: how likely \(x\) is given code \(z\); with a Gaussian decoder, \(-\log p_\theta(x \mid z)\) is a squared error
\(p(z)\)
the prior, a standard Gaussian; sampling starts here
  • Learns: an encoder and a decoder, trained together.
  • Likelihood: a lower bound (the ELBO), not the exact value.
  • Samples: one decoder call. Very fast.
  • Weakness: samples are often blurry. The squared-error reconstruction term is minimized by predicting the average of all images that fit a code, and an average of sharp images is a blurry image.

Today the VAE’s most important job is as a compressor for other models. Latent diffusion (Stable Diffusion) runs diffusion inside a VAE’s latent space: a \(512 \times 512 \times 3\) image becomes a \(64 \times 64 \times 4\) latent, 48 times fewer numbers. [Certain] The full story, including the reparameterization trick and posterior collapse, is in session 223.

Family 3: generative adversarial networks (GANs)

A GAN gives up on likelihood completely. A generator \(G\) turns noise \(z\) into an image \(G(z)\). A second network, the discriminator \(D\), sees real images and generated images and outputs the probability that its input is real. They play a game:

\[ \min_G \max_D \; \E_{x \sim p_{\text{data}}} \big[ \log D(x) \big] + \E_{z \sim p(z)} \big[ \log\big(1 - D(G(z))\big) \big] \tag{222.4} \]

The discriminator learns to tell real from fake; the generator learns to fool it. If \(D\) were perfect, this game would minimize the Jensen–Shannon divergence between \(p_{\text{data}}\) and the generator’s distribution (the value is \(2\,\mathrm{JSD} - \log 4\)). [Certain] In practice neither network is perfect, and training is a balancing act.

  • Learns: a generator (kept) and a discriminator (thrown away after training).
  • Likelihood: none. The generator defines a distribution only through its samples; you cannot ask “how probable is this image?”.
  • Samples: one generator call, and samples are sharp. StyleGAN faces are the classic example.
  • Weaknesses: unstable training, and mode collapse: the generator finds a few outputs that fool \(D\) and stops producing the rest of the data. Nothing in the loss punishes it for ignoring part of the data, because there is no likelihood term.

GANs are no longer the main tool for text-to-image, but the adversarial loss lives on as a helper: VQGAN tokenizers and many VAE decoders add a discriminator loss to make reconstructions sharp, and several few-step diffusion distillation methods use one too. [Likely]

Family 4: diffusion models

Diffusion splits the hard job “noise to image” into many easy jobs “slightly less noisy image”. First define a fixed forward process that slowly destroys data by adding Gaussian noise. At noise level \(t\) (from 0, clean, to 1, pure noise), a noisy version of a data point \(x_0\) is

\[ x_t = \sqrt{\bar\alpha_t}\; x_0 + \sqrt{1 - \bar\alpha_t}\; \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I) \tag{222.5} \]

where \(\bar\alpha_t\) goes from 1 (no noise) to about 0 (all noise). A network \(\varepsilon_\theta\) is trained to look at \(x_t\) and guess the noise that was added:

\[ \mathcal{L}_{\text{diff}}(\theta) = \E_{x_0,\, t,\, \varepsilon} \Big[ \big\lVert \varepsilon - \varepsilon_\theta(x_t, t) \big\rVert^2 \Big] \tag{222.6} \]
Read the symbols
\(x_0\)
a clean training example
\(x_t\)
the same example after adding noise of level \(t\)
\(\bar\alpha_t\)
the noise schedule: how much of the signal survives at level \(t\)
\(\varepsilon_\theta(x_t, t)\)
the denoising network’s guess of the noise; from it you also get a guess of \(x_0\)

To sample, start from pure noise and apply the denoiser again and again, each time removing a little noise. DDPM used \(T = 1000\) steps and reached a then state-of-the-art FID of 3.17 on CIFAR-10. [Certain] Faster samplers such as DDIM take larger steps; 20 to 50 steps are common, and distillation brings it down to 1 to 4 (session 226, session 230).

  • Learns: a denoiser (equivalently, the score \(\nabla_x \log p_t(x)\), the direction toward more probable data at each noise level).
  • Likelihood: a lower bound (an ELBO, like the VAE), or an exact value through an ODE that is expensive to compute.
  • Samples: many network calls, each on the full image. Slow, but samples are sharp and diverse.

The DDPM paper itself notes that its sampling can be seen as a generalization of autoregressive decoding: instead of “one more token”, each step adds “one more level of detail”. [Certain] Figure 3 lets you see why one big step is not enough.

data density \(p_{\text{data}}\)noised density \(p_t\)histogram of 2,000 generated samples
Figure 3. Diffusion on a 1-D toy: 30% of the data sits near −2 and 70% near +1.5. Top: the forward process of formula (222.5) blurs the two bumps into one Gaussian. Bottom: deterministic DDIM sampling from 2,000 fixed noise values with \(S\) steps, using the exact denoiser for this data (so all error comes from the step count, not from training).
How to read Figure 3

Move \(t\) to 1: the top curve becomes the standard Gaussian, which is where sampling starts. Now set \(S = 1\): every sample lands on the same point, 0.45, the average of the data. One step from pure noise can only predict the mean, the same reason VAE samples are blurry. With 2, 3, 5 steps the two bumps appear; by 20 to 50 steps the histogram matches the data, the fraction of samples left of 0 is close to 0.30, and the 1-D Fréchet distance (formula 222.10, explained later) is near 0.

Family 5: flow models

“Flow” names two related ideas. Both move samples from a simple distribution (a Gaussian) to the data distribution.

Normalizing flows

A normalizing flow is an invertible network \(f\) that maps data \(x\) to a code \(z = f(x)\) of the same size, where \(z\) should look Gaussian. Because \(f\) is invertible, the change-of-variables rule gives the exact likelihood:

\[ \log p_\theta(x) = \log p_Z\big(f(x)\big) + \log \left| \det \frac{\partial f(x)}{\partial x} \right| \tag{222.7} \]

The determinant term measures how much \(f\) stretches or squeezes space near \(x\). For a general network it would cost \(O(d^3)\) for \(d\) dimensions, so flow layers are designed to have cheap determinants (RealNVP, Glow use triangular “coupling” layers). [Certain] Sampling is one pass of \(f^{-1}\). The price: every layer must be invertible and keep the full size of \(x\), which limits the architecture, and image quality fell behind GANs and diffusion. [Likely]

Flow matching and rectified flow

The modern meaning of “flow” is closer to diffusion. Pick a noise sample \(x_0 \sim \mathcal{N}(0, I)\) and a data sample \(x_1\), and draw a straight line between them. A network \(v_\theta\) learns the velocity that moves points along such lines:

\[ x_t = (1 - t)\, x_0 + t\, x_1, \qquad \mathcal{L}_{\text{FM}}(\theta) = \E_{x_0,\, x_1,\, t} \Big[ \big\lVert v_\theta(x_t, t) - (x_1 - x_0) \big\rVert^2 \Big] \tag{222.8} \]

To sample, solve the ordinary differential equation \(dx/dt = v_\theta(x, t)\) from \(t = 0\) to \(1\) with a few Euler steps. Note that here \(t = 0\) is noise and \(t = 1\) is data, the opposite of formula (222.5); papers use both conventions. Stable Diffusion 3 is a rectified-flow transformer. [Certain] In this course “diffusion” and “flow matching” are close cousins: both learn a vector field from noise to data and both need several network calls. Details in session 229.

The trade-off triangle: likelihood, quality, speed

Put the five families next to each other and a pattern appears. We would like three things:

  1. Likelihood: an exact or tight \(\log p_\theta(x)\), so that we can score inputs, compress data, and know the model covers all of the data.
  2. Sample quality: sharp, realistic, diverse samples.
  3. Speed: few sequential network calls per sample.

No classic family gets all three. Xiao et al. call this the generative learning trilemma, with “mode coverage” in place of likelihood: GANs are fast and sharp but miss modes; VAEs and normalizing flows are fast and cover modes but have lower quality; diffusion has quality and coverage but is slow. [Certain] Autoregressive models add a fourth corner: exact likelihood and very high quality, paid for with one network call per token.

sequential network calls (latency)total work, in full-image passes (compute)
Figure 4. Left: a rough picture of where each family sits between the three goals (qualitative, from the trilemma paper and the sections above, not a measurement). Right: the cost of one sample when the image is \(L\) tokens or latent positions. Bars use a log scale.
How to read Figure 4

The two bars measure different things. Sequential calls set the latency floor: call \(k+1\) cannot start before call \(k\) ends. Total work counts how many times the network sees the whole image, which sets the compute bill. An autoregressive model with a KV cache makes \(L\) calls but each call processes one new token, so its total work is about one full pass (plus attention to the cache). A diffusion model makes only \(S\) calls, but each one processes all \(L\) positions, so its work is \(S\) passes. Change \(L\): only the autoregressive latency grows. Change \(S\): only diffusion and flow matching change.

Worked example: 256 image tokens, autoregressive vs diffusion

Take a \(256 \times 256\) image represented as 256 positions (a LlamaGen-style \(16 \times 16\) token grid, or a DiT-XL/2 latent: \(32 \times 32 \times 4\) with \(2 \times 2\) patches gives \(16 \times 16 = 256\) tokens [Certain]).

Autoregressive: 256 sequential calls, each processing 1 new token: \(256 \times 1 = 256\) token-positions in total, about one full pass.

Diffusion, 50 steps: 50 sequential calls, each processing 256 positions: \(50 \times 256 = 12{,}800\) token-positions, 50 times the autoregressive work. With classifier-free guidance each step runs two passes (with and without the prompt): 25,600 token-positions (and 512 for an autoregressive model that also uses guidance).

So diffusion is compute-heavy but parallel, and autoregression is compute-light but serial. Small-batch autoregressive decoding is limited by memory bandwidth, not FLOPs (session 82), which is why serving tricks from LLMs (KV cache, batching, speculative decoding) also speed up autoregressive image models: LlamaGen reports a 326% to 414% speedup from an LLM serving framework. [Certain]

Why likelihood and sample quality pull apart

Maximum likelihood (formula 222.1) minimizes \(\KL(p_{\text{data}} \| p_\theta)\). This divergence is huge wherever the data has mass and the model has almost none. So a likelihood-trained model is punished hard for missing any part of the data, and only mildly for putting a little mass in places where there is no data. The result is mass-covering: it spreads out, which can look like blur or odd samples. A GAN has no such term, so it can drop whole modes and still produce perfect-looking samples (mode-seeking). Theis et al. showed that log-likelihood and visual sample quality are largely independent in high dimensions: one does not predict the other. [Certain] Figure 5, in the metrics section, lets you see this with numbers.

Where LLMs and image generators meet

For a few years text used autoregressive transformers and images used diffusion U-Nets. Today the two worlds share almost everything:

  1. Images as tokens, generated by an LLM. A VQ tokenizer turns an image into discrete tokens; a decoder-only transformer predicts them one by one, exactly like text. Parti (up to 20B parameters, an encoder-decoder), LlamaGen (a plain Llama architecture, 2.18 FID on class-conditional ImageNet \(256 \times 256\)) and Chameleon (one model, one token stream for text and images) work this way. [Certain]
  2. Diffusion with a transformer backbone. DiT replaced the U-Net with a ViT-style transformer over latent patches; SD3’s MM-DiT also attends jointly over text and image tokens. [Certain] The scaling and serving know-how from LLMs carries over.
  3. One model, two losses. Transfusion trains a single transformer with next-token prediction on text and a diffusion loss on image patches, in one sequence. [Certain]
  4. Not strictly left to right. MaskGIT predicts many masked image tokens in parallel and refines them over a few steps; VAR predicts the image “next scale” by next scale, coarse to fine. [Certain] In the other direction, masked diffusion language models such as LLaDA generate text by iterative unmasking instead of left to right. [Certain]
  5. The language model as the brain. In most text-to-image systems, a text encoder (CLIP or a T5-style LLM) turns the prompt into vectors that condition the diffusion model (session 227). [Likely]

Measuring generated images: FID, CLIP score, human preference

For text LLMs we have perplexity (likelihood) and task benchmarks. For image generators the main question is “are the samples good?”, and there is no exact answer. Three kinds of metric are used, each measuring something different.

Likelihood in bits per dimension

Models with a likelihood (autoregressive, flows, VAE and diffusion bounds) report the negative log-likelihood in bits per dimension, so that images of different sizes can be compared:

\[ \text{bpd} = \frac{-\log p_\theta(x)}{D \cdot \ln 2} \tag{222.9} \]

Here \(D\) is the number of values in the image (\(32 \times 32 \times 3 = 3{,}072\) for CIFAR-10) and \(\ln 2\) converts nats to bits. Example: 3.75 bpd on CIFAR-10 means \(-\log p = 3.75 \times 3072 = 11{,}520\) bits, or \(11{,}520 \times \ln 2 = 7{,}985\) nats per image. This tells you how well the model would compress images, not how good its samples look.

FID: Fréchet Inception Distance

FID compares the set of generated images with a set of real images. Both sets are passed through an Inception-v3 classifier; each image becomes a 2,048-dimensional feature vector. Fit a Gaussian (mean \(\mu\), covariance \(\Sigma\)) to each set of features and compute the Fréchet distance between the two Gaussians:

\[ \text{FID} = \big\lVert \mu_r - \mu_g \big\rVert_2^2 + \operatorname{Tr}\Big( \Sigma_r + \Sigma_g - 2\big(\Sigma_r \Sigma_g\big)^{1/2} \Big) \tag{222.10} \]
Read the symbols
\(\mu_r, \Sigma_r\)
mean and covariance of the features of real images
\(\mu_g, \Sigma_g\)
the same for generated images
\(\operatorname{Tr}\)
the trace: the sum of the diagonal of a matrix
\((\cdot)^{1/2}\)
the matrix square root

Lower is better; 0 means the two Gaussians are identical. The first term catches samples that look different on average (low quality); the second catches a wrong spread, including missing diversity. FID needs many samples (50,000 is a common choice [Likely]) and it is biased upward when the sample count is small, so FIDs computed with different sample counts or different implementations are not comparable. [Certain]

Worked example: FID in one dimension

In 1-D the formula becomes \((\mu_r - \mu_g)^2 + (\sigma_r - \sigma_g)^2\), because \(\sigma_r^2 + \sigma_g^2 - 2\sigma_r\sigma_g = (\sigma_r - \sigma_g)^2\).

Real features: \(\mu_r = 0\), \(\sigma_r = 1\). Generated: \(\mu_g = 0.5\), \(\sigma_g = 0.6\). Then \(\text{FID} = 0.5^2 + 0.4^2 = 0.25 + 0.16 = 0.41\). The generator loses 0.25 for being shifted and 0.16 for having too little spread.

CLIP score: does the image match the prompt?

FID ignores the prompt. For text-to-image, we also want alignment: does the image show what was asked? CLIP (session 166) embeds the image and the prompt in the same space; their cosine similarity is a measure of match. CLIPScore defines

\[ \text{CLIPScore}(I, T) = w \cdot \max\big( \cos(\mathbf{e}_I, \mathbf{e}_T),\, 0 \big), \qquad w = 2.5 \tag{222.11} \]

where \(\mathbf{e}_I\) and \(\mathbf{e}_T\) are the CLIP image and text embeddings. [Likely: the value of \(w\) is from the paper body] Many text-to-image papers instead report \(100 \times \cos\). CLIP score is cheap and needs no reference images, but CLIP is weak at counting, spatial relations and attribute binding (“a red cube on a blue sphere”). Object-focused tests such as GenEval check those with a detector. [Certain]

Human preference

In the end, people decide. Human raters compare two images for the same prompt and pick the better one. Since this is slow and costly, preference models are trained on many such votes: PickScore (from the Pick-a-Pic dataset), HPSv2 and ImageReward. [Certain] They are trained like the reward models of RLHF (session 67), with a pairwise model of the form

\[ P(A \succ B \mid \text{prompt}) = \sigma\big( r(A) - r(B) \big) \tag{222.12} \]

where \(r\) is the learned score and \(\sigma\) the sigmoid. [Likely: the exact training losses differ by paper] These scores correlate better with people than FID does on modern text-to-image models, but they can be hacked: if you optimize a generator against one of them, it learns that model’s taste and quirks. Report several metrics, and always keep a human study for final claims.

real data: 50% at −2, 50% at +2, width 0.5model
Figure 5. Three numbers for one model. Likelihood: the average negative log-likelihood of real data under the model (lower is better; the best possible is 1.419 nats). Sample realism: the average log-density of model samples under the real distribution (higher means samples look more real). 1-D FID: formula (222.10) on the raw values.
How to read Figure 5

Press “Mode collapse”: the model puts everything on the right bump. Its samples are exactly as realistic as real data (realism −1.419, same as the perfect model), but the likelihood explodes to 16.7 nats because half of the real data now has almost zero probability. FID sees it too (6.44), mostly through the shifted mean. Press “Blurry model”: likelihood stays reasonable (2.03) but samples land between the bumps, so realism drops to −4.39. Now set \(\sigma = 0.2\) with \(\pi = 0.5\): samples are more “realistic” than real data and FID is only 0.003, but likelihood gets worse (3.13). Each metric can be fooled; together they tell the story.

Putting it together: one training step and one sample per family

Algorithm 1: one training step for each family
  1. input: a batch of real data \(x\)
  2. autoregressive: loss \(= -\sum_i \log p_\theta(x_i \mid x_{\lt i})\) // formula (222.2), all positions in parallel with a causal mask
  3. VAE: \(\mu, \sigma \leftarrow \text{enc}(x)\); \(z \leftarrow \mu + \sigma \odot \varepsilon\); loss \(= -\log p_\theta(x \mid z) + \KL\) // formula (222.3)
  4. GAN: update \(D\) to raise \(\log D(x) + \log(1 - D(G(z)))\); then update \(G\) to raise \(\log D(G(z))\) // formula (222.4), the common “non-saturating” form for G
  5. diffusion: \(t \sim U(0,1)\), \(\varepsilon \sim \mathcal{N}(0, I)\), \(x_t \leftarrow \sqrt{\bar\alpha_t} x + \sqrt{1 - \bar\alpha_t}\, \varepsilon\); loss \(= \lVert \varepsilon - \varepsilon_\theta(x_t, t) \rVert^2\) // formulas (222.5), (222.6)
  6. flow matching: \(x_0 \sim \mathcal{N}(0, I)\), \(t \sim U(0,1)\), \(x_t \leftarrow (1-t)x_0 + t x\); loss \(= \lVert v_\theta(x_t, t) - (x - x_0) \rVert^2\) // formula (222.8)
  7. backward pass and optimizer step, as in any network
Algorithm 2: drawing one sample
  1. autoregressive: for \(i = 1 \dots n\): \(x_i \sim p_\theta(\cdot \mid x_{\lt i})\) // \(n\) sequential calls, KV cache
  2. VAE: \(z \sim \mathcal{N}(0, I)\); \(x \leftarrow \text{dec}(z)\) // 1 call
  3. GAN: \(z \sim \mathcal{N}(0, I)\); \(x \leftarrow G(z)\) // 1 call
  4. normalizing flow: \(z \sim \mathcal{N}(0, I)\); \(x \leftarrow f^{-1}(z)\) // 1 call
  5. diffusion (DDIM): \(x \sim \mathcal{N}(0, I)\); for \(k = 1 \dots S\): \(\hat x_0 \leftarrow\) denoise\((x, t_k)\); \(x \leftarrow\) re-noise \(\hat x_0\) to level \(t_{k+1}\) // \(S\) calls
  6. flow matching: \(x \sim \mathcal{N}(0, I)\); for \(k = 0 \dots S-1\): \(x \leftarrow x + \tfrac{1}{S} v_\theta(x, k/S)\) // \(S\) Euler steps

The PyTorch version trains a diffusion model and a flow-matching model on the 1-D data of Figure 3 and samples both with 1, 5 and 50 steps. It runs on a CPU in under a minute.

import math, torch
torch.manual_seed(0)

# data: 1-D mixture, 30% near -2 and 70% near +1.5 (the data of Figure 3)
def sample_data(n):
    left = torch.rand(n) < 0.3
    return torch.where(left, -2.0 + 0.4 * torch.randn(n), 1.5 + 0.6 * torch.randn(n))

def mlp():   # input (x, t) -> one number
    return torch.nn.Sequential(torch.nn.Linear(2, 128), torch.nn.SiLU(),
                               torch.nn.Linear(128, 128), torch.nn.SiLU(),
                               torch.nn.Linear(128, 1))
abar = lambda t: torch.cos(math.pi * t / 2) ** 2          # noise schedule: abar(0) = 1, abar(1) = 0

def train(net, loss_fn, steps=2000):
    opt = torch.optim.Adam(net.parameters(), lr=2e-3)
    for _ in range(steps):
        loss = loss_fn(net, sample_data(512))
        opt.zero_grad(); loss.backward(); opt.step()
    return net

# 1) diffusion: noise the data (formula 222.5), learn to undo it.
#    DDPM predicts the noise eps; here the net predicts the clean x0, an equivalent
#    target that is more stable at t = 1 (x0 = (xt - sqrt(1-abar) eps) / sqrt(abar)).
def diff_loss(net, x0):
    t, eps = torch.rand_like(x0), torch.randn_like(x0)
    xt = abar(t).sqrt() * x0 + (1 - abar(t)).sqrt() * eps
    return ((net(torch.stack([xt, t], 1)).squeeze(1) - x0) ** 2).mean()

# 2) flow matching: predict the velocity x1 - x0 (formula 222.8); x0 = noise, x1 = data
def fm_loss(net, x1):
    x0, t = torch.randn_like(x1), torch.rand_like(x1)
    xt = (1 - t) * x0 + t * x1
    return ((net(torch.stack([xt, t], 1)).squeeze(1) - (x1 - x0)) ** 2).mean()

dnet, fnet = train(mlp(), diff_loss), train(mlp(), fm_loss)

@torch.no_grad()
def sample_ddim(net, n, S):                               # deterministic DDIM, S network calls
    x = torch.randn(n); ts = torch.linspace(1.0, 0.0, S + 1)
    for t, tn in zip(ts[:-1], ts[1:]):
        x0 = net(torch.stack([x, t.expand(n)], 1)).squeeze(1)
        eps = (x - abar(t).sqrt() * x0) / (1 - abar(t)).sqrt()
        x = abar(tn).sqrt() * x0 + (1 - abar(tn)).sqrt() * eps
    return x

@torch.no_grad()
def sample_fm(net, n, S):                                 # Euler steps on dx/dt = v(x, t)
    x = torch.randn(n)
    for k in range(S):
        t = torch.full((n,), k / S)
        x = x + net(torch.stack([x, t], 1)).squeeze(1) / S
    return x

def report(name, x):                                      # 1-D Frechet distance (formula 222.10 with d = 1)
    real = sample_data(100_000)
    fd = (x.mean() - real.mean()) ** 2 + (x.std() - real.std()) ** 2
    print(f"{name:22s} left-mode fraction {(x < 0).float().mean():.3f}   1-D FD {fd:.4f}")

report("data", sample_data(20_000))
for S in (1, 5, 50):
    report(f"diffusion DDIM S={S}", sample_ddim(dnet, 20_000, S))
    report(f"flow matching S={S}", sample_fm(fnet, 20_000, S))

In our run (PyTorch 2.8 on a CPU) one step gives a left-mode fraction of 0.029 for diffusion and 0.223 for flow matching, with 1-D FD above 2 for both: one step mostly lands near the data mean, as in Figure 3. With 50 steps the fractions are 0.276 and 0.315, close to the true 0.30, and the FD drops to 0.106 and 0.007. Exact values change with hardware and library version; the trend does not. The diffusion model is a little worse here only because this tiny network was trained for 2,000 steps; Figure 3 uses the exact denoiser.

Where this is used

ApplicationFamily usedWhy that family
Chat, code and reasoning LLMsAutoregressive over text tokensExact likelihood, simple training, text is naturally a sequence
Text-to-image (Stable Diffusion 3, DiT-based systems)VAE latent + diffusion or rectified-flow transformerBest quality and diversity; the VAE makes each step cheap (48× fewer values at 512 px)
Autoregressive image generation (Parti, LlamaGen) and native multimodal LLMs (Chameleon)VQ tokenizer + autoregressive transformerReuses LLM training and serving stacks; one token stream for text and images
Image tokenizers and VAE decoders (VQGAN, SD’s VAE)Autoencoder trained with reconstruction + adversarial (GAN) loss [Likely for every specific model]The GAN loss restores sharp detail that a squared error would blur
Speech and audioAutoregressive (WaveNet on raw samples; modern models over audio-codec tokens) [Likely for the modern part]Audio is a sequence; codec tokens make it short
Video generationDiffusion or flow-matching transformers over space-time latents (session 231)Parallel generation of all frames keeps them coherent
Video understanding: action recognition, video retrieval, key-moment detectionGenerated clips as extra training data; video-generation models used as feature extractors [Speculation for how much this helps]Rare actions and events are hard to collect; generated data can fill gaps
Anomaly detection in video or imagesLikelihood models (autoregressive, flows) or reconstruction error (VAE)Low likelihood suggests unusual input, but see the warning in Common mistakes
Robot control (vision-language-action models)Flow matching or diffusion over action chunks, or autoregressive action tokensSeveral good actions may exist; a generative model can represent all of them instead of averaging them

Common mistakes

  • Treating high likelihood as good samples, or the reverse. They are largely independent in high dimensions (Theis et al.). A model can have great bits per dimension and poor samples, and a GAN with great samples has no likelihood at all.
  • Using likelihood for out-of-distribution detection without checking. Flow, VAE and autoregressive models trained on CIFAR-10 can give higher likelihood to SVHN house numbers than to CIFAR test images (Nalisnick et al.). [Certain] Low-level statistics such as smoothness dominate the likelihood.
  • Comparing FIDs that are not comparable. Different sample counts, image resizing, JPEG settings or Inception implementations change FID. Compare only numbers computed with the same code and the same number of samples.
  • Counting steps instead of network calls. With classifier-free guidance a 50-step diffusion sampler makes 100 network evaluations (or 50 calls with a doubled batch). Report both.
  • Mixing up the two meanings of “flow”. A normalizing flow gives an exact likelihood in one pass; a flow-matching model needs an ODE solver.
  • Judging text-to-image alignment with CLIP score alone. CLIP misses counts and spatial relations, and if CLIP was used to guide generation, CLIP score is no longer an independent judge.
  • Calling autoregressive models “slow” without saying which kind of slow. Their total compute is small; their latency is high because the calls are sequential. Diffusion is the opposite.

Interview questions

Name the five families of generative models. For each, what does the network learn and how do you sample?basic
Autoregressive: learns \(p(x_i \mid x_{\lt i})\); sample one piece at a time. VAE: learns an encoder and a decoder with a Gaussian latent; sample \(z \sim \mathcal{N}(0, I)\) and decode once. GAN: learns a generator against a discriminator; sample \(z\) and run the generator once. Diffusion: learns to predict the noise in \(x_t\) (equivalently the score); sample by starting from noise and denoising over many steps. Flow: a normalizing flow learns an invertible map to a Gaussian (sample by inverting once); flow matching learns a velocity field (sample by integrating an ODE over several steps).
Which families give an exact likelihood, which give a bound, and which give none?core
Exact: autoregressive models (chain rule, formula 222.2) and normalizing flows (change of variables, 222.7). Lower bound: VAEs (ELBO, 222.3) and diffusion models (also an ELBO; an exact likelihood is possible through the probability-flow ODE, but it is expensive). None: GANs, because the generator defines a distribution only through samples; there is no formula for \(p(x)\). Flow-matching models can compute an exact likelihood with an ODE plus a divergence integral, again at high cost.
Why are VAE samples often blurry?core
With a Gaussian decoder the reconstruction term is a squared error. If several sharp images are consistent with the same latent region, the squared error is minimized by their average, which is blurry. In addition, the KL term pushes the encoder to keep codes near the prior, so codes carry limited information, and the decoder must fill in the rest by averaging. Fixes: perceptual and adversarial losses (as in VQGAN), more expressive decoders (autoregressive), or using the VAE only as a compressor and generating in its latent space with diffusion.
What is mode collapse, and why do likelihood-trained models not suffer from it in the same way?advanced
Mode collapse is when a GAN generator produces only a few kinds of outputs, ignoring parts of the data. The GAN loss only asks that samples look real to the discriminator; nothing directly punishes missing data. Maximum likelihood minimizes \(\KL(p_{\text{data}} \| p_\theta)\), which becomes infinite if the model gives near-zero probability to any real example, so the model must cover every mode (mass-covering). The opposite failure appears instead: spreading mass into regions without data, which looks like blur. In Figure 5, collapsing to one mode keeps sample realism perfect but raises the negative log-likelihood from 1.42 to 16.7 nats.
Why is diffusion sampling slow, and what are the main ways to speed it up?core
Each step only removes a little noise, because the denoiser is trained to predict an average (\(\E[x_0 \mid x_t]\)), and one large step from pure noise lands on the mean of the data (Figure 3 with \(S = 1\)). The original DDPM used 1,000 steps, each a full network pass, doubled with classifier-free guidance. Speed-ups: better ODE solvers (DDIM, DPM-Solver) with 20 to 50 steps; straighter paths (rectified flow); distillation into few-step students (progressive distillation, consistency models, adversarial distillation); working in a compressed latent space (latent diffusion); and system tricks such as caching features between steps.
Compare the inference cost of an autoregressive image model and a diffusion model for a 256-token image.advanced
Autoregressive: 256 sequential calls, each processing one new token with a KV cache, so total work is about one full pass but latency is 256 serial steps; small-batch decode is memory-bandwidth bound, so batching and speculative decoding help. Diffusion with 50 steps: 50 sequential calls, each processing all 256 tokens, so 12,800 token-positions of compute (25,600 with guidance) but only 50 serial steps, each compute bound and well parallelized. Which is faster in wall-clock depends on batch size and hardware.
What does FID measure, and what are its main problems?core
FID embeds real and generated images with Inception-v3 (2,048-D pool features), fits a Gaussian to each set, and computes the Fréchet distance (formula 222.10): a mean term for average difference and a covariance term for spread and diversity. Problems: it is biased with small sample counts; it depends on implementation details (resizing, compression); Inception was trained on ImageNet classes and may not capture what matters for modern text-to-image content; the Gaussian assumption is crude; and it ignores the prompt. CMMD (CLIP features with a kernel distance) was proposed as an alternative.
CLIP score and human-preference models: what do they measure and how can they mislead?core
CLIP score measures image-text alignment as the cosine similarity of CLIP embeddings (CLIPScore scales it by 2.5 and clips at 0). It is reference-free and cheap, but CLIP is weak at counting, spatial relations and attribute binding, and it can be gamed if CLIP was used during generation. Preference models (PickScore, HPSv2, ImageReward) are trained on human pairwise choices with a Bradley–Terry style loss, so they track overall human taste better, but optimizing a generator against one of them leads to reward hacking: the generator learns that scorer’s biases. Use several metrics plus a human study.
Where do LLMs and image diffusion models meet today?advanced
(1) Autoregressive transformers over VQ image tokens (Parti, LlamaGen, Chameleon) treat images as another language. (2) Diffusion models use transformer backbones (DiT, SD3’s MM-DiT), so LLM scaling and systems knowledge transfers. (3) Hybrid models such as Transfusion use one transformer with next-token loss for text and diffusion loss for images. (4) Non-left-to-right decoding: MaskGIT and VAR for images, masked diffusion language models such as LLaDA for text. (5) LLM or CLIP text encoders condition most diffusion models.
A model reports 3.0 bits per dimension on CIFAR-10. What is its negative log-likelihood per image in nats?basic
\(D = 32 \times 32 \times 3 = 3{,}072\). By formula (222.9), \(-\log p = 3.0 \times 3072 \times \ln 2 = 9{,}216 \times 0.693 = 6{,}388\) nats per image (9,216 bits). This is the compressed size the model would achieve with an ideal entropy coder; it says nothing directly about sample quality.
Why does a GAN with a perfect discriminator minimize the Jensen–Shannon divergence, and why is that not the full story in practice?advanced
For a fixed generator, the best discriminator is \(D^\star(x) = p_{\text{data}}(x) / (p_{\text{data}}(x) + p_g(x))\). Plugging it into formula (222.4) gives \(2\,\mathrm{JSD}(p_{\text{data}} \| p_g) - \log 4\), minimized when \(p_g = p_{\text{data}}\). In practice the discriminator is never optimal, the two distributions often lie on thin, non-overlapping manifolds where JSD is constant and gives no useful gradient, and the generator usually trains on the non-saturating loss \(-\log D(G(z))\) instead. That is why GAN training needs tricks such as spectral normalization, gradient penalties and careful learning rates.

Exercises

  1. In Figure 2, compute the probability of the image “top bar” \((1, 1, 0, 0)\) with the chain rule. Use \(p(x_1 = 1) = 0.5\), \(p(x_2 = 1 \mid 1) = 0.35\), \(p(x_3 = 0 \mid 1, 1) = 0.905\), \(p(x_4 = 0 \mid 1, 1, 0) = 0.947\).
  2. A \(512 \times 512\) image goes through a VAE with downsampling factor 8 and 4 latent channels, then a DiT with \(2 \times 2\) patches. How many tokens does the DiT see? How many sequential calls does a 30-step sampler with classifier-free guidance make, and how many network evaluations?
  3. Compute the 1-D FID between real features \(\mathcal{N}(1, 2^2)\) and generated features \(\mathcal{N}(1, 1^2)\). What failure does this number describe?
  4. In Figure 5, set \(\pi = 0.25\), \(\sigma = 0.5\). Read the three metrics. Which ones notice that the left mode is under-represented?
  5. Explain in two sentences why one DDIM step from pure noise gives the data mean in Figure 3.
Answerscheck
1. \(0.5 \times 0.35 \times 0.905 \times 0.947 = 0.150\), matching the 15% of the data distribution (exact values give 0.15). 2. Latent \(64 \times 64 \times 4\); patches \(2 \times 2\) give \(32 \times 32 = 1{,}024\) tokens. 30 sequential calls; with guidance 60 network evaluations (often run as 30 calls with a batch of 2). 3. \((1 - 1)^2 + (2 - 1)^2 = 1\). The means match, but the generated spread is half the real one: a lack of diversity. 4. Negative log-likelihood 1.563 (worse than 1.419), realism −1.419 (unchanged), FID 1.067. Likelihood and FID notice; sample realism does not, because every sample still looks real. 5. At \(t = 1\) the noisy input carries no information about \(x_0\), so the best (squared-error) prediction of \(x_0\) is \(\E[x_0] = 0.3 \times (-2) + 0.7 \times 1.5 = 0.45\). One step jumps straight to that prediction for every sample, so all 2,000 samples land on 0.45.

Key takeaways

  • Every generative model approximates \(p_{\text{data}}\); maximum likelihood (222.1) is the same as minimizing \(\KL(p_{\text{data}} \| p_\theta)\), the LLM training loss.
  • Autoregressive models factor the likelihood exactly (222.2) and sample one piece at a time; VAEs optimize a lower bound (222.3) and sample in one call but blur; GANs (222.4) are sharp and fast with no likelihood and risk mode collapse.
  • Diffusion (222.5, 222.6) and flow matching (222.8) learn a path from noise to data and need several steps; normalizing flows (222.7) give exact likelihood with constrained architectures.
  • The trade-off: likelihood, sample quality and speed. Autoregression is compute-light but serial; diffusion is compute-heavy but parallel.
  • Measure with several metrics: bits per dimension (222.9) for likelihood, FID (222.10) for quality and diversity, CLIP score (222.11) for prompt match, and preference models (222.12) plus human studies for taste.

Key formulas

\[ \log p_\theta(x) = \sum_i \log p_\theta(x_i \mid x_{\lt i}), \qquad \log p_\theta(x) \ge \E_q[\log p_\theta(x \mid z)] - \KL(q \,\|\, p) \]
\[ \mathcal{L}_{\text{diff}} = \E\big\lVert \varepsilon - \varepsilon_\theta(x_t, t) \big\rVert^2, \qquad \mathcal{L}_{\text{FM}} = \E\big\lVert v_\theta(x_t, t) - (x_1 - x_0) \big\rVert^2 \]
\[ \text{FID} = \lVert \mu_r - \mu_g \rVert^2 + \operatorname{Tr}\big(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2}\big) \]

References for this session

  1. Kingma & Welling (2013). Auto-Encoding Variational Bayes
  2. Goodfellow et al. (2014). Generative Adversarial Networks
  3. Ho et al. (2020). Denoising Diffusion Probabilistic Models
  4. Lipman et al. (2022). Flow Matching for Generative Modeling
  5. Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
  6. Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
  7. Peebles & Xie (2022). Scalable Diffusion Models with Transformers
  8. Esser et al. (2020). Taming Transformers for High-Resolution Image Synthesis
  9. Zhou et al. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
  10. Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models
  11. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  12. Theis, van den Oord & Bethge (2015). A note on the evaluation of generative models
  13. Xiao, Kreis & Vahdat (2021). Tackling the Generative Learning Trilemma with Denoising Diffusion GANs
  14. van den Oord, Kalchbrenner & Kavukcuoglu (2016). Pixel Recurrent Neural Networks
  15. van den Oord et al. (2016). WaveNet: A Generative Model for Raw Audio
  16. Dinh, Sohl-Dickstein & Bengio (2016). Density estimation using Real NVP
  17. Kingma & Dhariwal (2018). Glow: Generative Flow with Invertible 1x1 Convolutions
  18. Karras, Laine & Aila (2018). A Style-Based Generator Architecture for Generative Adversarial Networks
  19. Heusel et al. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (FID)
  20. Salimans et al. (2016). Improved Techniques for Training GANs (Inception Score)
  21. Hessel et al. (2021). CLIPScore: A Reference-free Evaluation Metric for Image Captioning
  22. Kirstain et al. (2023). Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation (PickScore)
  23. Wu et al. (2023). Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
  24. Xu et al. (2023). ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
  25. Ghosh, Hajishirzi & Schmidt (2023). GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
  26. Yu et al. (2022). Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (Parti)
  27. Sun et al. (2024). Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (LlamaGen)
  28. Tian et al. (2024). Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
  29. Chang et al. (2022). MaskGIT: Masked Generative Image Transformer
  30. Nie et al. (2025). Large Language Diffusion Models
  31. Jayasumana et al. (2024). Rethinking FID: Towards a Better Evaluation Metric for Image Generation (CMMD)
  32. Kynkäänniemi et al. (2019). Improved Precision and Recall Metric for Assessing Generative Models
  33. Nalisnick et al. (2018). Do Deep Generative Models Know What They Don't Know?
  34. Chong & Forsyth (2019). Effectively Unbiased FID and Inception Score and where to find them