train a tiny diffusion model and a tiny flow-matching model
In this lab you train a diffusion model and a flow-matching model with the same network, data and budget, first on 2-D toy data and then on MNIST digits. You sample both with 1 to 128 steps, plot their sample paths, and measure quality against cost. You will see that learned flow paths are curved until you apply reflow, that DDIM and flow matching trace the same curves in different coordinates, and which model to choose for a given number of steps.
What you will learn
- How to train a DDPM (noise prediction) and a flow-matching model (velocity prediction) so that only the noisy input and the target differ, and what one training example and one sampling step look like on real images
- Why both training losses stop far above zero, and how to compute the lowest possible loss on toy data
- How the DDPM, DDIM and Euler samplers work, and why one step fails differently for diffusion (an explosion) and for flow matching (the data mean)
- How to measure the straightness of sample paths, and why learned flow-matching paths are curved
- Why DDIM and flow matching integrate the same family of curves in different coordinates
- How reflow (2-rectified flow) straightens paths and enables one-step sampling, and what it costs
- How to compare sample quality against the number of steps with a fixed noise batch, on 2-D data and on images (Fréchet distance)
Why this matters
In session 225, session 226 and session 229 you met the two big families of modern image and video generators: diffusion models, which learn to predict the noise that was added to an image, and flow-matching models, which learn a velocity, a direction and speed that carries noise to data. In this lab you train both yourself, with the same network, the same data and the same budget, and then you measure the claims that you read about: that flow paths are straight, that straight paths need fewer sampling steps, and that “reflow” makes one-step generation possible.
The previous session, session 232, ended with models such as Transfusion that put a diffusion (or flow) loss for images next to the next-token loss for text. Before you build such a system, you should have trained the image part once yourself and seen where it fails. That is this lab.
You will do it in two settings. First a 2-D toy problem (eight Gaussian blobs on a ring), where you can see every sample and every path, and where a model trains in under a minute on a laptop CPU. Then small images (MNIST digits, 28 × 28 pixels) with a small U-Net, where the same ideas must survive in 784 dimensions. Every number on this page comes from our own run of the code below, unless it is tagged as coming from a paper; two of the figures run the trained 2-D networks live in your browser.
In short, what you will find: both models reach almost exactly the lowest possible training loss, so the loss tells you nothing about sample quality; with few sampling steps flow matching beats the diffusion model trained on the same budget (8 steps: 89.6% of samples on a mode against 64.3% for DDIM); the learned flow-matching paths are not straight (straightness 0.55, curved through the center), and the DDIM paths are the very same curves in other coordinates; only reflow makes them straight (0.999) and gives a good one-step sampler, at a small cost in final quality. On MNIST the same ranking appears: flow matching has the lowest Fréchet distance at every budget from 2 to 64 steps (8 steps: 37.5 against 55.2 for DDIM and 65.1 for DDPM), and the stochastic DDPM sampler closes the gap as the steps grow (64 steps: 14.7 against 13.1).
Setup
You need Python 3.10 or newer and PyTorch. Everything runs on a CPU. We used Python 3.15, torch 2.15.0, torchvision 0.29.2, NumPy 2.5.4 and SciPy 1.18.1 on an Apple-silicon laptop that was busy with other jobs; any recent versions should work. The 2-D part takes a few minutes. The MNIST part took about 87 minutes per model on our busy CPU (6,000 steps); a GPU should do it in a few minutes.
python -m venv venv && source venv/bin/activate pip install torch torchvision numpy scipy pillow python toy2d.py # trains DDPM, flow matching and 2-rectified flow on 2-D data python eval2d.py # quality vs steps, straightness of paths python clf.py # a small MNIST classifier, used only to measure image quality python mnist.py 6000 ddpm; python mnist.py 6000 fm; python mnist_eval.py
The plan of the lab is in Figure 1. The important design rule: change one thing at a time. The diffusion model and the flow-matching model share the network, the optimizer, the learning-rate schedule, the batch size, the number of steps and the random seed. They differ only in how a training example is made noisy and in what the network must predict. The samplers also start from the same noise, so you can compare two samples that began at the same point.
Step 1: one network, two targets
Both models start from the same idea: take a clean data point \(\mathbf{x}\), mix it with Gaussian noise \(\epsilon\), and train a network to undo the mix. They differ in the recipe of the mix and in what the network outputs.
Diffusion (DDPM). The DDPM paper of Ho et al. (2020) uses \(T = 1000\) discrete steps and a linear schedule: the noise added at step \(t\) has variance \(\beta_t\), growing linearly from \(10^{-4}\) to \(0.02\) [Certain]. With \(\bar\alpha_t = \prod_{s \le t} (1 - \beta_s)\) the noisy point has a closed form (the same equation as 225.3), and the network \(\epsilon_\theta\) is trained to predict the noise:
Flow matching. We use the same convention as session 229: \(t = 0\) is data and \(t = 1\) is noise, and the noisy point lies on the straight line between them. Its speed along the line is constant, \(\epsilon - \mathbf{x}\), and that is the target. (The flow-matching paper of Lipman et al. uses the opposite direction, \(t = 0\) for noise; be careful when you copy code.)
- Read the symbols
- \(\mathbf{x}\)
- a clean training example: a 2-D point, or a 28 × 28 image scaled to \([-1, 1]\)
- \(\epsilon\)
- standard Gaussian noise of the same shape, a new draw for every example
- \(t\)
- the noise level: an integer 1 … 1000 for DDPM (we feed \(t/1000\) to the network), a real number in \([0, 1]\) for flow matching
- \(\bar\alpha_t\)
- the fraction of signal variance left at step \(t\); 0.9999 at \(t = 1\) and \(4.04 \times 10^{-5}\) at \(t = 1000\) for the linear schedule
- \(\epsilon_\theta, \mathbf{v}_\theta\)
- the same network architecture, read as a noise prediction or as a velocity prediction
Look at the two paths for one fixed pair \((\mathbf{x}, \epsilon)\). The flow-matching point moves on a straight segment. The DDPM point has coefficients \((\sqrt{\bar\alpha_t}, \sqrt{1 - \bar\alpha_t})\) whose squares add to 1: it moves on a quarter of an ellipse, a curve. This is the origin of the slogan “diffusion paths are curved, flow paths are straight”. But it is a statement about one pair. The sampler does not know which pair it is on; it follows the average velocity over all pairs that pass through the current point. Step 4 shows what that does to the shape of the paths.
One training loop serves both models. The only lines that differ are marked.
- input: a mini-batch \(\mathbf{x}\) of clean data, the network, the kind (DDPM or FM)
- draw \(\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) with the shape of \(\mathbf{x}\)
- if DDPM: draw \(t \in \{1, \dots, 1000\}\) uniformly; \(\mathbf{x}_t \leftarrow \sqrt{\bar\alpha_t}\,\mathbf{x} + \sqrt{1-\bar\alpha_t}\,\epsilon\); target \(\leftarrow \epsilon\) // differs
- if FM: draw \(t \sim \mathcal{U}[0, 1]\); \(\mathbf{x}_t \leftarrow (1-t)\,\mathbf{x} + t\,\epsilon\); target \(\leftarrow \epsilon - \mathbf{x}\) // differs
- loss \(\leftarrow\) mean squared error between network\((\mathbf{x}_t, t)\) and target
- backpropagate, take one optimizer step, update the learning-rate schedule (and the EMA copy for images)
Before we go into the details, watch both trained models at work. Figure 2 shows real frames from the two small MNIST networks that you will train in Step 6. In “Training” mode you see one training example being made: a clean digit, a noise image, their mix at the chosen noise level, the target, the network’s actual output and the clean digit that this output implies. In “Sampling” mode you see the opposite direction: three samples that start as pure noise and become digits in 16 steps. In both modes the box on the right is the loop of Algorithm 1 or Algorithm 2 below, with the numbers of one pixel filled in.
How to read Figure 2
Press Play in training mode with “Diffusion (DDPM)”: the noisy input fades from the digit into noise as \(t\) grows. Watch the bottom-right tile, the network’s guess \(\hat{\mathbf{x}}\): at small \(t\) it is the digit itself, at large \(t\) it becomes a blurry average shape, because the input no longer says which digit it was. The small chart shows the loss of 512 test digits at each noise level; for DDPM it is largest at small \(t\), where the noise is hard to see. Switch to flow matching: now the target is \(\epsilon - \mathbf{x}\). Apart from \(t = 0\), where the input contains no trace of the noise and the loss is about 1 per pixel (the variance of \(\epsilon\)), the loss is now largest at large \(t\) (0.29 at \(t = 0.9\) against 0.10 at \(t = 0.35\)), where the network must guess which digit it is. Then switch to sampling and press Play. With flow matching the guess (bottom row) is a gray blob at step 1 and becomes a clear digit after a few steps; the chart shows how fast the guess approaches the final sample. With the DDPM sampler the current image (top row) stays noisy until the last steps, because fresh noise \(\sigma \mathbf{z}\) is added in every step; with DDIM \(\sigma = 0\). DDIM and flow matching draw the same three digits from the same noise, and the DDPM sampler draws other ones; over 500 samples with 32 steps, DDIM and flow matching gave the same class in 74% of cases. Step 4 explains why.
Step 2: train both on 2-D toy data
The data are eight Gaussian blobs: centers on a circle of radius 1.5, standard deviation 0.1. Each coordinate then has a standard deviation of about 1.07, close to the scale of the noise, which is what both methods expect. The network is a small multilayer perceptron (MLP): the input is the point \((x, y)\) and 16 time features \(\sin(2^k \pi t), \cos(2^k \pi t)\) for \(k = 0, \dots, 7\); then three hidden layers of 64 units with the SiLU activation; then 2 outputs. That is 9,666 parameters. Both models train for 20,000 steps of Adam (learning rate \(10^{-3}\) with cosine decay, batch 1,024, seed 0). The full script:
# toy2d.py: DDPM vs flow matching vs 2-rectified flow on 8 Gaussians import math, json, time, torch, torch.nn as nn, numpy as np torch.set_num_threads(4) DEV = "cpu" def ring(n, g): # 8 Gaussians on a circle of radius 1.5, std 0.1 k = torch.randint(0, 8, (n,), generator=g) a = k.float() * (2 * math.pi / 8) c = 1.5 * torch.stack([a.cos(), a.sin()], 1) return c + 0.1 * torch.randn(n, 2, generator=g) FREQ = (2.0 ** torch.arange(8)) * math.pi # 8 frequencies -> 16 time features def temb(t): a = t[:, None] * FREQ return torch.cat([a.sin(), a.cos()], 1) class MLP(nn.Module): def __init__(s, w=64): super().__init__() s.net = nn.Sequential(nn.Linear(2 + 16, w), nn.SiLU(), nn.Linear(w, w), nn.SiLU(), nn.Linear(w, w), nn.SiLU(), nn.Linear(w, 2)) def forward(s, x, t): # t in [0, 1] return s.net(torch.cat([x, temb(t)], 1)) T = 1000 beta = torch.linspace(1e-4, 0.02, T, dtype=torch.float64) abar = torch.cumprod(1 - beta, 0) # abar[i] is alpha-bar for step t = i + 1 def train(kind, steps=20000, bs=1024, seed=0, pairs=None, init=None): torch.manual_seed(seed); g = torch.Generator().manual_seed(seed + 1) m = MLP() if init is not None: m.load_state_dict(init.state_dict()) opt = torch.optim.Adam(m.parameters(), lr=1e-3) sch = torch.optim.lr_scheduler.CosineAnnealingLR(opt, steps) hist = [] for it in range(steps): if pairs is None: x = ring(bs, g); e = torch.randn(bs, 2, generator=g) else: idx = torch.randint(0, len(pairs[0]), (bs,), generator=g) e, x = pairs[0][idx], pairs[1][idx] if kind == "ddpm": ti = torch.randint(1, T + 1, (bs,), generator=g) ab = abar[ti - 1].float()[:, None] xt = ab.sqrt() * x + (1 - ab).sqrt() * e loss = ((m(xt, ti.float() / T) - e) ** 2).sum(1).mean() else: t = torch.rand(bs, generator=g) xt = (1 - t[:, None]) * x + t[:, None] * e loss = ((m(xt, t) - (e - x)) ** 2).sum(1).mean() opt.zero_grad(); loss.backward(); opt.step(); sch.step() if it % 500 == 0 or it == steps - 1: hist.append((it, loss.item())) return m, hist @torch.no_grad() def sample_ddpm(m, z, N, eta, keep=False): """DDIM family on N evenly spaced timesteps; eta=1 is DDPM (ancestral), eta=0 is DDIM (ODE).""" ts = torch.linspace(T, 0, N + 1).round().long() # T ... 0 x = z.clone(); path = [x.clone()] g = torch.Generator().manual_seed(123) for i in range(N): t, s = ts[i].item(), ts[i + 1].item() at = abar[t - 1].item(); as_ = abar[s - 1].item() if s > 0 else 1.0 eps = m(x, torch.full((len(x),), t / T)) x0 = (x - math.sqrt(1 - at) * eps) / math.sqrt(at) sig = eta * math.sqrt((1 - as_) / (1 - at) * (1 - at / as_)) x = math.sqrt(as_) * x0 + math.sqrt(max(1 - as_ - sig ** 2, 0)) * eps if sig > 0: x = x + sig * torch.randn(x.shape, generator=g) if keep: path.append(x.clone()) return (x, path) if keep else x @torch.no_grad() def sample_fm(m, z, N, keep=False): x = z.clone(); path = [x.clone()] for i in range(N): t = 1 - i / N; dt = 1 / N x = x - dt * m(x, torch.full((len(x),), t)) if keep: path.append(x.clone()) return (x, path) if keep else x def sw1(a, b, nproj=64): th = torch.arange(nproj) * math.pi / nproj d = torch.stack([th.cos(), th.sin()], 1) pa = (a @ d.T).sort(0).values; pb = (b @ d.T).sort(0).values return (pa - pb).abs().mean().item() def onmode(x): a = torch.arange(8) * (2 * math.pi / 8) c = 1.5 * torch.stack([a.cos(), a.sin()], 1) d = torch.cdist(x, c).min(1).values return (d < 0.3).float().mean().item() if __name__ == "__main__": t0 = time.time() res = {} mdd, hd = train("ddpm"); print("ddpm", time.time() - t0, hd[-1]) mfm, hf = train("fm"); print("fm", time.time() - t0, hf[-1]) # reflow pairs: noise -> 1-RF output with 100 Euler steps gz = torch.Generator().manual_seed(7) zp = torch.randn(50000, 2, generator=gz) xp = sample_fm(mfm, zp, 100) mrf, hr = train("fm", pairs=(zp, xp), init=mfm, steps=10000, seed=3); print("rf", time.time() - t0, hr[-1]) torch.save({"ddpm": mdd.state_dict(), "fm": mfm.state_dict(), "rf": mrf.state_dict()}, "toy2d.pt") json.dump({"ddpm": hd, "fm": hf, "rf": hr}, open("toy2d_hist.json", "w"))
On our laptop, busy with other jobs, one model trained in 45 seconds. Now the first surprise. The losses do not go to zero: on 400,000 fresh examples the DDPM model has loss 0.409 and the flow model 2.770. Is the flow model worse? No. Because every noisy point \(\mathbf{x}_t\) could have come from many pairs \((\mathbf{x}, \epsilon)\), even a perfect network cannot predict the target exactly. The best possible prediction is the average target given \(\mathbf{x}_t\), and the lowest possible loss is the average variance that is left:
For a mixture of Gaussians the posterior \(p(\mathbf{x} \mid \mathbf{x}_t)\) is known exactly, so we computed (233.3) by Monte Carlo with 400,000 samples: the minimum is 0.407 for DDPM and 2.757 for flow matching. So both networks are within 0.5% of perfect, and the two numbers cannot be compared with each other at all: they measure errors in different units (noise against velocity), and the flow loss is large near \(t = 0\), where the factor \(1/t^2\) explodes. The reflow model of Step 4, in contrast, reaches a loss of about 0.001, because its targets are a deterministic function of the input. A training loss near its minimum tells you that the optimizer worked. It does not tell you how good the samples are. For that you must sample.
Figure 3 lets you watch what one training step sees on the 2-D data, with our trained networks running live in your browser. Move the noise level and look at three things: how the colored modes melt into one Gaussian cloud, how the path of a single pair is a curve for DDPM and a straight line for flow matching, and how the network’s guess of the clean point (the orange cross) moves from the true point to the center of the data as the noise grows.
How to read Figure 3
Press Play with DDPM. The noisy points (blue dot for the highlighted pair) move from the ring to a round Gaussian cloud. At low noise the orange cross sits on the green ring (the network recovers the clean point); at high noise it moves toward the center, the average of all modes. The error chart shows where each loss puts its weight. For DDPM the error is largest at low noise (about 2.2 at \(t = 1\), where the noise is far too small to read from the input) and falls below 0.01 above \(t = 800\). Switch to flow matching: the dashed path becomes a straight line, and the error now peaks in the middle (about 4.6 near \(t = 0.5\) to \(0.6\), where the mode is still uncertain and the factor \(1/t^2\) of (233.3) is still large). At \(t = 1\) it ends near 2.3: the best prediction is \(\mathbf{z} - \E[\mathbf{x}]\), and what is left is the total variance of the data, \(2 \times 1.135 = 2.27\). Every number here is computed live from 400 points, so it moves a little with the sample.
Step 3: sample, with many steps and with few
Sampling runs the noise level backwards, from pure noise to data, in \(N\) steps. For the diffusion model we use the DDIM family of updates on \(N\) evenly spaced timesteps \(\tau_N = 1000 \gt \tau_{N-1} \gt \dots \gt \tau_0 = 0\). From the noise prediction, form the current guess of the clean point \(\hat{\mathbf{x}}\), then move to the next noise level:
- Read the symbols
- \(t, s\)
- the current and the next (smaller) timestep; \(\bar\alpha_0 = 1\)
- \(\hat{\mathbf{x}}\)
- the network’s current guess of the final clean point
- \(\eta\)
- 1 gives the stochastic DDPM sampler (“ancestral”, with fresh noise \(\mathbf{z}\) in every step); 0 gives the deterministic DDIM sampler
For the flow model, sampling is solving the ODE \(d\mathbf{x}/dt = \mathbf{v}_\theta(\mathbf{x}, t)\) from \(t = 1\) down to \(t = 0\). The simplest solver is Euler’s method: assume that the velocity stays constant for one step.
If the true path is a straight line traveled at constant speed, one Euler step lands exactly on the end. Every bend in the path costs error, and more steps reduce the error. Both samplers call the network once per step, so the number of steps is the cost.
- input: starting noise \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\), the number of steps \(N\), the method
- if flow (FM or 2-rectified flow): \(\mathbf{x} \leftarrow \mathbf{z}\); for \(i = 0, \dots, N-1\): \(t \leftarrow 1 - i/N\); \(\mathbf{x} \leftarrow \mathbf{x} - \frac{1}{N} \mathbf{v}_\theta(\mathbf{x}, t)\)
- if diffusion: timesteps \(\tau_i = \mathrm{round}(1000 - 1000\, i / N)\), \(i = 0, \dots, N\); \(\mathbf{x} \leftarrow \mathbf{z}\)
- for each pair \((t, s) = (\tau_i, \tau_{i+1})\): predict \(\epsilon_\theta(\mathbf{x}, t)\), form \(\hat{\mathbf{x}}\), apply (233.4) with \(\eta = 1\) (DDPM) or \(\eta = 0\) (DDIM) // in the last step \(s = 0\), so \(\sigma = 0\) and \(\mathbf{x} = \hat{\mathbf{x}}\)
- return \(\mathbf{x}\) // N network calls in every case
To measure quality on the toy data we use two numbers. The on-mode fraction is the share of samples that land within 0.3 (three standard deviations) of one of the eight centers; for real data it is 99%. The sliced Wasserstein distance \(\mathrm{SW}_1\) compares the whole distribution: project both sample sets on 64 directions, sort each projection, and average the absolute differences:
Here \(\mathbf{u}_k\) are unit directions at angles \(k\pi/64\), and \(a_{(i)}\) is the \(i\)-th smallest projection. With 4,000 samples on each side, two independent sets of real data are at \(\mathrm{SW}_1 \approx 0.030\) (from 0.017 to 0.045 over 10 seeds): differences below about 0.045 are noise. The on-mode fraction is more sensitive to blur and to samples stuck between modes.
The evaluation script loads the three trained models, samples all of them from one fixed batch of 4,000 noise points, and measures both numbers and the straightness of Step 4:
import torch, math, json
from toy2d import *
torch.set_num_threads(2)
S = torch.load("toy2d.pt")
M = {}
for k in S:
m = MLP(); m.load_state_dict(S[k]); m.eval(); M[k] = m
g = torch.Generator().manual_seed(2024)
ref = ring(4000, g)
ref2 = ring(4000, g)
print("noise floor sw1", round(sw1(ref, ref2), 4), "onmode data", round(onmode(ref), 4))
z = torch.randn(4000, 2, generator=torch.Generator().manual_seed(99))
out = {}
for N in [1, 2, 4, 8, 16, 32, 64, 128, 1000]:
row = {}
for name, f in [("ddpm", lambda N: sample_ddpm(M["ddpm"], z, N, 1.0)),
("ddim", lambda N: sample_ddpm(M["ddpm"], z, N, 0.0)),
("fm", lambda N: sample_fm(M["fm"], z, N)),
("rf", lambda N: sample_fm(M["rf"], z, N))]:
x = f(N)
row[name] = (round(sw1(x, ref), 3), round(onmode(x), 3))
out[N] = row
print(N, row)
# one-step mean
x1 = sample_fm(M["fm"], z, 1); print("fm 1-step mean", x1.mean(0), "radius mean", x1.norm(dim=1).mean())
x1 = sample_ddpm(M["ddpm"], z, 1, 0.0); print("ddim 1-step radius mean", x1.norm(dim=1).mean())
x1 = sample_fm(M["rf"], z, 1); print("rf 1-step radius mean", x1.norm(dim=1).mean())
# straightness at 64 steps on 500 paths
def straight(path):
P = torch.stack(path) # (N+1, n, 2)
L = (P[1:] - P[:-1]).norm(dim=2).sum(0)
D = (P[-1] - P[0]).norm(dim=1)
return (D / L).mean().item()
zz = z[:500]
for name, f in [("ddim", lambda: sample_ddpm(M["ddpm"], zz, 128, 0.0, keep=True)[1]),
("ddpm", lambda: sample_ddpm(M["ddpm"], zz, 128, 1.0, keep=True)[1]),
("fm", lambda: sample_fm(M["fm"], zz, 128, keep=True)[1]),
("rf", lambda: sample_fm(M["rf"], zz, 128, keep=True)[1])]:
print("straightness", name, round(straight(f()), 4))
json.dump(out, open("eval2d.json", "w"))
How to read Figure 4
It opens on flow matching with 8 steps: about 92% of the samples sit on a mode. Click 1 step: all samples collapse into a small cloud at the center (mean distance from the center about 0.07), because one Euler step from \(t = 1\) lands on the average of the data. Click DDIM with 1 step: the samples explode far outside the plot (Step 3’s worked example explains why). Increase the steps for DDIM: it needs about 16 steps for most samples to land on a mode, flow matching about 8. Now click 2-rectified flow: even 1 step puts about 89% of the samples on a mode. With 64 steps compare the four methods: the reflowed model is slightly worse than plain flow matching. Click “New noise” to see how much the numbers move between seeds.
The table below is the same experiment, measured in Python with 4,000 samples from one fixed noise batch (the noise is the same for all methods). The DDPM and DDIM rows at 1 step are identical: in the last step \(s = 0\), so \(\sigma = 0\) for any \(\eta\).
| Steps \(N\) | DDPM (on-mode / \(\mathrm{SW}_1\)) | DDIM | Flow matching | 2-rectified flow |
|---|---|---|---|---|
| 1 | 2.4% / 3.030 | 2.4% / 3.030 | 0.0% / 0.916 | 86.7% / 0.039 |
| 2 | 1.0% / 0.574 | 0.3% / 0.661 | 6.0% / 0.369 | 89.1% / 0.037 |
| 4 | 8.0% / 0.334 | 8.8% / 0.304 | 71.6% / 0.086 | 91.1% / 0.035 |
| 8 | 59.2% / 0.110 | 64.3% / 0.095 | 89.6% / 0.051 | 92.2% / 0.035 |
| 16 | 83.9% / 0.058 | 85.2% / 0.048 | 94.2% / 0.042 | 92.9% / 0.035 |
| 32 | 93.4% / 0.040 | 91.6% / 0.042 | 95.5% / 0.038 | 93.2% / 0.035 |
| 64 | 96.1% / 0.031 | 93.9% / 0.040 | 96.0% / 0.035 | 93.5% / 0.035 |
| 128 | 97.0% / 0.025 | 94.6% / 0.037 | 96.6% / 0.034 | 93.6% / 0.035 |
| 1000 | 97.4% / 0.018 | 95.1% / 0.037 | 96.7% / 0.034 | 93.7% / 0.035 |
Three things stand out. First, with few steps (4 to 16) flow matching is clearly better than the diffusion model trained with the same budget: at 8 steps 89.6% on a mode against 64.3% (DDIM) and 59.2% (DDPM). Second, with many steps the order changes: the stochastic DDPM sampler is the best of all at 128 and 1,000 steps (97.0% and 97.4%), because its fresh noise corrects errors made earlier on the path; the deterministic samplers carry their errors to the end. Third, the 2-rectified flow is good at every budget, even 1 step (86.7%), but it stops improving at about 93.7%: it can only be as good as the samples of the model it was distilled from, minus the errors of its own training.
Flow matching, one Euler step. Take the first noise point of Figure 4 (seed 1): \(\mathbf{z} = (0.966,\ 0.017)\). The trained network at \(t = 1\) outputs \(\mathbf{v} = (0.860,\ 0.056)\). One step of (233.6) with \(N = 1\) gives \(\mathbf{x} = \mathbf{z} - \mathbf{v} = (0.106,\ -0.040)\): near the center, far from every mode (the modes are at distance 1.5). This is not a bug. At \(t = 1\) the noisy point tells you nothing about which mode you will reach, so the best velocity is \(\mathbf{z} - \E[\mathbf{x}]\), and the step lands on \(\E[\mathbf{x}] = \mathbf{0}\), the mean of the data. Over 4,000 samples the mean distance from the center after one step was 0.070.
DDIM, one step. Now \(\hat{\mathbf{x}}\) of (233.4) at \(t = 1000\) divides by \(\sqrt{\bar\alpha_{1000}} = \sqrt{4.04 \times 10^{-5}} = 0.00635\), that is, it multiplies by 157.4. The best \(\hat{\mathbf{x}}\) is again the data mean, but every small error of the noise prediction is multiplied by 157. Our samples ended at a mean distance of 6.23 from the center, which means the numerator \(\mathbf{x}_t - \sqrt{1 - \bar\alpha_t}\,\epsilon_\theta\) was off by only \(6.23 / 157.4 \approx 0.040\) on average. This is the well-known problem of schedules that do not reach zero signal at the last step; Lin et al. (2023) propose to rescale the schedule to zero terminal signal-to-noise ratio and to always start sampling from the last timestep [Certain].
Step 4: plot the sample paths: curved or straight?
A deterministic sampler moves every noise point along a path. To measure how straight a path is, compare the straight-line distance between its two ends with its length:
It is 1 for a straight path and smaller for a curved one. It does not depend on how fast the sampler moves along the path, so it is fair to compare samplers with different time variables. Over 500 paths with 128 steps we measured: DDIM 0.779, flow matching 0.548, 2-rectified flow 0.999. (The stochastic DDPM sampler gives 0.044: its paths are random walks, and their length grows with the number of steps; the measure does not apply to it.)
So the flow-matching paths are more curved than the DDIM paths. Figure 5 shows why. A flow-matching path starts by moving toward the center (the average of the data, as in the worked example), and then turns outward to its mode. Over 4,000 paths the mean distance from the center falls from 1.25 at \(t = 1\) to 0.92 at \(t \approx 0.56\), then grows to 1.50 at \(t = 0\). The straight line of (233.2) is the path for one known pair; averaged over all pairs that cross the same point, the velocity bends.
And DDIM? Its paths look straighter only because of its coordinates. The DDIM update (233.4) with \(\eta = 0\) is a step of the probability-flow ODE of session 226. With Gaussian noise, all paths of the form \(a_t \mathbf{x} + b_t \epsilon\) share the same ODE solutions up to a rescaling of space and time; Karras et al. (2022) use this to compare samplers in one common form [Certain]. Divide a DDIM point by \(\sqrt{\bar\alpha_t} + \sqrt{1 - \bar\alpha_t}\):
This is exactly the flow-matching path with another clock \(t'\). So a perfect diffusion model and a perfect flow model trace the same curves, in different coordinates. We checked it: after this rescaling the DDIM paths (1,000 steps) have straightness 0.548, the flow-matching paths (1,000 steps) 0.549, and the end points of the two samplers from the same noise are a median distance of 0.012 apart. The two models are two estimates of one object.
Reflow. Liu et al. (2022) show how to straighten the paths: sample the trained flow to get pairs (noise \(\mathbf{z}\), its own output \(\hat{\mathbf{x}}\)), then train a new flow on straight lines between exactly these pairs [Certain]. The new pairs no longer cross (they come from an ODE, whose paths never cross), so the averaged velocity is the pair velocity, and the paths become straight. Applying the procedure again gives straighter flows; the model after one reflow is called the 2-rectified flow [Likely].
- input: a trained flow \(\mathbf{v}_\theta\); the number of pairs \(M\) (50,000 here); solver steps (100 Euler steps here)
- draw \(\mathbf{z}_1, \dots, \mathbf{z}_M \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\); compute \(\hat{\mathbf{x}}_m = \mathrm{ODESolve}_{\mathbf{v}_\theta}(\mathbf{z}_m)\) // the expensive part: M × steps network calls
- initialize \(\mathbf{v}_\phi \leftarrow \mathbf{v}_\theta\) // a warm start; training from scratch also works, more slowly
- repeat: pick a batch of pairs; \(t \sim \mathcal{U}[0,1]\); \(\mathbf{x}_t \leftarrow (1-t)\hat{\mathbf{x}} + t\mathbf{z}\); minimize \(\lVert \mathbf{v}_\phi(\mathbf{x}_t, t) - (\mathbf{z} - \hat{\mathbf{x}}) \rVert^2\)
- return \(\mathbf{v}_\phi\) // sample it with very few Euler steps
We trained the reflow model for 10,000 steps (half the first run), starting from the flow-matching weights. Its loss fell to about 0.001, against 2.77 for the original flow: the targets are now almost a function of the input, so almost nothing is irreducible.
How to read Figure 5
Start with flow matching and move the slider from 0 to 64. In the first half the dots contract toward the center; then they turn outward to the modes. Straightness is about 0.57. Click “DDIM”: the dots hardly move at first (at high noise, the velocity of the variance-preserving ODE is close to zero) and then go to their modes; straightness about 0.81. Click “DDIM, rescaled”: the same paths, divided by \(\sqrt{\bar\alpha_t} + \sqrt{1-\bar\alpha_t}\), now look like the flow-matching paths (about 0.56). Click “2-rectified flow”: the paths are straight lines (about 0.9996) and each point goes to the mode in its own direction. Click “DDPM” to see the jagged random paths of the stochastic sampler.
Look also at where the points go. Reflow straightens the paths but keeps the assignment of noise to modes: 99.4% of 4,000 noise points reach the same mode with the 2-rectified flow as with the original flow, because it was trained on the original flow’s own pairs. Its paths are also shorter (mean length about 0.60 against 1.04 for flow matching in Figure 5): reflow never increases the transport cost, a guarantee proved in the rectified-flow paper for every convex cost [Certain].
Step 5: quality against the number of steps
Figure 6 puts all the measurements together. Read it as a budget question: if your product can afford \(N\) network calls per sample, which model should you ship?
How to read Figure 6
On the on-mode view the 2-rectified flow (top-left) is good from 1 step on but flat; flow matching rises fast between 2 and 8 steps; DDIM and DDPM rise later and need about 16 to 32 steps. At 64 steps and more the curves cross: DDPM ends on top. The \(\mathrm{SW}_1\) view shows the same story on a log scale; remember that values below about 0.045 are within noise. The MNIST view shows the same order for images: flow matching is better at small budgets. The readout names the best method at four budgets.
Do not over-read the comparison between DDIM and flow matching. Two things are mixed in it: the model (noise against velocity prediction) and the time grid of the sampler. Our DDIM uses steps that are evenly spaced in \(t\) for the linear schedule, which spends many steps where little happens; the DDIM paper already used a quadratic spacing for CIFAR-10 [Likely], and modern samplers pick the grid carefully (session 226). By (233.9), the same ODE in flow coordinates with an even grid in \(t'\) behaves like our flow model. Part of the advantage of “flow matching” in practice is this better default parameterization.
Step 6: small images (MNIST)
Now the same experiment in 784 dimensions. The data are the 60,000 MNIST training digits, scaled to \([-1, 1]\). The network is a small U-Net (502,913 parameters): two resolutions down (28 → 14 → 7) and back up with skip connections, residual blocks with group normalization, and the time features added inside every block. Both models train for 6,000 steps of AdamW (one-cycle learning rate up to \(2 \times 10^{-3}\), batch 128, seed 0), and we sample from an exponential moving average (EMA) of the weights with decay 0.999, as is usual for diffusion models. Only the marked lines differ between the two models, as in Algorithm 1.
# mnist.py: a tiny U-Net trained as DDPM (eps-prediction) and as flow matching (v-prediction) on MNIST import math, json, time, sys, torch, torch.nn as nn, torch.nn.functional as F, torchvision torch.set_num_threads(6) ROOT = "data" tr = torchvision.datasets.MNIST(ROOT, train=True) X = (tr.data.float() / 127.5 - 1).unsqueeze(1) # 60000 x 1 x 28 x 28 in [-1, 1] def temb(t, d=64): # t in [0, 1] -> d sinusoidal features f = torch.exp(-math.log(1000) * torch.arange(d // 2) / (d // 2)) * 1000 a = t[:, None] * f return torch.cat([a.sin(), a.cos()], 1) class Res(nn.Module): def __init__(s, ci, co, td=128): super().__init__() s.n1, s.c1 = nn.GroupNorm(8, ci), nn.Conv2d(ci, co, 3, padding=1) s.t = nn.Linear(td, co) s.n2, s.c2 = nn.GroupNorm(8, co), nn.Conv2d(co, co, 3, padding=1) s.skip = nn.Conv2d(ci, co, 1) if ci != co else nn.Identity() def forward(s, x, te): h = s.c1(F.silu(s.n1(x))) + s.t(te)[:, :, None, None] h = s.c2(F.silu(s.n2(h))) return h + s.skip(x) class UNet(nn.Module): # 28 -> 14 -> 7 and back, with skips def __init__(s, c=32): super().__init__() s.tm = nn.Sequential(nn.Linear(64, 128), nn.SiLU(), nn.Linear(128, 128)) s.inp = nn.Conv2d(1, c, 3, padding=1) s.d1, s.d2 = Res(c, c), Res(c, 2 * c) s.down1, s.down2 = nn.Conv2d(c, c, 3, 2, 1), nn.Conv2d(2 * c, 2 * c, 3, 2, 1) s.mid = Res(2 * c, 2 * c) s.up2, s.u2 = nn.ConvTranspose2d(2 * c, 2 * c, 4, 2, 1), Res(4 * c, 2 * c) s.up1, s.u1 = nn.ConvTranspose2d(2 * c, c, 4, 2, 1), Res(2 * c, c) s.out = nn.Sequential(nn.GroupNorm(8, c), nn.SiLU(), nn.Conv2d(c, 1, 3, padding=1)) def forward(s, x, t): te = s.tm(temb(t)) h0 = s.d1(s.inp(x), te) # 28 x 28, c h1 = s.d2(s.down1(h0), te) # 14 x 14, 2c h = s.mid(s.down2(h1), te) # 7 x 7, 2c h = s.u2(torch.cat([s.up2(h), h1], 1), te) # 14 x 14 h = s.u1(torch.cat([s.up1(h), h0], 1), te) # 28 x 28 return s.out(h) T = 1000 beta = torch.linspace(1e-4, 0.02, T, dtype=torch.float64) abar = torch.cumprod(1 - beta, 0) def train(kind, steps, bs=128, seed=0): torch.manual_seed(seed); g = torch.Generator().manual_seed(seed + 1) m = UNet(); opt = torch.optim.AdamW(m.parameters(), lr=2e-3, weight_decay=0.0) sch = torch.optim.lr_scheduler.OneCycleLR(opt, 2e-3, total_steps=steps, pct_start=0.05) ema = UNet(); ema.load_state_dict(m.state_dict()) hist = []; t0 = time.time() for it in range(steps): x = X[torch.randint(0, len(X), (bs,), generator=g)] e = torch.randn(x.shape, generator=g) if kind == "ddpm": ti = torch.randint(1, T + 1, (bs,), generator=g) ab = abar[ti - 1].float()[:, None, None, None] loss = F.mse_loss(m(ab.sqrt() * x + (1 - ab).sqrt() * e, ti.float() / T), e) else: t = torch.rand(bs, generator=g); tt = t[:, None, None, None] loss = F.mse_loss(m((1 - tt) * x + tt * e, t), e - x) opt.zero_grad(); loss.backward(); opt.step(); sch.step() with torch.no_grad(): for pe, pm in zip(ema.parameters(), m.parameters()): pe.lerp_(pm, 1 - 0.999) if it % 100 == 0 or it == steps - 1: hist.append((it, loss.item())); print(kind, it, round(loss.item(), 4), round(time.time() - t0), flush=True) return ema, hist if __name__ == "__main__": steps = int(sys.argv[1]); kind = sys.argv[2] m, h = train(kind, steps) print("params", sum(p.numel() for p in m.parameters())) torch.save(m.state_dict(), f"mnist_{kind}.pt"); json.dump(h, open(f"mnist_{kind}_hist.json", "w"))
How to measure image quality. The standard metric is FID, the Fréchet distance between Gaussian fits to Inception-network features of real and generated images. Inception features mean little for 28 × 28 digits, so we use the same formula with the features of a small MNIST classifier (two convolution layers, 128 features, 98.4% test accuracy) [Certain for the formula; our numbers are not comparable to published FID values]:
- Read the symbols
- \(\mu_r, \Sigma_r\)
- mean and covariance of the 128 classifier features over the 10,000 real test images
- \(\mu_g, \Sigma_g\)
- the same for 2,000 generated images
- \((\cdot)^{1/2}\)
- the matrix square root; FD is 0 when the two Gaussian fits are identical
As a floor, 2,000 real training images score 3.55 against the test set; part of any FD is just the noise of using 2,000 samples. The metric code:
import math, json, time, torch, numpy as np, scipy.linalg, torchvision, base64, io
torch.set_num_threads(12)
from mnist import UNet, abar, T
from clf import Clf
from PIL import Image
c = Clf(); c.load_state_dict(torch.load("clf.pt")); c.eval()
te = torchvision.datasets.MNIST("data", train=False); tr = torchvision.datasets.MNIST("data", train=True)
Xt = (te.data.float() / 127.5 - 1).unsqueeze(1)
@torch.no_grad()
def feats(x): return torch.cat([c.f(x[i:i+1000]) for i in range(0, len(x), 1000)]).double().numpy()
def stats(f): return f.mean(0), np.cov(f, rowvar=False)
def fd(a, b):
(m1, c1), (m2, c2) = a, b
s = scipy.linalg.sqrtm(c1 @ c2).real
return float(((m1 - m2) ** 2).sum() + np.trace(c1 + c2 - 2 * s))
R = stats(feats(Xt))
g = torch.Generator().manual_seed(11)
Xtr = (tr.data.float() / 127.5 - 1).unsqueeze(1)[torch.randperm(60000, generator=g)[:2000]]
print("floor FD (2000 train vs test)", round(fd(stats(feats(Xtr)), R), 2))
M = {}
for k in ["ddpm", "fm"]:
m = UNet(); m.load_state_dict(torch.load(f"mnist_{k}.pt")); m.eval(); M[k] = m
z = torch.randn(2000, 1, 28, 28, generator=torch.Generator().manual_seed(5))
@torch.no_grad()
def samp(kind, N):
out = []
for b in range(0, 2000, 500):
x = z[b:b+500].clone(); gn = torch.Generator().manual_seed(1000 + b)
if kind == "fm":
for i in range(N):
t = 1 - i / N; x = x - (1 / N) * M["fm"](x, torch.full((len(x),), t))
else:
eta = 1.0 if kind == "ddpm" else 0.0
ts = torch.linspace(T, 0, N + 1).round().long()
for i in range(N):
t, s = ts[i].item(), ts[i+1].item()
at = abar[t-1].item(); as_ = abar[s-1].item() if s > 0 else 1.0
eps = M["ddpm"](x, torch.full((len(x),), t / T))
x0 = (x - math.sqrt(1 - at) * eps) / math.sqrt(at)
sig = eta * math.sqrt((1 - as_) / (1 - at) * (1 - at / as_))
x = math.sqrt(as_) * x0 + math.sqrt(max(1 - as_ - sig**2, 0)) * eps
if sig > 0: x = x + sig * torch.randn(x.shape, generator=gn)
out.append(x.clamp(-1, 1))
return torch.cat(out)
res = {}; tiles = {}
for kind in ["ddim", "fm", "ddpm"]:
for N in [1, 2, 4, 8, 16, 32, 64]:
t0 = time.time(); x = samp(kind, N)
with torch.no_grad(): p = c(x).softmax(1)
f = feats(x); hist = torch.bincount(p.argmax(1), minlength=10).tolist()
res[f"{kind}|{N}"] = {"fd": round(fd(stats(f), R), 2), "conf": round(p.max(1).values.mean().item(), 3), "hist": hist}
tiles[f"{kind}|{N}"] = ((x[:6] + 1) * 127.5).round().clamp(0, 255).byte().squeeze(1).numpy()
print(kind, N, res[f"{kind}|{N}"], round(time.time() - t0), flush=True)
json.dump(res, open("mnist_eval.json", "w"))
np.savez("tiles.npz", **{k.replace("|", "_"): v for k, v in tiles.items()})
The results, for 2,000 samples per setting from one fixed batch of noise (lower is better; bold is the best at each budget):
| Steps \(N\) | DDPM sampler | DDIM sampler | Flow matching (Euler) |
|---|---|---|---|
| 1 | 730.8 | 730.8 | 1816.3 |
| 2 | 783.7 | 684.5 | 535.4 |
| 4 | 239.7 | 172.1 | 106.7 |
| 8 | 65.1 | 55.2 | 37.5 |
| 16 | 29.4 | 32.2 | 20.7 |
| 32 | 20.5 | 24.9 | 15.2 |
| 64 | 14.7 | 22.2 | 13.1 |
The picture is the same as in 2-D, with one difference. With one step both models fail, in the two ways of the worked example: the flow model outputs the blurry average digit (our classifier called 1,843 of the 2,000 one-step samples a “5”), and the diffusion model outputs over-amplified noise that is clipped to black and white. From 2 to 64 steps flow matching has the lowest distance at every budget. The stochastic DDPM sampler is the worst of the three at 2 to 8 steps but improves fastest: it passes DDIM at 16 steps and is close to flow matching at 64 (14.7 against 13.1). Unlike in 2-D, it did not pass flow matching within 64 steps; whether it would at 1,000 steps we did not test. All values are still far above the floor of 3.55, which is expected for a 0.5-million-parameter network trained for only 6,000 steps.
How to read Figure 7
It opens at 8 steps. Each row is one sampler and each column one starting noise, the same in every row. Click 1 step: the flow-matching row shows the same gray blob in every column (the average digit), and the two diffusion rows show identical clipped noise (with one step, DDPM and DDIM are the same update). Click 2 and 4 steps: flow matching forms digits first. At 16 steps and more, compare DDIM and flow matching in the same column: they often draw the same digit from the same noise, as the shared ODE of Step 4 suggests; the DDPM row draws different digits, because it adds fresh noise. The readout gives the Fréchet distance of each sampler at the chosen budget.
Where this is used
| Application | What from this lab carries over |
|---|---|
| Text-to-image models (Stable Diffusion 3, FLUX.1) | The rectified-flow objective (233.2) with a transformer instead of our MLP; SD3 samples \(t\) from a logit-normal distribution instead of uniformly [Certain] (session 229) |
| Fast or one-step image generation | Reflow (InstaFlow) and distillation (session 230): the 1-step collapse of Step 3 is the problem they solve |
| Video generation | The same losses on space-time latents; the cost per step is far higher, so the step-count curve of Figure 6 decides the product (session 231). Meta’s Movie Gen is trained with flow matching [Certain] |
| Latent diffusion for images and video | Everything here runs unchanged on the latent of an autoencoder instead of pixels (session 228) |
| Synthetic data for video tasks (action recognition, segmentation, retrieval) | Generated clips are used to augment rare classes; the evaluation habit of this lab (a fixed noise batch, a feature-space distance, a sample grid) is how you check them before training on them |
| Robot policies (vision-language-action models) | Some policies predict continuous actions with a flow-matching head instead of tokens [Likely]; the few-step trade-off matters for control frequency |
Common mistakes
- Mixing the two time conventions. Lipman et al. put noise at \(t = 0\); rectified flow and SD3 code often put it at \(t = 1\). A flipped sign or a flipped \(t\) still trains to a reasonable loss, but sampling goes the wrong way. Check: one Euler step from \(t = 1\) should land near the data mean.
- Judging a model by its training loss. Our flow loss is 2.770 and our DDPM loss 0.409, and both are within 0.5% of their minimum (233.3). Neither number says anything about samples. Sample, and measure.
- Comparing step counts with different time grids. Evenly spaced DDIM steps on a linear schedule are a weak baseline. Say which grid you use, or the comparison measures the grid, not the model.
- Sampling DDIM with one or two steps from a schedule that never reaches zero signal. \(1/\sqrt{\bar\alpha_{1000}} = 157\) multiplies every error; our 1-step samples landed 6.2 units from the center.
- Believing that flow-matching paths are straight. Only the per-pair path is straight. The learned paths measured 0.55 here; you need reflow or distillation for few-step sampling.
- Forgetting that reflow costs quality. The 2-rectified flow was best at 1 to 8 steps but plateaued at 93.7% on a mode against 96.7% for the original flow. Keep both models, or distill.
- Comparing models with different starting noise and few samples. With 500 samples the on-mode fraction moved by up to 4 points between two seeds. Use one fixed noise batch for all models, and know the noise floor of your metric.
- Using a numerical FID on tiny images without saying which network made the features. Our MNIST FD uses a small classifier; its values cannot be compared with Inception-based FID in papers.
Interview questions
Write the training losses of DDPM and of flow matching. What is different?basic
Why does the flow-matching loss not go to zero, and how do you know your model is close to the best possible?core
One Euler step of a flow-matching model gives a blurry average. Why?core
Are the sampling paths of a flow-matching model straight?core
Is a DDIM sampler on a DDPM model the same as an Euler sampler on a flow-matching model?advanced
Why does DDIM with one step from a linear schedule explode, and how do you fix it?advanced
What is reflow, and what does it cost?core
Your product allows 4 network calls per image. Which model do you choose?core
Why can a stochastic sampler beat a deterministic one at many steps but lose at few?advanced
How do you evaluate a generative model on a toy 2-D problem and on images?basic
Exercises
- Compute \(1/\sqrt{\bar\alpha_t}\) at \(t = 1000\) for the linear schedule, and predict where one DDIM step lands if the noise prediction has an error of norm 0.01.
- For the straight path (233.2), show that \(\hat{\mathbf{x}} = \mathbf{x}_t - t\,\mathbf{v}\) and \(\hat{\epsilon} = \mathbf{x}_t + (1-t)\,\mathbf{v}\). Then show that one Euler step from \(t = 1\) with the best velocity gives \(\E[\mathbf{x}]\).
- A path goes from \((0, 0)\) to \((1, 0)\) and then to \((1, 1)\). What is its straightness (233.8)? What is the straightness of a half circle?
- Show (233.9): divide \(\sqrt{\bar\alpha}\mathbf{x} + \sqrt{1-\bar\alpha}\epsilon\) by \(\sqrt{\bar\alpha} + \sqrt{1-\bar\alpha}\). What is \(t'\) when \(\bar\alpha = 0.5\)?
- Run
toy2d.pywith the flow loss but sample \(t\) from a logit-normal distribution (the SD3 choice): \(t = \sigma(u)\), \(u \sim \mathcal{N}(0, 1)\). Does the 4-step and 64-step quality change? - Train a 3-rectified flow (reflow the 2-rectified model). Predict the change in straightness and in 1-step quality, then measure it.
- On MNIST, add a Heun (second-order) sampler to the flow model: two network calls per step. Compare it with Euler at the same number of network calls.
Answerscheck
Key takeaways
- Diffusion and flow matching differ in two lines of training code: the noisy input (a curve or a straight line between data and noise) and the target (noise or velocity). Same network, same data, same budget.
- Both trained models reached their minimum possible loss within 0.5% (2-D). The loss says that the optimizer worked; only samples say how good the model is.
- At small step budgets flow matching beat the DDPM-trained model with DDIM (8 steps: 89.6% against 64.3% on a mode); at large budgets the stochastic DDPM sampler was the best (97.4%).
- Learned flow-matching paths are curved (straightness 0.55): they head for the data mean and then turn. One Euler step lands on the mean.
- DDIM and flow matching integrate the same family of curves in different coordinates (rescaled straightness 0.548 against 0.549).
- Reflow makes the paths straight (0.999) and makes 1-step sampling work (86.7% on a mode), at the cost of a lower ceiling (93.7% against 96.7%).
- The same order appeared on MNIST images: flow matching had the lowest Fréchet distance from 2 to 64 steps (8 steps: 37.5 against 55.2 for DDIM), and the DDPM sampler improved fastest with more steps (64 steps: 14.7 against 13.1).
Key formulas
References for this session
- Ho et al. (2020). Denoising Diffusion Probabilistic Models
- Lipman et al. (2022). Flow Matching for Generative Modeling
- Ho et al. (2020). Denoising Diffusion Probabilistic Models
- Lipman et al. (2022). Flow Matching for Generative Modeling
- Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- Song et al. (2020). Denoising Diffusion Implicit Models
- Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
- Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Lin, Liu, Li & Yang (2023). Common Diffusion Noise Schedules and Sample Steps are Flawed
- Karras, Aittala, Aila & Laine (2022). Elucidating the Design Space of Diffusion-Based Generative Models
- LeCun, Cortes & Burges (1998). The MNIST database of handwritten digits