LLM & VLM courseBlog
233Track 9: Generative models · session 12 of 12

train a tiny diffusion model and a tiny flow-matching model

In this lab you train a diffusion model and a flow-matching model with the same network, data and budget, first on 2-D toy data and then on MNIST digits. You sample both with 1 to 128 steps, plot their sample paths, and measure quality against cost. You will see that learned flow paths are curved until you apply reflow, that DDIM and flow matching trace the same curves in different coordinates, and which model to choose for a given number of steps.

Time: 3 to 4 hours (mostly hands-on; the 2-D part runs in a few minutes on a CPU, the MNIST part needs a GPU or about three hours of CPU)Level: Intermediate · hands-on labBefore this: Sessions 225 and 226 (diffusion), Session 229 (flow matching and rectified flow), Session 4 (backpropagation and AdamW); Python and PyTorch

What you will learn

  • How to train a DDPM (noise prediction) and a flow-matching model (velocity prediction) so that only the noisy input and the target differ, and what one training example and one sampling step look like on real images
  • Why both training losses stop far above zero, and how to compute the lowest possible loss on toy data
  • How the DDPM, DDIM and Euler samplers work, and why one step fails differently for diffusion (an explosion) and for flow matching (the data mean)
  • How to measure the straightness of sample paths, and why learned flow-matching paths are curved
  • Why DDIM and flow matching integrate the same family of curves in different coordinates
  • How reflow (2-rectified flow) straightens paths and enables one-step sampling, and what it costs
  • How to compare sample quality against the number of steps with a fixed noise batch, on 2-D data and on images (Fréchet distance)

Why this matters

In session 225, session 226 and session 229 you met the two big families of modern image and video generators: diffusion models, which learn to predict the noise that was added to an image, and flow-matching models, which learn a velocity, a direction and speed that carries noise to data. In this lab you train both yourself, with the same network, the same data and the same budget, and then you measure the claims that you read about: that flow paths are straight, that straight paths need fewer sampling steps, and that “reflow” makes one-step generation possible.

The previous session, session 232, ended with models such as Transfusion that put a diffusion (or flow) loss for images next to the next-token loss for text. Before you build such a system, you should have trained the image part once yourself and seen where it fails. That is this lab.

You will do it in two settings. First a 2-D toy problem (eight Gaussian blobs on a ring), where you can see every sample and every path, and where a model trains in under a minute on a laptop CPU. Then small images (MNIST digits, 28 × 28 pixels) with a small U-Net, where the same ideas must survive in 784 dimensions. Every number on this page comes from our own run of the code below, unless it is tagged as coming from a paper; two of the figures run the trained 2-D networks live in your browser.

In short, what you will find: both models reach almost exactly the lowest possible training loss, so the loss tells you nothing about sample quality; with few sampling steps flow matching beats the diffusion model trained on the same budget (8 steps: 89.6% of samples on a mode against 64.3% for DDIM); the learned flow-matching paths are not straight (straightness 0.55, curved through the center), and the DDIM paths are the very same curves in other coordinates; only reflow makes them straight (0.999) and gives a good one-step sampler, at a small cost in final quality. On MNIST the same ranking appears: flow matching has the lowest Fréchet distance at every budget from 2 to 64 steps (8 steps: 37.5 against 55.2 for DDIM and 65.1 for DDPM), and the stochastic DDPM sampler closes the gap as the steps grow (64 steps: 14.7 against 13.1).

Setup

You need Python 3.10 or newer and PyTorch. Everything runs on a CPU. We used Python 3.15, torch 2.15.0, torchvision 0.29.2, NumPy 2.5.4 and SciPy 1.18.1 on an Apple-silicon laptop that was busy with other jobs; any recent versions should work. The 2-D part takes a few minutes. The MNIST part took about 87 minutes per model on our busy CPU (6,000 steps); a GPU should do it in a few minutes.

python -m venv venv && source venv/bin/activate
pip install torch torchvision numpy scipy pillow
python toy2d.py            # trains DDPM, flow matching and 2-rectified flow on 2-D data
python eval2d.py           # quality vs steps, straightness of paths
python clf.py              # a small MNIST classifier, used only to measure image quality
python mnist.py 6000 ddpm; python mnist.py 6000 fm; python mnist_eval.py

The plan of the lab is in Figure 1. The important design rule: change one thing at a time. The diffusion model and the flow-matching model share the network, the optimizer, the learning-rate schedule, the batch size, the number of steps and the random seed. They differ only in how a training example is made noisy and in what the network must predict. The samplers also start from the same noise, so you can compare two samples that began at the same point.

data8 Gaussians (2-D)MNIST 28 × 28 one networkMLP (2-D), U-Net (MNIST) diffusion (DDPM)xt = √ᾱt·x + √(1 − ᾱt)·ε, curved pathpredict the noise εt = 1 … 1000 flow matchingxt = (1 − t)·x + t·ε, straight linepredict the velocity ε − xt in [0, 1] reflowtrain on itsown pairs sample with N steps DDPM: stochasticDDIM: deterministicEuler: deterministic measurequality vs N (1 … 128)path straightness2-D: on-mode %, SW1MNIST: Fréchet distance
Figure 1. The lab. One architecture is trained in two ways that differ only in the noisy input and the target. The flow model is then trained once more on pairs of (noise, its own sample): this is reflow, and it gives the 2-rectified flow. All models are sampled with 1 to 128 steps from the same starting noise and measured in the same way.

Step 1: one network, two targets

Both models start from the same idea: take a clean data point \(\mathbf{x}\), mix it with Gaussian noise \(\epsilon\), and train a network to undo the mix. They differ in the recipe of the mix and in what the network outputs.

Diffusion (DDPM). The DDPM paper of Ho et al. (2020) uses \(T = 1000\) discrete steps and a linear schedule: the noise added at step \(t\) has variance \(\beta_t\), growing linearly from \(10^{-4}\) to \(0.02\) [Certain]. With \(\bar\alpha_t = \prod_{s \le t} (1 - \beta_s)\) the noisy point has a closed form (the same equation as 225.3), and the network \(\epsilon_\theta\) is trained to predict the noise:

\[ \mathbf{x}_t = \sqrt{\bar\alpha_t}\, \mathbf{x} + \sqrt{1 - \bar\alpha_t}\, \epsilon, \qquad \mathcal{L}_\text{DDPM} = \E_{t, \mathbf{x}, \epsilon} \big\lVert \epsilon_\theta(\mathbf{x}_t, t) - \epsilon \big\rVert^2 \tag{233.1} \]

Flow matching. We use the same convention as session 229: \(t = 0\) is data and \(t = 1\) is noise, and the noisy point lies on the straight line between them. Its speed along the line is constant, \(\epsilon - \mathbf{x}\), and that is the target. (The flow-matching paper of Lipman et al. uses the opposite direction, \(t = 0\) for noise; be careful when you copy code.)

\[ \mathbf{x}_t = (1 - t)\, \mathbf{x} + t\, \epsilon, \qquad \mathcal{L}_\text{FM} = \E_{t \sim \mathcal{U}[0,1],\, \mathbf{x}, \epsilon} \big\lVert \mathbf{v}_\theta(\mathbf{x}_t, t) - (\epsilon - \mathbf{x}) \big\rVert^2 \tag{233.2} \]
Read the symbols
\(\mathbf{x}\)
a clean training example: a 2-D point, or a 28 × 28 image scaled to \([-1, 1]\)
\(\epsilon\)
standard Gaussian noise of the same shape, a new draw for every example
\(t\)
the noise level: an integer 1 … 1000 for DDPM (we feed \(t/1000\) to the network), a real number in \([0, 1]\) for flow matching
\(\bar\alpha_t\)
the fraction of signal variance left at step \(t\); 0.9999 at \(t = 1\) and \(4.04 \times 10^{-5}\) at \(t = 1000\) for the linear schedule
\(\epsilon_\theta, \mathbf{v}_\theta\)
the same network architecture, read as a noise prediction or as a velocity prediction

Look at the two paths for one fixed pair \((\mathbf{x}, \epsilon)\). The flow-matching point moves on a straight segment. The DDPM point has coefficients \((\sqrt{\bar\alpha_t}, \sqrt{1 - \bar\alpha_t})\) whose squares add to 1: it moves on a quarter of an ellipse, a curve. This is the origin of the slogan “diffusion paths are curved, flow paths are straight”. But it is a statement about one pair. The sampler does not know which pair it is on; it follows the average velocity over all pairs that pass through the current point. Step 4 shows what that does to the shape of the paths.

One training loop serves both models. The only lines that differ are marked.

Algorithm 1: one training step, diffusion or flow matching
  1. input: a mini-batch \(\mathbf{x}\) of clean data, the network, the kind (DDPM or FM)
  2. draw \(\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) with the shape of \(\mathbf{x}\)
  3. if DDPM: draw \(t \in \{1, \dots, 1000\}\) uniformly; \(\mathbf{x}_t \leftarrow \sqrt{\bar\alpha_t}\,\mathbf{x} + \sqrt{1-\bar\alpha_t}\,\epsilon\); target \(\leftarrow \epsilon\) // differs
  4. if FM: draw \(t \sim \mathcal{U}[0, 1]\); \(\mathbf{x}_t \leftarrow (1-t)\,\mathbf{x} + t\,\epsilon\); target \(\leftarrow \epsilon - \mathbf{x}\) // differs
  5. loss \(\leftarrow\) mean squared error between network\((\mathbf{x}_t, t)\) and target
  6. backpropagate, take one optimizer step, update the learning-rate schedule (and the EMA copy for images)

Before we go into the details, watch both trained models at work. Figure 2 shows real frames from the two small MNIST networks that you will train in Step 6. In “Training” mode you see one training example being made: a clean digit, a noise image, their mix at the chosen noise level, the target, the network’s actual output and the clean digit that this output implies. In “Sampling” mode you see the opposite direction: three samples that start as pure noise and become digits in 16 steps. In both modes the box on the right is the loop of Algorithm 1 or Algorithm 2 below, with the numbers of one pixel filled in.

Figure 2. Real frames from our two trained MNIST U-Nets (Step 6). Training mode: one test digit and one fixed noise image at 21 noise levels; the prediction, the implied clean guess and the losses were computed by the trained networks. Sampling mode: 16 steps from the same three noise images for each sampler. Gray tiles show values from −1 (black) to 1 (white) for images and from −3 to 3 for noise, targets and predictions. Play animates the loop line by line; the highlighted line is the one being executed.
How to read Figure 2

Press Play in training mode with “Diffusion (DDPM)”: the noisy input fades from the digit into noise as \(t\) grows. Watch the bottom-right tile, the network’s guess \(\hat{\mathbf{x}}\): at small \(t\) it is the digit itself, at large \(t\) it becomes a blurry average shape, because the input no longer says which digit it was. The small chart shows the loss of 512 test digits at each noise level; for DDPM it is largest at small \(t\), where the noise is hard to see. Switch to flow matching: now the target is \(\epsilon - \mathbf{x}\). Apart from \(t = 0\), where the input contains no trace of the noise and the loss is about 1 per pixel (the variance of \(\epsilon\)), the loss is now largest at large \(t\) (0.29 at \(t = 0.9\) against 0.10 at \(t = 0.35\)), where the network must guess which digit it is. Then switch to sampling and press Play. With flow matching the guess (bottom row) is a gray blob at step 1 and becomes a clear digit after a few steps; the chart shows how fast the guess approaches the final sample. With the DDPM sampler the current image (top row) stays noisy until the last steps, because fresh noise \(\sigma \mathbf{z}\) is added in every step; with DDIM \(\sigma = 0\). DDIM and flow matching draw the same three digits from the same noise, and the DDPM sampler draws other ones; over 500 samples with 32 steps, DDIM and flow matching gave the same class in 74% of cases. Step 4 explains why.

Step 2: train both on 2-D toy data

The data are eight Gaussian blobs: centers on a circle of radius 1.5, standard deviation 0.1. Each coordinate then has a standard deviation of about 1.07, close to the scale of the noise, which is what both methods expect. The network is a small multilayer perceptron (MLP): the input is the point \((x, y)\) and 16 time features \(\sin(2^k \pi t), \cos(2^k \pi t)\) for \(k = 0, \dots, 7\); then three hidden layers of 64 units with the SiLU activation; then 2 outputs. That is 9,666 parameters. Both models train for 20,000 steps of Adam (learning rate \(10^{-3}\) with cosine decay, batch 1,024, seed 0). The full script:

# toy2d.py: DDPM vs flow matching vs 2-rectified flow on 8 Gaussians
import math, json, time, torch, torch.nn as nn, numpy as np
torch.set_num_threads(4)
DEV = "cpu"

def ring(n, g):                      # 8 Gaussians on a circle of radius 1.5, std 0.1
    k = torch.randint(0, 8, (n,), generator=g)
    a = k.float() * (2 * math.pi / 8)
    c = 1.5 * torch.stack([a.cos(), a.sin()], 1)
    return c + 0.1 * torch.randn(n, 2, generator=g)

FREQ = (2.0 ** torch.arange(8)) * math.pi       # 8 frequencies -> 16 time features
def temb(t):
    a = t[:, None] * FREQ
    return torch.cat([a.sin(), a.cos()], 1)

class MLP(nn.Module):
    def __init__(s, w=64):
        super().__init__()
        s.net = nn.Sequential(nn.Linear(2 + 16, w), nn.SiLU(), nn.Linear(w, w), nn.SiLU(),
                              nn.Linear(w, w), nn.SiLU(), nn.Linear(w, 2))
    def forward(s, x, t):                 # t in [0, 1]
        return s.net(torch.cat([x, temb(t)], 1))

T = 1000
beta = torch.linspace(1e-4, 0.02, T, dtype=torch.float64)
abar = torch.cumprod(1 - beta, 0)                # abar[i] is alpha-bar for step t = i + 1

def train(kind, steps=20000, bs=1024, seed=0, pairs=None, init=None):
    torch.manual_seed(seed); g = torch.Generator().manual_seed(seed + 1)
    m = MLP()
    if init is not None: m.load_state_dict(init.state_dict())
    opt = torch.optim.Adam(m.parameters(), lr=1e-3)
    sch = torch.optim.lr_scheduler.CosineAnnealingLR(opt, steps)
    hist = []
    for it in range(steps):
        if pairs is None:
            x = ring(bs, g); e = torch.randn(bs, 2, generator=g)
        else:
            idx = torch.randint(0, len(pairs[0]), (bs,), generator=g)
            e, x = pairs[0][idx], pairs[1][idx]
        if kind == "ddpm":
            ti = torch.randint(1, T + 1, (bs,), generator=g)
            ab = abar[ti - 1].float()[:, None]
            xt = ab.sqrt() * x + (1 - ab).sqrt() * e
            loss = ((m(xt, ti.float() / T) - e) ** 2).sum(1).mean()
        else:
            t = torch.rand(bs, generator=g)
            xt = (1 - t[:, None]) * x + t[:, None] * e
            loss = ((m(xt, t) - (e - x)) ** 2).sum(1).mean()
        opt.zero_grad(); loss.backward(); opt.step(); sch.step()
        if it % 500 == 0 or it == steps - 1: hist.append((it, loss.item()))
    return m, hist

@torch.no_grad()
def sample_ddpm(m, z, N, eta, keep=False):
    """DDIM family on N evenly spaced timesteps; eta=1 is DDPM (ancestral), eta=0 is DDIM (ODE)."""
    ts = torch.linspace(T, 0, N + 1).round().long()          # T ... 0
    x = z.clone(); path = [x.clone()]
    g = torch.Generator().manual_seed(123)
    for i in range(N):
        t, s = ts[i].item(), ts[i + 1].item()
        at = abar[t - 1].item(); as_ = abar[s - 1].item() if s > 0 else 1.0
        eps = m(x, torch.full((len(x),), t / T))
        x0 = (x - math.sqrt(1 - at) * eps) / math.sqrt(at)
        sig = eta * math.sqrt((1 - as_) / (1 - at) * (1 - at / as_))
        x = math.sqrt(as_) * x0 + math.sqrt(max(1 - as_ - sig ** 2, 0)) * eps
        if sig > 0: x = x + sig * torch.randn(x.shape, generator=g)
        if keep: path.append(x.clone())
    return (x, path) if keep else x

@torch.no_grad()
def sample_fm(m, z, N, keep=False):
    x = z.clone(); path = [x.clone()]
    for i in range(N):
        t = 1 - i / N; dt = 1 / N
        x = x - dt * m(x, torch.full((len(x),), t))
        if keep: path.append(x.clone())
    return (x, path) if keep else x

def sw1(a, b, nproj=64):
    th = torch.arange(nproj) * math.pi / nproj
    d = torch.stack([th.cos(), th.sin()], 1)
    pa = (a @ d.T).sort(0).values; pb = (b @ d.T).sort(0).values
    return (pa - pb).abs().mean().item()

def onmode(x):
    a = torch.arange(8) * (2 * math.pi / 8)
    c = 1.5 * torch.stack([a.cos(), a.sin()], 1)
    d = torch.cdist(x, c).min(1).values
    return (d < 0.3).float().mean().item()

if __name__ == "__main__":
    t0 = time.time()
    res = {}
    mdd, hd = train("ddpm"); print("ddpm", time.time() - t0, hd[-1])
    mfm, hf = train("fm"); print("fm", time.time() - t0, hf[-1])
    # reflow pairs: noise -> 1-RF output with 100 Euler steps
    gz = torch.Generator().manual_seed(7)
    zp = torch.randn(50000, 2, generator=gz)
    xp = sample_fm(mfm, zp, 100)
    mrf, hr = train("fm", pairs=(zp, xp), init=mfm, steps=10000, seed=3); print("rf", time.time() - t0, hr[-1])
    torch.save({"ddpm": mdd.state_dict(), "fm": mfm.state_dict(), "rf": mrf.state_dict()}, "toy2d.pt")
    json.dump({"ddpm": hd, "fm": hf, "rf": hr}, open("toy2d_hist.json", "w"))

On our laptop, busy with other jobs, one model trained in 45 seconds. Now the first surprise. The losses do not go to zero: on 400,000 fresh examples the DDPM model has loss 0.409 and the flow model 2.770. Is the flow model worse? No. Because every noisy point \(\mathbf{x}_t\) could have come from many pairs \((\mathbf{x}, \epsilon)\), even a perfect network cannot predict the target exactly. The best possible prediction is the average target given \(\mathbf{x}_t\), and the lowest possible loss is the average variance that is left:

\[ \mathcal{L}_\text{min} = \E_{t, \mathbf{x}_t} \Big[ \operatorname{tr} \operatorname{Cov}\big(\text{target} \mid \mathbf{x}_t\big) \Big], \qquad \text{for FM: } \operatorname{Cov}(\epsilon - \mathbf{x} \mid \mathbf{x}_t) = \frac{\operatorname{Cov}(\mathbf{x} \mid \mathbf{x}_t)}{t^2} \tag{233.3} \]

For a mixture of Gaussians the posterior \(p(\mathbf{x} \mid \mathbf{x}_t)\) is known exactly, so we computed (233.3) by Monte Carlo with 400,000 samples: the minimum is 0.407 for DDPM and 2.757 for flow matching. So both networks are within 0.5% of perfect, and the two numbers cannot be compared with each other at all: they measure errors in different units (noise against velocity), and the flow loss is large near \(t = 0\), where the factor \(1/t^2\) explodes. The reflow model of Step 4, in contrast, reaches a loss of about 0.001, because its targets are a deterministic function of the input. A training loss near its minimum tells you that the optimizer worked. It does not tell you how good the samples are. For that you must sample.

Figure 3 lets you watch what one training step sees on the 2-D data, with our trained networks running live in your browser. Move the noise level and look at three things: how the colored modes melt into one Gaussian cloud, how the path of a single pair is a curve for DDPM and a straight line for flow matching, and how the network’s guess of the clean point (the orange cross) moves from the true point to the center of the data as the noise grows.

Figure 3. The training step of (233.1) and (233.2) on 400 points of the 2-D data, computed live with our trained networks. The box shows the formulas with the numbers of the highlighted pair; the small chart shows the mean error over the 400 points at 21 noise levels (orange line: the current level).
How to read Figure 3

Press Play with DDPM. The noisy points (blue dot for the highlighted pair) move from the ring to a round Gaussian cloud. At low noise the orange cross sits on the green ring (the network recovers the clean point); at high noise it moves toward the center, the average of all modes. The error chart shows where each loss puts its weight. For DDPM the error is largest at low noise (about 2.2 at \(t = 1\), where the noise is far too small to read from the input) and falls below 0.01 above \(t = 800\). Switch to flow matching: the dashed path becomes a straight line, and the error now peaks in the middle (about 4.6 near \(t = 0.5\) to \(0.6\), where the mode is still uncertain and the factor \(1/t^2\) of (233.3) is still large). At \(t = 1\) it ends near 2.3: the best prediction is \(\mathbf{z} - \E[\mathbf{x}]\), and what is left is the total variance of the data, \(2 \times 1.135 = 2.27\). Every number here is computed live from 400 points, so it moves a little with the sample.

Step 3: sample, with many steps and with few

Sampling runs the noise level backwards, from pure noise to data, in \(N\) steps. For the diffusion model we use the DDIM family of updates on \(N\) evenly spaced timesteps \(\tau_N = 1000 \gt \tau_{N-1} \gt \dots \gt \tau_0 = 0\). From the noise prediction, form the current guess of the clean point \(\hat{\mathbf{x}}\), then move to the next noise level:

\[ \hat{\mathbf{x}} = \frac{\mathbf{x}_t - \sqrt{1 - \bar\alpha_t}\, \epsilon_\theta(\mathbf{x}_t, t)}{\sqrt{\bar\alpha_t}}, \qquad \mathbf{x}_s = \sqrt{\bar\alpha_s}\, \hat{\mathbf{x}} + \sqrt{1 - \bar\alpha_s - \sigma^2}\; \epsilon_\theta(\mathbf{x}_t, t) + \sigma \mathbf{z} \tag{233.4} \]
\[ \sigma = \eta \sqrt{\frac{1 - \bar\alpha_s}{1 - \bar\alpha_t} \Big( 1 - \frac{\bar\alpha_t}{\bar\alpha_s} \Big)}, \qquad \mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) \tag{233.5} \]
Read the symbols
\(t, s\)
the current and the next (smaller) timestep; \(\bar\alpha_0 = 1\)
\(\hat{\mathbf{x}}\)
the network’s current guess of the final clean point
\(\eta\)
1 gives the stochastic DDPM sampler (“ancestral”, with fresh noise \(\mathbf{z}\) in every step); 0 gives the deterministic DDIM sampler

For the flow model, sampling is solving the ODE \(d\mathbf{x}/dt = \mathbf{v}_\theta(\mathbf{x}, t)\) from \(t = 1\) down to \(t = 0\). The simplest solver is Euler’s method: assume that the velocity stays constant for one step.

\[ \mathbf{x}_{t - 1/N} = \mathbf{x}_t - \tfrac{1}{N}\, \mathbf{v}_\theta(\mathbf{x}_t, t), \qquad t = 1, \tfrac{N-1}{N}, \dots, \tfrac{1}{N} \tag{233.6} \]

If the true path is a straight line traveled at constant speed, one Euler step lands exactly on the end. Every bend in the path costs error, and more steps reduce the error. Both samplers call the network once per step, so the number of steps is the cost.

Algorithm 2: sampling with N steps
  1. input: starting noise \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\), the number of steps \(N\), the method
  2. if flow (FM or 2-rectified flow): \(\mathbf{x} \leftarrow \mathbf{z}\); for \(i = 0, \dots, N-1\): \(t \leftarrow 1 - i/N\); \(\mathbf{x} \leftarrow \mathbf{x} - \frac{1}{N} \mathbf{v}_\theta(\mathbf{x}, t)\)
  3. if diffusion: timesteps \(\tau_i = \mathrm{round}(1000 - 1000\, i / N)\), \(i = 0, \dots, N\); \(\mathbf{x} \leftarrow \mathbf{z}\)
  4.   for each pair \((t, s) = (\tau_i, \tau_{i+1})\): predict \(\epsilon_\theta(\mathbf{x}, t)\), form \(\hat{\mathbf{x}}\), apply (233.4) with \(\eta = 1\) (DDPM) or \(\eta = 0\) (DDIM) // in the last step \(s = 0\), so \(\sigma = 0\) and \(\mathbf{x} = \hat{\mathbf{x}}\)
  5. return \(\mathbf{x}\) // N network calls in every case

To measure quality on the toy data we use two numbers. The on-mode fraction is the share of samples that land within 0.3 (three standard deviations) of one of the eight centers; for real data it is 99%. The sliced Wasserstein distance \(\mathrm{SW}_1\) compares the whole distribution: project both sample sets on 64 directions, sort each projection, and average the absolute differences:

\[ \mathrm{SW}_1(P, Q) \approx \frac{1}{64} \sum_{k=1}^{64} \frac{1}{n} \sum_{i=1}^{n} \big| a^{(k)}_{(i)} - b^{(k)}_{(i)} \big|, \qquad a^{(k)}_i = \mathbf{u}_k^\top \mathbf{x}_i, \quad b^{(k)}_i = \mathbf{u}_k^\top \mathbf{y}_i \tag{233.7} \]

Here \(\mathbf{u}_k\) are unit directions at angles \(k\pi/64\), and \(a_{(i)}\) is the \(i\)-th smallest projection. With 4,000 samples on each side, two independent sets of real data are at \(\mathrm{SW}_1 \approx 0.030\) (from 0.017 to 0.045 over 10 seeds): differences below about 0.045 are noise. The on-mode fraction is more sensitive to blur and to samples stuck between modes.

The evaluation script loads the three trained models, samples all of them from one fixed batch of 4,000 noise points, and measures both numbers and the straightness of Step 4:

import torch, math, json
from toy2d import *
torch.set_num_threads(2)
S = torch.load("toy2d.pt")
M = {}
for k in S:
    m = MLP(); m.load_state_dict(S[k]); m.eval(); M[k] = m
g = torch.Generator().manual_seed(2024)
ref = ring(4000, g)
ref2 = ring(4000, g)
print("noise floor sw1", round(sw1(ref, ref2), 4), "onmode data", round(onmode(ref), 4))
z = torch.randn(4000, 2, generator=torch.Generator().manual_seed(99))
out = {}
for N in [1, 2, 4, 8, 16, 32, 64, 128, 1000]:
    row = {}
    for name, f in [("ddpm", lambda N: sample_ddpm(M["ddpm"], z, N, 1.0)),
                    ("ddim", lambda N: sample_ddpm(M["ddpm"], z, N, 0.0)),
                    ("fm", lambda N: sample_fm(M["fm"], z, N)),
                    ("rf", lambda N: sample_fm(M["rf"], z, N))]:
        x = f(N)
        row[name] = (round(sw1(x, ref), 3), round(onmode(x), 3))
    out[N] = row
    print(N, row)
# one-step mean
x1 = sample_fm(M["fm"], z, 1); print("fm 1-step mean", x1.mean(0), "radius mean", x1.norm(dim=1).mean())
x1 = sample_ddpm(M["ddpm"], z, 1, 0.0); print("ddim 1-step radius mean", x1.norm(dim=1).mean())
x1 = sample_fm(M["rf"], z, 1); print("rf 1-step radius mean", x1.norm(dim=1).mean())
# straightness at 64 steps on 500 paths
def straight(path):
    P = torch.stack(path)                      # (N+1, n, 2)
    L = (P[1:] - P[:-1]).norm(dim=2).sum(0)
    D = (P[-1] - P[0]).norm(dim=1)
    return (D / L).mean().item()
zz = z[:500]
for name, f in [("ddim", lambda: sample_ddpm(M["ddpm"], zz, 128, 0.0, keep=True)[1]),
                ("ddpm", lambda: sample_ddpm(M["ddpm"], zz, 128, 1.0, keep=True)[1]),
                ("fm", lambda: sample_fm(M["fm"], zz, 128, keep=True)[1]),
                ("rf", lambda: sample_fm(M["rf"], zz, 128, keep=True)[1])]:
    print("straightness", name, round(straight(f()), 4))
json.dump(out, open("eval2d.json", "w"))
Steps
Figure 4. Live samples from our trained 2-D networks (weights stored in this page as 16-bit floats), 500 samples per setting. Dashed circles: the eight modes of the real data, radius 0.3. The readout gives the on-mode fraction and the mean distance from the center; the computation runs in your browser.
How to read Figure 4

It opens on flow matching with 8 steps: about 92% of the samples sit on a mode. Click 1 step: all samples collapse into a small cloud at the center (mean distance from the center about 0.07), because one Euler step from \(t = 1\) lands on the average of the data. Click DDIM with 1 step: the samples explode far outside the plot (Step 3’s worked example explains why). Increase the steps for DDIM: it needs about 16 steps for most samples to land on a mode, flow matching about 8. Now click 2-rectified flow: even 1 step puts about 89% of the samples on a mode. With 64 steps compare the four methods: the reflowed model is slightly worse than plain flow matching. Click “New noise” to see how much the numbers move between seeds.

The table below is the same experiment, measured in Python with 4,000 samples from one fixed noise batch (the noise is the same for all methods). The DDPM and DDIM rows at 1 step are identical: in the last step \(s = 0\), so \(\sigma = 0\) for any \(\eta\).

Steps \(N\)DDPM (on-mode / \(\mathrm{SW}_1\))DDIMFlow matching2-rectified flow
12.4% / 3.0302.4% / 3.0300.0% / 0.91686.7% / 0.039
21.0% / 0.5740.3% / 0.6616.0% / 0.36989.1% / 0.037
48.0% / 0.3348.8% / 0.30471.6% / 0.08691.1% / 0.035
859.2% / 0.11064.3% / 0.09589.6% / 0.05192.2% / 0.035
1683.9% / 0.05885.2% / 0.04894.2% / 0.04292.9% / 0.035
3293.4% / 0.04091.6% / 0.04295.5% / 0.03893.2% / 0.035
6496.1% / 0.03193.9% / 0.04096.0% / 0.03593.5% / 0.035
12897.0% / 0.02594.6% / 0.03796.6% / 0.03493.6% / 0.035
100097.4% / 0.01895.1% / 0.03796.7% / 0.03493.7% / 0.035

Three things stand out. First, with few steps (4 to 16) flow matching is clearly better than the diffusion model trained with the same budget: at 8 steps 89.6% on a mode against 64.3% (DDIM) and 59.2% (DDPM). Second, with many steps the order changes: the stochastic DDPM sampler is the best of all at 128 and 1,000 steps (97.0% and 97.4%), because its fresh noise corrects errors made earlier on the path; the deterministic samplers carry their errors to the end. Third, the 2-rectified flow is good at every budget, even 1 step (86.7%), but it stops improving at about 93.7%: it can only be as good as the samples of the model it was distilled from, minus the errors of its own training.

Worked example: why one step fails in two different ways

Flow matching, one Euler step. Take the first noise point of Figure 4 (seed 1): \(\mathbf{z} = (0.966,\ 0.017)\). The trained network at \(t = 1\) outputs \(\mathbf{v} = (0.860,\ 0.056)\). One step of (233.6) with \(N = 1\) gives \(\mathbf{x} = \mathbf{z} - \mathbf{v} = (0.106,\ -0.040)\): near the center, far from every mode (the modes are at distance 1.5). This is not a bug. At \(t = 1\) the noisy point tells you nothing about which mode you will reach, so the best velocity is \(\mathbf{z} - \E[\mathbf{x}]\), and the step lands on \(\E[\mathbf{x}] = \mathbf{0}\), the mean of the data. Over 4,000 samples the mean distance from the center after one step was 0.070.

DDIM, one step. Now \(\hat{\mathbf{x}}\) of (233.4) at \(t = 1000\) divides by \(\sqrt{\bar\alpha_{1000}} = \sqrt{4.04 \times 10^{-5}} = 0.00635\), that is, it multiplies by 157.4. The best \(\hat{\mathbf{x}}\) is again the data mean, but every small error of the noise prediction is multiplied by 157. Our samples ended at a mean distance of 6.23 from the center, which means the numerator \(\mathbf{x}_t - \sqrt{1 - \bar\alpha_t}\,\epsilon_\theta\) was off by only \(6.23 / 157.4 \approx 0.040\) on average. This is the well-known problem of schedules that do not reach zero signal at the last step; Lin et al. (2023) propose to rescale the schedule to zero terminal signal-to-noise ratio and to always start sampling from the last timestep [Certain].

Step 4: plot the sample paths: curved or straight?

A deterministic sampler moves every noise point along a path. To measure how straight a path is, compare the straight-line distance between its two ends with its length:

\[ \text{straightness} = \frac{\lVert \mathbf{x}^{(N)} - \mathbf{x}^{(0)} \rVert}{\sum_{i=1}^{N} \lVert \mathbf{x}^{(i)} - \mathbf{x}^{(i-1)} \rVert} \in (0, 1] \tag{233.8} \]

It is 1 for a straight path and smaller for a curved one. It does not depend on how fast the sampler moves along the path, so it is fair to compare samplers with different time variables. Over 500 paths with 128 steps we measured: DDIM 0.779, flow matching 0.548, 2-rectified flow 0.999. (The stochastic DDPM sampler gives 0.044: its paths are random walks, and their length grows with the number of steps; the measure does not apply to it.)

So the flow-matching paths are more curved than the DDIM paths. Figure 5 shows why. A flow-matching path starts by moving toward the center (the average of the data, as in the worked example), and then turns outward to its mode. Over 4,000 paths the mean distance from the center falls from 1.25 at \(t = 1\) to 0.92 at \(t \approx 0.56\), then grows to 1.50 at \(t = 0\). The straight line of (233.2) is the path for one known pair; averaged over all pairs that cross the same point, the velocity bends.

And DDIM? Its paths look straighter only because of its coordinates. The DDIM update (233.4) with \(\eta = 0\) is a step of the probability-flow ODE of session 226. With Gaussian noise, all paths of the form \(a_t \mathbf{x} + b_t \epsilon\) share the same ODE solutions up to a rescaling of space and time; Karras et al. (2022) use this to compare samplers in one common form [Certain]. Divide a DDIM point by \(\sqrt{\bar\alpha_t} + \sqrt{1 - \bar\alpha_t}\):

\[ \frac{\mathbf{x}_t}{\sqrt{\bar\alpha_t} + \sqrt{1 - \bar\alpha_t}} = (1 - t')\, \mathbf{x} + t'\, \epsilon, \qquad t' = \frac{\sqrt{1 - \bar\alpha_t}}{\sqrt{\bar\alpha_t} + \sqrt{1 - \bar\alpha_t}} \tag{233.9} \]

This is exactly the flow-matching path with another clock \(t'\). So a perfect diffusion model and a perfect flow model trace the same curves, in different coordinates. We checked it: after this rescaling the DDIM paths (1,000 steps) have straightness 0.548, the flow-matching paths (1,000 steps) 0.549, and the end points of the two samplers from the same noise are a median distance of 0.012 apart. The two models are two estimates of one object.

Reflow. Liu et al. (2022) show how to straighten the paths: sample the trained flow to get pairs (noise \(\mathbf{z}\), its own output \(\hat{\mathbf{x}}\)), then train a new flow on straight lines between exactly these pairs [Certain]. The new pairs no longer cross (they come from an ODE, whose paths never cross), so the averaged velocity is the pair velocity, and the paths become straight. Applying the procedure again gives straighter flows; the model after one reflow is called the 2-rectified flow [Likely].

\[ \mathcal{L}_\text{reflow} = \E_{t,\, \mathbf{z}} \big\lVert \mathbf{v}_\phi\big((1 - t)\, \hat{\mathbf{x}}(\mathbf{z}) + t\, \mathbf{z},\ t\big) - \big(\mathbf{z} - \hat{\mathbf{x}}(\mathbf{z})\big) \big\rVert^2, \qquad \hat{\mathbf{x}}(\mathbf{z}) = \mathrm{ODESolve}_{\mathbf{v}_\theta}(\mathbf{z}) \tag{233.10} \]
Algorithm 3: reflow (from 1-rectified to 2-rectified flow)
  1. input: a trained flow \(\mathbf{v}_\theta\); the number of pairs \(M\) (50,000 here); solver steps (100 Euler steps here)
  2. draw \(\mathbf{z}_1, \dots, \mathbf{z}_M \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\); compute \(\hat{\mathbf{x}}_m = \mathrm{ODESolve}_{\mathbf{v}_\theta}(\mathbf{z}_m)\) // the expensive part: M × steps network calls
  3. initialize \(\mathbf{v}_\phi \leftarrow \mathbf{v}_\theta\) // a warm start; training from scratch also works, more slowly
  4. repeat: pick a batch of pairs; \(t \sim \mathcal{U}[0,1]\); \(\mathbf{x}_t \leftarrow (1-t)\hat{\mathbf{x}} + t\mathbf{z}\); minimize \(\lVert \mathbf{v}_\phi(\mathbf{x}_t, t) - (\mathbf{z} - \hat{\mathbf{x}}) \rVert^2\)
  5. return \(\mathbf{v}_\phi\) // sample it with very few Euler steps

We trained the reflow model for 10,000 steps (half the first run), starting from the flow-matching weights. Its loss fell to about 0.001, against 2.77 for the original flow: the targets are now almost a function of the input, so almost nothing is irreducible.

Figure 5. Forty sample paths with 64 steps, computed live from the same 40 noise points for every method. Open circles: the starting noise; filled dots: the position at the chosen step. The readout gives the mean straightness (233.8) and the mean path length.
How to read Figure 5

Start with flow matching and move the slider from 0 to 64. In the first half the dots contract toward the center; then they turn outward to the modes. Straightness is about 0.57. Click “DDIM”: the dots hardly move at first (at high noise, the velocity of the variance-preserving ODE is close to zero) and then go to their modes; straightness about 0.81. Click “DDIM, rescaled”: the same paths, divided by \(\sqrt{\bar\alpha_t} + \sqrt{1-\bar\alpha_t}\), now look like the flow-matching paths (about 0.56). Click “2-rectified flow”: the paths are straight lines (about 0.9996) and each point goes to the mode in its own direction. Click “DDPM” to see the jagged random paths of the stochastic sampler.

Look also at where the points go. Reflow straightens the paths but keeps the assignment of noise to modes: 99.4% of 4,000 noise points reach the same mode with the 2-rectified flow as with the original flow, because it was trained on the original flow’s own pairs. Its paths are also shorter (mean length about 0.60 against 1.04 for flow matching in Figure 5): reflow never increases the transport cost, a guarantee proved in the rectified-flow paper for every convex cost [Certain].

Step 5: quality against the number of steps

Figure 6 puts all the measurements together. Read it as a budget question: if your product can afford \(N\) network calls per sample, which model should you ship?

Figure 6. Measured sample quality against the number of steps (log scale), for 2-D data (4,000 samples, one fixed noise batch) and MNIST (2,000 samples, Step 6). The dashed line is real data: 99% on a mode, \(\mathrm{SW}_1 \approx 0.030\) between two real sets, and the Fréchet distance of 2,000 real training images.
How to read Figure 6

On the on-mode view the 2-rectified flow (top-left) is good from 1 step on but flat; flow matching rises fast between 2 and 8 steps; DDIM and DDPM rise later and need about 16 to 32 steps. At 64 steps and more the curves cross: DDPM ends on top. The \(\mathrm{SW}_1\) view shows the same story on a log scale; remember that values below about 0.045 are within noise. The MNIST view shows the same order for images: flow matching is better at small budgets. The readout names the best method at four budgets.

Do not over-read the comparison between DDIM and flow matching. Two things are mixed in it: the model (noise against velocity prediction) and the time grid of the sampler. Our DDIM uses steps that are evenly spaced in \(t\) for the linear schedule, which spends many steps where little happens; the DDIM paper already used a quadratic spacing for CIFAR-10 [Likely], and modern samplers pick the grid carefully (session 226). By (233.9), the same ODE in flow coordinates with an even grid in \(t'\) behaves like our flow model. Part of the advantage of “flow matching” in practice is this better default parameterization.

Step 6: small images (MNIST)

Now the same experiment in 784 dimensions. The data are the 60,000 MNIST training digits, scaled to \([-1, 1]\). The network is a small U-Net (502,913 parameters): two resolutions down (28 → 14 → 7) and back up with skip connections, residual blocks with group normalization, and the time features added inside every block. Both models train for 6,000 steps of AdamW (one-cycle learning rate up to \(2 \times 10^{-3}\), batch 128, seed 0), and we sample from an exponential moving average (EMA) of the weights with decay 0.999, as is usual for diffusion models. Only the marked lines differ between the two models, as in Algorithm 1.

# mnist.py: a tiny U-Net trained as DDPM (eps-prediction) and as flow matching (v-prediction) on MNIST
import math, json, time, sys, torch, torch.nn as nn, torch.nn.functional as F, torchvision
torch.set_num_threads(6)
ROOT = "data"
tr = torchvision.datasets.MNIST(ROOT, train=True)
X = (tr.data.float() / 127.5 - 1).unsqueeze(1)          # 60000 x 1 x 28 x 28 in [-1, 1]

def temb(t, d=64):                                       # t in [0, 1] -> d sinusoidal features
    f = torch.exp(-math.log(1000) * torch.arange(d // 2) / (d // 2)) * 1000
    a = t[:, None] * f
    return torch.cat([a.sin(), a.cos()], 1)

class Res(nn.Module):
    def __init__(s, ci, co, td=128):
        super().__init__()
        s.n1, s.c1 = nn.GroupNorm(8, ci), nn.Conv2d(ci, co, 3, padding=1)
        s.t = nn.Linear(td, co)
        s.n2, s.c2 = nn.GroupNorm(8, co), nn.Conv2d(co, co, 3, padding=1)
        s.skip = nn.Conv2d(ci, co, 1) if ci != co else nn.Identity()
    def forward(s, x, te):
        h = s.c1(F.silu(s.n1(x))) + s.t(te)[:, :, None, None]
        h = s.c2(F.silu(s.n2(h)))
        return h + s.skip(x)

class UNet(nn.Module):                                   # 28 -> 14 -> 7 and back, with skips
    def __init__(s, c=32):
        super().__init__()
        s.tm = nn.Sequential(nn.Linear(64, 128), nn.SiLU(), nn.Linear(128, 128))
        s.inp = nn.Conv2d(1, c, 3, padding=1)
        s.d1, s.d2 = Res(c, c), Res(c, 2 * c)
        s.down1, s.down2 = nn.Conv2d(c, c, 3, 2, 1), nn.Conv2d(2 * c, 2 * c, 3, 2, 1)
        s.mid = Res(2 * c, 2 * c)
        s.up2, s.u2 = nn.ConvTranspose2d(2 * c, 2 * c, 4, 2, 1), Res(4 * c, 2 * c)
        s.up1, s.u1 = nn.ConvTranspose2d(2 * c, c, 4, 2, 1), Res(2 * c, c)
        s.out = nn.Sequential(nn.GroupNorm(8, c), nn.SiLU(), nn.Conv2d(c, 1, 3, padding=1))
    def forward(s, x, t):
        te = s.tm(temb(t))
        h0 = s.d1(s.inp(x), te)                          # 28 x 28, c
        h1 = s.d2(s.down1(h0), te)                       # 14 x 14, 2c
        h = s.mid(s.down2(h1), te)                       # 7 x 7, 2c
        h = s.u2(torch.cat([s.up2(h), h1], 1), te)       # 14 x 14
        h = s.u1(torch.cat([s.up1(h), h0], 1), te)       # 28 x 28
        return s.out(h)

T = 1000
beta = torch.linspace(1e-4, 0.02, T, dtype=torch.float64)
abar = torch.cumprod(1 - beta, 0)

def train(kind, steps, bs=128, seed=0):
    torch.manual_seed(seed); g = torch.Generator().manual_seed(seed + 1)
    m = UNet(); opt = torch.optim.AdamW(m.parameters(), lr=2e-3, weight_decay=0.0)
    sch = torch.optim.lr_scheduler.OneCycleLR(opt, 2e-3, total_steps=steps, pct_start=0.05)
    ema = UNet(); ema.load_state_dict(m.state_dict())
    hist = []; t0 = time.time()
    for it in range(steps):
        x = X[torch.randint(0, len(X), (bs,), generator=g)]
        e = torch.randn(x.shape, generator=g)
        if kind == "ddpm":
            ti = torch.randint(1, T + 1, (bs,), generator=g)
            ab = abar[ti - 1].float()[:, None, None, None]
            loss = F.mse_loss(m(ab.sqrt() * x + (1 - ab).sqrt() * e, ti.float() / T), e)
        else:
            t = torch.rand(bs, generator=g); tt = t[:, None, None, None]
            loss = F.mse_loss(m((1 - tt) * x + tt * e, t), e - x)
        opt.zero_grad(); loss.backward(); opt.step(); sch.step()
        with torch.no_grad():
            for pe, pm in zip(ema.parameters(), m.parameters()): pe.lerp_(pm, 1 - 0.999)
        if it % 100 == 0 or it == steps - 1:
            hist.append((it, loss.item())); print(kind, it, round(loss.item(), 4), round(time.time() - t0), flush=True)
    return ema, hist

if __name__ == "__main__":
    steps = int(sys.argv[1]); kind = sys.argv[2]
    m, h = train(kind, steps)
    print("params", sum(p.numel() for p in m.parameters()))
    torch.save(m.state_dict(), f"mnist_{kind}.pt"); json.dump(h, open(f"mnist_{kind}_hist.json", "w"))

How to measure image quality. The standard metric is FID, the Fréchet distance between Gaussian fits to Inception-network features of real and generated images. Inception features mean little for 28 × 28 digits, so we use the same formula with the features of a small MNIST classifier (two convolution layers, 128 features, 98.4% test accuracy) [Certain for the formula; our numbers are not comparable to published FID values]:

\[ \mathrm{FD} = \lVert \mu_r - \mu_g \rVert^2 + \operatorname{tr}\Big( \Sigma_r + \Sigma_g - 2 \big( \Sigma_r \Sigma_g \big)^{1/2} \Big) \tag{233.11} \]
Read the symbols
\(\mu_r, \Sigma_r\)
mean and covariance of the 128 classifier features over the 10,000 real test images
\(\mu_g, \Sigma_g\)
the same for 2,000 generated images
\((\cdot)^{1/2}\)
the matrix square root; FD is 0 when the two Gaussian fits are identical

As a floor, 2,000 real training images score 3.55 against the test set; part of any FD is just the noise of using 2,000 samples. The metric code:

import math, json, time, torch, numpy as np, scipy.linalg, torchvision, base64, io
torch.set_num_threads(12)
from mnist import UNet, abar, T
from clf import Clf
from PIL import Image
c = Clf(); c.load_state_dict(torch.load("clf.pt")); c.eval()
te = torchvision.datasets.MNIST("data", train=False); tr = torchvision.datasets.MNIST("data", train=True)
Xt = (te.data.float() / 127.5 - 1).unsqueeze(1)
@torch.no_grad()
def feats(x): return torch.cat([c.f(x[i:i+1000]) for i in range(0, len(x), 1000)]).double().numpy()
def stats(f): return f.mean(0), np.cov(f, rowvar=False)
def fd(a, b):
    (m1, c1), (m2, c2) = a, b
    s = scipy.linalg.sqrtm(c1 @ c2).real
    return float(((m1 - m2) ** 2).sum() + np.trace(c1 + c2 - 2 * s))
R = stats(feats(Xt))
g = torch.Generator().manual_seed(11)
Xtr = (tr.data.float() / 127.5 - 1).unsqueeze(1)[torch.randperm(60000, generator=g)[:2000]]
print("floor FD (2000 train vs test)", round(fd(stats(feats(Xtr)), R), 2))
M = {}
for k in ["ddpm", "fm"]:
    m = UNet(); m.load_state_dict(torch.load(f"mnist_{k}.pt")); m.eval(); M[k] = m
z = torch.randn(2000, 1, 28, 28, generator=torch.Generator().manual_seed(5))
@torch.no_grad()
def samp(kind, N):
    out = []
    for b in range(0, 2000, 500):
        x = z[b:b+500].clone(); gn = torch.Generator().manual_seed(1000 + b)
        if kind == "fm":
            for i in range(N):
                t = 1 - i / N; x = x - (1 / N) * M["fm"](x, torch.full((len(x),), t))
        else:
            eta = 1.0 if kind == "ddpm" else 0.0
            ts = torch.linspace(T, 0, N + 1).round().long()
            for i in range(N):
                t, s = ts[i].item(), ts[i+1].item()
                at = abar[t-1].item(); as_ = abar[s-1].item() if s > 0 else 1.0
                eps = M["ddpm"](x, torch.full((len(x),), t / T))
                x0 = (x - math.sqrt(1 - at) * eps) / math.sqrt(at)
                sig = eta * math.sqrt((1 - as_) / (1 - at) * (1 - at / as_))
                x = math.sqrt(as_) * x0 + math.sqrt(max(1 - as_ - sig**2, 0)) * eps
                if sig > 0: x = x + sig * torch.randn(x.shape, generator=gn)
        out.append(x.clamp(-1, 1))
    return torch.cat(out)
res = {}; tiles = {}
for kind in ["ddim", "fm", "ddpm"]:
    for N in [1, 2, 4, 8, 16, 32, 64]:
        t0 = time.time(); x = samp(kind, N)
        with torch.no_grad(): p = c(x).softmax(1)
        f = feats(x); hist = torch.bincount(p.argmax(1), minlength=10).tolist()
        res[f"{kind}|{N}"] = {"fd": round(fd(stats(f), R), 2), "conf": round(p.max(1).values.mean().item(), 3), "hist": hist}
        tiles[f"{kind}|{N}"] = ((x[:6] + 1) * 127.5).round().clamp(0, 255).byte().squeeze(1).numpy()
        print(kind, N, res[f"{kind}|{N}"], round(time.time() - t0), flush=True)
json.dump(res, open("mnist_eval.json", "w"))
np.savez("tiles.npz", **{k.replace("|", "_"): v for k, v in tiles.items()})

The results, for 2,000 samples per setting from one fixed batch of noise (lower is better; bold is the best at each budget):

Steps \(N\)DDPM samplerDDIM samplerFlow matching (Euler)
1730.8730.81816.3
2783.7684.5535.4
4239.7172.1106.7
865.155.237.5
1629.432.220.7
3220.524.915.2
6414.722.213.1

The picture is the same as in 2-D, with one difference. With one step both models fail, in the two ways of the worked example: the flow model outputs the blurry average digit (our classifier called 1,843 of the 2,000 one-step samples a “5”), and the diffusion model outputs over-amplified noise that is clipped to black and white. From 2 to 64 steps flow matching has the lowest distance at every budget. The stochastic DDPM sampler is the worst of the three at 2 to 8 steps but improves fastest: it passes DDIM at 16 steps and is close to flow matching at 64 (14.7 against 13.1). Unlike in 2-D, it did not pass flow matching within 64 steps; whether it would at 1,000 steps we did not test. All values are still far above the floor of 3.55, which is expected for a 0.5-million-parameter network trained for only 6,000 steps.

Steps
DDIMFlow matchingDDPM
Figure 7. MNIST samples from our two U-Nets (6,000 training steps each). Every column starts from the same noise image in all rows and at all step counts. The readout gives the Fréchet distance (233.11) of 2,000 samples for each sampler.
How to read Figure 7

It opens at 8 steps. Each row is one sampler and each column one starting noise, the same in every row. Click 1 step: the flow-matching row shows the same gray blob in every column (the average digit), and the two diffusion rows show identical clipped noise (with one step, DDPM and DDIM are the same update). Click 2 and 4 steps: flow matching forms digits first. At 16 steps and more, compare DDIM and flow matching in the same column: they often draw the same digit from the same noise, as the shared ODE of Step 4 suggests; the DDPM row draws different digits, because it adds fresh noise. The readout gives the Fréchet distance of each sampler at the chosen budget.

Where this is used

ApplicationWhat from this lab carries over
Text-to-image models (Stable Diffusion 3, FLUX.1)The rectified-flow objective (233.2) with a transformer instead of our MLP; SD3 samples \(t\) from a logit-normal distribution instead of uniformly [Certain] (session 229)
Fast or one-step image generationReflow (InstaFlow) and distillation (session 230): the 1-step collapse of Step 3 is the problem they solve
Video generationThe same losses on space-time latents; the cost per step is far higher, so the step-count curve of Figure 6 decides the product (session 231). Meta’s Movie Gen is trained with flow matching [Certain]
Latent diffusion for images and videoEverything here runs unchanged on the latent of an autoencoder instead of pixels (session 228)
Synthetic data for video tasks (action recognition, segmentation, retrieval)Generated clips are used to augment rare classes; the evaluation habit of this lab (a fixed noise batch, a feature-space distance, a sample grid) is how you check them before training on them
Robot policies (vision-language-action models)Some policies predict continuous actions with a flow-matching head instead of tokens [Likely]; the few-step trade-off matters for control frequency

Common mistakes

  • Mixing the two time conventions. Lipman et al. put noise at \(t = 0\); rectified flow and SD3 code often put it at \(t = 1\). A flipped sign or a flipped \(t\) still trains to a reasonable loss, but sampling goes the wrong way. Check: one Euler step from \(t = 1\) should land near the data mean.
  • Judging a model by its training loss. Our flow loss is 2.770 and our DDPM loss 0.409, and both are within 0.5% of their minimum (233.3). Neither number says anything about samples. Sample, and measure.
  • Comparing step counts with different time grids. Evenly spaced DDIM steps on a linear schedule are a weak baseline. Say which grid you use, or the comparison measures the grid, not the model.
  • Sampling DDIM with one or two steps from a schedule that never reaches zero signal. \(1/\sqrt{\bar\alpha_{1000}} = 157\) multiplies every error; our 1-step samples landed 6.2 units from the center.
  • Believing that flow-matching paths are straight. Only the per-pair path is straight. The learned paths measured 0.55 here; you need reflow or distillation for few-step sampling.
  • Forgetting that reflow costs quality. The 2-rectified flow was best at 1 to 8 steps but plateaued at 93.7% on a mode against 96.7% for the original flow. Keep both models, or distill.
  • Comparing models with different starting noise and few samples. With 500 samples the on-mode fraction moved by up to 4 points between two seeds. Use one fixed noise batch for all models, and know the noise floor of your metric.
  • Using a numerical FID on tiny images without saying which network made the features. Our MNIST FD uses a small classifier; its values cannot be compared with Inception-based FID in papers.

Interview questions

Write the training losses of DDPM and of flow matching. What is different?basic
DDPM: \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\mathbf{x} + \sqrt{1-\bar\alpha_t}\epsilon\), loss \(\lVert \epsilon_\theta(\mathbf{x}_t, t) - \epsilon \rVert^2\) with \(t\) uniform in 1..1000. Flow matching: \(\mathbf{x}_t = (1-t)\mathbf{x} + t\epsilon\), loss \(\lVert \mathbf{v}_\theta(\mathbf{x}_t, t) - (\epsilon - \mathbf{x}) \rVert^2\) with \(t\) uniform in [0, 1]. Three differences: the path (a curve on which signal and noise variances add to 1, against a straight line), the target (noise against velocity) and therefore the effective weighting over noise levels. The code differs in two lines; in our lab everything else was identical.
Why does the flow-matching loss not go to zero, and how do you know your model is close to the best possible?core
A noisy point can come from many (data, noise) pairs, so the target is random given the input. The best prediction is the conditional mean, and the minimum loss is the expected conditional variance of the target, (233.3). For a toy mixture of Gaussians the posterior is exact, so you can compute it: 2.757 for our flow loss and 0.407 for DDPM, against 2.770 and 0.409 for our trained networks. On real data you cannot compute it, so you compare sample metrics instead. A loss near zero would rather be a warning sign, for example of a leak of the target into the input (or a reflow model, whose targets are deterministic: 0.001).
One Euler step of a flow-matching model gives a blurry average. Why?core
At \(t = 1\) the input is pure noise, independent of the data, so the best velocity is \(\E[\epsilon - \mathbf{x} \mid \mathbf{x}_1 = \mathbf{z}] = \mathbf{z} - \E[\mathbf{x}]\). One step gives \(\mathbf{z} - \mathbf{v} = \E[\mathbf{x}]\), the mean of the data: in our 2-D lab every sample landed near the center (mean radius 0.07, 0% on a mode); on images this is a blurry average digit. More steps let the path bend toward one mode. Reflow or distillation fixes it by making the target depend on the noise.
Are the sampling paths of a flow-matching model straight?core
The training path for one pair is straight, but the sampler follows the average velocity over all pairs that pass through a point, and those pairs point to different modes. So the learned paths first move toward the data mean and later turn. In our lab their straightness (straight distance / length) was 0.55. They become straight only after reflow (0.999), which retrains on pairs that do not cross. Straight paths in the per-pair sense are a property of the loss; straight paths of the sampler are a property of the coupling between noise and data.
Is a DDIM sampler on a DDPM model the same as an Euler sampler on a flow-matching model?advanced
Up to the network and the coordinates, yes. DDIM with \(\eta = 0\) is a first-order step of the probability-flow ODE. Dividing \(\mathbf{x}_t\) by \(\sqrt{\bar\alpha_t} + \sqrt{1-\bar\alpha_t}\) turns the variance-preserving path into the straight path \((1-t')\mathbf{x} + t'\epsilon\) with a different clock \(t'\), so for perfect networks both samplers integrate the same family of curves. In our lab the rescaled DDIM paths had straightness 0.548 and the flow paths 0.549, and their end points from the same noise were a median of 0.012 apart. Differences in practice come from the time grid, the loss weighting over \(t\) and the training error, not from a fundamentally different ODE.
Why does DDIM with one step from a linear schedule explode, and how do you fix it?advanced
The clean-point estimate divides by \(\sqrt{\bar\alpha_T}\); with the linear schedule \(\bar\alpha_{1000} = 4.04 \times 10^{-5}\), so every error of the noise prediction is multiplied by 157. Our samples landed at radius 6.2 though the data sit at 1.5. Fixes: a schedule with zero terminal signal (Lin et al., 2023) together with velocity or clean-point prediction, which does not divide by a tiny number; sampling grids that start exactly at the last timestep; or simply more steps. The more general lesson: the parameterization of the network output matters most at the extremes of the noise range.
What is reflow, and what does it cost?core
Sample the trained flow to get pairs (noise, its own output), then train a new flow on straight lines between these pairs (233.10). Because the pairs come from an ODE, their straight lines rarely cross, so the averaged velocity equals the pair velocity and the new paths are almost straight; few Euler steps suffice. Costs: generating the pairs (50,000 × 100 network calls in our lab; millions of samples for a real image model), a second training run, and quality: the new model copies the old model’s samples and adds its own error. In our lab the 2-rectified flow had 86.7% on a mode with 1 step (the original: 0%) but plateaued at 93.7% against 96.7%.
Your product allows 4 network calls per image. Which model do you choose?core
Measure the quality-against-steps curve on your own data with a fixed noise batch. In our lab, at 4 steps a reflowed flow was clearly best (91.1% on a mode), plain flow matching second (71.6%) and the DDPM-trained model with DDIM far behind (8.8%). At 64 steps or more the order reverses, and the stochastic DDPM sampler was the best of all. So for a tight budget take a flow model with reflow or a distilled few-step model, and keep a many-step model for offline, quality-critical use. Also try a better solver (Heun, DPM-Solver) and a better time grid before you retrain anything.
Why can a stochastic sampler beat a deterministic one at many steps but lose at few?advanced
A deterministic ODE sampler carries every error it makes to the end. The stochastic DDPM sampler adds fresh noise and removes part of the old state in every step, so the distribution is pulled back toward the right marginal at each noise level; this corrects earlier errors when there are many small steps (our best result: 97.4% at 1,000 steps against 95.1% for DDIM). With few large steps the injected noise is large and the network has few chances to remove it, so the samples stay noisy (DDPM 59.2% at 8 steps). Methods like EDM’s stochastic sampler add a controlled amount of “churn” to get some of both.
How do you evaluate a generative model on a toy 2-D problem and on images?basic
On 2-D data you can look at all samples and use exact metrics: the fraction of samples on a mode (precision), whether every mode is covered (recall), and a distribution distance such as the sliced Wasserstein distance; know the noise floor (0.030 for two real sets of 4,000 points here). On images use a feature-space distance such as FID (233.11) with a fixed number of samples, plus precision and recall in feature space, plus a sample grid from fixed noise. Always fix the starting noise across models and report how many samples you used.

Exercises

  1. Compute \(1/\sqrt{\bar\alpha_t}\) at \(t = 1000\) for the linear schedule, and predict where one DDIM step lands if the noise prediction has an error of norm 0.01.
  2. For the straight path (233.2), show that \(\hat{\mathbf{x}} = \mathbf{x}_t - t\,\mathbf{v}\) and \(\hat{\epsilon} = \mathbf{x}_t + (1-t)\,\mathbf{v}\). Then show that one Euler step from \(t = 1\) with the best velocity gives \(\E[\mathbf{x}]\).
  3. A path goes from \((0, 0)\) to \((1, 0)\) and then to \((1, 1)\). What is its straightness (233.8)? What is the straightness of a half circle?
  4. Show (233.9): divide \(\sqrt{\bar\alpha}\mathbf{x} + \sqrt{1-\bar\alpha}\epsilon\) by \(\sqrt{\bar\alpha} + \sqrt{1-\bar\alpha}\). What is \(t'\) when \(\bar\alpha = 0.5\)?
  5. Run toy2d.py with the flow loss but sample \(t\) from a logit-normal distribution (the SD3 choice): \(t = \sigma(u)\), \(u \sim \mathcal{N}(0, 1)\). Does the 4-step and 64-step quality change?
  6. Train a 3-rectified flow (reflow the 2-rectified model). Predict the change in straightness and in 1-step quality, then measure it.
  7. On MNIST, add a Heun (second-order) sampler to the flow model: two network calls per step. Compare it with Euler at the same number of network calls.
Answerscheck
1. \(\bar\alpha_{1000} = 4.04 \times 10^{-5}\), \(1/\sqrt{\bar\alpha} = 157.4\). An error of 0.01 in \(\epsilon_\theta\) moves \(\hat{\mathbf{x}}\) by \(0.01 \times \sqrt{1-\bar\alpha} \times 157.4 \approx 1.57\), the radius of the whole data ring. 2. From \(\mathbf{x}_t = (1-t)\mathbf{x} + t\epsilon\) and \(\mathbf{v} = \epsilon - \mathbf{x}\): \(\mathbf{x}_t - t\mathbf{v} = (1-t)\mathbf{x} + t\epsilon - t\epsilon + t\mathbf{x} = \mathbf{x}\), and similarly for \(\epsilon\). At \(t = 1\), \(\mathbf{x}_1 = \epsilon\) is independent of \(\mathbf{x}\), so the best \(\mathbf{v}\) is \(\mathbf{z} - \E[\mathbf{x}]\) and \(\mathbf{z} - \mathbf{v} = \E[\mathbf{x}]\). 3. Distance \(\sqrt{2} = 1.414\), length 2: straightness 0.707. Half circle of radius \(r\): \(2r / (\pi r) = 2/\pi = 0.637\). (Our flow paths, 0.55 on average, are more curved than a half circle.) 4. The coefficients become \(\sqrt{\bar\alpha}/(\sqrt{\bar\alpha}+\sqrt{1-\bar\alpha})\) and \(\sqrt{1-\bar\alpha}/(\sqrt{\bar\alpha}+\sqrt{1-\bar\alpha})\), which add to 1. At \(\bar\alpha = 0.5\) both square roots are equal, so \(t' = 0.5\). 5 to 7: open-ended; run them. Expect logit-normal sampling to spend more training on middle noise levels; expect reflow again to raise straightness further but to lower the plateau; and expect Heun to beat Euler at the same number of calls when steps are few, because it corrects the curvature of the path.

Key takeaways

  • Diffusion and flow matching differ in two lines of training code: the noisy input (a curve or a straight line between data and noise) and the target (noise or velocity). Same network, same data, same budget.
  • Both trained models reached their minimum possible loss within 0.5% (2-D). The loss says that the optimizer worked; only samples say how good the model is.
  • At small step budgets flow matching beat the DDPM-trained model with DDIM (8 steps: 89.6% against 64.3% on a mode); at large budgets the stochastic DDPM sampler was the best (97.4%).
  • Learned flow-matching paths are curved (straightness 0.55): they head for the data mean and then turn. One Euler step lands on the mean.
  • DDIM and flow matching integrate the same family of curves in different coordinates (rescaled straightness 0.548 against 0.549).
  • Reflow makes the paths straight (0.999) and makes 1-step sampling work (86.7% on a mode), at the cost of a lower ceiling (93.7% against 96.7%).
  • The same order appeared on MNIST images: flow matching had the lowest Fréchet distance from 2 to 64 steps (8 steps: 37.5 against 55.2 for DDIM), and the DDPM sampler improved fastest with more steps (64 steps: 14.7 against 13.1).

Key formulas

\[ \text{DDPM: } \mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x} + \sqrt{1-\bar\alpha_t}\,\epsilon,\ \ \lVert \epsilon_\theta - \epsilon \rVert^2 \qquad \text{FM: } \mathbf{x}_t = (1-t)\,\mathbf{x} + t\,\epsilon,\ \ \lVert \mathbf{v}_\theta - (\epsilon - \mathbf{x}) \rVert^2 \]
\[ \hat{\mathbf{x}} = \frac{\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta}{\sqrt{\bar\alpha_t}}, \qquad \mathbf{x}_{t-1/N} = \mathbf{x}_t - \tfrac{1}{N}\,\mathbf{v}_\theta(\mathbf{x}_t, t) \]
\[ \text{straightness} = \frac{\lVert \mathbf{x}^{(N)} - \mathbf{x}^{(0)} \rVert}{\sum_i \lVert \mathbf{x}^{(i)} - \mathbf{x}^{(i-1)} \rVert}, \qquad \frac{\mathbf{x}_t}{\sqrt{\bar\alpha_t} + \sqrt{1-\bar\alpha_t}} = (1-t')\,\mathbf{x} + t'\,\epsilon \]
\[ \mathcal{L}_\text{reflow} = \E \big\lVert \mathbf{v}_\phi\big((1-t)\hat{\mathbf{x}}(\mathbf{z}) + t\mathbf{z}, t\big) - (\mathbf{z} - \hat{\mathbf{x}}(\mathbf{z})) \big\rVert^2, \qquad \mathrm{FD} = \lVert \mu_r - \mu_g \rVert^2 + \operatorname{tr}\big(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2}\big) \]

References for this session

  1. Ho et al. (2020). Denoising Diffusion Probabilistic Models
  2. Lipman et al. (2022). Flow Matching for Generative Modeling
  3. Ho et al. (2020). Denoising Diffusion Probabilistic Models
  4. Lipman et al. (2022). Flow Matching for Generative Modeling
  5. Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
  6. Song et al. (2020). Denoising Diffusion Implicit Models
  7. Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
  8. Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
  9. Lin, Liu, Li & Yang (2023). Common Diffusion Noise Schedules and Sample Steps are Flawed
  10. Karras, Aittala, Aila & Laine (2022). Elucidating the Design Space of Diffusion-Based Generative Models
  11. LeCun, Cortes & Burges (1998). The MNIST database of handwritten digits