LLM & VLM courseBlog
226Track 9: Generative models · session 5 of 12

Diffusion 2: score matching, DDIM and fast samplers

A diffusion model secretly learns the score: the direction in which an image becomes more likely. This session shows why removing noise and learning the score are the same task, how diffusion becomes a differential equation (a random one and a deterministic one), why DDIM can sample in 10 to 50 steps instead of 1000 with the same trained network, and how modern solvers such as Heun and DPM-Solver++ trade quality against the number of network calls.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Diffusion 1 (session 225), Gaussians and the chain rule, basic ODEs (Euler's method); basic PyTorch

What you will learn

  • What the score of a distribution is, and why it does not need the normalizing constant
  • Why score matching on clean data is hard, and how denoising score matching turns it into plain noise prediction (Tweedie's formula)
  • Langevin dynamics, why one noise level fails, and how annealing over many noise levels (NCSN) fixes it
  • The forward SDE, the reverse-time SDE and the probability-flow ODE, and why they all share the same marginals
  • DDIM: the update rule, the η knob between DDIM and DDPM, why it is Euler's method on the ODE, and DDIM inversion
  • Higher-order and multistep solvers (Heun, DPM-Solver++, UniPC), the Karras noise schedule, and how to read error versus NFE
  • How to choose a sampler and a step count in practice, and where the network error, not the solver, sets the limit

Why this matters

In session 225 you trained a network to predict the noise in a noisy image, and you sampled by undoing the noise one small step at a time, about 1000 times. This session answers two questions. What has the network really learned? And why can we sample with the same network in 20 steps instead of 1000?

The answer to the first question is the score: for every noisy image, the direction in which it becomes more likely. A noise predictor is a score estimator in disguise. Once you see this, diffusion turns into a differential equation that runs from noise to data, and sampling becomes a problem of solving that equation numerically. Then the second question has a clean answer: a better numerical solver needs fewer network calls. DDIM is the simplest such solver (Euler's method); DPM-Solver++, UniPC and the Heun sampler of Karras et al. are better ones.

The cost is real. Each sampling step is one full forward pass of a large U-Net or diffusion transformer. For a video model, one step processes every frame of the clip. Going from 1000 steps to 25 is a 40 times cheaper sample with no retraining. This is why every image and video product exposes a “sampler” and a “steps” setting, and why the choice matters for latency budgets.

The plan: the score (what it is), score matching (how to learn it), Langevin dynamics and many noise levels (how to sample with it, the NCSN view), the SDE and ODE view (the unified picture), DDIM, and finally modern solvers and the step-count trade-off. The next session, session 227, adds a text prompt to the score.

The score: which way is “more likely”

Let \(p(\mathbf{x})\) be a probability density over images (or over any vectors). Its score is the gradient of its logarithm with respect to the input:

\[ \mathbf{s}(\mathbf{x}) = \nabla_{\mathbf{x}} \log p(\mathbf{x}), \qquad p(\mathbf{x}) = \frac{e^{-E(\mathbf{x})}}{Z} \ \Rightarrow\ \mathbf{s}(\mathbf{x}) = -\nabla_{\mathbf{x}} E(\mathbf{x}) \tag{226.1} \]
Read the symbols
\(\mathbf{s}(\mathbf{x})\)
the score: a vector of the same shape as \(\mathbf{x}\) (for an image, one number per pixel and channel)
\(E(\mathbf{x})\)
an energy: any function that is low where data is common; \(e^{-E}\) is the unnormalized density
\(Z\)
the normalizing constant \(\int e^{-E(\mathbf{x})}\,d\mathbf{x}\), which makes the density integrate to 1; for images it is an integral over millions of dimensions and cannot be computed

Two facts make the score useful. First, it points uphill in probability: a small move \(\mathbf{x} + \delta\,\mathbf{s}(\mathbf{x})\) makes \(\mathbf{x}\) more likely. Second, \(\log Z\) is a constant, so its gradient is zero and the score never needs \(Z\). A model that outputs a score can be any neural network with an output of the same shape as its input; nothing has to integrate to 1. This is the main reason the score is easier to model than the density itself.

The simplest example: for a Gaussian \(\mathcal{N}(\mu, v)\) in one dimension, \(\log p(x) = -(x-\mu)^2/(2v) + \text{const}\), so \(s(x) = -(x-\mu)/v\). The score points toward the mean, and it is stronger far from the mean and for narrow Gaussians. For a mixture of Gaussians the score is a weighted average of the scores of the components, weighted by how likely each component is to have produced \(\mathbf{x}\).

Diffusion never uses the score of the clean data alone. It uses the score of the data after Gaussian noise of level \(\sigma\) has been added, written \(p_\sigma\). Figure 1 shows this for a 2-D mixture of three Gaussians. With little noise, the score is huge near each mode and close to zero (and unreliable) in the empty space between them. With a lot of noise, \(p_\sigma\) is one wide blob, and the score points smoothly toward the data from everywhere.

density \(p_\sigma\) (darker = more likely)score directionprobe \(\mathbf{x}\)denoised guess \(\mathbf{x} + \sigma^2 \mathbf{s}\)
Figure 1. The score of a noisy distribution. The clean data is a mixture of three round Gaussians (standard deviation 0.3, weights 0.6, 0.25, 0.15). Adding noise of level \(\sigma\) widens each one to variance \(0.09 + \sigma^2\). Arrows show the direction of \(\nabla \log p_\sigma\) (length grows with the square root of its size). All values are exact.
How to read Figure 1

Click anywhere to place the orange probe. The readout gives the exact score there and the green point \(\mathbf{x} + \sigma^2\,\mathbf{s}(\mathbf{x})\), which is the best guess of the clean point (formula 226.5 below). Move the slider down to \(\sigma = 0.05\): between the modes the arrows still point somewhere, but the density there is almost zero, so a network trained on data would almost never see these points and could not learn these arrows. Move it up to \(\sigma = 2\) or more: one smooth blob, and every arrow points toward the center of mass. A sampler that starts at high noise and follows the score down through the noise levels uses the easy, smooth field first and the sharp field only at the end.

Learning the score: score matching and denoising

We want a network \(\mathbf{s}_\theta(\mathbf{x})\) with \(\mathbf{s}_\theta \approx \nabla \log p\). The natural loss is the squared distance between the two, averaged over the data. It is called the Fisher divergence or explicit score matching:

\[ J_{\text{ESM}}(\theta) = \tfrac12\, \E_{\mathbf{x} \sim p} \big\lVert \mathbf{s}_\theta(\mathbf{x}) - \nabla_{\mathbf{x}} \log p(\mathbf{x}) \big\rVert^2 \tag{226.2} \]

But we do not know \(\nabla \log p\); we only have samples. Hyvärinen (2005) showed with integration by parts that (226.2) equals, up to a constant that does not depend on \(\theta\),

\[ J_{\text{ISM}}(\theta) = \E_{\mathbf{x} \sim p} \Big[ \operatorname{tr}\big(\nabla_{\mathbf{x}} \mathbf{s}_\theta(\mathbf{x})\big) + \tfrac12 \lVert \mathbf{s}_\theta(\mathbf{x}) \rVert^2 \Big] \tag{226.3} \]

This needs no true score, but the trace of the Jacobian \(\nabla_{\mathbf{x}} \mathbf{s}_\theta\) costs one backward pass per input dimension. For a 64 × 64 × 3 image that is 12,288 backward passes per training example. It does not scale.

Denoising score matching

Vincent (2011) found the way out. Add Gaussian noise to a data point, \(\tilde{\mathbf{x}} = \mathbf{x} + \sigma\varepsilon\) with \(\varepsilon \sim \mathcal{N}(\mathbf{0}, I)\), and match the score of the noisy distribution \(p_\sigma\). For one fixed clean \(\mathbf{x}\), the noisy point has the simple density \(q(\tilde{\mathbf{x}} \mid \mathbf{x}) = \mathcal{N}(\tilde{\mathbf{x}}; \mathbf{x}, \sigma^2 I)\), whose score we know: \(-(\tilde{\mathbf{x}} - \mathbf{x})/\sigma^2 = -\varepsilon/\sigma\). Vincent proved that regressing on this known, per-example target gives the same minimizer as regressing on the unknown \(\nabla \log p_\sigma\):

\[ J_{\text{DSM}}(\theta) = \tfrac12\, \E_{\mathbf{x},\, \varepsilon} \Big\lVert \mathbf{s}_\theta(\mathbf{x} + \sigma\varepsilon,\ \sigma) + \frac{\varepsilon}{\sigma} \Big\rVert^2 = \frac{1}{2\sigma^2}\, \E_{\mathbf{x},\, \varepsilon} \big\lVert \sigma\, \mathbf{s}_\theta(\tilde{\mathbf{x}}, \sigma) + \varepsilon \big\rVert^2 \tag{226.4} \]
Read the symbols
\(\tilde{\mathbf{x}} = \mathbf{x} + \sigma\varepsilon\)
a training image with noise of level \(\sigma\) added
\(-\varepsilon/\sigma\)
the target: the score of the noise distribution around this one clean image; it points from \(\tilde{\mathbf{x}}\) back to \(\mathbf{x}\)
\(\mathbf{s}_\theta(\tilde{\mathbf{x}}, \sigma)\)
the network: it sees only the noisy image and \(\sigma\), never \(\mathbf{x}\), so the best it can do is average the target over all clean images that could have produced \(\tilde{\mathbf{x}}\); that average is exactly \(\nabla \log p_\sigma(\tilde{\mathbf{x}})\)

Look at the right-hand side: \(\sigma\,\mathbf{s}_\theta\) must predict \(-\varepsilon\). Define \(\varepsilon_\theta = -\sigma\,\mathbf{s}_\theta\) and the loss becomes \(\lVert \varepsilon_\theta(\tilde{\mathbf{x}}, \sigma) - \varepsilon \rVert^2\): the DDPM loss of session 225 is denoising score matching, with a weight of \(\sigma^2\) per noise level. Ho et al. (2020) made this link explicit. [Certain]

Tweedie’s formula: score = denoiser

There is a third view of the same object. The best estimate of the clean image given the noisy one, in the squared-error sense, is the posterior mean \(\E[\mathbf{x} \mid \tilde{\mathbf{x}}]\). Tweedie’s formula (Robbins 1956, popularized by Efron 2011) links it to the score:

\[ D(\tilde{\mathbf{x}}; \sigma) := \E[\mathbf{x} \mid \tilde{\mathbf{x}}] = \tilde{\mathbf{x}} + \sigma^2\, \nabla_{\tilde{\mathbf{x}}} \log p_\sigma(\tilde{\mathbf{x}}) \tag{226.5} \]

So the score, the noise and the clean image are three views of one network output. The conversions, in the plain noise-added form \(\tilde{\mathbf{x}} = \mathbf{x} + \sigma\varepsilon\) and in the DDPM form \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon\), are:

\[ \mathbf{s}_\theta = \frac{D_\theta - \tilde{\mathbf{x}}}{\sigma^2} = -\frac{\varepsilon_\theta}{\sigma}, \qquad \text{DDPM form: } \ \mathbf{s}_\theta(\mathbf{x}_t, t) = -\frac{\varepsilon_\theta(\mathbf{x}_t, t)}{\sqrt{1 - \bar\alpha_t}}, \quad \hat{\mathbf{x}}_0 = \frac{\mathbf{x}_t - \sqrt{1 - \bar\alpha_t}\,\varepsilon_\theta}{\sqrt{\bar\alpha_t}} \tag{226.6} \]
Read the symbols
\(D_\theta\)
the denoiser: the network’s guess of the clean image (also written \(\hat{\mathbf{x}}_0\))
\(\varepsilon_\theta\)
the network’s guess of the noise that was added
\(\bar\alpha_t\)
the DDPM signal level at step \(t\); the two forms are the same thing with \(\sigma = \sqrt{(1 - \bar\alpha_t)/\bar\alpha_t}\) and \(\tilde{\mathbf{x}} = \mathbf{x}_t / \sqrt{\bar\alpha_t}\)

The last point is worth remembering: a DDPM noisy image is just a scaled version of “clean image plus noise of level \(\sigma\)”. For the linear schedule of DDPM (\(\beta\) from \(10^{-4}\) to 0.02 over 1000 steps), \(\bar\alpha_{1000} = 4.04 \times 10^{-5}\), which is \(\sigma \approx 157\), and \(\bar\alpha_1 = 0.9999\), which is \(\sigma = 0.01\). We will use the \(\sigma\) form for the math, because it is the simplest; every formula maps back to the \(\bar\alpha\) form by this rescaling.

Worked example: score, noise and denoiser for Gaussian data

Let the data be \(\mathcal{N}(0, 1)\) in one dimension. Adding noise of level \(\sigma = 2\) gives \(p_\sigma = \mathcal{N}(0,\ 1 + 4) = \mathcal{N}(0, 5)\). At \(\tilde{x} = 3\):

Score: \(s = -3/5 = -0.6\). Tweedie (226.5): \(D = 3 + 4 \cdot (-0.6) = 0.6\). Check against the direct posterior mean of a Gaussian, \(\tfrac{1}{1 + 4} \cdot 3 = 0.6\). Predicted noise: \(\varepsilon = (\tilde{x} - D)/\sigma = (3 - 0.6)/2 = 1.2\), and indeed \(-\varepsilon/\sigma = -0.6 = s\).

In DDPM form this point has \(\bar\alpha = 1/(1 + \sigma^2) = 0.2\) and \(x_t = 3 \cdot \sqrt{0.2} = 1.342\). The three numbers \(s = -0.6\), \(D = 0.6\), \(\varepsilon = 1.2\) are what a perfect network would output, in its three possible parameterizations.

Sampling with the score: Langevin dynamics and many noise levels

Suppose we have the score. How do we turn it into samples? Langevin dynamics is a random walk that climbs the score and adds noise at each step:

\[ \mathbf{x}_{k+1} = \mathbf{x}_k + \frac{\alpha}{2}\, \mathbf{s}(\mathbf{x}_k) + \sqrt{\alpha}\, \mathbf{z}_k, \qquad \mathbf{z}_k \sim \mathcal{N}(\mathbf{0}, I) \tag{226.7} \]

Here \(\alpha \gt 0\) is the step size. The drift term alone would climb to the nearest peak and stop (that is gradient ascent on \(\log p\)). The noise term keeps the walker moving, and the two balance exactly so that, as \(\alpha \to 0\) and the number of steps grows, the walker’s position is distributed as \(p\). [Certain]

In practice plain Langevin dynamics on the clean score fails, for two reasons that Song and Ermon (2019) pointed out:

  • The score is wrong where there is no data. The network is trained on samples, and the empty space between modes has almost none. It is exactly the space a walker must cross.
  • Mixing between modes is extremely slow. Even with the true score, a walker in one mode must cross a region of near-zero probability to reach another. In finite time, walkers stay in whichever mode they fell into first, so the share of samples per mode follows the size of each mode’s basin of attraction, not the true weights.

The fix of their noise-conditional score network (NCSN) is to train one network for many noise levels \(\sigma_1 \gt \sigma_2 \gt \dots \gt \sigma_L\) at once, with loss (226.4) summed over levels (weighted by \(\sigma^2\)), and to sample with annealed Langevin dynamics: run Langevin at the largest noise level, where \(p_\sigma\) is one smooth blob and all modes are connected, then use the result as the start for the next, smaller noise level, and so on.

Algorithm 1: annealed Langevin dynamics (NCSN)
  1. input: score network \(\mathbf{s}_\theta(\mathbf{x}, \sigma)\); noise levels \(\sigma_1 \gt \dots \gt \sigma_L\) (geometric); steps per level \(T\); base step \(\epsilon\)
  2. \(\mathbf{x} \leftarrow\) a random start (uniform noise or \(\mathcal{N}(\mathbf{0}, \sigma_1^2 I)\))
  3. for \(i = 1, \dots, L\): // from coarse to fine
  4.   \(\alpha_i \leftarrow \epsilon\, \sigma_i^2 / \sigma_L^2\) // step size scales with the variance of the level
  5.   repeat \(T\) times: \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\); \(\ \mathbf{x} \leftarrow \mathbf{x} + \tfrac{\alpha_i}{2}\, \mathbf{s}_\theta(\mathbf{x}, \sigma_i) + \sqrt{\alpha_i}\, \mathbf{z}\) // formula (226.7)
  6. return \(\mathbf{x}\) // later work adds one final denoising step with Tweedie (226.5)

The step size rule \(\alpha_i \propto \sigma_i^2\) keeps the “signal to noise ratio” of each update the same at every level. Figure 2 runs both versions on the mixture of Figure 1, with the exact score.

Figure 2. 500 particles, started uniformly over the box, run 400 Langevin steps (10 stages of 40). Annealed: 10 geometric noise levels from 3 to 0.05 with \(\alpha_i = 0.3\,\sigma_i^2\). Plain: all 400 steps at \(\sigma = 0.05\) with \(\alpha = 0.02\). Fixed random seed; dashed circles mark the three modes.
How to read Figure 2

Step through the stages. In the plain run, every particle slides into the nearest mode within the first stage and never leaves, so the shares end near the shares of the start box closest to each mode (0.30 / 0.29 / 0.41), far from the true 0.6 / 0.25 / 0.15. In the annealed run, the first stages pull all particles into one wide blob near the weighted center; as \(\sigma\) shrinks, the blob splits, and each part gets a share close to the true weight (0.55 / 0.30 / 0.16). The remaining gap comes from 500 particles and the short run; it shrinks with more steps per level.

Diffusion as a differential equation

DDPM has 1000 discrete steps; NCSN has 10 or so noise levels. Song et al. (2021) took the limit of infinitely many tiny steps. The noising process then becomes a stochastic differential equation (SDE): a differential equation with a random term.

\[ d\mathbf{x} = \mathbf{f}(\mathbf{x}, t)\, dt + g(t)\, d\mathbf{w} \tag{226.8} \]
Read the symbols
\(t \in [0, T]\)
continuous time; \(t = 0\) is clean data, \(t = T\) is (almost) pure noise
\(\mathbf{f}(\mathbf{x}, t)\)
the drift: a deterministic push per unit time
\(g(t)\, d\mathbf{w}\)
the diffusion term: \(d\mathbf{w}\) is a tiny Gaussian step with variance \(dt\) (Brownian motion), and \(g(t)\) sets how much noise is added per unit time
VP SDE
\(\mathbf{f} = -\tfrac12 \beta(t)\,\mathbf{x}\), \(g = \sqrt{\beta(t)}\): the continuous version of DDPM; the signal shrinks while noise is added, so the variance stays bounded (“variance preserving”)
VE SDE
\(\mathbf{f} = \mathbf{0}\), \(g = \sqrt{d\sigma^2(t)/dt}\): the continuous version of NCSN; only noise is added, so \(\mathbf{x}_t = \mathbf{x}_0 + \sigma(t)\varepsilon\) and the variance explodes (“variance exploding”)

A classic result of Anderson (1982) says that this process can be run backward in time, and the reverse is again an SDE. Its drift needs only one thing we do not already know: the score of the noisy data at each time.

\[ d\mathbf{x} = \big[ \mathbf{f}(\mathbf{x}, t) - g(t)^2\, \nabla_{\mathbf{x}} \log p_t(\mathbf{x}) \big]\, dt + g(t)\, d\bar{\mathbf{w}} \tag{226.9} \]

Here time runs from \(T\) down to 0 (so \(dt\) is negative) and \(d\bar{\mathbf{w}}\) is Brownian motion in reverse time. Replace the score by the network \(\mathbf{s}_\theta(\mathbf{x}, t)\), start from noise, integrate down to \(t = 0\), and you have a sample. DDPM’s sampler is one discretization of this equation; adding Langevin “corrector” steps after each step gives the predictor–corrector sampler of the same paper.

The probability-flow ODE

The most useful result in the paper is this: there is a deterministic process, an ordinary differential equation (ODE) with no random term, whose solutions have exactly the same distribution \(p_t\) at every time \(t\) as the SDE:

\[ \frac{d\mathbf{x}}{dt} = \mathbf{f}(\mathbf{x}, t) - \tfrac12\, g(t)^2\, \nabla_{\mathbf{x}} \log p_t(\mathbf{x}) \tag{226.10} \]

Note the factor \(\tfrac12\) where the reverse SDE has 1, and the missing noise term. Intuitively, the reverse SDE moves twice as far up the score and then adds noise that spreads the samples back out by half that amount on average; the ODE takes only the net effect. For the VE form with \(\sigma\) as the time variable (Karras et al., 2022), and using Tweedie (226.5), the ODE becomes very simple:

\[ \frac{d\mathbf{x}}{d\sigma} = -\sigma\, \nabla_{\mathbf{x}} \log p_\sigma(\mathbf{x}) = \frac{\mathbf{x} - D(\mathbf{x}; \sigma)}{\sigma} \tag{226.11} \]
Read the symbols
\(\mathbf{x} - D(\mathbf{x}; \sigma)\)
the part of \(\mathbf{x}\) the denoiser thinks is noise
\((\mathbf{x} - D)/\sigma\)
the predicted noise \(\hat\varepsilon\): the ODE says “as \(\sigma\) goes down by \(d\sigma\), remove \(\hat\varepsilon\, d\sigma\) of noise”

Why the ODE matters so much:

  • Big steps. An ODE has smooth solutions, and numerical analysis offers many accurate solvers for it. An SDE solver must also simulate the noise, which limits how large the steps can be.
  • A fixed map from noise to images. Each starting noise gives one image. This makes inversion possible: run the ODE forward from a real image to get its noise, then edit and run it back.
  • Exact likelihoods. With the change-of-variables formula of continuous normalizing flows, the ODE gives \(\log p(\mathbf{x})\); Song et al. report 2.99 bits per dimension on CIFAR-10. [Certain]

Figure 3 shows both on a 1-D mixture of three Gaussians, with the exact denoiser. The knob \(\eta\) blends between them (this is the DDIM \(\eta\), next section): \(\eta = 0\) is the ODE, \(\eta = 1\) is the DDPM-like reverse SDE.

sampling paths (scaled by \(1/\sqrt{1 + \sigma^2}\))true data density
Figure 3. 2,000 samples of a 1-D mixture (weights 0.35, 0.45, 0.2; means −2, 0.5, 2.5; standard deviations 0.3, 0.4, 0.2), each run for 100 steps from \(\sigma = 80\) to 0 with the exact denoiser. Paths are drawn for 40 of them, scaled by \(1/\sqrt{1 + \sigma^2}\) (the DDPM view) so that the start fits on the plot. Right: histogram of all end points against the true density.
How to read Figure 3

At \(\eta = 0\) the paths are smooth and never cross (the readout counts 0 swapped pairs): in one dimension the ODE keeps the order of its starting points, so it maps the \(k\)-th smallest noise to the \(k\)-th smallest sample. Raise \(\eta\): the paths become jittery and cross, and “New noise” changes the result. Yet the histogram on the right stays close to the true density at every \(\eta\), and the share per mode stays close to 0.35 / 0.45 / 0.2. Same marginals, different paths: that is the whole content of (226.9) versus (226.10). Notice also that the paths decide their mode in the middle, around \(\sigma\) between 0.3 and 3.

DDIM: deterministic sampling with far fewer steps

DDIM (Song, Meng and Ermon, 2020) was found from a different angle, before the ODE view was common. The DDPM training loss only uses the marginals \(q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}(\sqrt{\bar\alpha_t}\,\mathbf{x}_0,\ (1 - \bar\alpha_t) I)\). It never uses the step-by-step chain \(q(\mathbf{x}_t \mid \mathbf{x}_{t-1})\). So any other forward process with the same marginals has the same training loss and can reuse the same trained network. DDIM picks a family of non-Markovian forward processes (each step may also look at \(\mathbf{x}_0\)) with these marginals, and derives their reverse step:

\[ \mathbf{x}_{t'} = \sqrt{\bar\alpha_{t'}}\; \hat{\mathbf{x}}_0 + \sqrt{1 - \bar\alpha_{t'} - \sigma_t^2}\;\, \varepsilon_\theta(\mathbf{x}_t, t) + \sigma_t\, \mathbf{z}, \qquad \sigma_t = \eta \sqrt{\frac{1 - \bar\alpha_{t'}}{1 - \bar\alpha_t}} \sqrt{1 - \frac{\bar\alpha_t}{\bar\alpha_{t'}}} \tag{226.12} \]
Read the symbols
\(t' \lt t\)
the next time step; it does not need to be \(t - 1\): any decreasing sub-sequence of the 1000 training steps works
\(\hat{\mathbf{x}}_0\)
the predicted clean image from (226.6)
\(\sqrt{1 - \bar\alpha_{t'} - \sigma_t^2}\; \varepsilon_\theta\)
the “direction pointing to \(\mathbf{x}_t\)”: the predicted noise, re-used at the lower noise level
\(\sigma_t \mathbf{z}\)
fresh random noise; \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\)
\(\eta\)
\(\eta = 0\): deterministic DDIM. \(\eta = 1\): the DDPM posterior variance, so with \(t' = t - 1\) this is DDPM ancestral sampling

Read (226.12) as two moves. First, jump to the network’s guess of the clean image. Then, put back exactly the noise that belongs at level \(t'\), using the same noise direction the network just predicted (plus fresh noise if \(\eta \gt 0\)). The paper reports samples of high quality 10 to 50 times faster in wall-clock time than DDPM. [Certain]

DDIM is Euler’s method on the probability-flow ODE

Divide (226.12) with \(\eta = 0\) by \(\sqrt{\bar\alpha_{t'}}\) and switch to the \(\sigma\) form, \(\mathbf{x} = \mathbf{x}_t / \sqrt{\bar\alpha_t}\) and \(\sigma = \sqrt{(1 - \bar\alpha_t)/\bar\alpha_t}\). With \(D = \hat{\mathbf{x}}_0\) the step becomes:

\[ \mathbf{x}_{\text{next}} = D(\mathbf{x}; \sigma) + \frac{\sigma_{\text{next}}}{\sigma} \big( \mathbf{x} - D(\mathbf{x}; \sigma) \big) = \mathbf{x} + (\sigma_{\text{next}} - \sigma)\, \frac{\mathbf{x} - D(\mathbf{x}; \sigma)}{\sigma} \tag{226.13} \]

The right-hand form is one step of Euler’s method on the ODE (226.11): current point plus step size times current slope. [Certain] So DDIM is the simplest possible ODE solver, applied in the coordinates where the ODE is simplest. The left-hand form has a nice picture, shown in Figure 4: the next point lies on the straight line from the current point to the current guess of the clean image, at the fraction \(\sigma_{\text{next}}/\sigma\) of the way back from the guess.

clean data x at level σ D(x; σ) = guess of x₀ DDIM: xnext exact ODE at σnext xnext lies on the dashed line, at the fractionσnext/σ of the way from D back toward x error of one Euler step
Figure 4. One DDIM step, drawn as a sketch (not to scale). DDIM jumps along the straight line (dashed) toward the current denoised guess and stops at the next noise level. The exact ODE path (green) curves, because the guess \(D\) changes as the noise goes down. The gap between the two is the error of one Euler step; it shrinks when steps are small or when the path is nearly straight.
Worked example: one DDIM step and one Heun step, with exact numbers

Data \(\mathcal{N}(0, 1)\), start at \(x = 3\) with \(\sigma = 2\), step to \(\sigma_{\text{next}} = 1\). From the previous worked example, \(D = 0.6\) and \(\hat\varepsilon = 1.2\).

DDIM (226.13): \(x_{\text{next}} = 0.6 + \tfrac12 (3 - 0.6) = 1.8\). In DDPM form: \(\bar\alpha\) goes from 0.2 to 0.5, and \(\sqrt{0.5}\cdot 0.6 + \sqrt{0.5}\cdot 1.2 = 1.273 = 1.8/\sqrt{2}\). The two forms agree.

Exact ODE: for Gaussian data \(D = x/(1 + \sigma^2)\), so (226.11) reads \(dx/d\sigma = \sigma x/(1 + \sigma^2)\), whose solution is \(x \propto \sqrt{1 + \sigma^2}\). From \(x = 3\) at \(\sigma = 2\): \(x(1) = 3\sqrt{2/5} = 1.897\). DDIM is off by 0.097.

Heun (226.14 below): the Euler guess 1.8 gives a second slope at \(\sigma = 1\): \(D = 1.8/2 = 0.9\), slope \((1.8 - 0.9)/1 = 0.9\). Average slope \((1.2 + 0.9)/2 = 1.05\), so \(x_{\text{next}} = 3 - 1 \cdot 1.05 = 1.95\). Error 0.053, about half, for one extra network call.

Algorithm 2: DDIM sampling with \(S\) steps (DDPM-trained network)
  1. input: noise network \(\varepsilon_\theta\); schedule \(\bar\alpha_1, \dots, \bar\alpha_{1000}\); a decreasing sub-sequence \(\tau_S \gt \dots \gt \tau_1\) of training steps (for example every 50th step for \(S = 20\)); \(\eta\)
  2. \(\mathbf{x} \sim \mathcal{N}(\mathbf{0}, I)\) // at time \(\tau_S\)
  3. for \(i = S, \dots, 1\): \(t \leftarrow \tau_i\); \(t' \leftarrow \tau_{i-1}\) (with \(\bar\alpha_{\tau_0} := 1\))
  4.   \(\hat\varepsilon \leftarrow \varepsilon_\theta(\mathbf{x}, t)\); \(\ \hat{\mathbf{x}}_0 \leftarrow (\mathbf{x} - \sqrt{1 - \bar\alpha_t}\,\hat\varepsilon)/\sqrt{\bar\alpha_t}\) // formula (226.6); optionally clip to the data range
  5.   \(\sigma_t \leftarrow\) formula (226.12) with \(\eta\); \(\ \mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\) if \(\eta \gt 0\)
  6.   \(\mathbf{x} \leftarrow \sqrt{\bar\alpha_{t'}}\,\hat{\mathbf{x}}_0 + \sqrt{1 - \bar\alpha_{t'} - \sigma_t^2}\,\hat\varepsilon + \sigma_t \mathbf{z}\) // \(S\) network calls in total
  7. return \(\mathbf{x}\)

Figure 5 puts the two samplers side by side on a picture. The setup is the one of Figures 4 and 5 in session 225: 96 × 96 color pictures with values in \([-1, 1]\), the linear DDPM schedule, and in place of a trained network the ideal prediction \(\varepsilon^\star = \E[\varepsilon \mid \mathbf{x}_t]\) of a perfectly trained network, for a training set of one or of three pictures. Both samplers start from the same noise image \(\mathbf{x}_{1000}\). DDPM runs 1000 steps of its update with fresh noise \(\mathbf{z}\) at every step (\(\eta = 1\)). DDIM runs \(S\) steps of (226.12) with \(\eta = 0\) at the steps \(\tau_i = i \cdot 1000/S\), so it also starts at \(t = 1000\). The algorithm box is Algorithm 2 with \(\eta = 0\), split into one line per formula.

DDIM steps \(S\)Perfect model of
DDIM sampling, \(\eta = 0\) (Algorithm 2)
  1. \(\mathbf{x}_{1000} \sim \mathcal{N}(\mathbf{0}, I)\)
  2. for \(i = S, \dots, 1\): \(t \leftarrow \tau_i,\ t' \leftarrow \tau_{i-1}\)
  3. \(\hat\varepsilon \leftarrow \varepsilon_\theta(\mathbf{x}_t, t)\) one network call
  4. \(\hat{\mathbf{x}}_0 \leftarrow \big(\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\hat\varepsilon\big) / \sqrt{\bar\alpha_t}\) (226.6)
  5. \(\mathbf{x}_{t'} \leftarrow\) \(\sqrt{\bar\alpha_{t'}}\,\hat{\mathbf{x}}_0 + \sqrt{1-\bar\alpha_{t'}}\,\hat\varepsilon\) (226.12), \(\eta = 0\)
  6. return \(\mathbf{x}_0\)
DDPM \(\mathbf{x}_t\)
DDIM \(\mathbf{x}_t\)
DDIM \(\hat{\mathbf{x}}_0\)
Figure 5. DDPM (1000 stochastic steps) and DDIM (\(S\) = 50, 20 or 10 deterministic steps) on a 96 × 96 picture, from the same seeded noise image, with the linear schedule. Both use the ideal prediction \(\varepsilon^\star\) of a perfectly trained network for a training set of one or three pictures, not a real model. The DDPM panel always shows the same noise level \(t\) as the DDIM panel; its label counts the network calls DDPM needed to get there.
How to read Figure 5

Press Play, or Next line to go one formula at a time. With one picture, the ideal \(\hat\varepsilon\) is exactly the noise inside \(\mathbf{x}_t\), and line 5 puts that same noise back at the lower level, so \(\hat\varepsilon\) never changes. DDIM therefore walks back along the forward path \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\hat\varepsilon\) (the top row of Figure 4 in session 225, in reverse), and its \(\hat{\mathbf{x}}_0\) panel shows the picture from the first step. In the \(\sigma\) form this path is a straight line, so Euler’s method has no error, and \(S\) = 50, 20 and 10 all end exactly on the picture (RMS distance 0.0000) after 50, 20 or 10 network calls. DDPM ends on the same picture after 1000 calls, but its noise pattern changes at every step because each step adds fresh noise \(\mathbf{z}\). Switch to three pictures. DDIM’s \(\hat{\mathbf{x}}_0\) now starts as a blend of all three, and the weights move smoothly: with \(S = 50\) they go from 0.37, 0.31, 0.32 at \(t = 1000\) to 0.94 for sunny hills at \(t = 800\) and 1.000 by \(t = 760\). \(S = 20\) and \(S = 10\) follow almost the same path with fewer, larger steps, and all three end on sunny hills. DDPM, from the same starting noise, ends on the night picture: its fresh noise changed the decision, which is made in roughly the first 200 steps (this is the same run as Figure 5 of session 225). A deterministic sampler is a fixed map from noise to picture; here the same \(\mathbf{x}_{1000}\) gives the same picture at every \(S\), and the DDIM paper reports this consistency for real models too [Likely]. It is what makes DDIM inversion (next) possible. A stochastic sampler fixes only the distribution of pictures, not which one you get. With a real network, fewer steps cost quality because the ODE path curves (Figure 6); a perfect model of a tiny training set hides that cost, because its path bends only during the short time when the decision is made.

DDIM inversion

Because DDIM with \(\eta = 0\) is a deterministic ODE solver, it can also run in the other direction: from a real image, step up in noise with the same rule, and you get a noise \(\mathbf{x}_T\) that DDIM maps back to (almost) the same image. Editing methods then change the prompt or the noise and decode again. The inversion is only approximate: each Euler step uses the slope at the start of the step, which is slightly different going up than going down, and with classifier-free guidance the error grows. [Likely] Methods such as null-text inversion were built to fix this.

Modern solvers and the step-count trade-off

Once sampling is “solve the ODE (226.11) from \(\sigma_{\max}\) to 0”, the cost is counted in NFE, the number of function (network) evaluations. Everything else is cheap. Three ideas push quality per NFE up.

1. Higher-order steps (Heun)

Euler’s error per step is proportional to the step size squared, so its total error is proportional to the step size: halving the step size halves the error. Heun’s method takes an Euler step, evaluates the slope again at the end, and uses the average of the two slopes:

\[ \mathbf{d}_i = \frac{\mathbf{x}_i - D(\mathbf{x}_i; \sigma_i)}{\sigma_i}, \quad \tilde{\mathbf{x}} = \mathbf{x}_i + (\sigma_{i+1} - \sigma_i)\, \mathbf{d}_i, \quad \mathbf{x}_{i+1} = \mathbf{x}_i + (\sigma_{i+1} - \sigma_i)\, \frac{\mathbf{d}_i + \tilde{\mathbf{d}}}{2} \tag{226.14} \]

where \(\tilde{\mathbf{d}}\) is the slope at \((\tilde{\mathbf{x}}, \sigma_{i+1})\). Its total error falls with the square of the step size, but each step costs 2 NFE. Karras et al. (2022, “EDM”) use it with no extra step at the very end (the slope at \(\sigma = 0\) is undefined), so \(N\) steps cost \(2N - 1\) NFE; their CIFAR-10 result of FID 1.79 used 35 NFE [Certain], which is 18 steps. [Likely]

2. Where to put the steps: the noise schedule for sampling

The ODE path is nearly straight at high noise and bends strongly at low noise (Figure 3: the paths are flat at high noise and bend between \(\sigma \approx 3\) and 0.3). So steps should be dense at small \(\sigma\). EDM proposes:

\[ \sigma_i = \Big( \sigma_{\max}^{1/\rho} + \tfrac{i}{N-1} \big( \sigma_{\min}^{1/\rho} - \sigma_{\max}^{1/\rho} \big) \Big)^{\rho}, \quad i = 0, \dots, N-1, \qquad \sigma_N = 0 \tag{226.15} \]

with \(\sigma_{\max} = 80\), \(\sigma_{\min} = 0.002\), \(\rho = 7\). For \(N = 10\) this gives 80, 42.4, 21.1, 9.72, 4.07, 1.50, 0.470, 0.117, 0.0204, 0.002, then 0: half of the steps are below \(\sigma = 1.5\).

3. Exact linear part and multistep reuse (DPM-Solver, DPM-Solver++, UniPC)

Lu et al. (2022) noticed that the diffusion ODE has a linear part that can be integrated exactly, leaving only the network term to approximate. Written with \(\lambda = -\log\sigma\) and the data prediction \(D\), the first-order DPM-Solver++ step is exactly DDIM (226.13). The second-order multistep version, DPM-Solver++(2M), reuses the previous step’s \(D\) to estimate how \(D\) changes, so it gets second-order accuracy for one NFE per step:

\[ h_i = \log\frac{\sigma_i}{\sigma_{i+1}}, \qquad \hat D_i = D_i + \frac{h_i}{2 h_{i-1}} \big( D_i - D_{i-1} \big), \qquad \mathbf{x}_{i+1} = \frac{\sigma_{i+1}}{\sigma_i}\, \mathbf{x}_i + \Big( 1 - \frac{\sigma_{i+1}}{\sigma_i} \Big) \hat D_i \tag{226.16} \]
Read the symbols
\(D_i\)
the denoiser output at step \(i\), \(D(\mathbf{x}_i; \sigma_i)\); \(D_{i-1}\) is kept from the previous step, so no extra network call
\(h_i\)
the step size in log-noise; \(h_i / h_{i-1}\) corrects for unequal steps
\(\hat D_i\)
a linear extrapolation of the clean-image guess toward the middle of the step; with \(\hat D_i = D_i\) the update is DDIM again

The DPM-Solver paper reports FID 4.70 at 10 NFE and 2.87 at 20 NFE on CIFAR-10 [Certain]; DPM-Solver++ targets guided sampling, where it reports high quality in 15 to 20 steps, while DDIM “generally needs 100 to 250 steps” in that setting [Certain]. UniPC adds a corrector that reuses the new network output, and reports FID 3.87 on CIFAR-10 with 10 NFE. [Certain]

Algorithm 3: a generic ODE sampler in \(\sigma\) form (DDIM, Heun or DPM-Solver++(2M))
  1. input: denoiser \(D_\theta(\mathbf{x}; \sigma)\); number of steps \(N\); schedule (226.15) \(\sigma_0 \gt \dots \gt \sigma_{N-1} \gt \sigma_N = 0\)
  2. \(\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \sigma_0^2 I)\); \(\ D_{\text{prev}} \leftarrow\) none
  3. for \(i = 0, \dots, N - 1\): \(\ D_i \leftarrow D_\theta(\mathbf{x}; \sigma_i)\) // 1 NFE
  4.   if \(\sigma_{i+1} = 0\): \(\mathbf{x} \leftarrow D_i\); break // last step: just denoise
  5.   DDIM: \(\mathbf{x} \leftarrow D_i + (\sigma_{i+1}/\sigma_i)(\mathbf{x} - D_i)\) // (226.13)
  6.   Heun: Euler guess \(\tilde{\mathbf{x}}\), one more call \(D_\theta(\tilde{\mathbf{x}}; \sigma_{i+1})\), average slopes // (226.14), +1 NFE
  7.   DPM-Solver++(2M): extrapolate \(\hat D_i\) from \(D_i, D_{\text{prev}}\); update with (226.16); \(D_{\text{prev}} \leftarrow D_i\)
  8. return \(\mathbf{x}\) // for a latent model, decode with the VAE afterwards

Figure 6 runs all of these on the 1-D mixture of Figure 3 with the exact denoiser, so the only error left is the solver’s. Because the exact ODE in 1-D maps the \(k\)-th smallest starting noise to the \(k\)-th smallest data quantile, we can start from 1,000 evenly spaced quantiles of \(p_{80}\) and measure the error as the average distance between the sorted outputs and the true data quantiles (the 1-D Wasserstein-1 distance \(W_1\)).

DDIMHeunDPM-Solver++(2M)ancestral, \(\eta = 1\) (dashed)
Figure 6. The step-count trade-off with a perfect network. Sampling the 1-D mixture of Figure 3 with the schedule (226.15) (\(\sigma_{\max} = 80\), \(\sigma_{\min} = 0.002\), \(\rho = 7\)). Heun gets \(N = \lfloor(\text{NFE} + 1)/2\rfloor\) steps, so it uses \(2N - 1\) NFE. The ancestral sampler uses a fixed seed; with 1,000 random samples its error cannot go much below about 0.05 even with perfect steps.
How to read Figure 6

Move the budget. At 10 NFE: DDIM 0.29, Heun 1.05 (only 5 steps, too coarse for a second-order method), DPM-Solver++(2M) 0.23. At 50 NFE: DDIM 0.054, Heun 0.019, DPM-Solver++(2M) 0.0047. On the log-log plot, the DDIM line has slope about −1 (error \(\propto 1/\text{NFE}\)) and the two second-order methods have slope about −2: from 31 to 51 NFE, DDIM’s error falls by 1.67 times, DPM-Solver++(2M)’s by 2.9 times. Below about 8 NFE nothing is good, and Heun is the worst: its extra evaluations buy nothing when each step is huge. Look at the histogram at low NFE: the error shows up as modes that are too wide and too close together, the 1-D version of blurry, washed-out images.

Choosing a sampler in practice

  • 20 to 30 steps of a second-order multistep solver (DPM-Solver++(2M), UniPC) is the usual default for latent image models; DDIM at 50 steps is the classic safe choice. [Likely]
  • Stochastic samplers (\(\eta \gt 0\), ancestral, SDE variants) need more steps, but the injected noise can correct earlier errors, and with many steps they sometimes give better quality than the ODE. EDM found that a modest amount of noise helps some models and hurts others. [Likely]
  • Below about 8 steps, solvers alone fail; you need a model trained or distilled for few steps: progressive distillation, consistency models, or straight flows (session 229, session 230).
  • The network sets a floor. A perfect solver of the wrong ODE is still wrong. Once the solver error is below the network’s own error, more steps change little (the PyTorch run below shows this).
SamplerTypeNFE per stepOrderTypical steps
DDPM ancestralreverse SDE11 (weak)1000 in the original paper
DDIM (\(\eta = 0\))ODE, Euler1150 to 100 [Likely]
Heun (EDM)ODE, Runge–Kutta2218 steps = 35 NFE on CIFAR-10
DPM-Solver-2 / -3 (single-step)ODE, exponential integrator2 / 32 / 310 to 20 NFE
DPM-Solver++(2M)ODE, multistep1215 to 25
UniPCODE, predictor–corrector12 to 310 to 20
Predictor–corrector (score SDE)SDE + Langevin21hundreds to thousands

Score-based sampling in PyTorch

A complete, deterministic example on the 1-D mixture of Figures 3 and 6. It trains a small denoiser with denoising score matching (in the “predict the clean value” form, with the input and output scaling of EDM, which keeps the network’s inputs and targets near unit variance at every \(\sigma\)), then samples with DDIM, Heun and DPM-Solver++(2M), using both the exact denoiser and the trained network. It runs in about half a minute on a CPU.

import math, torch, torch.nn as nn
torch.manual_seed(0)

# toy data: a 1-D mixture of three Gaussians (weights, means, standard deviations)
W  = torch.tensor([0.35, 0.45, 0.20])
MU = torch.tensor([-2.0, 0.5, 2.5])
SD = torch.tensor([0.3, 0.4, 0.2])
SIG_D = math.sqrt(float((W * (SD**2 + MU**2)).sum() - (W * MU).sum()**2))   # data std

def sample_data(n):
    c = torch.multinomial(W, n, replacement=True)
    return MU[c] + SD[c] * torch.randn(n)

def exact_D(x, s):
    """Exact denoiser E[x0 | x_sigma = x] of the mixture (Tweedie): what a perfect network outputs."""
    v = SD**2 + s[:, None]**2                                   # variance of each noisy component
    logw = torch.log(W) - 0.5 * torch.log(v) - 0.5 * (x[:, None] - MU)**2 / v
    r = torch.softmax(logw, dim=1)                              # which component produced x?
    return (r * (MU + SD**2 / v * (x[:, None] - MU))).sum(1)

class Denoiser(nn.Module):                                      # D(x; sigma) with EDM preconditioning
    def __init__(self, h=128):
        super().__init__()
        self.f = nn.Sequential(nn.Linear(2, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 1))
    def forward(self, x, s):
        c_in = 1 / torch.sqrt(s**2 + SIG_D**2)
        c_skip = SIG_D**2 / (s**2 + SIG_D**2)
        c_out = s * SIG_D / torch.sqrt(s**2 + SIG_D**2)
        F = self.f(torch.stack([c_in * x, torch.log(s) / 4], dim=1)).squeeze(1)
        return c_skip * x + c_out * F

# denoising score matching = train D to remove the noise (formula 226.4)
net = Denoiser()
opt = torch.optim.Adam(net.parameters(), lr=2e-3)
for step in range(6000):
    x0 = sample_data(1024)
    s = torch.exp(-1.2 + 1.2 * torch.randn(1024))              # log-normal noise levels (EDM)
    x = x0 + s * torch.randn(1024)
    lam = (s**2 + SIG_D**2) / (s * SIG_D)**2                    # weight so every sigma counts equally
    loss = (lam * (net(x, s) - x0)**2).mean()
    opt.zero_grad(); loss.backward(); opt.step()

def karras(n, smin=0.002, smax=80.0, rho=7.0):                  # formula (226.15), then a final 0
    i = torch.arange(n) / max(n - 1, 1)
    s = (smax**(1 / rho) + i * (smin**(1 / rho) - smax**(1 / rho)))**rho
    return torch.cat([s, torch.zeros(1)])

@torch.no_grad()
def sample(D, x, n, method):
    sig = karras(n)
    D_prev = h_prev = None
    for i in range(n):
        s, s2 = sig[i], sig[i + 1]
        Dx = D(x, s.expand_as(x))
        if method == "ddim" or s2 == 0:                         # DDIM = Euler on the PF-ODE
            x = Dx + (s2 / s) * (x - Dx)
        elif method == "heun":                                  # EDM 2nd order: one extra evaluation
            d = (x - Dx) / s
            xe = x + (s2 - s) * d
            d2 = (xe - D(xe, s2.expand_as(x))) / s2
            x = x + (s2 - s) * (d + d2) / 2
        elif method == "dpm2m":                                 # DPM-Solver++(2M): reuse the last D
            h = torch.log(s / s2)
            Dd = Dx if D_prev is None else Dx + (Dx - D_prev) * h / (2 * h_prev)
            x = (s2 / s) * x + (1 - s2 / s) * Dd
            D_prev, h_prev = Dx, h
    return x

def mixture_quantiles(u, s):                                    # exact quantiles of p_sigma by bisection
    lo, hi = torch.full_like(u, -500.), torch.full_like(u, 500.)
    sd = torch.sqrt(SD**2 + s**2)
    for _ in range(100):
        mid = (lo + hi) / 2
        cdf = (W * torch.special.ndtr((mid[:, None] - MU) / sd)).sum(1)
        lo, hi = torch.where(cdf < u, mid, lo), torch.where(cdf < u, hi, mid)
    return (lo + hi) / 2

M = 1000
u = (torch.arange(M, dtype=torch.float64) + 0.5) / M
torch.set_default_dtype(torch.float32)
xT = mixture_quantiles(u, 80.0).float()                         # start noise, stratified
target = mixture_quantiles(u, 0.0).float()                      # where the exact ODE must land
w1 = lambda x: (x.sort().values - target).abs().mean().item()   # 1-D Wasserstein-1 distance

print(f"data std {SIG_D:.3f}, final training loss {loss.item():.3f}")
xs = torch.tensor([-2.0, 0.0, 1.5]); ss = torch.tensor([0.5, 0.5, 0.5])
print("D_net  ", [round(v, 3) for v in net(xs, ss).tolist()])
print("D_exact", [round(v, 3) for v in exact_D(xs, ss).tolist()])
for nfe in [5, 10, 20, 50]:
    row = []
    for D in (exact_D, net):
        row += [w1(sample(D, xT, nfe, "ddim")), w1(sample(D, xT, (nfe + 1) // 2, "heun")),
                w1(sample(D, xT, nfe, "dpm2m"))]
    print(f"NFE {nfe:2d} | exact D: ddim {row[0]:.3f} heun {row[1]:.3f} dpm2m {row[2]:.3f}"
          f" | network: ddim {row[3]:.3f} heun {row[4]:.3f} dpm2m {row[5]:.3f}")

On PyTorch 2.15 (CPU) this prints a data standard deviation of 1.695. The trained network is close to the exact denoiser: at \(x = -2, 0, 1.5\) and \(\sigma = 0.5\) it outputs −1.990, 0.288, 1.297 against the exact −1.999, 0.299, 1.246. The \(W_1\) errors:

NFEDDIM, exact DHeun, exact DDPM++(2M), exact DDDIM, networkHeun, networkDPM++(2M), network
50.6635.6950.3460.6683.3640.354
100.2911.0530.2280.2580.9900.318
200.1390.1670.0360.1640.2560.165
500.0540.0190.0050.1520.1640.161

With the exact denoiser the numbers match Figure 6 to three decimals: this is pure solver error. With the trained network, every solver ends near 0.15 to 0.16 from 20 NFE on: the network’s own error is now the floor, and a better solver cannot remove it. Other versions or hardware may print slightly different numbers.

Where this is used

SettingWhat is usedWhat matters
Text-to-image (Stable Diffusion, SDXL, SD3 and others)Swappable samplers: libraries such as diffusers ship DDIM, Euler, Heun, DPM-Solver++ and UniPC schedulers for the same trained modelSteps set latency directly; 20 to 30 steps of a multistep solver is common [Likely]
Video generationThe same ODE solvers on spatio-temporal latents; one NFE processes the whole clipNFE dominates cost, so solver order and distillation matter even more; see session 231
Image and video editingDDIM inversion: map a real image or each video frame to noise, change the prompt, sample againInversion is approximate; errors grow with guidance; per-frame inversion can flicker [Likely]
Inverse problems (inpainting, super-resolution, deblurring)Score SDE with the measurement added to the score, or a DDIM loop that projects onto the known pixelsUses the score as a learned prior without retraining
Likelihood and anomaly scoresProbability-flow ODE plus the change-of-variables formulaExact \(\log p(\mathbf{x})\) at the cost of an ODE solve with a trace estimator
Robot and video-agent action generationDiffusion policies sample action sequences with few DDIM-style steps [Likely]Control loops need low latency, so step count is a hard budget
Few-step generationThe ODE trajectory is the training target for distillation and consistency modelssession 230 builds on the ODE of this session

Common mistakes

  • Mixing parameterizations. Using an \(\varepsilon\)-prediction model in a sampler written for \(\mathbf{x}_0\)- or \(v\)-prediction (or the reverse) gives garbage. Convert with (226.6) and check one step by hand.
  • Forgetting the input scaling. In the \(\sigma\) form, the network of a DDPM model expects \(\mathbf{x}/\sqrt{1 + \sigma^2}\), not \(\mathbf{x}\). Samplers that work in \(\sigma\) form must scale before every call.
  • Picking the DDIM sub-sequence badly. Evenly spaced steps that skip the last training steps (for example, ending at \(t = 50\) instead of near 0) leave visible noise; a schedule that is too sparse at low noise blurs details.
  • Expecting determinism with \(\eta \gt 0\). Only \(\eta = 0\) is a fixed map from noise to image. Inversion, interpolation and caching of noises need \(\eta = 0\).
  • Comparing samplers by steps instead of NFE. A Heun step costs 2 NFE. Compare at equal NFE, as in Figure 6.
  • Using a high-order single-step solver at very few steps or with strong guidance. It can be worse than DDIM (Heun in Figure 6 below 10 NFE); DPM-Solver++ uses the data prediction and a multistep method for this reason.
  • Blaming the sampler for a network problem. If quality stops improving above 20 to 30 steps, the floor is the model, not the solver.

Interview questions

What is the score of a distribution, and why is it easier to model than the density?basic
The score is \(\nabla_{\mathbf{x}} \log p(\mathbf{x})\), a vector field that points toward higher probability. For \(p = e^{-E}/Z\), the score is \(-\nabla E\): the normalizing constant \(Z\), an intractable integral over all images, drops out. So a network that outputs a vector of the same shape as its input can represent a score with no constraint, while a density model must integrate to 1.
Why is plain score matching impractical for images, and how does denoising score matching fix it?core
The direct loss \(\lVert \mathbf{s}_\theta - \nabla \log p \rVert^2\) needs the unknown true score. Hyvärinen’s form removes it but needs the trace of the Jacobian of \(\mathbf{s}_\theta\), one backward pass per input dimension (12,288 for 64 × 64 × 3). Denoising score matching adds noise \(\tilde{\mathbf{x}} = \mathbf{x} + \sigma\varepsilon\) and regresses on the score of the noise kernel, \(-\varepsilon/\sigma\), which is known per example. Its minimizer is the score of the noisy distribution \(p_\sigma\). The loss is a plain regression, the same as DDPM’s noise prediction.
How are the noise prediction, the score and the denoised image related?core
In the form \(\tilde{\mathbf{x}} = \mathbf{x} + \sigma\varepsilon\): \(\mathbf{s} = -\varepsilon_\theta/\sigma\), and by Tweedie’s formula \(D = \E[\mathbf{x} \mid \tilde{\mathbf{x}}] = \tilde{\mathbf{x}} + \sigma^2 \mathbf{s} = \tilde{\mathbf{x}} - \sigma\varepsilon_\theta\). In DDPM form, \(\mathbf{s} = -\varepsilon_\theta/\sqrt{1 - \bar\alpha_t}\) and \(\hat{\mathbf{x}}_0 = (\mathbf{x}_t - \sqrt{1 - \bar\alpha_t}\varepsilon_\theta)/\sqrt{\bar\alpha_t}\). Example: data \(\mathcal{N}(0,1)\), \(\sigma = 2\), \(\tilde{x} = 3\): score −0.6, denoised 0.6, noise 1.2.
Why does NCSN use many noise levels instead of one?core
At low noise, the score is only learned near the data; in the empty space between modes the network has seen no samples, and Langevin dynamics cannot cross low-density regions in reasonable time, so samples get the wrong share per mode (they follow basins of attraction, not weights). Large noise fills the space and connects the modes, so the score is accurate everywhere and mixing is fast. Annealing from large to small noise gets both: correct global proportions from the high levels and sharp details from the low ones.
What is the probability-flow ODE, and how does it differ from the reverse SDE?core
For a forward SDE \(d\mathbf{x} = \mathbf{f}\,dt + g\,d\mathbf{w}\), the ODE \(d\mathbf{x}/dt = \mathbf{f} - \tfrac12 g^2 \nabla \log p_t\) has the same marginal distribution \(p_t\) at every time as the reverse SDE \(d\mathbf{x} = [\mathbf{f} - g^2 \nabla\log p_t]dt + g\,d\bar{\mathbf{w}}\). The ODE is deterministic (one image per noise), allows large steps with standard solvers, can be inverted, and gives exact likelihoods. The SDE has twice the score drift plus fresh noise; the noise can correct accumulated errors but needs more, smaller steps.
Why can DDIM sample with 50 steps from a model trained with 1000 steps, without retraining?core
The DDPM loss depends only on the marginals \(q(\mathbf{x}_t \mid \mathbf{x}_0)\), never on the step-by-step chain. DDIM defines non-Markovian forward processes with the same marginals, so the same network is the optimal denoiser for them too, and their reverse step can jump between any two time steps. With \(\eta = 0\), DDIM is exactly Euler’s method on the probability-flow ODE in the \(\sigma\) coordinates, and an ODE can be integrated with large steps.
What does \(\eta\) do in DDIM?basic
It sets the variance of fresh noise added at each step: \(\sigma_t = \eta \sqrt{(1 - \bar\alpha_{t'})/(1 - \bar\alpha_t)}\sqrt{1 - \bar\alpha_t/\bar\alpha_{t'}}\). \(\eta = 0\): deterministic DDIM (the ODE). \(\eta = 1\): the DDPM posterior variance, so with consecutive steps it is DDPM ancestral sampling (a discretized reverse SDE). In between, the samples have the same target distribution; only the paths and the step-size sensitivity change.
DDIM inversion of a real image does not give back exactly the same image. Why?advanced
Inversion runs the Euler step upward in noise, using the slope at the current (lower-noise) point, while sampling later uses the slope at the higher-noise point; the two differ by the local curvature of the ODE path, so each step adds an error of order step size squared. With few steps or strong classifier-free guidance (which makes the effective ODE less smooth and not exactly invertible), the errors add up. Fixes: more steps, higher-order or exactly invertible schemes, or optimizing the null-text embedding per step (null-text inversion).
Heun is second order, yet at 10 NFE it was worse than DDIM in Figure 6. Explain.advanced
Order describes how the error shrinks as the step size goes to zero; it says nothing about large steps. With a budget of 10 NFE, Heun gets only 5 steps, each spanning a large range of \(\sigma\) where the path bends, and its second slope is evaluated at a poor Euler guess. Multistep methods such as DPM-Solver++(2M) get second order for one NFE per step by reusing the previous denoiser output, so they keep the full number of steps. At 50 NFE, both second-order methods beat DDIM (0.019 and 0.0047 vs 0.054 in Figure 6).
A latent image model looks good at 50 DDIM steps. Product wants 10 steps. What do you do?advanced
First, without retraining: switch to a multistep solver (DPM-Solver++(2M) or UniPC), use a schedule dense at low noise (Karras \(\rho = 7\) or the solver’s recommended one), make sure the last step denoises fully, and compare at equal NFE, including guidance (each guided step may cost 2 NFE unless batched). Measure FID or human preference at 10, 15, 20 NFE. If 10 is still not good enough, train for few steps: progressive distillation, consistency distillation, or a rectified-flow fine-tune; these change the model, not just the sampler.
How many NFE does a 18-step EDM Heun sampler use? And 20 steps of DPM-Solver++(2M) with classifier-free guidance?basic
Heun: 2 per step except the last, \(2 \cdot 18 - 1 = 35\). DPM-Solver++(2M): 1 per step, 20 NFE without guidance; classifier-free guidance evaluates the network with and without the prompt, so 40 network evaluations (often run as one batch of 2, so about 20 batched calls).

Exercises

  1. The data is \(\mathcal{N}(1, 0.5^2)\). At noise level \(\sigma = 1\) and \(\tilde{x} = 2\), compute the score, the Tweedie denoised value and the predicted noise.
  2. Continue exercise 1: take one DDIM step to \(\sigma_{\text{next}} = 0.5\). Compare with the exact ODE solution (hint: for Gaussian data, \(x - \mu\) grows like \(\sqrt{s^2 + \sigma^2}\)).
  3. A DDPM point has \(\bar\alpha_t = 0.25\). What is \(\sigma\) in the \(\sigma\) form, and by what factor must you scale \(\mathbf{x}_t\)?
  4. One Langevin step (226.7) for the target \(\mathcal{N}(2, 1)\): \(x = 0\), \(\alpha = 0.1\), noise sample \(z = 0.5\). Where does \(x\) go?
  5. Show that \(\eta = 0\) in (226.12) gives a deterministic update, and that with \(\eta = 1\) the noise variance equals the DDPM posterior variance \(\tilde\beta_t = \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\beta_t\) when \(t' = t - 1\).
  6. In Figure 6, find the smallest budget at which DPM-Solver++(2M) reaches \(W_1 \lt 0.05\). How many NFE does DDIM need for the same error?
Answerscheck
1. \(p_\sigma = \mathcal{N}(1, 0.25 + 1) = \mathcal{N}(1, 1.25)\). Score \(= -(2 - 1)/1.25 = -0.8\). Denoised \(D = 2 + 1 \cdot (-0.8) = 1.2\) (check: \(1 + \tfrac{0.25}{1.25} \cdot 1 = 1.2\)). Noise \(\hat\varepsilon = (2 - 1.2)/1 = 0.8\). 2. DDIM: \(1.2 + 0.5 \cdot 0.8 = 1.6\). Exact: \(1 + 1 \cdot \sqrt{0.5/1.25} = 1.632\). Error 0.032. 3. \(\sigma = \sqrt{0.75/0.25} = \sqrt{3} = 1.732\); divide \(\mathbf{x}_t\) by \(\sqrt{0.25} = 0.5\) (multiply by 2). 4. Score at 0 is \(-(0 - 2)/1 = 2\). \(x = 0 + 0.05 \cdot 2 + \sqrt{0.1} \cdot 0.5 = 0.1 + 0.158 = 0.258\). 5. With \(\eta = 0\), \(\sigma_t = 0\), the \(\mathbf{z}\) term disappears and the update is a fixed function of \(\mathbf{x}_t\). With \(\eta = 1\), \(t' = t - 1\): \(\sigma_t^2 = \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\big(1 - \frac{\bar\alpha_t}{\bar\alpha_{t-1}}\big)\) and \(\bar\alpha_t/\bar\alpha_{t-1} = 1 - \beta_t\), so \(\sigma_t^2 = \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\beta_t = \tilde\beta_t\). 6. Read it from the figure: DPM-Solver++(2M) needs 18 NFE; DDIM needs 54 NFE, so the second-order multistep solver is about 3 times cheaper at this quality.

Key takeaways

  • The score \(\nabla \log p\) (226.1) points toward more likely data and does not need the normalizing constant.
  • Denoising score matching (226.4) learns the score of noisy data by plain regression; it is the DDPM noise loss. Score, noise and clean-image predictions convert into each other with Tweedie’s formula (226.5, 226.6).
  • Langevin dynamics (226.7) samples with the score, but only across many noise levels (NCSN, annealing) does it get the mode proportions right.
  • Diffusion is an SDE (226.8); its reverse (226.9) and the deterministic probability-flow ODE (226.10, 226.11) have the same marginals.
  • DDIM (226.12) reuses a DDPM network with any sub-sequence of steps; with \(\eta = 0\) it is Euler’s method on the ODE (226.13).
  • Better solvers (Heun 226.14, DPM-Solver++(2M) 226.16, UniPC) plus a schedule dense at low noise (226.15) give second-order error decay; compare at equal NFE, and remember the network’s own error is the floor.

Key formulas

\[ \mathbf{s} = \nabla_{\mathbf{x}} \log p_\sigma = \frac{D - \mathbf{x}}{\sigma^2} = -\frac{\varepsilon}{\sigma}, \qquad J_{\text{DSM}} = \tfrac{1}{2\sigma^2}\,\E\lVert \sigma\mathbf{s}_\theta(\mathbf{x} + \sigma\varepsilon, \sigma) + \varepsilon \rVert^2 \]
\[ \frac{d\mathbf{x}}{dt} = \mathbf{f} - \tfrac12 g^2 \nabla \log p_t, \qquad \frac{d\mathbf{x}}{d\sigma} = \frac{\mathbf{x} - D(\mathbf{x}; \sigma)}{\sigma}, \qquad \text{DDIM: } \mathbf{x}_{\text{next}} = D + \frac{\sigma_{\text{next}}}{\sigma}(\mathbf{x} - D) \]

References for this session

  1. Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
  2. Song et al. (2020). Denoising Diffusion Implicit Models
  3. Song & Ermon (2019). Generative Modeling by Estimating Gradients of the Data Distribution
  4. Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
  5. Song & Ermon (2019). Generative Modeling by Estimating Gradients of the Data Distribution
  6. Song et al. (2020). Denoising Diffusion Implicit Models
  7. Ho et al. (2020). Denoising Diffusion Probabilistic Models
  8. Hyvärinen (2005). Estimation of Non-Normalized Statistical Models by Score Matching
  9. Vincent (2011). A Connection Between Score Matching and Denoising Autoencoders
  10. Anderson (1982). Reverse-Time Diffusion Equation Models
  11. Efron (2011). Tweedie's Formula and Selection Bias
  12. Karras, Aittala, Aila & Laine (2022). Elucidating the Design Space of Diffusion-Based Generative Models
  13. Lu, Zhou, Bao, Chen, Li & Zhu (2022). DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
  14. Lu, Zhou, Bao, Chen, Li & Zhu (2022). DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
  15. Zhao, Bai, Rao, Zhou & Lu (2023). UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models
  16. Song & Ermon (2020). Improved Techniques for Training Score-Based Generative Models
  17. Salimans & Ho (2022). Progressive Distillation for Fast Sampling of Diffusion Models
  18. Song et al. (2023). Consistency Models
  19. Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow