Diffusion 2: score matching, DDIM and fast samplers
A diffusion model secretly learns the score: the direction in which an image becomes more likely. This session shows why removing noise and learning the score are the same task, how diffusion becomes a differential equation (a random one and a deterministic one), why DDIM can sample in 10 to 50 steps instead of 1000 with the same trained network, and how modern solvers such as Heun and DPM-Solver++ trade quality against the number of network calls.
What you will learn
- What the score of a distribution is, and why it does not need the normalizing constant
- Why score matching on clean data is hard, and how denoising score matching turns it into plain noise prediction (Tweedie's formula)
- Langevin dynamics, why one noise level fails, and how annealing over many noise levels (NCSN) fixes it
- The forward SDE, the reverse-time SDE and the probability-flow ODE, and why they all share the same marginals
- DDIM: the update rule, the η knob between DDIM and DDPM, why it is Euler's method on the ODE, and DDIM inversion
- Higher-order and multistep solvers (Heun, DPM-Solver++, UniPC), the Karras noise schedule, and how to read error versus NFE
- How to choose a sampler and a step count in practice, and where the network error, not the solver, sets the limit
Why this matters
In session 225 you trained a network to predict the noise in a noisy image, and you sampled by undoing the noise one small step at a time, about 1000 times. This session answers two questions. What has the network really learned? And why can we sample with the same network in 20 steps instead of 1000?
The answer to the first question is the score: for every noisy image, the direction in which it becomes more likely. A noise predictor is a score estimator in disguise. Once you see this, diffusion turns into a differential equation that runs from noise to data, and sampling becomes a problem of solving that equation numerically. Then the second question has a clean answer: a better numerical solver needs fewer network calls. DDIM is the simplest such solver (Euler's method); DPM-Solver++, UniPC and the Heun sampler of Karras et al. are better ones.
The cost is real. Each sampling step is one full forward pass of a large U-Net or diffusion transformer. For a video model, one step processes every frame of the clip. Going from 1000 steps to 25 is a 40 times cheaper sample with no retraining. This is why every image and video product exposes a “sampler” and a “steps” setting, and why the choice matters for latency budgets.
The plan: the score (what it is), score matching (how to learn it), Langevin dynamics and many noise levels (how to sample with it, the NCSN view), the SDE and ODE view (the unified picture), DDIM, and finally modern solvers and the step-count trade-off. The next session, session 227, adds a text prompt to the score.
The score: which way is “more likely”
Let \(p(\mathbf{x})\) be a probability density over images (or over any vectors). Its score is the gradient of its logarithm with respect to the input:
- Read the symbols
- \(\mathbf{s}(\mathbf{x})\)
- the score: a vector of the same shape as \(\mathbf{x}\) (for an image, one number per pixel and channel)
- \(E(\mathbf{x})\)
- an energy: any function that is low where data is common; \(e^{-E}\) is the unnormalized density
- \(Z\)
- the normalizing constant \(\int e^{-E(\mathbf{x})}\,d\mathbf{x}\), which makes the density integrate to 1; for images it is an integral over millions of dimensions and cannot be computed
Two facts make the score useful. First, it points uphill in probability: a small move \(\mathbf{x} + \delta\,\mathbf{s}(\mathbf{x})\) makes \(\mathbf{x}\) more likely. Second, \(\log Z\) is a constant, so its gradient is zero and the score never needs \(Z\). A model that outputs a score can be any neural network with an output of the same shape as its input; nothing has to integrate to 1. This is the main reason the score is easier to model than the density itself.
The simplest example: for a Gaussian \(\mathcal{N}(\mu, v)\) in one dimension, \(\log p(x) = -(x-\mu)^2/(2v) + \text{const}\), so \(s(x) = -(x-\mu)/v\). The score points toward the mean, and it is stronger far from the mean and for narrow Gaussians. For a mixture of Gaussians the score is a weighted average of the scores of the components, weighted by how likely each component is to have produced \(\mathbf{x}\).
Diffusion never uses the score of the clean data alone. It uses the score of the data after Gaussian noise of level \(\sigma\) has been added, written \(p_\sigma\). Figure 1 shows this for a 2-D mixture of three Gaussians. With little noise, the score is huge near each mode and close to zero (and unreliable) in the empty space between them. With a lot of noise, \(p_\sigma\) is one wide blob, and the score points smoothly toward the data from everywhere.
How to read Figure 1
Click anywhere to place the orange probe. The readout gives the exact score there and the green point \(\mathbf{x} + \sigma^2\,\mathbf{s}(\mathbf{x})\), which is the best guess of the clean point (formula 226.5 below). Move the slider down to \(\sigma = 0.05\): between the modes the arrows still point somewhere, but the density there is almost zero, so a network trained on data would almost never see these points and could not learn these arrows. Move it up to \(\sigma = 2\) or more: one smooth blob, and every arrow points toward the center of mass. A sampler that starts at high noise and follows the score down through the noise levels uses the easy, smooth field first and the sharp field only at the end.
Learning the score: score matching and denoising
We want a network \(\mathbf{s}_\theta(\mathbf{x})\) with \(\mathbf{s}_\theta \approx \nabla \log p\). The natural loss is the squared distance between the two, averaged over the data. It is called the Fisher divergence or explicit score matching:
But we do not know \(\nabla \log p\); we only have samples. Hyvärinen (2005) showed with integration by parts that (226.2) equals, up to a constant that does not depend on \(\theta\),
This needs no true score, but the trace of the Jacobian \(\nabla_{\mathbf{x}} \mathbf{s}_\theta\) costs one backward pass per input dimension. For a 64 × 64 × 3 image that is 12,288 backward passes per training example. It does not scale.
Denoising score matching
Vincent (2011) found the way out. Add Gaussian noise to a data point, \(\tilde{\mathbf{x}} = \mathbf{x} + \sigma\varepsilon\) with \(\varepsilon \sim \mathcal{N}(\mathbf{0}, I)\), and match the score of the noisy distribution \(p_\sigma\). For one fixed clean \(\mathbf{x}\), the noisy point has the simple density \(q(\tilde{\mathbf{x}} \mid \mathbf{x}) = \mathcal{N}(\tilde{\mathbf{x}}; \mathbf{x}, \sigma^2 I)\), whose score we know: \(-(\tilde{\mathbf{x}} - \mathbf{x})/\sigma^2 = -\varepsilon/\sigma\). Vincent proved that regressing on this known, per-example target gives the same minimizer as regressing on the unknown \(\nabla \log p_\sigma\):
- Read the symbols
- \(\tilde{\mathbf{x}} = \mathbf{x} + \sigma\varepsilon\)
- a training image with noise of level \(\sigma\) added
- \(-\varepsilon/\sigma\)
- the target: the score of the noise distribution around this one clean image; it points from \(\tilde{\mathbf{x}}\) back to \(\mathbf{x}\)
- \(\mathbf{s}_\theta(\tilde{\mathbf{x}}, \sigma)\)
- the network: it sees only the noisy image and \(\sigma\), never \(\mathbf{x}\), so the best it can do is average the target over all clean images that could have produced \(\tilde{\mathbf{x}}\); that average is exactly \(\nabla \log p_\sigma(\tilde{\mathbf{x}})\)
Look at the right-hand side: \(\sigma\,\mathbf{s}_\theta\) must predict \(-\varepsilon\). Define \(\varepsilon_\theta = -\sigma\,\mathbf{s}_\theta\) and the loss becomes \(\lVert \varepsilon_\theta(\tilde{\mathbf{x}}, \sigma) - \varepsilon \rVert^2\): the DDPM loss of session 225 is denoising score matching, with a weight of \(\sigma^2\) per noise level. Ho et al. (2020) made this link explicit. [Certain]
Tweedie’s formula: score = denoiser
There is a third view of the same object. The best estimate of the clean image given the noisy one, in the squared-error sense, is the posterior mean \(\E[\mathbf{x} \mid \tilde{\mathbf{x}}]\). Tweedie’s formula (Robbins 1956, popularized by Efron 2011) links it to the score:
So the score, the noise and the clean image are three views of one network output. The conversions, in the plain noise-added form \(\tilde{\mathbf{x}} = \mathbf{x} + \sigma\varepsilon\) and in the DDPM form \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon\), are:
- Read the symbols
- \(D_\theta\)
- the denoiser: the network’s guess of the clean image (also written \(\hat{\mathbf{x}}_0\))
- \(\varepsilon_\theta\)
- the network’s guess of the noise that was added
- \(\bar\alpha_t\)
- the DDPM signal level at step \(t\); the two forms are the same thing with \(\sigma = \sqrt{(1 - \bar\alpha_t)/\bar\alpha_t}\) and \(\tilde{\mathbf{x}} = \mathbf{x}_t / \sqrt{\bar\alpha_t}\)
The last point is worth remembering: a DDPM noisy image is just a scaled version of “clean image plus noise of level \(\sigma\)”. For the linear schedule of DDPM (\(\beta\) from \(10^{-4}\) to 0.02 over 1000 steps), \(\bar\alpha_{1000} = 4.04 \times 10^{-5}\), which is \(\sigma \approx 157\), and \(\bar\alpha_1 = 0.9999\), which is \(\sigma = 0.01\). We will use the \(\sigma\) form for the math, because it is the simplest; every formula maps back to the \(\bar\alpha\) form by this rescaling.
Let the data be \(\mathcal{N}(0, 1)\) in one dimension. Adding noise of level \(\sigma = 2\) gives \(p_\sigma = \mathcal{N}(0,\ 1 + 4) = \mathcal{N}(0, 5)\). At \(\tilde{x} = 3\):
Score: \(s = -3/5 = -0.6\). Tweedie (226.5): \(D = 3 + 4 \cdot (-0.6) = 0.6\). Check against the direct posterior mean of a Gaussian, \(\tfrac{1}{1 + 4} \cdot 3 = 0.6\). Predicted noise: \(\varepsilon = (\tilde{x} - D)/\sigma = (3 - 0.6)/2 = 1.2\), and indeed \(-\varepsilon/\sigma = -0.6 = s\).
In DDPM form this point has \(\bar\alpha = 1/(1 + \sigma^2) = 0.2\) and \(x_t = 3 \cdot \sqrt{0.2} = 1.342\). The three numbers \(s = -0.6\), \(D = 0.6\), \(\varepsilon = 1.2\) are what a perfect network would output, in its three possible parameterizations.
Sampling with the score: Langevin dynamics and many noise levels
Suppose we have the score. How do we turn it into samples? Langevin dynamics is a random walk that climbs the score and adds noise at each step:
Here \(\alpha \gt 0\) is the step size. The drift term alone would climb to the nearest peak and stop (that is gradient ascent on \(\log p\)). The noise term keeps the walker moving, and the two balance exactly so that, as \(\alpha \to 0\) and the number of steps grows, the walker’s position is distributed as \(p\). [Certain]
In practice plain Langevin dynamics on the clean score fails, for two reasons that Song and Ermon (2019) pointed out:
- The score is wrong where there is no data. The network is trained on samples, and the empty space between modes has almost none. It is exactly the space a walker must cross.
- Mixing between modes is extremely slow. Even with the true score, a walker in one mode must cross a region of near-zero probability to reach another. In finite time, walkers stay in whichever mode they fell into first, so the share of samples per mode follows the size of each mode’s basin of attraction, not the true weights.
The fix of their noise-conditional score network (NCSN) is to train one network for many noise levels \(\sigma_1 \gt \sigma_2 \gt \dots \gt \sigma_L\) at once, with loss (226.4) summed over levels (weighted by \(\sigma^2\)), and to sample with annealed Langevin dynamics: run Langevin at the largest noise level, where \(p_\sigma\) is one smooth blob and all modes are connected, then use the result as the start for the next, smaller noise level, and so on.
- input: score network \(\mathbf{s}_\theta(\mathbf{x}, \sigma)\); noise levels \(\sigma_1 \gt \dots \gt \sigma_L\) (geometric); steps per level \(T\); base step \(\epsilon\)
- \(\mathbf{x} \leftarrow\) a random start (uniform noise or \(\mathcal{N}(\mathbf{0}, \sigma_1^2 I)\))
- for \(i = 1, \dots, L\): // from coarse to fine
- \(\alpha_i \leftarrow \epsilon\, \sigma_i^2 / \sigma_L^2\) // step size scales with the variance of the level
- repeat \(T\) times: \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\); \(\ \mathbf{x} \leftarrow \mathbf{x} + \tfrac{\alpha_i}{2}\, \mathbf{s}_\theta(\mathbf{x}, \sigma_i) + \sqrt{\alpha_i}\, \mathbf{z}\) // formula (226.7)
- return \(\mathbf{x}\) // later work adds one final denoising step with Tweedie (226.5)
The step size rule \(\alpha_i \propto \sigma_i^2\) keeps the “signal to noise ratio” of each update the same at every level. Figure 2 runs both versions on the mixture of Figure 1, with the exact score.
How to read Figure 2
Step through the stages. In the plain run, every particle slides into the nearest mode within the first stage and never leaves, so the shares end near the shares of the start box closest to each mode (0.30 / 0.29 / 0.41), far from the true 0.6 / 0.25 / 0.15. In the annealed run, the first stages pull all particles into one wide blob near the weighted center; as \(\sigma\) shrinks, the blob splits, and each part gets a share close to the true weight (0.55 / 0.30 / 0.16). The remaining gap comes from 500 particles and the short run; it shrinks with more steps per level.
Diffusion as a differential equation
DDPM has 1000 discrete steps; NCSN has 10 or so noise levels. Song et al. (2021) took the limit of infinitely many tiny steps. The noising process then becomes a stochastic differential equation (SDE): a differential equation with a random term.
- Read the symbols
- \(t \in [0, T]\)
- continuous time; \(t = 0\) is clean data, \(t = T\) is (almost) pure noise
- \(\mathbf{f}(\mathbf{x}, t)\)
- the drift: a deterministic push per unit time
- \(g(t)\, d\mathbf{w}\)
- the diffusion term: \(d\mathbf{w}\) is a tiny Gaussian step with variance \(dt\) (Brownian motion), and \(g(t)\) sets how much noise is added per unit time
- VP SDE
- \(\mathbf{f} = -\tfrac12 \beta(t)\,\mathbf{x}\), \(g = \sqrt{\beta(t)}\): the continuous version of DDPM; the signal shrinks while noise is added, so the variance stays bounded (“variance preserving”)
- VE SDE
- \(\mathbf{f} = \mathbf{0}\), \(g = \sqrt{d\sigma^2(t)/dt}\): the continuous version of NCSN; only noise is added, so \(\mathbf{x}_t = \mathbf{x}_0 + \sigma(t)\varepsilon\) and the variance explodes (“variance exploding”)
A classic result of Anderson (1982) says that this process can be run backward in time, and the reverse is again an SDE. Its drift needs only one thing we do not already know: the score of the noisy data at each time.
Here time runs from \(T\) down to 0 (so \(dt\) is negative) and \(d\bar{\mathbf{w}}\) is Brownian motion in reverse time. Replace the score by the network \(\mathbf{s}_\theta(\mathbf{x}, t)\), start from noise, integrate down to \(t = 0\), and you have a sample. DDPM’s sampler is one discretization of this equation; adding Langevin “corrector” steps after each step gives the predictor–corrector sampler of the same paper.
The probability-flow ODE
The most useful result in the paper is this: there is a deterministic process, an ordinary differential equation (ODE) with no random term, whose solutions have exactly the same distribution \(p_t\) at every time \(t\) as the SDE:
Note the factor \(\tfrac12\) where the reverse SDE has 1, and the missing noise term. Intuitively, the reverse SDE moves twice as far up the score and then adds noise that spreads the samples back out by half that amount on average; the ODE takes only the net effect. For the VE form with \(\sigma\) as the time variable (Karras et al., 2022), and using Tweedie (226.5), the ODE becomes very simple:
- Read the symbols
- \(\mathbf{x} - D(\mathbf{x}; \sigma)\)
- the part of \(\mathbf{x}\) the denoiser thinks is noise
- \((\mathbf{x} - D)/\sigma\)
- the predicted noise \(\hat\varepsilon\): the ODE says “as \(\sigma\) goes down by \(d\sigma\), remove \(\hat\varepsilon\, d\sigma\) of noise”
Why the ODE matters so much:
- Big steps. An ODE has smooth solutions, and numerical analysis offers many accurate solvers for it. An SDE solver must also simulate the noise, which limits how large the steps can be.
- A fixed map from noise to images. Each starting noise gives one image. This makes inversion possible: run the ODE forward from a real image to get its noise, then edit and run it back.
- Exact likelihoods. With the change-of-variables formula of continuous normalizing flows, the ODE gives \(\log p(\mathbf{x})\); Song et al. report 2.99 bits per dimension on CIFAR-10. [Certain]
Figure 3 shows both on a 1-D mixture of three Gaussians, with the exact denoiser. The knob \(\eta\) blends between them (this is the DDIM \(\eta\), next section): \(\eta = 0\) is the ODE, \(\eta = 1\) is the DDPM-like reverse SDE.
How to read Figure 3
At \(\eta = 0\) the paths are smooth and never cross (the readout counts 0 swapped pairs): in one dimension the ODE keeps the order of its starting points, so it maps the \(k\)-th smallest noise to the \(k\)-th smallest sample. Raise \(\eta\): the paths become jittery and cross, and “New noise” changes the result. Yet the histogram on the right stays close to the true density at every \(\eta\), and the share per mode stays close to 0.35 / 0.45 / 0.2. Same marginals, different paths: that is the whole content of (226.9) versus (226.10). Notice also that the paths decide their mode in the middle, around \(\sigma\) between 0.3 and 3.
DDIM: deterministic sampling with far fewer steps
DDIM (Song, Meng and Ermon, 2020) was found from a different angle, before the ODE view was common. The DDPM training loss only uses the marginals \(q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}(\sqrt{\bar\alpha_t}\,\mathbf{x}_0,\ (1 - \bar\alpha_t) I)\). It never uses the step-by-step chain \(q(\mathbf{x}_t \mid \mathbf{x}_{t-1})\). So any other forward process with the same marginals has the same training loss and can reuse the same trained network. DDIM picks a family of non-Markovian forward processes (each step may also look at \(\mathbf{x}_0\)) with these marginals, and derives their reverse step:
- Read the symbols
- \(t' \lt t\)
- the next time step; it does not need to be \(t - 1\): any decreasing sub-sequence of the 1000 training steps works
- \(\hat{\mathbf{x}}_0\)
- the predicted clean image from (226.6)
- \(\sqrt{1 - \bar\alpha_{t'} - \sigma_t^2}\; \varepsilon_\theta\)
- the “direction pointing to \(\mathbf{x}_t\)”: the predicted noise, re-used at the lower noise level
- \(\sigma_t \mathbf{z}\)
- fresh random noise; \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\)
- \(\eta\)
- \(\eta = 0\): deterministic DDIM. \(\eta = 1\): the DDPM posterior variance, so with \(t' = t - 1\) this is DDPM ancestral sampling
Read (226.12) as two moves. First, jump to the network’s guess of the clean image. Then, put back exactly the noise that belongs at level \(t'\), using the same noise direction the network just predicted (plus fresh noise if \(\eta \gt 0\)). The paper reports samples of high quality 10 to 50 times faster in wall-clock time than DDPM. [Certain]
DDIM is Euler’s method on the probability-flow ODE
Divide (226.12) with \(\eta = 0\) by \(\sqrt{\bar\alpha_{t'}}\) and switch to the \(\sigma\) form, \(\mathbf{x} = \mathbf{x}_t / \sqrt{\bar\alpha_t}\) and \(\sigma = \sqrt{(1 - \bar\alpha_t)/\bar\alpha_t}\). With \(D = \hat{\mathbf{x}}_0\) the step becomes:
The right-hand form is one step of Euler’s method on the ODE (226.11): current point plus step size times current slope. [Certain] So DDIM is the simplest possible ODE solver, applied in the coordinates where the ODE is simplest. The left-hand form has a nice picture, shown in Figure 4: the next point lies on the straight line from the current point to the current guess of the clean image, at the fraction \(\sigma_{\text{next}}/\sigma\) of the way back from the guess.
Data \(\mathcal{N}(0, 1)\), start at \(x = 3\) with \(\sigma = 2\), step to \(\sigma_{\text{next}} = 1\). From the previous worked example, \(D = 0.6\) and \(\hat\varepsilon = 1.2\).
DDIM (226.13): \(x_{\text{next}} = 0.6 + \tfrac12 (3 - 0.6) = 1.8\). In DDPM form: \(\bar\alpha\) goes from 0.2 to 0.5, and \(\sqrt{0.5}\cdot 0.6 + \sqrt{0.5}\cdot 1.2 = 1.273 = 1.8/\sqrt{2}\). The two forms agree.
Exact ODE: for Gaussian data \(D = x/(1 + \sigma^2)\), so (226.11) reads \(dx/d\sigma = \sigma x/(1 + \sigma^2)\), whose solution is \(x \propto \sqrt{1 + \sigma^2}\). From \(x = 3\) at \(\sigma = 2\): \(x(1) = 3\sqrt{2/5} = 1.897\). DDIM is off by 0.097.
Heun (226.14 below): the Euler guess 1.8 gives a second slope at \(\sigma = 1\): \(D = 1.8/2 = 0.9\), slope \((1.8 - 0.9)/1 = 0.9\). Average slope \((1.2 + 0.9)/2 = 1.05\), so \(x_{\text{next}} = 3 - 1 \cdot 1.05 = 1.95\). Error 0.053, about half, for one extra network call.
- input: noise network \(\varepsilon_\theta\); schedule \(\bar\alpha_1, \dots, \bar\alpha_{1000}\); a decreasing sub-sequence \(\tau_S \gt \dots \gt \tau_1\) of training steps (for example every 50th step for \(S = 20\)); \(\eta\)
- \(\mathbf{x} \sim \mathcal{N}(\mathbf{0}, I)\) // at time \(\tau_S\)
- for \(i = S, \dots, 1\): \(t \leftarrow \tau_i\); \(t' \leftarrow \tau_{i-1}\) (with \(\bar\alpha_{\tau_0} := 1\))
- \(\hat\varepsilon \leftarrow \varepsilon_\theta(\mathbf{x}, t)\); \(\ \hat{\mathbf{x}}_0 \leftarrow (\mathbf{x} - \sqrt{1 - \bar\alpha_t}\,\hat\varepsilon)/\sqrt{\bar\alpha_t}\) // formula (226.6); optionally clip to the data range
- \(\sigma_t \leftarrow\) formula (226.12) with \(\eta\); \(\ \mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\) if \(\eta \gt 0\)
- \(\mathbf{x} \leftarrow \sqrt{\bar\alpha_{t'}}\,\hat{\mathbf{x}}_0 + \sqrt{1 - \bar\alpha_{t'} - \sigma_t^2}\,\hat\varepsilon + \sigma_t \mathbf{z}\) // \(S\) network calls in total
- return \(\mathbf{x}\)
Figure 5 puts the two samplers side by side on a picture. The setup is the one of Figures 4 and 5 in session 225: 96 × 96 color pictures with values in \([-1, 1]\), the linear DDPM schedule, and in place of a trained network the ideal prediction \(\varepsilon^\star = \E[\varepsilon \mid \mathbf{x}_t]\) of a perfectly trained network, for a training set of one or of three pictures. Both samplers start from the same noise image \(\mathbf{x}_{1000}\). DDPM runs 1000 steps of its update with fresh noise \(\mathbf{z}\) at every step (\(\eta = 1\)). DDIM runs \(S\) steps of (226.12) with \(\eta = 0\) at the steps \(\tau_i = i \cdot 1000/S\), so it also starts at \(t = 1000\). The algorithm box is Algorithm 2 with \(\eta = 0\), split into one line per formula.
- \(\mathbf{x}_{1000} \sim \mathcal{N}(\mathbf{0}, I)\)
- for \(i = S, \dots, 1\): \(t \leftarrow \tau_i,\ t' \leftarrow \tau_{i-1}\)
- \(\hat\varepsilon \leftarrow \varepsilon_\theta(\mathbf{x}_t, t)\) one network call
- \(\hat{\mathbf{x}}_0 \leftarrow \big(\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\hat\varepsilon\big) / \sqrt{\bar\alpha_t}\) (226.6)
- \(\mathbf{x}_{t'} \leftarrow\) \(\sqrt{\bar\alpha_{t'}}\,\hat{\mathbf{x}}_0 + \sqrt{1-\bar\alpha_{t'}}\,\hat\varepsilon\) (226.12), \(\eta = 0\)
- return \(\mathbf{x}_0\)
How to read Figure 5
Press Play, or Next line to go one formula at a time. With one picture, the ideal \(\hat\varepsilon\) is exactly the noise inside \(\mathbf{x}_t\), and line 5 puts that same noise back at the lower level, so \(\hat\varepsilon\) never changes. DDIM therefore walks back along the forward path \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\hat\varepsilon\) (the top row of Figure 4 in session 225, in reverse), and its \(\hat{\mathbf{x}}_0\) panel shows the picture from the first step. In the \(\sigma\) form this path is a straight line, so Euler’s method has no error, and \(S\) = 50, 20 and 10 all end exactly on the picture (RMS distance 0.0000) after 50, 20 or 10 network calls. DDPM ends on the same picture after 1000 calls, but its noise pattern changes at every step because each step adds fresh noise \(\mathbf{z}\). Switch to three pictures. DDIM’s \(\hat{\mathbf{x}}_0\) now starts as a blend of all three, and the weights move smoothly: with \(S = 50\) they go from 0.37, 0.31, 0.32 at \(t = 1000\) to 0.94 for sunny hills at \(t = 800\) and 1.000 by \(t = 760\). \(S = 20\) and \(S = 10\) follow almost the same path with fewer, larger steps, and all three end on sunny hills. DDPM, from the same starting noise, ends on the night picture: its fresh noise changed the decision, which is made in roughly the first 200 steps (this is the same run as Figure 5 of session 225). A deterministic sampler is a fixed map from noise to picture; here the same \(\mathbf{x}_{1000}\) gives the same picture at every \(S\), and the DDIM paper reports this consistency for real models too [Likely]. It is what makes DDIM inversion (next) possible. A stochastic sampler fixes only the distribution of pictures, not which one you get. With a real network, fewer steps cost quality because the ODE path curves (Figure 6); a perfect model of a tiny training set hides that cost, because its path bends only during the short time when the decision is made.
DDIM inversion
Because DDIM with \(\eta = 0\) is a deterministic ODE solver, it can also run in the other direction: from a real image, step up in noise with the same rule, and you get a noise \(\mathbf{x}_T\) that DDIM maps back to (almost) the same image. Editing methods then change the prompt or the noise and decode again. The inversion is only approximate: each Euler step uses the slope at the start of the step, which is slightly different going up than going down, and with classifier-free guidance the error grows. [Likely] Methods such as null-text inversion were built to fix this.
Modern solvers and the step-count trade-off
Once sampling is “solve the ODE (226.11) from \(\sigma_{\max}\) to 0”, the cost is counted in NFE, the number of function (network) evaluations. Everything else is cheap. Three ideas push quality per NFE up.
1. Higher-order steps (Heun)
Euler’s error per step is proportional to the step size squared, so its total error is proportional to the step size: halving the step size halves the error. Heun’s method takes an Euler step, evaluates the slope again at the end, and uses the average of the two slopes:
where \(\tilde{\mathbf{d}}\) is the slope at \((\tilde{\mathbf{x}}, \sigma_{i+1})\). Its total error falls with the square of the step size, but each step costs 2 NFE. Karras et al. (2022, “EDM”) use it with no extra step at the very end (the slope at \(\sigma = 0\) is undefined), so \(N\) steps cost \(2N - 1\) NFE; their CIFAR-10 result of FID 1.79 used 35 NFE [Certain], which is 18 steps. [Likely]
2. Where to put the steps: the noise schedule for sampling
The ODE path is nearly straight at high noise and bends strongly at low noise (Figure 3: the paths are flat at high noise and bend between \(\sigma \approx 3\) and 0.3). So steps should be dense at small \(\sigma\). EDM proposes:
with \(\sigma_{\max} = 80\), \(\sigma_{\min} = 0.002\), \(\rho = 7\). For \(N = 10\) this gives 80, 42.4, 21.1, 9.72, 4.07, 1.50, 0.470, 0.117, 0.0204, 0.002, then 0: half of the steps are below \(\sigma = 1.5\).
3. Exact linear part and multistep reuse (DPM-Solver, DPM-Solver++, UniPC)
Lu et al. (2022) noticed that the diffusion ODE has a linear part that can be integrated exactly, leaving only the network term to approximate. Written with \(\lambda = -\log\sigma\) and the data prediction \(D\), the first-order DPM-Solver++ step is exactly DDIM (226.13). The second-order multistep version, DPM-Solver++(2M), reuses the previous step’s \(D\) to estimate how \(D\) changes, so it gets second-order accuracy for one NFE per step:
- Read the symbols
- \(D_i\)
- the denoiser output at step \(i\), \(D(\mathbf{x}_i; \sigma_i)\); \(D_{i-1}\) is kept from the previous step, so no extra network call
- \(h_i\)
- the step size in log-noise; \(h_i / h_{i-1}\) corrects for unequal steps
- \(\hat D_i\)
- a linear extrapolation of the clean-image guess toward the middle of the step; with \(\hat D_i = D_i\) the update is DDIM again
The DPM-Solver paper reports FID 4.70 at 10 NFE and 2.87 at 20 NFE on CIFAR-10 [Certain]; DPM-Solver++ targets guided sampling, where it reports high quality in 15 to 20 steps, while DDIM “generally needs 100 to 250 steps” in that setting [Certain]. UniPC adds a corrector that reuses the new network output, and reports FID 3.87 on CIFAR-10 with 10 NFE. [Certain]
- input: denoiser \(D_\theta(\mathbf{x}; \sigma)\); number of steps \(N\); schedule (226.15) \(\sigma_0 \gt \dots \gt \sigma_{N-1} \gt \sigma_N = 0\)
- \(\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \sigma_0^2 I)\); \(\ D_{\text{prev}} \leftarrow\) none
- for \(i = 0, \dots, N - 1\): \(\ D_i \leftarrow D_\theta(\mathbf{x}; \sigma_i)\) // 1 NFE
- if \(\sigma_{i+1} = 0\): \(\mathbf{x} \leftarrow D_i\); break // last step: just denoise
- DDIM: \(\mathbf{x} \leftarrow D_i + (\sigma_{i+1}/\sigma_i)(\mathbf{x} - D_i)\) // (226.13)
- Heun: Euler guess \(\tilde{\mathbf{x}}\), one more call \(D_\theta(\tilde{\mathbf{x}}; \sigma_{i+1})\), average slopes // (226.14), +1 NFE
- DPM-Solver++(2M): extrapolate \(\hat D_i\) from \(D_i, D_{\text{prev}}\); update with (226.16); \(D_{\text{prev}} \leftarrow D_i\)
- return \(\mathbf{x}\) // for a latent model, decode with the VAE afterwards
Figure 6 runs all of these on the 1-D mixture of Figure 3 with the exact denoiser, so the only error left is the solver’s. Because the exact ODE in 1-D maps the \(k\)-th smallest starting noise to the \(k\)-th smallest data quantile, we can start from 1,000 evenly spaced quantiles of \(p_{80}\) and measure the error as the average distance between the sorted outputs and the true data quantiles (the 1-D Wasserstein-1 distance \(W_1\)).
How to read Figure 6
Move the budget. At 10 NFE: DDIM 0.29, Heun 1.05 (only 5 steps, too coarse for a second-order method), DPM-Solver++(2M) 0.23. At 50 NFE: DDIM 0.054, Heun 0.019, DPM-Solver++(2M) 0.0047. On the log-log plot, the DDIM line has slope about −1 (error \(\propto 1/\text{NFE}\)) and the two second-order methods have slope about −2: from 31 to 51 NFE, DDIM’s error falls by 1.67 times, DPM-Solver++(2M)’s by 2.9 times. Below about 8 NFE nothing is good, and Heun is the worst: its extra evaluations buy nothing when each step is huge. Look at the histogram at low NFE: the error shows up as modes that are too wide and too close together, the 1-D version of blurry, washed-out images.
Choosing a sampler in practice
- 20 to 30 steps of a second-order multistep solver (DPM-Solver++(2M), UniPC) is the usual default for latent image models; DDIM at 50 steps is the classic safe choice. [Likely]
- Stochastic samplers (\(\eta \gt 0\), ancestral, SDE variants) need more steps, but the injected noise can correct earlier errors, and with many steps they sometimes give better quality than the ODE. EDM found that a modest amount of noise helps some models and hurts others. [Likely]
- Below about 8 steps, solvers alone fail; you need a model trained or distilled for few steps: progressive distillation, consistency models, or straight flows (session 229, session 230).
- The network sets a floor. A perfect solver of the wrong ODE is still wrong. Once the solver error is below the network’s own error, more steps change little (the PyTorch run below shows this).
| Sampler | Type | NFE per step | Order | Typical steps |
|---|---|---|---|---|
| DDPM ancestral | reverse SDE | 1 | 1 (weak) | 1000 in the original paper |
| DDIM (\(\eta = 0\)) | ODE, Euler | 1 | 1 | 50 to 100 [Likely] |
| Heun (EDM) | ODE, Runge–Kutta | 2 | 2 | 18 steps = 35 NFE on CIFAR-10 |
| DPM-Solver-2 / -3 (single-step) | ODE, exponential integrator | 2 / 3 | 2 / 3 | 10 to 20 NFE |
| DPM-Solver++(2M) | ODE, multistep | 1 | 2 | 15 to 25 |
| UniPC | ODE, predictor–corrector | 1 | 2 to 3 | 10 to 20 |
| Predictor–corrector (score SDE) | SDE + Langevin | 2 | 1 | hundreds to thousands |
Score-based sampling in PyTorch
A complete, deterministic example on the 1-D mixture of Figures 3 and 6. It trains a small denoiser with denoising score matching (in the “predict the clean value” form, with the input and output scaling of EDM, which keeps the network’s inputs and targets near unit variance at every \(\sigma\)), then samples with DDIM, Heun and DPM-Solver++(2M), using both the exact denoiser and the trained network. It runs in about half a minute on a CPU.
import math, torch, torch.nn as nn torch.manual_seed(0) # toy data: a 1-D mixture of three Gaussians (weights, means, standard deviations) W = torch.tensor([0.35, 0.45, 0.20]) MU = torch.tensor([-2.0, 0.5, 2.5]) SD = torch.tensor([0.3, 0.4, 0.2]) SIG_D = math.sqrt(float((W * (SD**2 + MU**2)).sum() - (W * MU).sum()**2)) # data std def sample_data(n): c = torch.multinomial(W, n, replacement=True) return MU[c] + SD[c] * torch.randn(n) def exact_D(x, s): """Exact denoiser E[x0 | x_sigma = x] of the mixture (Tweedie): what a perfect network outputs.""" v = SD**2 + s[:, None]**2 # variance of each noisy component logw = torch.log(W) - 0.5 * torch.log(v) - 0.5 * (x[:, None] - MU)**2 / v r = torch.softmax(logw, dim=1) # which component produced x? return (r * (MU + SD**2 / v * (x[:, None] - MU))).sum(1) class Denoiser(nn.Module): # D(x; sigma) with EDM preconditioning def __init__(self, h=128): super().__init__() self.f = nn.Sequential(nn.Linear(2, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 1)) def forward(self, x, s): c_in = 1 / torch.sqrt(s**2 + SIG_D**2) c_skip = SIG_D**2 / (s**2 + SIG_D**2) c_out = s * SIG_D / torch.sqrt(s**2 + SIG_D**2) F = self.f(torch.stack([c_in * x, torch.log(s) / 4], dim=1)).squeeze(1) return c_skip * x + c_out * F # denoising score matching = train D to remove the noise (formula 226.4) net = Denoiser() opt = torch.optim.Adam(net.parameters(), lr=2e-3) for step in range(6000): x0 = sample_data(1024) s = torch.exp(-1.2 + 1.2 * torch.randn(1024)) # log-normal noise levels (EDM) x = x0 + s * torch.randn(1024) lam = (s**2 + SIG_D**2) / (s * SIG_D)**2 # weight so every sigma counts equally loss = (lam * (net(x, s) - x0)**2).mean() opt.zero_grad(); loss.backward(); opt.step() def karras(n, smin=0.002, smax=80.0, rho=7.0): # formula (226.15), then a final 0 i = torch.arange(n) / max(n - 1, 1) s = (smax**(1 / rho) + i * (smin**(1 / rho) - smax**(1 / rho)))**rho return torch.cat([s, torch.zeros(1)]) @torch.no_grad() def sample(D, x, n, method): sig = karras(n) D_prev = h_prev = None for i in range(n): s, s2 = sig[i], sig[i + 1] Dx = D(x, s.expand_as(x)) if method == "ddim" or s2 == 0: # DDIM = Euler on the PF-ODE x = Dx + (s2 / s) * (x - Dx) elif method == "heun": # EDM 2nd order: one extra evaluation d = (x - Dx) / s xe = x + (s2 - s) * d d2 = (xe - D(xe, s2.expand_as(x))) / s2 x = x + (s2 - s) * (d + d2) / 2 elif method == "dpm2m": # DPM-Solver++(2M): reuse the last D h = torch.log(s / s2) Dd = Dx if D_prev is None else Dx + (Dx - D_prev) * h / (2 * h_prev) x = (s2 / s) * x + (1 - s2 / s) * Dd D_prev, h_prev = Dx, h return x def mixture_quantiles(u, s): # exact quantiles of p_sigma by bisection lo, hi = torch.full_like(u, -500.), torch.full_like(u, 500.) sd = torch.sqrt(SD**2 + s**2) for _ in range(100): mid = (lo + hi) / 2 cdf = (W * torch.special.ndtr((mid[:, None] - MU) / sd)).sum(1) lo, hi = torch.where(cdf < u, mid, lo), torch.where(cdf < u, hi, mid) return (lo + hi) / 2 M = 1000 u = (torch.arange(M, dtype=torch.float64) + 0.5) / M torch.set_default_dtype(torch.float32) xT = mixture_quantiles(u, 80.0).float() # start noise, stratified target = mixture_quantiles(u, 0.0).float() # where the exact ODE must land w1 = lambda x: (x.sort().values - target).abs().mean().item() # 1-D Wasserstein-1 distance print(f"data std {SIG_D:.3f}, final training loss {loss.item():.3f}") xs = torch.tensor([-2.0, 0.0, 1.5]); ss = torch.tensor([0.5, 0.5, 0.5]) print("D_net ", [round(v, 3) for v in net(xs, ss).tolist()]) print("D_exact", [round(v, 3) for v in exact_D(xs, ss).tolist()]) for nfe in [5, 10, 20, 50]: row = [] for D in (exact_D, net): row += [w1(sample(D, xT, nfe, "ddim")), w1(sample(D, xT, (nfe + 1) // 2, "heun")), w1(sample(D, xT, nfe, "dpm2m"))] print(f"NFE {nfe:2d} | exact D: ddim {row[0]:.3f} heun {row[1]:.3f} dpm2m {row[2]:.3f}" f" | network: ddim {row[3]:.3f} heun {row[4]:.3f} dpm2m {row[5]:.3f}")
On PyTorch 2.15 (CPU) this prints a data standard deviation of 1.695. The trained network is close to the exact denoiser: at \(x = -2, 0, 1.5\) and \(\sigma = 0.5\) it outputs −1.990, 0.288, 1.297 against the exact −1.999, 0.299, 1.246. The \(W_1\) errors:
| NFE | DDIM, exact D | Heun, exact D | DPM++(2M), exact D | DDIM, network | Heun, network | DPM++(2M), network |
|---|---|---|---|---|---|---|
| 5 | 0.663 | 5.695 | 0.346 | 0.668 | 3.364 | 0.354 |
| 10 | 0.291 | 1.053 | 0.228 | 0.258 | 0.990 | 0.318 |
| 20 | 0.139 | 0.167 | 0.036 | 0.164 | 0.256 | 0.165 |
| 50 | 0.054 | 0.019 | 0.005 | 0.152 | 0.164 | 0.161 |
With the exact denoiser the numbers match Figure 6 to three decimals: this is pure solver error. With the trained network, every solver ends near 0.15 to 0.16 from 20 NFE on: the network’s own error is now the floor, and a better solver cannot remove it. Other versions or hardware may print slightly different numbers.
Where this is used
| Setting | What is used | What matters |
|---|---|---|
| Text-to-image (Stable Diffusion, SDXL, SD3 and others) | Swappable samplers: libraries such as diffusers ship DDIM, Euler, Heun, DPM-Solver++ and UniPC schedulers for the same trained model | Steps set latency directly; 20 to 30 steps of a multistep solver is common [Likely] |
| Video generation | The same ODE solvers on spatio-temporal latents; one NFE processes the whole clip | NFE dominates cost, so solver order and distillation matter even more; see session 231 |
| Image and video editing | DDIM inversion: map a real image or each video frame to noise, change the prompt, sample again | Inversion is approximate; errors grow with guidance; per-frame inversion can flicker [Likely] |
| Inverse problems (inpainting, super-resolution, deblurring) | Score SDE with the measurement added to the score, or a DDIM loop that projects onto the known pixels | Uses the score as a learned prior without retraining |
| Likelihood and anomaly scores | Probability-flow ODE plus the change-of-variables formula | Exact \(\log p(\mathbf{x})\) at the cost of an ODE solve with a trace estimator |
| Robot and video-agent action generation | Diffusion policies sample action sequences with few DDIM-style steps [Likely] | Control loops need low latency, so step count is a hard budget |
| Few-step generation | The ODE trajectory is the training target for distillation and consistency models | session 230 builds on the ODE of this session |
Common mistakes
- Mixing parameterizations. Using an \(\varepsilon\)-prediction model in a sampler written for \(\mathbf{x}_0\)- or \(v\)-prediction (or the reverse) gives garbage. Convert with (226.6) and check one step by hand.
- Forgetting the input scaling. In the \(\sigma\) form, the network of a DDPM model expects \(\mathbf{x}/\sqrt{1 + \sigma^2}\), not \(\mathbf{x}\). Samplers that work in \(\sigma\) form must scale before every call.
- Picking the DDIM sub-sequence badly. Evenly spaced steps that skip the last training steps (for example, ending at \(t = 50\) instead of near 0) leave visible noise; a schedule that is too sparse at low noise blurs details.
- Expecting determinism with \(\eta \gt 0\). Only \(\eta = 0\) is a fixed map from noise to image. Inversion, interpolation and caching of noises need \(\eta = 0\).
- Comparing samplers by steps instead of NFE. A Heun step costs 2 NFE. Compare at equal NFE, as in Figure 6.
- Using a high-order single-step solver at very few steps or with strong guidance. It can be worse than DDIM (Heun in Figure 6 below 10 NFE); DPM-Solver++ uses the data prediction and a multistep method for this reason.
- Blaming the sampler for a network problem. If quality stops improving above 20 to 30 steps, the floor is the model, not the solver.
Interview questions
What is the score of a distribution, and why is it easier to model than the density?basic
Why is plain score matching impractical for images, and how does denoising score matching fix it?core
How are the noise prediction, the score and the denoised image related?core
Why does NCSN use many noise levels instead of one?core
What is the probability-flow ODE, and how does it differ from the reverse SDE?core
Why can DDIM sample with 50 steps from a model trained with 1000 steps, without retraining?core
What does \(\eta\) do in DDIM?basic
DDIM inversion of a real image does not give back exactly the same image. Why?advanced
Heun is second order, yet at 10 NFE it was worse than DDIM in Figure 6. Explain.advanced
A latent image model looks good at 50 DDIM steps. Product wants 10 steps. What do you do?advanced
How many NFE does a 18-step EDM Heun sampler use? And 20 steps of DPM-Solver++(2M) with classifier-free guidance?basic
Exercises
- The data is \(\mathcal{N}(1, 0.5^2)\). At noise level \(\sigma = 1\) and \(\tilde{x} = 2\), compute the score, the Tweedie denoised value and the predicted noise.
- Continue exercise 1: take one DDIM step to \(\sigma_{\text{next}} = 0.5\). Compare with the exact ODE solution (hint: for Gaussian data, \(x - \mu\) grows like \(\sqrt{s^2 + \sigma^2}\)).
- A DDPM point has \(\bar\alpha_t = 0.25\). What is \(\sigma\) in the \(\sigma\) form, and by what factor must you scale \(\mathbf{x}_t\)?
- One Langevin step (226.7) for the target \(\mathcal{N}(2, 1)\): \(x = 0\), \(\alpha = 0.1\), noise sample \(z = 0.5\). Where does \(x\) go?
- Show that \(\eta = 0\) in (226.12) gives a deterministic update, and that with \(\eta = 1\) the noise variance equals the DDPM posterior variance \(\tilde\beta_t = \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\beta_t\) when \(t' = t - 1\).
- In Figure 6, find the smallest budget at which DPM-Solver++(2M) reaches \(W_1 \lt 0.05\). How many NFE does DDIM need for the same error?
Answerscheck
Key takeaways
- The score \(\nabla \log p\) (226.1) points toward more likely data and does not need the normalizing constant.
- Denoising score matching (226.4) learns the score of noisy data by plain regression; it is the DDPM noise loss. Score, noise and clean-image predictions convert into each other with Tweedie’s formula (226.5, 226.6).
- Langevin dynamics (226.7) samples with the score, but only across many noise levels (NCSN, annealing) does it get the mode proportions right.
- Diffusion is an SDE (226.8); its reverse (226.9) and the deterministic probability-flow ODE (226.10, 226.11) have the same marginals.
- DDIM (226.12) reuses a DDPM network with any sub-sequence of steps; with \(\eta = 0\) it is Euler’s method on the ODE (226.13).
- Better solvers (Heun 226.14, DPM-Solver++(2M) 226.16, UniPC) plus a schedule dense at low noise (226.15) give second-order error decay; compare at equal NFE, and remember the network’s own error is the floor.
Key formulas
References for this session
- Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
- Song et al. (2020). Denoising Diffusion Implicit Models
- Song & Ermon (2019). Generative Modeling by Estimating Gradients of the Data Distribution
- Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
- Song & Ermon (2019). Generative Modeling by Estimating Gradients of the Data Distribution
- Song et al. (2020). Denoising Diffusion Implicit Models
- Ho et al. (2020). Denoising Diffusion Probabilistic Models
- Hyvärinen (2005). Estimation of Non-Normalized Statistical Models by Score Matching
- Vincent (2011). A Connection Between Score Matching and Denoising Autoencoders
- Anderson (1982). Reverse-Time Diffusion Equation Models
- Efron (2011). Tweedie's Formula and Selection Bias
- Karras, Aittala, Aila & Laine (2022). Elucidating the Design Space of Diffusion-Based Generative Models
- Lu, Zhou, Bao, Chen, Li & Zhu (2022). DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
- Lu, Zhou, Bao, Chen, Li & Zhu (2022). DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
- Zhao, Bai, Rao, Zhou & Lu (2023). UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models
- Song & Ermon (2020). Improved Techniques for Training Score-Based Generative Models
- Salimans & Ho (2022). Progressive Distillation for Fast Sampling of Diffusion Models
- Song et al. (2023). Consistency Models
- Liu et al. (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow