LLM & VLM courseBlog
225Track 9: Generative models · session 4 of 12

Diffusion 1: the forward noising process and the reverse denoiser (DDPM)

A diffusion model learns to generate by learning to undo noise. This session builds DDPM from the ground up: the forward process that adds Gaussian noise step by step (and its closed form for any step), the reverse process and why training reduces to predicting the noise, ancestral sampling from pure noise, and the noise schedules that decide where the model spends its effort.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Variational autoencoders and the ELBO (session 223), Gaussian distributions and KL divergence (session 3), backpropagation and Adam (session 4); basic PyTorch

What you will learn

  • The forward noising step, why it scales the signal by the square root of 1 minus beta, and how to jump to any step t in one line
  • Signal-to-noise ratio as the one number that describes how noisy step t is
  • The reverse process, the exact posterior q(x_{t-1} | x_t, x_0), and how the ELBO turns into a sum of Gaussian KL terms
  • Why DDPM trains the network to predict the noise, and what the simple loss L_simple drops from the true bound
  • Watch a real 78,451-weight U-Net learn to predict noise in your browser, with a hand-written backward pass and Adam
  • Ancestral sampling: start from pure noise and run 1000 small denoising steps, with the exact optimal denoiser as a reference
  • Linear, cosine and scaled-linear schedules, zero terminal SNR, and why the schedule must change with image resolution
  • The link between noise prediction and the score of NCSN, which the next session develops

Why this matters

Most modern image and video generators are diffusion models or close relatives: Stable Diffusion, Imagen, DALL·E 2 and many video generators are diffusion models, and SD3 and FLUX use flow matching, a close cousin (session 229). They all rest on one simple trick. Destroying an image with noise is easy. If a network learns to undo a small amount of noise, we can chain many small undo steps and turn pure noise into a new image.

This session builds that idea in the form of DDPM (denoising diffusion probabilistic models, Ho, Jain and Abbeel, 2020), the paper that made diffusion competitive. With 1000 steps and a 35.7M-parameter U-Net it reached an FID of 3.17 on unconditional CIFAR-10, better than most GANs of the time [Certain] (DDPM paper, Table 1). The core ideas are older: Sohl-Dickstein et al. (2015) introduced diffusion probabilistic models, and Song and Ermon (2019) trained a network to estimate the score of noisy data at many noise levels (NCSN). DDPM showed that the two views lead to the same, very simple training loss.

Four ideas carry the whole field, and each is a classic interview question:

  • The forward process adds Gaussian noise in many small steps, and we can jump to any step \(t\) in one line (formula 225.3).
  • Training is regression: show the network a noisy image and ask it to predict the noise that was added (formula 225.11).
  • Sampling starts from pure noise and applies the learned denoiser step by step (Algorithm 2).
  • The noise schedule decides how fast the signal disappears, and so which noise levels the model practices on. A bad schedule caused a real bug in Stable Diffusion: it could not make very dark or very bright images.

In session 223 you met the ELBO, a lower bound on \(\log p(\mathbf{x})\). A diffusion model is a VAE with a fixed encoder and 1000 latent layers, and its loss is the same kind of bound. session 224 showed the other road, discrete tokens. The next session, session 226, looks at the same model through the score and differential equations and cuts sampling from 1000 steps to 20–50.

The idea: destroy slowly, learn to repair

Take a data point \(\mathbf{x}_0\), for example an image with pixel values scaled to \([-1, 1]\). Add a little Gaussian noise to get \(\mathbf{x}_1\). Add a little more to get \(\mathbf{x}_2\). After \(T\) steps (DDPM uses \(T = 1000\)), \(\mathbf{x}_T\) is pure noise: a sample from \(\mathcal{N}(\mathbf{0}, I)\) that carries no trace of the image. This is the forward process (also called the diffusion process). It has no learned parameters.

Now run the film backwards. Each backward step only has to remove a tiny amount of noise. When the noise added in one step is small, the best guess for the previous, slightly cleaner image is almost a Gaussian around a point we can learn to predict. So the reverse process is also a chain of Gaussians, and a neural network gives each Gaussian its mean. To generate, we draw \(\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, I)\) and apply the learned reverse step \(T\) times.

Figure 1 shows both chains on a tiny 12 × 12 image. The top arrows are the fixed forward process \(q\); the bottom arrows are the learned reverse process \(p_\theta\).

forward process \(q(\mathbf{x}_t \mid \mathbf{x}_{t-1})\): fixed, adds noisereverse process \(p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t)\): learned, removes noise
Figure 1. The two Markov chains of DDPM on a 12 × 12 image with pixel values in \([-1, 1]\), using the linear schedule. Each panel is \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon\) with the same noise \(\varepsilon\); the numbers under each panel are the two coefficients. By \(t = 500\) only 28% of the signal amplitude is left; by \(t = 1000\) only 0.6%.

A Markov chain is a sequence where each step depends only on the step just before it. Both chains here are Markov chains. That is what makes the math tractable: we only ever need to model one step at a time.

The forward process: one step

Pick a sequence of small numbers \(\beta_1, \dots, \beta_T\) between 0 and 1, the noise schedule. One forward step is:

\[ q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = \mathcal{N}\!\big(\mathbf{x}_t;\ \sqrt{1-\beta_t}\,\mathbf{x}_{t-1},\ \beta_t I\big) \quad\Longleftrightarrow\quad \mathbf{x}_t = \sqrt{1-\beta_t}\,\mathbf{x}_{t-1} + \sqrt{\beta_t}\,\varepsilon_t \tag{225.1} \]
Read the symbols
\(\mathbf{x}_t\)
the data after \(t\) noise steps; same shape as the image (\(\mathbf{x}_0\) is the clean image)
\(\beta_t\)
the variance of the noise added at step \(t\); DDPM uses values from \(10^{-4}\) up to \(0.02\)
\(\varepsilon_t\)
fresh standard Gaussian noise, \(\mathcal{N}(\mathbf{0}, I)\), drawn independently at every step and for every pixel
\(\mathcal{N}(\mathbf{x};\ \mu,\ \Sigma)\)
the Gaussian density of \(\mathbf{x}\) with mean \(\mu\) and covariance \(\Sigma\)

Why multiply the old value by \(\sqrt{1-\beta_t}\) instead of just adding noise? Because it keeps the total variance fixed. If \(\mathbf{x}_{t-1}\) has variance 1 in each coordinate, then \(\mathbf{x}_t\) has variance \((1-\beta_t) \cdot 1 + \beta_t = 1\). The signal shrinks a little, noise fills the gap, and the numbers never blow up. This version is called variance preserving. (NCSN uses the other choice, variance exploding: it only adds noise, so the variance grows to the largest noise level. session 226 compares them.)

The whole forward chain is the product of its steps:

\[ q(\mathbf{x}_{1:T} \mid \mathbf{x}_0) = \prod_{t=1}^{T} q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) \tag{225.2} \]

In VAE language, this is the encoder. But unlike a VAE encoder it is fixed: nothing in it is learned. All the learning happens in the reverse direction.

The closed form for any step t

To train, we need noisy images at random steps \(t\). Running \(t\) single steps each time would be slow. Luckily the sum of independent Gaussians is a Gaussian, so the chain collapses into one line. Define

\[ \alpha_t = 1 - \beta_t, \qquad \bar\alpha_t = \prod_{s=1}^{t} \alpha_s \]

Look at two steps first. \(\mathbf{x}_2 = \sqrt{\alpha_2}\,\mathbf{x}_1 + \sqrt{1-\alpha_2}\,\varepsilon_2\) and \(\mathbf{x}_1 = \sqrt{\alpha_1}\,\mathbf{x}_0 + \sqrt{1-\alpha_1}\,\varepsilon_1\). Substitute:

\[ \mathbf{x}_2 = \sqrt{\alpha_1\alpha_2}\,\mathbf{x}_0 + \underbrace{\sqrt{\alpha_2(1-\alpha_1)}\,\varepsilon_1 + \sqrt{1-\alpha_2}\,\varepsilon_2}_{\text{Gaussian, variance } \alpha_2(1-\alpha_1) + 1 - \alpha_2 \;=\; 1 - \alpha_1\alpha_2} \]

The two noise terms are independent, so their variances add, and the total is exactly \(1 - \alpha_1\alpha_2 = 1 - \bar\alpha_2\). The same argument repeats for every step (induction), and gives the most used formula in diffusion:

\[ q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}\!\big(\mathbf{x}_t;\ \sqrt{\bar\alpha_t}\,\mathbf{x}_0,\ (1-\bar\alpha_t) I\big) \quad\Longleftrightarrow\quad \mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon,\ \ \varepsilon \sim \mathcal{N}(\mathbf{0}, I) \tag{225.3} \]
Read the symbols
\(\bar\alpha_t\)
“alpha bar”: the product of all \(\alpha_s\) up to step \(t\); it starts near 1 and falls toward 0
\(\sqrt{\bar\alpha_t}\)
the signal coefficient: how much of the clean image is still in \(\mathbf{x}_t\)
\(\sqrt{1-\bar\alpha_t}\)
the noise coefficient; the squares of the two coefficients always add to 1
\(\varepsilon\)
one standard Gaussian draw that stands for all the noise added in steps 1 to \(t\)

A noisy image at step \(t\) is a weighted mix of the clean image and one noise image. The schedule only decides how the weights move. A single number summarizes how noisy step \(t\) is, the signal-to-noise ratio:

\[ \mathrm{SNR}(t) = \frac{\bar\alpha_t}{1-\bar\alpha_t} \tag{225.4} \]

It is the ratio of the signal variance to the noise variance (for data with unit variance). We often plot \(\log \mathrm{SNR}(t)\): it is positive when the signal dominates, 0 when they are equal, and negative when noise dominates. With the linear DDPM schedule, \(\log\mathrm{SNR}\) goes from 9.2 at \(t = 1\) to −10.1 at \(t = 1000\), and crosses 0 at \(t = 260\) (computed below).

Worked example: jumping to step \(t\)

Two steps. Take \(\beta_1 = 0.1\), \(\beta_2 = 0.2\). Then \(\bar\alpha_2 = 0.9 \cdot 0.8 = 0.72\). The noise variance from running both steps is \(\alpha_2\beta_1 + \beta_2 = 0.8 \cdot 0.1 + 0.2 = 0.28 = 1 - 0.72\). The closed form agrees.

DDPM's schedule. \(\beta_t\) rises linearly from \(10^{-4}\) to \(0.02\) over \(T = 1000\) steps. For one pixel with clean value \(x_0 = 0.8\) and noise draw \(\varepsilon = -0.5\):

  • \(t = 250\): \(\bar\alpha_t = 0.5241\), so \(x_t = 0.7239 \cdot 0.8 + 0.6899 \cdot (-0.5) = 0.2342\). SNR \(= 1.10\): signal and noise are about equal.
  • \(t = 500\): \(\bar\alpha_t = 0.0786\), so \(x_t = 0.2803 \cdot 0.8 + 0.9599 \cdot (-0.5) = -0.2557\). The noise now dominates.
  • \(t = 1000\): \(\bar\alpha_T = 4.04 \times 10^{-5}\), so \(x_T = 0.0064 \cdot 0.8 + 1.0000 \cdot (-0.5) = -0.4949\). Almost pure noise: only 0.6% of the signal amplitude is left.

The DDPM paper chose these \(\beta\) values so that \(\mathbf{x}_T\) is very close to \(\mathcal{N}(\mathbf{0}, I)\): the KL divergence between \(q(\mathbf{x}_T \mid \mathbf{x}_0)\) and \(\mathcal{N}(\mathbf{0}, I)\) is about \(10^{-5}\) bits per dimension [Certain].

Figure 2 applies the closed form to a 2-D data set: 400 points drawn from 8 small blobs on a circle of radius 1. Think of each point as a tiny “image” with two pixels. Move \(t\) and watch the blobs shrink toward the center (the signal is scaled by \(\sqrt{\bar\alpha_t}\)) while the noise spreads them out. The readout also runs the chain one step at a time, to show that the result has the same spread as the one-line formula.

Schedule
noisy points \(\mathbf{x}_t\)blob centers scaled by \(\sqrt{\bar\alpha_t}\)circle of radius 2 = 2 standard deviations of \(\mathcal{N}(\mathbf{0}, I)\)
Figure 2. The forward process on a toy 2-D data set (8 blobs of standard deviation 0.08 on a circle of radius 1). Each point uses the closed form (225.3) with its own fixed noise draw, so moving \(t\) shows one continuous path per point. The right side shows the two coefficients; their squares always add to 1.
How to read Figure 2

At \(t = 0\) you see the 8 blobs. As \(t\) grows, the orange rings (the blob centers times \(\sqrt{\bar\alpha_t}\)) move toward the origin, and the noise blurs the blobs together. By \(t \approx 700\) on the linear schedule the cloud looks like a plain Gaussian; switch to cosine and the same happens much later. The readout compares three spreads: from the closed form, from actually running \(t\) single steps (225.1) with fresh noise, and from theory \(\sqrt{\bar\alpha_t \operatorname{Var}(x_0) + 1 - \bar\alpha_t}\). They agree up to sampling noise, which is the point of (225.3).

Algorithm 1: make a noisy training example (closed form)
  1. precompute once: \(\beta_{1..T}\) from the schedule; \(\alpha_t \leftarrow 1-\beta_t\); \(\bar\alpha_t \leftarrow \prod_{s \le t}\alpha_s\) (a cumulative product)
  2. input: a clean example \(\mathbf{x}_0\) (scaled to \([-1, 1]\))
  3. \(t \sim \text{Uniform}\{1, \dots, T\}\) // a random noise level
  4. \(\varepsilon \sim \mathcal{N}(\mathbf{0}, I)\) // same shape as \(\mathbf{x}_0\)
  5. \(\mathbf{x}_t \leftarrow \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon\) // formula (225.3): one line, no loop over steps
  6. return \((\mathbf{x}_t, t, \varepsilon)\) // \(\varepsilon\) is the regression target

The reverse process and the exact posterior

The generative model runs the other way. It starts from the prior \(p(\mathbf{x}_T) = \mathcal{N}(\mathbf{0}, I)\) and applies learned Gaussian steps:

\[ p_\theta(\mathbf{x}_{0:T}) = p(\mathbf{x}_T) \prod_{t=1}^{T} p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t), \qquad p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t) = \mathcal{N}\!\big(\mathbf{x}_{t-1};\ \mu_\theta(\mathbf{x}_t, t),\ \sigma_t^2 I\big) \tag{225.5} \]
Read the symbols
\(\mu_\theta(\mathbf{x}_t, t)\)
the mean of the reverse step, computed by one neural network (weights \(\theta\)) shared by all steps; the step \(t\) is an input
\(\sigma_t^2\)
the variance of the reverse step; DDPM fixes it to a constant per step (not learned)
\(p(\mathbf{x}_T)\)
the prior: standard Gaussian noise, where sampling starts

Why is a Gaussian reverse step good enough? The true reverse step \(q(\mathbf{x}_{t-1} \mid \mathbf{x}_t)\) is not Gaussian in general: it depends on the whole data distribution. But when \(\beta_t\) is small, \(\mathbf{x}_{t-1}\) is pinned to a small region around \(\mathbf{x}_t\), and inside a small region any smooth density looks like a Gaussian. This is why we need many small steps, and why sampling is slow [Certain] (Sohl-Dickstein et al. 2015; DDPM Section 2).

There is one reverse step we can write down exactly: the one that also knows the clean image. Conditioned on \(\mathbf{x}_0\), Bayes’ rule combines two Gaussians, \(q(\mathbf{x}_t \mid \mathbf{x}_{t-1})\) and \(q(\mathbf{x}_{t-1} \mid \mathbf{x}_0)\), into a third Gaussian:

\[ q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0) = \mathcal{N}\!\big(\mathbf{x}_{t-1};\ \tilde\mu_t(\mathbf{x}_t, \mathbf{x}_0),\ \tilde\beta_t I\big), \quad \tilde\mu_t = \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1-\bar\alpha_t}\,\mathbf{x}_0 + \frac{\sqrt{\alpha_t}\,(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\,\mathbf{x}_t, \quad \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\,\beta_t \tag{225.6} \]
Read the symbols
\(\tilde\mu_t\)
the posterior mean: a weighted average of the clean image and the current noisy image
\(\tilde\beta_t\)
the posterior variance; slightly smaller than \(\beta_t\), because knowing \(\mathbf{x}_0\) removes some uncertainty

Where does it come from? Multiply the two Gaussian densities and collect the terms in \(\mathbf{x}_{t-1}\) (“complete the square”). The precisions (inverse variances) add: \(1/\tilde\beta_t = \alpha_t/\beta_t + 1/(1-\bar\alpha_{t-1})\), which simplifies to the \(\tilde\beta_t\) above. The mean is the precision-weighted average of the two guesses for \(\mathbf{x}_{t-1}\).

The plan is now clear. At sampling time we do not know \(\mathbf{x}_0\), so we train \(p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t)\) to match this posterior as well as possible, averaged over all the images that could have produced \(\mathbf{x}_t\).

Training: from the ELBO to “predict the noise”

A diffusion model is a latent-variable model: \(\mathbf{x}_1, \dots, \mathbf{x}_T\) are latents with the same shape as the data. As for the VAE, \(\log p_\theta(\mathbf{x}_0)\) is intractable, so we maximize the ELBO, with the forward process as the (fixed) encoder. After some algebra (DDPM Appendix A), the negative ELBO splits into one term per step:

\[ L = \E_q\Big[ \underbrace{\KL\big(q(\mathbf{x}_T \mid \mathbf{x}_0) \,\|\, p(\mathbf{x}_T)\big)}_{L_T} + \sum_{t=2}^{T} \underbrace{\KL\big(q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0) \,\|\, p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t)\big)}_{L_{t-1}} \underbrace{- \log p_\theta(\mathbf{x}_0 \mid \mathbf{x}_1)}_{L_0} \Big] \tag{225.7} \]
Read the symbols
\(L_T\)
how far the end of the forward chain is from the prior; it has no learned parameters, and the schedule makes it about 0
\(L_{t-1}\)
for each step: how well the learned reverse step matches the exact posterior (225.6)
\(L_0\)
the last decoding step back to discrete pixel values

Every \(L_{t-1}\) is a KL between two Gaussians with fixed variances, so only the means matter:

\[ L_{t-1} = \E_q\Big[ \frac{1}{2\sigma_t^2} \big\lVert \tilde\mu_t(\mathbf{x}_t, \mathbf{x}_0) - \mu_\theta(\mathbf{x}_t, t) \big\rVert^2 \Big] + C \tag{225.8} \]

We could train the network to output \(\tilde\mu_t\) directly. DDPM found a better choice. Use (225.3) to write \(\mathbf{x}_0 = (\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\varepsilon)/\sqrt{\bar\alpha_t}\) and put it into \(\tilde\mu_t\). The posterior mean becomes \(\frac{1}{\sqrt{\alpha_t}}\big(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\varepsilon\big)\). The network sees \(\mathbf{x}_t\), so the only unknown is the noise \(\varepsilon\). Let the network predict it:

\[ \mu_\theta(\mathbf{x}_t, t) = \frac{1}{\sqrt{\alpha_t}} \Big( \mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\, \varepsilon_\theta(\mathbf{x}_t, t) \Big) \tag{225.9} \]

Put (225.9) into (225.8). The \(\mathbf{x}_t\) parts cancel and only the noise error is left:

\[ L_{t-1} - C = \E_{\mathbf{x}_0, \varepsilon}\Big[ \frac{\beta_t^2}{2\sigma_t^2\,\alpha_t\,(1-\bar\alpha_t)} \big\lVert \varepsilon - \varepsilon_\theta\big(\sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon,\ t\big) \big\rVert^2 \Big] \tag{225.10} \]

This is the true bound: a weighted sum of noise-prediction errors. DDPM then makes one bold change: drop the weight. The result is the loss almost every diffusion model is trained with:

\[ L_\text{simple}(\theta) = \E_{t \sim \mathcal{U}\{1..T\},\ \mathbf{x}_0,\ \varepsilon \sim \mathcal{N}(\mathbf{0}, I)} \Big[ \big\lVert \varepsilon - \varepsilon_\theta\big(\sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon,\ t\big) \big\rVert^2 \Big] \tag{225.11} \]
Read the symbols
\(\varepsilon_\theta(\mathbf{x}_t, t)\)
the network: input a noisy image and its step, output a guess of the noise, same shape as the image
\(\mathcal{U}\{1..T\}\)
the step is drawn uniformly at random, so one training step trains one random term of the sum
\(\lVert\cdot\rVert^2\)
plain squared error: the whole method is a regression problem

What does dropping the weight do? Figure 3 plots the weight in (225.10) for the linear schedule with \(\sigma_t^2 = \beta_t\), rescaled to an average of 1 so it can be compared with \(L_\text{simple}\) (weight 1 for every step).

true bound (225.10), rescaled to mean 1\(L_\text{simple}\): weight 1 everywhere
Figure 3. Relative weight of each step’s noise-prediction error. The true bound puts about 51 times the average weight on \(t = 1\), and its smallest weight (0.50 of the average) is near \(t = 349\). \(L_\text{simple}\) is flat. Linear schedule, \(T = 1000\), \(\sigma_t^2 = \beta_t\).
How to read Figure 3

The y-axis is logarithmic. Move the slider to read the two weights at any step. The orange curve is above 1 only for about the first 100 steps, and far above it (7.5 at \(t = 10\), 51 at \(t = 1\)) where the noise is tiny. Those steps fix invisible details (noise smaller than one pixel level), so \(L_\text{simple}\) gives them less weight and spends more of the model’s effort on the harder, noisier steps that decide the content of the image. 17% of the true bound’s total weight sits on \(t \le 10\) alone.

The DDPM paper reports the effect directly (Table 2, CIFAR-10) [Certain]: noise prediction trained on the true bound with fixed variance gave FID 13.51; the same network trained with \(L_\text{simple}\) gave FID 3.17. The price is likelihood: the \(L_\text{simple}\) model has a slightly worse bound (≤ 3.75 bits per dimension vs ≤ 3.70). Predicting the mean \(\tilde\mu\) with an unweighted squared error was unstable. The paper also tried predicting \(\mathbf{x}_0\) directly and found worse samples early on.

Because \(\varepsilon\) and \(\mathbf{x}_0\) are tied by (225.3), a noise prediction is also a prediction of the clean image:

\[ \hat{\mathbf{x}}_0 = \frac{\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\varepsilon_\theta(\mathbf{x}_t, t)}{\sqrt{\bar\alpha_t}} \tag{225.12} \]

This is handy for previews during sampling, and it explains a weakness: at large \(t\), \(\sqrt{\bar\alpha_t}\) is tiny, so a small error in \(\varepsilon_\theta\) becomes a huge error in \(\hat{\mathbf{x}}_0\). With the linear schedule, a noise error of 0.1 becomes an image error of 0.004 at \(t = 10\), 0.34 at \(t = 500\), and 15.7 at \(t = 1000\). Later models use other targets, such as v-prediction \(\mathbf{v} = \sqrt{\bar\alpha_t}\,\varepsilon - \sqrt{1-\bar\alpha_t}\,\mathbf{x}_0\), which behave well at both ends (session 226).

Algorithm 2: one DDPM training step (\(L_\text{simple}\))
  1. input: a mini-batch \(\mathbf{x}_0^{(1)}, \dots, \mathbf{x}_0^{(B)}\); the table \(\bar\alpha_{1..T}\); the network \(\varepsilon_\theta\)
  2. for each example \(i\): draw \(t_i \sim \mathcal{U}\{1..T\}\) and \(\varepsilon_i \sim \mathcal{N}(\mathbf{0}, I)\)
  3. \(\mathbf{x}_{t_i}^{(i)} \leftarrow \sqrt{\bar\alpha_{t_i}}\,\mathbf{x}_0^{(i)} + \sqrt{1-\bar\alpha_{t_i}}\,\varepsilon_i\) // Algorithm 1
  4. \(\text{loss} \leftarrow \frac1B \sum_i \lVert \varepsilon_i - \varepsilon_\theta(\mathbf{x}_{t_i}^{(i)}, t_i) \rVert^2\) // formula (225.11)
  5. backpropagate and update \(\theta\) with Adam; keep an EMA copy of the weights for sampling // DDPM: EMA decay 0.9999

The network must know \(t\), because the right amount of denoising depends on how noisy the input is. DDPM turns \(t\) into a vector with the sinusoidal position embedding of the Transformer and adds it inside every residual block of a U-Net (an encoder–decoder convolutional network with skip connections between matching resolutions, Ronneberger et al. 2015) [Certain]. Its CIFAR-10 model has 35.7M parameters and trained for 800k steps at batch size 128 with learning rate \(2 \times 10^{-4}\) [Certain] (DDPM Appendix B). session 228 replaces the U-Net with a transformer.

Figure 4 shows the training loop on real pictures, with a real network. The top row applies the closed form (225.3) to a 96 × 96 color picture (drawn by the page’s code, values scaled to \([-1, 1]\)) at six steps, with the same noise image \(\varepsilon\) in every panel. The bottom part trains a network inside your browser. The training set is the three pictures of the top row, shrunk to 16 × 16 pixels so that one pass (forward, backward and update) takes about 12 ms on a laptop. The network \(\varepsilon_\theta\) is a tiny U-Net written in plain JavaScript, with hand-written backpropagation:

  • Down path: two 3 × 3 convolutions with 16 channels at 16 × 16, average pooling to 8 × 8, two convolutions with 32 channels, pooling to 4 × 4, two more with 32 channels.
  • Up path: copy each pixel to a 2 × 2 block (upsampling) back to 8 × 8, join it with the 8 × 8 features of the down path (a skip connection), two convolutions with 32 channels; upsample to 16 × 16, join the 16 × 16 features, one convolution with 16 channels; a last convolution gives the 3 output channels \(\hat\varepsilon\). Every convolution except the last is followed by the SiLU activation \(z \cdot \sigma(z)\).
  • Time input: \(t\) becomes 16 sine and cosine features (as in the Transformer), then a small layer with 32 outputs; each convolution adds its own learned linear map of these 32 numbers, one number per channel. This is how DDPM’s U-Net receives \(t\), at a much smaller size.
  • Size and training: 78,451 weights. One pass is the loop in the figure (the DDPM paper’s Algorithm 1; Algorithm 2 above is the same loop for a batch) with a batch of one picture, followed by one Adam step on all weights, with the gradient norm clipped at 1. The last layer starts at 0, so the first prediction is \(\hat\varepsilon = \mathbf{0}\) and the first loss is about 1 (the mean of \(\varepsilon^2\)). A second copy of the weights, an exponential moving average (EMA, decay 0.999), is used for sampling, as Algorithm 2 says.

We checked the hand-written backward pass against finite differences on a smaller copy of the same network (in 64-bit floats, every weight): the largest relative difference was \(4 \times 10^{-5}\).

PictureSchedule
Training a real network on 16 × 16 copies of the three pictures
Learning rateScheduleSeed
Training loop (DDPM paper, Algorithm 1)
  1. repeat
  2. \(\mathbf{x}_0 \sim q(\mathbf{x}_0)\) pick a training picture
  3. \(t \sim \text{Uniform}(\{1, \dots, T\})\)
  4. \(\varepsilon \sim \mathcal{N}(\mathbf{0}, I)\) same shape as the picture
  5. gradient step on \(\nabla_\theta \big\lVert \varepsilon - \varepsilon_\theta(\mathbf{x}_t, t) \big\rVert^2\), where \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon\)
  6. until converged
\(\mathbf{x}_0\) (line 2)
\(\varepsilon\) (line 4)
\(\mathbf{x}_t\), input
\(\hat\varepsilon = \varepsilon_\theta(\mathbf{x}_t, t)\)
\(\lvert \varepsilon - \hat\varepsilon \rvert\)
\(\hat{\mathbf{x}}_0\) from \(\hat\varepsilon\)
loss of each pass (dots)average of the last 200 passes
Samples from the current network
Figure 4. Training on pictures. Top: the forward process (225.3) on one 96 × 96 picture, one fixed noise draw \(\varepsilon\), real schedules with \(T = 1000\); under each panel are \(\sqrt{\bar\alpha_t}\) and \(\sqrt{1-\bar\alpha_t}\). Bottom: a real training run of a 78,451-weight U-Net on 16 × 16 copies of the three pictures, in your browser, from random weights (seeded, so the same settings give the same run). Line 5 is a real forward pass, backward pass and Adam step. The loss shown for a pass is computed before that pass’s update. The dashed part of the curve after loading the checkpoint is its 100-pass average. Samples are drawn every 1,000 passes with the EMA weights and a 40-step deterministic sampler (DDIM, session 226). “Load checkpoint” loads weights that we trained offline with this same code and default settings (seed 1, learning rate 0.001, linear), rounded to 8 bits; use it on slow devices. Noise images are shown at half contrast.
How to read Figure 4

Top row: on the linear schedule the picture is still clear at \(t = 100\) (\(\sqrt{\bar\alpha_t} = 0.947\)), half gone at \(t = 250\) (0.724 signal, 0.690 noise), faint at \(t = 500\) (0.280), and invisible from \(t = 750\) (0.058) to \(t = 1000\) (0.0064). Switch to cosine: at \(t = 500\) the two coefficients are about equal (0.703 and 0.711).

Bottom: press Next line to walk through one pass. Lines 2–4 draw a picture, a step and a noise image; line 5 builds \(\mathbf{x}_t\), runs the network, and changes all 78,451 weights by one Adam step. The six panels show what the network saw and said: its guess \(\hat\varepsilon\), the error map \(\lvert \varepsilon - \hat\varepsilon \rvert\) (bright = large error), and the clean picture implied by the guess, \(\hat{\mathbf{x}}_0\) (225.12). At the start the guess is 0, the error map is just the noise, and the loss is about 1. Then press Fast-forward and watch three things. First, the loss curve. With the default settings (seed 1) the average loss falls from 1 to 0.15 after 1,000 passes, 0.07 after 4,000, 0.05 after 8,000, and 0.025 after 20,000 (the checkpoint run). The dots spread over a wide range because the task depends on \(t\): small \(t\) is the hardest step for noise prediction. With the checkpoint weights the average loss is 0.64 for \(t \le 10\), 0.078 for \(t\) from 11 to 100, and below 0.01 for \(t \gt 100\). At tiny \(t\) the noise is smaller than the details of the picture, so noise and detail are hard to tell apart; at large \(t\), \(\mathbf{x}_t\) is almost all noise and \(\hat\varepsilon \approx \mathbf{x}_t\) is a good answer. Second, the samples: the same 6 noise seeds go from colored noise, to blurry blobs with the right colors, to the three pictures. Third, the panels: \(\hat{\mathbf{x}}_0\) becomes a clean picture whenever \(t\) is not too large. On our laptop (Chrome) one pass takes about 12 ms, so about 85 passes per second: the samples were recognizable after 4,000 passes (47 s of training) and clear after 8,000 passes (95 s). Phones are slower; there, use the checkpoint. The learning rate can be changed at any time; changing the schedule or the seed starts a new run, because a network trained under one schedule does not fit another (see the “Watch out” note below). A run with three pictures can at best copy them: like the ideal predictor of Figure 5, a perfectly trained network on three pictures only reproduces its training set.

The link to score matching (NCSN)

The score of a distribution is the gradient of its log-density, \(\nabla_{\mathbf{x}} \log p(\mathbf{x})\): an arrow that points toward more likely data. For the noisy distribution \(q(\mathbf{x}_t)\), the best possible noise predictor (the one that minimizes \(L_\text{simple}\)) is the conditional mean \(\E[\varepsilon \mid \mathbf{x}_t]\), and a short calculation (Tweedie’s formula) gives

\[ \varepsilon^\star(\mathbf{x}_t, t) = \E[\varepsilon \mid \mathbf{x}_t] = -\sqrt{1-\bar\alpha_t}\ \nabla_{\mathbf{x}_t} \log q_t(\mathbf{x}_t) \tag{225.13} \]

So a noise predictor is a score estimator, scaled by \(-\sqrt{1-\bar\alpha_t}\). Song and Ermon’s NCSN trained exactly such a score network at 10 noise levels (standard deviations from 1 down to 0.01, a geometric sequence) and sampled with annealed Langevin dynamics: 100 small noisy gradient steps per level, from the largest noise to the smallest [Certain]. It reached an Inception score of 8.87 on CIFAR-10 [Certain]. DDPM’s sampling step (Algorithm 3 below) looks just like one Langevin step with \(\varepsilon_\theta\) as the learned gradient. session 226 makes this link exact with differential equations.

A practical consequence: the optimal denoiser does not reach zero loss. Several clean images can produce the same \(\mathbf{x}_t\), so the best guess of the noise is an average, and \(L_\text{simple}\) has a floor above 0. In the PyTorch example below, the exact optimal denoiser of the toy data scores 0.339, and the trained network 0.345.

Sampling: from pure noise to data

Once \(\varepsilon_\theta\) is trained, generation follows the reverse chain (225.5) with the mean (225.9). This is called ancestral sampling, because each sample is drawn from its “parent” step.

\[ \mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}} \Big( \mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\, \varepsilon_\theta(\mathbf{x}_t, t) \Big) + \sigma_t\, \mathbf{z}, \qquad \mathbf{z} \sim \mathcal{N}(\mathbf{0}, I) \tag{225.14} \]
Read the symbols
\(\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\varepsilon_\theta\)
the small part of the predicted noise removed in this one step
\(1/\sqrt{\alpha_t}\)
undoes the slight shrinking of the forward step
\(\sigma_t \mathbf{z}\)
fresh noise added back; DDPM uses \(\sigma_t^2 = \beta_t\) or \(\sigma_t^2 = \tilde\beta_t\), which gave similar results [Certain]
Algorithm 3: DDPM ancestral sampling
  1. \(\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, I)\) // pure noise
  2. for \(t = T, T-1, \dots, 1\):
  3.   \(\hat\varepsilon \leftarrow \varepsilon_\theta(\mathbf{x}_t, t)\) // one network call per step
  4.   \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\) if \(t \gt 1\), else \(\mathbf{z} = \mathbf{0}\) // no noise on the last step
  5.   \(\mathbf{x}_{t-1} \leftarrow \frac{1}{\sqrt{\alpha_t}}\big(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\hat\varepsilon\big) + \sigma_t\mathbf{z}\) // formula (225.14)
  6. return \(\mathbf{x}_0\) // clip to [−1, 1] and map back to pixels

Why add noise back at every step? The reverse step is a distribution, not a single point. Without the \(\sigma_t\mathbf{z}\) term, (225.14) is not a correct sampler for this model: errors pile up and samples become over-smoothed. (DDIM in session 226 shows how to build a different, deterministic update that is correct.)

Worked example: one reverse step at \(t = 500\)

Linear schedule: \(\beta_{500} = 0.010040\), \(\alpha_{500} = 0.989960\), \(\bar\alpha_{500} = 0.078587\), \(\bar\alpha_{499} = 0.079384\). Continue the pixel from before: \(x_{500} = -0.2557\), made from \(x_0 = 0.8\) and \(\varepsilon = -0.5\).

Exact posterior (225.6): \(\tilde\mu = 0.003070 \cdot 0.8 + 0.994107 \cdot (-0.2557) = -0.2517\), and \(\tilde\beta_{500} = 0.010031\), just 0.09% below \(\beta_{500}\).

Noise form (225.9) with a perfect prediction \(\varepsilon_\theta = -0.5\): \(\frac{1}{\sqrt{0.98996}}\big(-0.2557 - 0.010459 \cdot (-0.5)\big) = 1.005058 \cdot (-0.2505) = -0.2517\). The same number, as it must be. The step then adds noise with \(\sigma = \sqrt{\beta_{500}} = 0.1002\).

Notice how small one step is: the mean moves by only 0.004, while the added noise has a standard deviation of 0.1. A single step barely changes the image; the 1000 steps together do the work. Near the start the posterior variance is much smaller than \(\beta_t\): at \(t = 2\), \(\tilde\beta_2/\beta_2 = 0.455\); by \(t = 100\) the ratio is 0.982.

Figure 5 runs Algorithm 3 on the pictures of Figure 4. It starts from a pure noise image \(\mathbf{x}_{1000}\) and applies the real update (225.14) 1000 times, with \(\sigma_t^2 = \beta_t\) and seeded noise \(\mathbf{z}\). You can choose what plays the role of \(\varepsilon_\theta\). The reference is the ideal predictor: the output of a perfectly trained network for a training set of one or three 96 × 96 pictures, which is \(\E[\varepsilon \mid \mathbf{x}_t]\) (225.13 above). For a training set of a few pictures \(\mathbf{x}^{(1)}, \dots, \mathbf{x}^{(K)}\) it has a closed form. First weigh each picture by how well it explains \(\mathbf{x}_t\):

\[ w_k = \softmax_k\!\Big( -\frac{\lVert \mathbf{x}_t - \sqrt{\bar\alpha_t}\,\mathbf{x}^{(k)} \rVert^2}{2\,(1-\bar\alpha_t)} \Big), \qquad \hat{\mathbf{x}}_0^\star = \sum_{k=1}^{K} w_k\, \mathbf{x}^{(k)} \]

Then read off the noise with (225.3): \(\varepsilon^\star = (\mathbf{x}_t - \sqrt{\bar\alpha_t}\,\hat{\mathbf{x}}_0^\star)/\sqrt{1-\bar\alpha_t}\). The weights \(w_k\) are Bayes’ rule: \(q(\mathbf{x}_t \mid \mathbf{x}^{(k)})\) is the Gaussian (225.3), and each picture is equally likely. With a single picture (\(K = 1\)) the weight is 1 and \(\varepsilon^\star = (\mathbf{x}_t - \sqrt{\bar\alpha_t}\,\mathbf{x}_0)/\sqrt{1-\bar\alpha_t}\), which is exactly the noise inside \(\mathbf{x}_t\). Then the mean in (225.14) equals the exact posterior mean (225.6), as in the worked example, and the chain must end at that picture. With three pictures the sampler must also decide which picture to draw, and the readout shows the weights \(w_k\) while it decides. The third choice is the network you trained in Figure 4 (at 16 × 16): a real, imperfect \(\varepsilon_\theta\) in the same 1000-step loop.

\(\varepsilon_\theta\) isSchedule

Changing \(\varepsilon_\theta\) or the schedule recomputes the whole run from the same seeded \(\mathbf{x}_{1000}\) and the same \(\mathbf{z}\) draws, up to the current step.

Sampling loop (Algorithm 3; DDPM paper, Algorithm 2)
  1. \(\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, I)\) pure noise
  2. for \(t = T, \dots, 1\):
  3. \(\hat\varepsilon \leftarrow \varepsilon_\theta(\mathbf{x}_t, t)\) one network call
  4. \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, I)\) if \(t \gt 1\), else \(\mathbf{z} = \mathbf{0}\)
  5. \(\mathbf{x}_{t-1} \leftarrow\) \(\frac{1}{\sqrt{\alpha_t}}\Big(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\hat\varepsilon\Big) + \sigma_t\,\mathbf{z}\)
  6. return \(\mathbf{x}_0\)
\(\mathbf{x}_t\)
\(\hat\varepsilon\) (line 3)
\(\hat{\mathbf{x}}_0\) (225.12)
The same run at key steps (filled in as the run passes them)
Figure 5. DDPM ancestral sampling (Algorithm 3): 1000 steps of (225.14) from pure noise, \(\sigma_t^2 = \beta_t\), all noise seeded. The two “ideal” choices are not a trained model: they use \(\varepsilon^\star = \E[\varepsilon \mid \mathbf{x}_t]\) for a training set of one or three 96 × 96 pictures, which is what a perfectly trained \(\varepsilon\)-predictor would output. The third choice uses a snapshot of the EMA weights of the network trained in Figure 4 (16 × 16 pixels, the schedule it was trained with). The middle panel shows \(\hat\varepsilon\) at half contrast; the right panel shows the clean-picture guess \(\hat{\mathbf{x}}_0\) from (225.12). All numbers in the box and the readout belong to the same step \(t\).
How to read Figure 5

Each reverse step removes only a small part of the noise, so use the step buttons: Next line walks through one step line by line (after line 5 the left panel shows the new \(\mathbf{x}_{t-1}\)), +1, +10 and +100 jump ahead, the slider jumps anywhere, and Run to t = 0 plays the rest. The two thumbnail rows record \(\mathbf{x}_t\) and \(\hat{\mathbf{x}}_0\) at \(t\) = 1000, 900, 800, 700, 500, 300, 100 and 0. On the linear schedule \(\mathbf{x}_t\) still looks like noise at \(t = 500\) (signal coefficient 0.280), and the picture appears only in the last few hundred steps.

With one picture, \(\hat{\mathbf{x}}_0\) is that picture from the very first step: a model that knows only one picture always guesses it. Switch to three pictures. Now \(\hat{\mathbf{x}}_0\) starts as a mix of all three. With the seeded noise used here (linear schedule) the weights are about 0.07, 0.16, 0.77 at \(t = 950\), swing to 0.15, 0.52, 0.32 at \(t = 900\), and the night picture has weight 1.000 by \(t = 800\). The choice is made in the first 200 of the 1000 steps, while \(\mathbf{x}_t\) still looks like pure noise; the other 800 steps only clean up. The run ends exactly on a training picture (RMS distance 0.0000): a perfect minimizer of \(L_\text{simple}\) on three pictures can only copy them. Real models trained on huge data sets do not reach that exact minimum, and they produce new images [Likely].

Choose network trained in Figure 4 after training there (or loading the checkpoint). Now \(\hat{\mathbf{x}}_0\) at large \(t\) is a blurry average, and the end result is close to one of the training pictures but not equal to it: with the 20,000-pass checkpoint, the seeded run ends on the night picture, blurred, at RMS distance 0.060 (the ideal predictor ends at exactly 0). Look at the \(\hat{\mathbf{x}}_0\) row: at \(t\) = 1000, 900 and 800 it is colored noise, because (225.12) multiplies every error in \(\hat\varepsilon\) by \(\sqrt{1-\bar\alpha_t}/\sqrt{\bar\alpha_t}\), which is 157 at \(t = 1000\), 60 at \(t = 900\) and 3.4 at \(t = 500\). The ideal predictor has no error to amplify. The network’s \(\hat{\mathbf{x}}_0\) becomes a picture only around \(t = 500\). An untrained network gives noise.

Figure 6 runs Algorithm 3 on the toy data from Figure 2, with one difference: instead of a trained network it uses the exact optimal denoiser \(\varepsilon^\star\). For a mixture of Gaussian blobs, \(q_t(\mathbf{x}_t)\) is also a mixture of Gaussians, so \(\E[\varepsilon \mid \mathbf{x}_t]\) has a closed form. This shows what a perfectly trained model would do.

Schedule
\(\sigma_t^2\)Show
samples (or their \(\hat{\mathbf{x}}_0\))the 8 data blobs (radius 3 standard deviations)
Figure 6. DDPM ancestral sampling (Algorithm 3) of 400 points with \(T = 1000\) and the exact optimal noise predictor for the 8-blob data. All randomness is seeded. The right side shows how many samples are near each blob.
How to read Figure 6

Drag the slider or press Play. For the first few hundred steps (linear schedule) the cloud only shrinks a little: at that noise level the data is almost invisible, and the denoiser can only pull points toward the overall center. Then the points split and choose a blob. Switch to “prediction \(\hat{\mathbf{x}}_0\)”: early on, every point predicts nearly the same blurry average (the middle of the circle), because many clean points are consistent with a very noisy \(\mathbf{x}_t\). Later the predictions snap onto blobs. This is the “coarse first, details later” behavior of diffusion models. At the end, about 99% of the samples sit within 3 standard deviations of a blob, spread evenly across the 8 blobs.

Sampling cost is the main weakness of DDPM. Each step is one full network call, so one image costs 1000 forward passes. The DDPM paper reports 17 seconds to sample 256 CIFAR-10 images and 300 seconds for 128 images at 256 × 256 on a TPU v3-8 [Certain]. The rest of this track is largely about cutting this cost: better variances (Nichol and Dhariwal learn \(\sigma_t\) and reach good samples with about 10 times fewer steps [Certain]), deterministic samplers and ODE solvers (session 226), running in a small latent space (session 228), and distillation to 1–4 steps (session 230).

Noise schedules and why they matter

The schedule \(\beta_{1..T}\) (or, equivalently, the curve \(\bar\alpha_t\)) decides which noise levels the model sees during training, because \(t\) is drawn uniformly. If many steps are nearly pure noise, the model wastes effort on them; if the last step still contains signal, training and sampling disagree. Three schedules are common:

  • Linear (DDPM): \(\beta_t\) linear from \(10^{-4}\) to \(0.02\). Simple, but the signal disappears early. Nichol and Dhariwal observed that the last quarter of the chain is almost pure noise, and that a linear-schedule model on ImageNet 64 × 64 barely got worse when up to 20% of the reverse process was skipped [Certain].
  • Cosine (Nichol and Dhariwal, 2021): define \(\bar\alpha_t\) directly, so that it falls slowly at both ends and roughly linearly in the middle:
\[ \bar\alpha_t = \frac{f(t)}{f(0)}, \qquad f(t) = \cos^2\!\Big( \frac{t/T + s}{1 + s} \cdot \frac{\pi}{2} \Big), \qquad \beta_t = \min\Big(1 - \frac{\bar\alpha_t}{\bar\alpha_{t-1}},\ 0.999\Big) \tag{225.15} \]
Read the symbols
\(s = 0.008\)
a small offset so that \(\beta_t\) is not too tiny near \(t = 0\); chosen so \(\sqrt{\beta_1}\) is just below one pixel step, 1/127.5 [Certain]
0.999
a clip on \(\beta_t\) that avoids a division by zero at \(t = T\), where \(\bar\alpha_T\) reaches 0
  • Scaled linear (Stable Diffusion): \(\sqrt{\beta_t}\) linear from \(\sqrt{0.00085}\) to \(\sqrt{0.012}\). Its noise ends far too early, as we see next.

Figure 7 plots the signal coefficient \(\sqrt{\bar\alpha_t}\) and \(\log \mathrm{SNR}(t)\) for all three.

linear (DDPM)cosinescaled linear (Stable Diffusion)SD rescaled (dashed)
Figure 7. Three noise schedules with \(T = 1000\). Top: how much signal is left. Bottom: \(\log\mathrm{SNR}(t)\), cut at −14 for display. The readout gives the exact values at the chosen step and at \(t = T\).
How to read Figure 7

Look at where each bottom curve crosses 0 (signal equals noise): \(t = 260\) for linear, \(t = 355\) for scaled linear, \(t = 497\) for cosine. The cosine schedule spends many more steps at medium noise, where the image content is decided. Now look at the right end of the top plot: the linear schedule leaves \(\sqrt{\bar\alpha_T} = 0.0064\), the cosine schedule \(0.00005\), but Stable Diffusion’s leaves \(0.068\). Tick the box to see the fix of Lin et al.: rescale \(\sqrt{\bar\alpha_t}\) linearly so that it ends at exactly 0.

Zero terminal SNR

Training uses \(\mathbf{x}_T = \sqrt{\bar\alpha_T}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_T}\,\varepsilon\), but sampling starts from pure noise. If \(\bar\alpha_T\) is not 0, the two do not match. Lin et al. (2023) showed that the Stable Diffusion schedule gives \(\mathbf{x}_T = 0.068265\,\mathbf{x}_0 + 0.997667\,\varepsilon\) [Certain]. The leaked signal is mostly the lowest frequency, the mean brightness of each channel. The model learns to keep the mean it sees at \(t = T\); at sampling time pure noise has mean 0, so Stable Diffusion (up to version 2.1) made images of medium brightness and failed on prompts such as “Solid black color” or “A white background” [Certain]. Their fix: rescale the schedule so \(\bar\alpha_T = 0\), switch to v-prediction (noise prediction is useless when there is no signal at all, see (225.12)), and start sampling from the last step [Certain].

\[ \sqrt{\bar\alpha_t}^{\,\text{new}} = \big(\sqrt{\bar\alpha_t} - \sqrt{\bar\alpha_T}\big) \cdot \frac{\sqrt{\bar\alpha_1}}{\sqrt{\bar\alpha_1} - \sqrt{\bar\alpha_T}} \tag{225.16} \]

This keeps the first step unchanged, sets the last to exactly 0, and moves everything in between linearly.

Resolution changes the schedule you need

The same \(\bar\alpha_t\) does not destroy the same amount of information at every image size. Neighboring pixels of a large image are strongly correlated, and the noise is independent per pixel, so averaging nearby pixels removes much of the noise. Chen (2023) shows images at 64 × 64 and 1024 × 1024 with the same noise level: the large one is far more recognizable [Certain]. So a schedule tuned for 64 × 64 is too easy at high resolution, and the model trains too little at the high-noise steps that set the global layout. The fixes shift the schedule toward more noise (lower \(\log\mathrm{SNR}\)) for larger images, either by changing the schedule or by scaling the input down [Certain] (Chen 2023). Latent diffusion (session 228) and flow models (session 229) use the same idea under names like “resolution-dependent shift”.

DDPM in PyTorch

A complete, deterministic DDPM on the 2-D toy data of Figures 2 and 6: 8 blobs on a circle. It checks the closed form against step-by-step noising, trains a small MLP noise predictor with \(L_\text{simple}\), compares its loss with the exact optimal denoiser, and samples with Algorithm 3. It runs in about one minute on a laptop CPU.

import math, torch, torch.nn as nn
torch.manual_seed(0)

# ---- data: 8 small Gaussian blobs on a circle of radius 1 (2-D "images" with values in about [-1, 1])
K, R, S = 8, 1.0, 0.08
centers = torch.tensor([[R * math.cos(2 * math.pi * k / K), R * math.sin(2 * math.pi * k / K)] for k in range(K)])
def sample_data(n):
    return centers[torch.randint(0, K, (n,))] + S * torch.randn(n, 2)

# ---- noise schedule (DDPM linear): beta_t from 1e-4 to 0.02, T = 1000
T = 1000
betas = torch.linspace(1e-4, 0.02, T, dtype=torch.float64)
alphas = 1 - betas
abar = torch.cumprod(alphas, 0)                       # alpha-bar_t for t = 1..T (index t-1)
print(f"abar_T = {abar[-1]:.3e}, signal left sqrt(abar_T) = {abar[-1].sqrt():.4f}")

# ---- check: the closed form q(x_t | x_0) matches running t single steps
x0 = sample_data(100000).double()
x = x0.clone()
for t in range(500):                                  # 500 single steps of q(x_t | x_{t-1})
    x = alphas[t].sqrt() * x + betas[t].sqrt() * torch.randn_like(x)
xc = abar[499].sqrt() * x0 + (1 - abar[499]).sqrt() * torch.randn_like(x0)
print(f"t=500 std: step by step {x.std():.4f}, closed form {xc.std():.4f}, "
      f"theory {(abar[499] * x0.var() + 1 - abar[499]).sqrt():.4f}")

abar_f, betas_f, alphas_f = abar.float(), betas.float(), alphas.float()

# ---- the noise-prediction network eps_theta(x_t, t): an MLP with a sinusoidal time embedding
def time_embed(t, dim=32):
    half = dim // 2
    freqs = torch.exp(-math.log(10000) * torch.arange(half) / half)
    a = t.float()[:, None] * freqs[None]
    return torch.cat([a.sin(), a.cos()], dim=1)

class EpsNet(nn.Module):
    def __init__(self, h=128):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(2 + 32, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(),
                                 nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))
    def forward(self, x, t):
        return self.net(torch.cat([x, time_embed(t)], dim=1))

model = EpsNet()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=8000)

# ---- training: Algorithm 1 (L_simple)
for step in range(8000):
    x0 = sample_data(512)
    t = torch.randint(1, T + 1, (512,))               # t ~ Uniform{1..T}
    eps = torch.randn_like(x0)
    ab = abar_f[t - 1][:, None]
    xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps       # closed form, one line
    loss = ((eps - model(xt, t)) ** 2).sum(1).mean()  # || eps - eps_theta ||^2
    opt.zero_grad(); loss.backward(); opt.step(); sched.step()

# ---- the best possible loss: the exact denoiser of this mixture, E[eps | x_t]
def eps_star(xt, t):
    ab = abar_f[t - 1][:, None]
    var = ab * S**2 + (1 - ab)                        # variance of each blob after noising
    d2 = ((xt[:, None, :] - ab.sqrt()[:, :, None] * centers[None]) ** 2).sum(-1)
    w = torch.softmax(-d2 / (2 * var), dim=1)         # posterior over blobs
    mean_x0 = centers[None] + (ab.sqrt() * S**2 / var)[:, :, None] * (xt[:, None, :] - ab.sqrt()[:, :, None] * centers[None])
    x0_hat = (w[:, :, None] * mean_x0).sum(1)         # E[x_0 | x_t]
    return (xt - ab.sqrt() * x0_hat) / (1 - ab).sqrt()

with torch.no_grad():
    torch.manual_seed(1)
    x0 = sample_data(200000); t = torch.randint(1, T + 1, (200000,)); eps = torch.randn_like(x0)
    ab = abar_f[t - 1][:, None]; xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps
    print(f"test loss: network {((eps - model(xt, t))**2).sum(1).mean():.4f}, "
          f"exact denoiser {((eps - eps_star(xt, t))**2).sum(1).mean():.4f}")

# ---- sampling: Algorithm 2 (ancestral sampling), sigma_t^2 = beta_t
@torch.no_grad()
def sample(n, eps_fn):
    g = torch.Generator().manual_seed(2)
    x = torch.randn(n, 2, generator=g)                # x_T ~ N(0, I)
    for t in range(T, 0, -1):
        tt = torch.full((n,), t)
        e = eps_fn(x, tt)
        mean = (x - betas_f[t - 1] / (1 - abar_f[t - 1]).sqrt() * e) / alphas_f[t - 1].sqrt()
        z = torch.randn(n, 2, generator=g) if t > 1 else torch.zeros(n, 2)
        x = mean + betas_f[t - 1].sqrt() * z
    return x

for name, fn in [("network", model), ("exact denoiser", eps_star)]:
    xs = sample(4000, fn)
    d = torch.cdist(xs, centers)
    near = (d.min(1).values < 3 * S).float().mean()
    counts = torch.bincount(d.argmin(1), minlength=K)
    print(f"{name}: {100 * near:.1f}% of samples within 3 std of a blob, per-blob counts {counts.tolist()}")

On PyTorch 2.8 (CPU) this prints \(\bar\alpha_T = 4.036 \times 10^{-5}\) (signal left 0.0064). At \(t = 500\) the spread is 0.9823 step by step, 0.9793 from the closed form and 0.9804 in theory: the same up to Monte Carlo noise. The test loss is 0.3452 for the network and 0.3386 for the exact denoiser, which is the floor no network can beat. With the trained network, 89.3% of 4000 samples land within 3 standard deviations of a blob, with 452 to 525 samples per blob (perfect balance would be 500). With the exact denoiser the same sampler gives 99.0%, which is what a perfect model would score (a 2-D Gaussian puts \(1 - e^{-4.5} = 98.9\%\) of its mass within 3 standard deviations). Other versions or hardware may print slightly different numbers.

Where this is used

SettingHow DDPM ideas are usedWhat matters
Text-to-image (Stable Diffusion, SDXL, Imagen, DALL·E 2)Noise prediction with a time-conditioned U-Net or transformer; SD runs in a VAE latent, Imagen in pixels with a cascade of super-resolution models [Likely]The schedule and the prediction target (ε or v); conditioning on text (session 227)
Video generationThe same forward process on a video tensor (frames × height × width), denoised jointly so frames stay consistent [Likely]Many more pixels per sample, so latent compression and fewer steps matter more (session 231)
Video editing, inpainting and frame interpolationNoise the known video to step \(t\) with (225.3), then run the reverse chain while keeping known regions or frames fixed [Likely]The start step \(t\) trades faithfulness to the input against freedom to change it
Video and image super-resolution, restorationA denoiser conditioned on the low-quality input; the reverse chain adds plausible detail [Likely]Can invent details that were not there; risky for forensic or medical use
Audio and speech (DiffWave and others)The waveform or spectrogram is the data vector [Likely]Fast samplers are needed for real time
Robot control (Diffusion Policy)The reverse chain generates a short sequence of future actions, conditioned on camera frames [Likely]Handles several valid ways to act (multi-modal actions) without averaging them
Detection and segmentation (for example DiffusionDet)Boxes or masks are noised and the model learns to denoise them toward the objects in an image [Likely]The same detector can run with more or fewer reverse steps
Data augmentation and synthetic data for video tasks (action recognition, retrieval)Generate extra clips or frames for rare classes [Speculation]Generated data can shift the distribution; check gains on real test data

Common mistakes

  • Off-by-one between \(t\) and the array index. Papers count steps from 1 to \(T\); code arrays start at 0. Using \(\bar\alpha_{t}\) where \(\bar\alpha_{t-1}\) is meant (or the reverse) gives a model that is quietly worse. Write one helper that takes \(t\) and returns all coefficients.
  • Data not scaled to \([-1, 1]\) (or to unit variance for latents). The schedule assumes the signal has variance near 1; with pixels in \([0, 255]\) the noise is negligible and \(\mathbf{x}_T\) is not close to \(\mathcal{N}(\mathbf{0}, I)\).
  • Forgetting \(\sqrt{\cdot}\) on the coefficients. \(\mathbf{x}_t = \bar\alpha_t \mathbf{x}_0 + (1-\bar\alpha_t)\varepsilon\) is wrong: the coefficients are the square roots, and their squares add to 1.
  • Adding noise on the last sampling step, or not adding it on the others. Algorithm 3 sets \(\mathbf{z} = \mathbf{0}\) only at \(t = 1\).
  • Sampling with a different schedule or \(T\) than training without converting through \(\bar\alpha_t\). The network’s idea of “step \(t\)” is tied to its training schedule.
  • A non-zero terminal SNR. Training never sees pure noise but sampling starts from it; the result is the medium-brightness bug. Check \(\sqrt{\bar\alpha_T}\) of your schedule.
  • Reading the loss as quality. \(L_\text{simple}\) has a floor above 0 (the optimal denoiser cannot know the exact noise), and it is averaged over very different steps. Watch the loss per range of \(t\), and judge samples with FID or similar metrics.
  • Sampling with the raw weights instead of the EMA copy. Diffusion samples are very sensitive to weight noise; DDPM samples with an exponential moving average (decay 0.9999) of the weights [Certain].

Interview questions

Derive \(q(\mathbf{x}_t \mid \mathbf{x}_0)\) from the single-step forward process.basic
\(\mathbf{x}_t = \sqrt{\alpha_t}\mathbf{x}_{t-1} + \sqrt{1-\alpha_t}\varepsilon_t\). Substitute \(\mathbf{x}_{t-1}\) the same way. Independent Gaussian noise terms add their variances: \(\alpha_t(1-\alpha_{t-1}) + (1-\alpha_t) = 1 - \alpha_t\alpha_{t-1}\). By induction \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\varepsilon\) with \(\bar\alpha_t = \prod_{s \le t}\alpha_s\). Example: \(\beta_1 = 0.1, \beta_2 = 0.2\) gives \(\bar\alpha_2 = 0.72\) and noise variance \(0.8 \cdot 0.1 + 0.2 = 0.28\). This is what makes training cheap: any noise level in one line.
Why is the old sample multiplied by \(\sqrt{1-\beta_t}\)?basic
To keep the variance fixed (variance preserving). If \(\mathbf{x}_{t-1}\) has unit variance, \((1-\beta_t) + \beta_t = 1\). Without the factor the variance would grow to \(1 + \sum\beta_t\), the network inputs would change scale with \(t\), and the end of the chain would not be a standard Gaussian. The alternative, used by NCSN, is variance exploding: add noise only, and start sampling from a very wide Gaussian.
Why does DDPM predict the noise instead of the clean image or the posterior mean?core
Writing \(\mu_\theta\) in terms of \(\varepsilon_\theta\) (225.9) turns each ELBO term into a weighted noise-regression loss (225.10), and the target \(\varepsilon\) has unit variance at every \(t\), so one network can serve all noise levels without its output scale changing. Empirically (DDPM Table 2, CIFAR-10): \(\varepsilon\)-prediction with \(L_\text{simple}\) gave FID 3.17; \(\tilde\mu\)-prediction was only good with the true bound (13.22) and unstable with plain MSE; \(\mathbf{x}_0\)-prediction gave worse samples. The weakness: at very high noise, \(\hat{\mathbf{x}}_0 = (\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\varepsilon_\theta)/\sqrt{\bar\alpha_t}\) amplifies errors, which is why v-prediction is used with zero-terminal-SNR schedules.
What does \(L_\text{simple}\) drop from the true variational bound, and why does that help?core
The per-step weight \(\beta_t^2/(2\sigma_t^2\alpha_t(1-\bar\alpha_t))\) (and the exact \(L_0\) decoder term). With the linear schedule and \(\sigma_t^2 = \beta_t\), that weight, rescaled to mean 1, is about 51 at \(t = 1\), 7.5 at \(t = 10\) and 0.5 near \(t = 349\). Dropping it down-weights the tiny-noise steps, which only fix imperceptible details, and moves effort to the harder high-noise steps. Result: much better FID (3.17 vs 13.51) and a slightly worse likelihood bound (3.75 vs 3.70 bits/dim). It is a weighted ELBO, not the ELBO.
Write the posterior \(q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0)\). Why is it needed?core
Gaussian with mean \(\tilde\mu_t = \frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}\mathbf{x}_0 + \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\mathbf{x}_t\) and variance \(\tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t\), from Bayes’ rule on two Gaussians. The ELBO compares each learned reverse step with this posterior, so every term is a KL between Gaussians in closed form, with low variance (no Monte Carlo over the whole chain). \(\tilde\beta_t\) is also one of the two standard choices for the sampling variance.
How is a noise predictor related to the score? Connect DDPM to NCSN.advanced
The minimizer of \(L_\text{simple}\) is \(\E[\varepsilon \mid \mathbf{x}_t]\), and Tweedie’s formula gives \(\E[\varepsilon \mid \mathbf{x}_t] = -\sqrt{1-\bar\alpha_t}\nabla\log q_t(\mathbf{x}_t)\). So \(-\varepsilon_\theta/\sqrt{1-\bar\alpha_t}\) estimates the score of the noisy data. NCSN trains a score network with denoising score matching at 10 noise levels and samples with annealed Langevin dynamics; the DDPM update (225.14) has the same form as a Langevin step with the learned score. DDPM’s \(L_\text{simple}\) is (up to weighting) denoising score matching over many noise levels. The SDE view in the next session unifies both.
Why does DDPM sampling need about 1000 steps, and what are the ways to reduce it?core
The reverse step is modeled as a Gaussian, which is only accurate when each step adds very little noise; large steps make the true reverse distribution non-Gaussian (multi-modal), so the model would be wrong. Each step is one network call, so cost is linear in the number of steps (DDPM: 17 s for 256 CIFAR-10 images on a TPU v3-8). Remedies: learned variances (Improved DDPM, about 10× fewer steps), deterministic DDIM and ODE solvers (20–50 steps), latent diffusion (cheaper steps), and distillation or consistency models (1–4 steps).
Linear vs cosine schedule: what is the difference and why does it matter?core
Linear \(\beta_t\) (\(10^{-4}\) to 0.02) destroys the signal fast: \(\log\mathrm{SNR}\) crosses 0 at \(t = 260\), and the last quarter is almost pure noise, so up to 20% of the reverse process can be skipped with little FID loss (on ImageNet 64). Cosine defines \(\bar\alpha_t = \cos^2(\cdot)\) so it falls slowly at both ends and linearly in the middle; it crosses 0 near \(t = 497\). Since training samples \(t\) uniformly, cosine gives more training to medium noise levels, which matter most at low resolution (32 and 64 pixels).
What is zero terminal SNR, and what goes wrong without it?advanced
The SNR at \(t = T\) should be 0, so that training at \(t = T\) sees pure noise, exactly like sampling. Stable Diffusion’s scaled-linear schedule has \(\sqrt{\bar\alpha_T} = 0.068\), so \(\mathbf{x}_T\) still contains the mean brightness of the image. The model learns to trust it; at sampling time pure noise has mean 0, so images come out with medium brightness and the model cannot make very dark or bright images. Fix (Lin et al. 2023): rescale \(\sqrt{\bar\alpha_t}\) to end at 0, use v-prediction (ε-prediction is meaningless with no signal), and make the sampler start at \(t = T\).
Why should the schedule change when you move from 64 × 64 to 1024 × 1024 images?advanced
Noise is independent per pixel, but neighboring pixels of a large image are highly correlated, so local averaging removes much of the noise: at the same \(\bar\alpha_t\), a large image keeps far more recognizable information. A schedule tuned for small images is effectively too easy at high resolution; the model spends little training on the noise levels that set global structure, and samples lose coherence. Shift \(\log\mathrm{SNR}\) down for larger images (more noise), by changing the schedule or scaling the input (Chen 2023).
Your diffusion loss is 0.34 and does not go down further. Is the model broken?core
Not necessarily. \(L_\text{simple}\) has a floor: given \(\mathbf{x}_t\), several clean images are possible, so the best prediction is \(\E[\varepsilon \mid \mathbf{x}_t]\), not the true \(\varepsilon\), and the remaining variance cannot be removed. In the toy example the exact optimal denoiser scores 0.339 and the trained network 0.345. Compare against a baseline, look at the loss per \(t\) range (high-\(t\) losses are near the floor early), and measure sample quality directly (FID, precision/recall).
DDPM fixes \(\sigma_t^2\) instead of learning it. What are the two choices and what did later work do?advanced
\(\sigma_t^2 = \beta_t\) and \(\sigma_t^2 = \tilde\beta_t\). They are the upper and lower bounds of the true reverse variance: \(\beta_t\) is optimal if the data were \(\mathcal{N}(\mathbf{0}, I)\), \(\tilde\beta_t\) if the data were a single point. With 1000 steps they give similar samples, because they differ only at small \(t\) (\(\tilde\beta_2/\beta_2 = 0.455\), but 0.98 by \(t = 100\)). Learning \(\Sigma_\theta\) directly was unstable in DDPM. Improved DDPM learns an interpolation \(\exp(v\log\beta_t + (1-v)\log\tilde\beta_t)\) with \(L_\text{hybrid} = L_\text{simple} + 0.001 L_\text{vlb}\), which improves likelihood and allows far fewer sampling steps.

Exercises

  1. Three forward steps with \(\beta_1 = 0.1, \beta_2 = 0.2, \beta_3 = 0.5\). Compute \(\bar\alpha_3\) and check, by running the steps, that the noise variance is \(1 - \bar\alpha_3\).
  2. Linear DDPM schedule, \(t = 250\) (\(\bar\alpha_{250} = 0.5241\)). Compute the SNR and \(\log\mathrm{SNR}\). Is the image dominated by signal or by noise?
  3. At \(t = 500\) on the linear schedule (\(\bar\alpha = 0.0786\)), the network’s noise prediction is off by 0.1 in one pixel. How large is the resulting error in \(\hat{\mathbf{x}}_0\)? Repeat for \(t = 10\) (\(\bar\alpha = 0.99811\)).
  4. Stable Diffusion’s schedule leaves \(\mathbf{x}_T = 0.068265\,\mathbf{x}_0 + 0.997667\,\varepsilon\). For a 64 × 64 latent channel whose clean mean is 0.5, compare the signal in the channel mean of \(\mathbf{x}_T\) with the standard deviation of the noise in that mean. Can the network see the brightness?
  5. Show that with \(\sigma_t^2 = \beta_t\) the weight in (225.10) equals \(\beta_t / (2\alpha_t(1-\bar\alpha_t))\), and evaluate it at \(t = 1\) for the linear schedule.
  6. A model was trained with \(T = 1000\) and the linear schedule. You want to sample with 250 steps by using every 4th step. Which \(\bar\alpha\) values must the sampler use, and what goes wrong if you instead build a new linear schedule with \(T = 250\)?
Answerscheck
1. \(\bar\alpha_3 = 0.9 \cdot 0.8 \cdot 0.5 = 0.36\). Running the steps: after two steps the noise variance is 0.28; step 3 scales it by \(\alpha_3 = 0.5\) and adds \(\beta_3 = 0.5\): \(0.5 \cdot 0.28 + 0.5 = 0.64 = 1 - 0.36\). 2. SNR \(= 0.5241/0.4759 = 1.10\), \(\log\mathrm{SNR} = 0.096\): signal and noise are almost equal, slightly more signal. 3. \(\Delta\hat x_0 = 0.1 \cdot \sqrt{1-\bar\alpha}/\sqrt{\bar\alpha}\): at \(t = 500\), \(0.1 \cdot 0.9599/0.2803 = 0.342\), a third of the whole pixel range; at \(t = 10\), \(0.1 \cdot 0.0435/0.9991 = 0.0044\). 4. Signal in the mean: \(0.068265 \cdot 0.5 = 0.0341\). The noise mean over \(64 \cdot 64 = 4096\) independent values has standard deviation \(0.997667/\sqrt{4096} = 0.0156\). The signal is 2.2 standard deviations, so yes, the brightness leaks through, and the model learns to rely on it. 5. Put \(\sigma_t^2 = \beta_t\): \(\beta_t^2/(2\beta_t\alpha_t(1-\bar\alpha_t)) = \beta_t/(2\alpha_t(1-\bar\alpha_t))\). At \(t = 1\), \(\bar\alpha_1 = \alpha_1\), so it is \(\beta_1/(2\alpha_1\beta_1) = 1/(2\alpha_1) = 0.50005\). 6. Use the trained \(\bar\alpha_t\) at \(t = 1000, 996, \dots, 4\) (passing those \(t\) values to the network), and define each jump’s \(\beta\) as \(1 - \bar\alpha_t/\bar\alpha_{t'}\) for consecutive kept steps \(t' \lt t\). A fresh \(T = 250\) linear schedule from \(10^{-4}\) to 0.02 has a different \(\bar\alpha\) curve: it ends at \(\bar\alpha_{250} = 0.080\) (signal coefficient 0.28) instead of \(4.0 \times 10^{-5}\), so sampling would start from noise that the network believes still holds a lot of signal, and the step numbers mean different noise levels than the network learned.

Key takeaways

  • The forward process adds Gaussian noise with a variance-preserving step (225.1); the chain collapses to \(\mathbf{x}_t = \sqrt{\bar\alpha_t}\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\varepsilon\) (225.3), so training can jump to any \(t\) in one line.
  • \(\mathrm{SNR}(t) = \bar\alpha_t/(1-\bar\alpha_t)\) (225.4) describes a noise level in one number; the schedule is just the curve of \(\log\mathrm{SNR}\) over \(t\).
  • The reverse process is a chain of learned Gaussians (225.5). The ELBO (225.7) compares each step with the exact posterior (225.6), which gives squared errors between means (225.8).
  • Writing the mean through a noise prediction (225.9) gives a weighted noise-regression loss (225.10); dropping the weight gives \(L_\text{simple}\) (225.11), with much better samples (FID 3.17 vs 13.51 on CIFAR-10) and a slightly worse likelihood.
  • A noise predictor is a scaled score estimator (225.13), which links DDPM to NCSN; the optimal predictor still has a loss floor above 0.
  • Sampling runs (225.14) from pure noise for \(T = 1000\) steps, one network call each; that cost drives the rest of this track.
  • Schedules matter: cosine (225.15) keeps more steps at medium noise than linear; terminal SNR must be exactly 0 (225.16); larger images need noisier schedules.

Key formulas

\[ \mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon, \qquad \bar\alpha_t = \prod_{s=1}^{t}(1-\beta_s), \qquad \mathrm{SNR}(t) = \frac{\bar\alpha_t}{1-\bar\alpha_t} \]
\[ L_\text{simple} = \E_{t, \mathbf{x}_0, \varepsilon}\big\lVert \varepsilon - \varepsilon_\theta(\mathbf{x}_t, t) \big\rVert^2, \qquad \hat{\mathbf{x}}_0 = \frac{\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\varepsilon_\theta}{\sqrt{\bar\alpha_t}} \]
\[ \mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\Big(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\varepsilon_\theta(\mathbf{x}_t, t)\Big) + \sigma_t\mathbf{z}, \qquad \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t \]

References for this session

  1. Ho et al. (2020). Denoising Diffusion Probabilistic Models
  2. Song & Ermon (2019). Generative Modeling by Estimating Gradients of the Data Distribution
  3. Ho et al. (2020). Denoising Diffusion Probabilistic Models
  4. Song & Ermon (2019). Generative Modeling by Estimating Gradients of the Data Distribution
  5. Sohl-Dickstein, Weiss, Maheswaranathan & Ganguli (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics
  6. Nichol & Dhariwal (2021). Improved Denoising Diffusion Probabilistic Models
  7. Lin, Liu, Li & Yang (2023). Common Diffusion Noise Schedules and Sample Steps are Flawed
  8. Chen (2023). On the Importance of Noise Scheduling for Diffusion Models
  9. Ronneberger, Fischer & Brox (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation
  10. Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations
  11. Song et al. (2020). Denoising Diffusion Implicit Models
  12. Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models
  13. Kingma & Welling (2013). Auto-Encoding Variational Bayes